We've gotten very close to physical limits. Quick back-of-the-envelope estimate: if electrons traveled at the speed of light through silicon (they don't), then in 3GHz, an electron could travel .1 meters. In reality, electrons in silicon travel quite a bit slower than that. Net result: electrons can barely cross the diameter of the chip in one cycle, even without gate propagation delays and other factors limiting work done per cycle.
So, if you want data from a tiny distance away, such as a local register, you can grab it and do something simple with it in one cycle. If you need data from any further away, forget about it; cache takes longer, another core takes even longer, and memory takes far longer.
Fun fact: electrons in an metallic wire under a potential difference move only in the order of mm/s. See e.g.[1]
As electrical signals still travel at about half the speed of light, that seems besides the point. However, electrical signals do not need to span the entire width of a chip in order to have an effect. That is true for your example of accessing memory, but not for the actual calculations happening inside the CPU.
On each clock cycle, all transistors are 'fed' by the transistors before it. The state of a transistor only depends on the states of the immediately connecting transistors on the clock cycle before it. That means the signal only has to travel the length of a single transistor on each clock cycle. We could have CPU's working at 3THz and a much larger focus on the difference between CPU-bound and IO-bound tasks.
The easiest to understand analogy to illustrate this that I've found to date is to imagine a tube full of marbles, push one more marble in on one end and another one will pop out instantly on the other end. As long as the marbles are all the same colour it is as though the marble you pushed in has miraculously teleported to the far end of the tube. It appeared to have moved at the speed of light even though in reality all the marbles have shifted only an amount equal to the diameter of a single marble.
There is a problem with that analogy: if you only apply pressure to a single marble, the marble at the other end will not be pushed out instantly. In fact, the 'signal' will travel with the speed of sound in the material of the marbles and you can measure the time difference it takes for the last marble to move after the first one has moved. The marbles will be slightly compressed and decompressed while moving, accounting for the extra length needed to accommodate the 'not moving instantly'. [1]
A better analogy would be one where all marbles are pushed simultaneously. That is more like what happens in logic circuits.
[1] People doubting relativity often try this thought experiment: I have an incompressible metal bar of a lightyear long. I press on one side. The other side must move instantaneously and not after 1 second (or longer). In fact, this proves the reverse: relativity is incompatible with the existence of incompressible metal bars. As we know, all metal bars are in fact compressible, so that is not a problem. And with compressible metal bars, the thought experiment fails, because the push will travel with the speed of sound.
That's true, but it is only an analogy, and like every other analogy it breaks down at some level (after all, it isn't the 'real thing').
The fact that there is a pressure wave set up in the materials is possible because marbles are made of some material (glass, stone, metal, whatever). If you wanted a 'perfect' picture you'd have to explain about electron migration in detail and then we're looking at a completely different picture.
You'd not have a pressure wave in an electron to begin with, and they're not 'pushing' against adjacent electrons either.
But it serves well to show how a slow move can have an apparent instantaneous effect at a distance.
Edit: I had a thing written here, but I figure I'm not understanding what you mean by 'all marbles are pushed simultaneously'. Can you elaborate? Other than the conflation of the speeds of light and sound (which is not such a big deal, really), it's a good analogy.
As for the 'all marbles are pushed simultaneously': a key characteristic of useful transistors is that they operate as amplifiers[1]. This is possible because effectively, every transistor has its own power source. As such, on each clock cycle, then can all get a simultaneous push from their power sources.
BTW, I'm not conflating the speeds of light and sound. If you bang on one side of an iron bar, a wave travels through the material, quickly compressing and decompressing the bar where the wave passes. Such waves travel at the speed of sound in that material, no at the speed of light. That makes sense, because sound is nothing more than the physical modulation of the density of a medium, most commonly air.
You're not, the person you're replying to was. I find it odd that you would object to an analogy where the signal propagates through the marbles at the speed of light (i.e. 'instantly'), and offer up instead one where each marble moves simultaneously. If the exit event is simultaneous to the entry of the other marble, they are separated by a spacelike interval despite the causal relationship between the two events. This violates relativity.
On the other hand, in the original analogy the two events are separated by a lightlike interval, which while impossible in the case of marbles, at least does not violate relativity when the two events are causally linked. This is what I meant when I said the conflation of the speed of sound and the speed of light is not such a big deal, in this case.
Anyway, it's perfectly reasonable to talk about light traveling instantly and it's time that has propagation delays. The idea being what separates the present from the future is the ability to interact with each other.
The electrons are only moving at ~ mm/s. With an AC current, the net effect is that they're not moving at all. Electrical power is not provided by the movement of electrons. The force carriers of the electrical field, the particles responsible for actually exchanging energy, are photons.
I initially was mostly interested in the word instantly, but if we're going to analyze this analogy further, I'd have to say you just can't treat electrons inside bulk materials as particles. It may sound sensible and give you the idea you understand what's going on, but it's just not even wrong.
Umm, yea just like the marbles in the example. If you have a marble pushed into a pipe with 1,000 marbles the one popped out at the end is pushed out a lot faster than any single marble moves.
Ummm... no? It'll come out at roughly the same speed (less if the moment it came out there was still energy trapped in the form of compressed marbles when it emerges). Why would it be otherwise?
Have you ever seen those desktop toy's where steel balls 10+ steel balls are strung up next to each other. You drop one and it fall hit's one end of the chain and in less time than it would have taken the first ball to move that distance the last one pops off the end? Same concept the final ball moves a little slower than the first ball that hit the chain, but it start moving sooner than it would have taken the first ball to move the distance of the chain.
Ok that's true, but not how I read your post. Yes the energy is transfered at the speed of sound in the material, but I read your post as saying the velocity of the marble on the end is greater than the velocity of any other marble, including the one you push in, which is certainly not true.
I've been told by chip designers that chip-crossings are on the order of 5-10 cycles these days. This is for data; they can arrange faster crossings for some signals, such as clocks, but they don't like doing this.
GPUs are getting really interesting. It's not general purpose yet -- you still have to worry a lot about contention and memory bandwidth -- but you can do a jaw-dropping amount of computation in a second on a recent GPU.
One thing Knuth doesn't mention is that he may not explicitly use that extra core on his workstation more than once a week for computation, but he /does/ use it for parallelism at the OS level, where it's helping reduce the latency of his interactions. I can definitely tell a single CPU box from a multicore box, given a couple of Emacs sessions.
I was going to chime in and say, "ahh, but GPUs are worthless for everyday tasks!" when I realized that my 4-yr old MacBook is just fine for every day tasks. For scientific computing GPUs seem to be a great way forward. The pain and complexity of writing parallel code is mitigated by the fact that, hey, scientists are writing it.
For normal, everyday usage I see no reason why 3.5ghz is any better or worse than 20ghz. Or 2ghz, for that matter. The 'computer' my mom is most excited about now is her iPad 2, which has less power than the workstation I built in high school.
A nitpick - electrons travel very slowly in silicon (iirc on the order of a few metres per second), but electrical signals travel rather more quickly - around a half or two-thirds the speed of light (again, iirc.)
Think of electrical signals like waves of sound through the air - the air particles don't move by any significant amount, but the signal propagation, is rather quick because of the speed at which movement is transmitted to neighbouring particles (and so on.)
So, if you want data from a tiny distance away, such as a local register, you can grab it and do something simple with it in one cycle. If you need data from any further away, forget about it; cache takes longer, another core takes even longer, and memory takes far longer.