I thought this is because everyone else is interested in N3E while Apple is happy to book on N3B?

Density doesn't always matter. I'm reminded of Apple's 5nm M1 Ultra struggling to keep up with Nvidia's 10nm RTX 3080 in standard use. Having such a minor node advantage won't necessarily save them here, especially since Nvidia's currently occupying the TSMC 4nm supply.

You're comparing a pickup truck with a Main Battle Tank. An RTX 3080 is an electricity hog and produces heat like a monster. No wonder it performs better than an M1 Ultra with a worse node tech.

The RTX 3080 consumes ~300w at load, the M1 Ultra consumes ~200w. If you extrapolate the M1 Ultra's performance to match the 3080, it would also consume roughly the same amount of power.

Is this not a battle-tank-to-battle-tank comparison?

Since when is there a full fledged CPU in an RTX 3080?

You can run an RTX 3080 off anything with enough PCI bandwidth to handle it. Presumably the same goes for Apple's GPU. We could adjust for CPU wattage, but at-load it amounts to +/-40w on either side and when we're only testing the GPU it's like +/-10w maximum.

The larger point is that Apple's lead doesn't extrapolate very far here, even with a generous comparison to a last-gen GPU. It will be great at inferencing, but so are most machines with AVX2 and 8 gigs of DRAM. If you're convinced Apple hardware is the apex of inferencing performance, you should Runpod a 40-series card and prove yourself wrong real quick. It's less than $1 and well worth the reality check.

My point was mostly that the 200W TDP you quote is for the whole package (CPU, GPU, RAM, plus the Neural network thingy and the whole IO stuff). A 120W figure for the GPU is more realistic.

I'm not pretending the Apple chips are the be-all-end-all of performance. They certainly have limitations and are not able to compete with proper high end chips. However I can confidently say that on mobile devices and laptops, competition is largely behind. Sure a 1000+$ standalone GPU will be faster, but it doesn't fit in my jeans. It's the same as comparing a Hasselblad camera with the iPhone 14 pro...

The competition is all fine, though. They have enough memory to run the models, they have hardware acceleration (ARMnn, SNPE, etc.) and both OSes can run it fine. Apple's difference is... their own set of APIs and hardware options?

How can you justify your claim that they're "largely behind"? It sounds to me like the competition is neck-and-neck in the consumer market, and blowing them out at-scale. It's simply hard to entertain that argument for a platform without CUDA, much less the performance crown or performance-per-watt crown.

Nvidia is somewhat encumbered by their need to optimize for raster performance. Ideally, all those transistors should be going toward tensor cores. Apple has never really taken the gaming market seriously. If they wanted to, they could ship their next M3 chip with identical GPU performance and use all that new 3nm die space for AI accelerators.

> Having such a minor node advantage

Is that a minor advantage? I would think that, the smaller the nodes get, the larger the impact of a 1nm difference. Because transistors have area, I think the math, in ≈transistor count would be 3nm:4nm = ⅓²:¼², and that’s 1,777… so a 3nm node could have 75% more transistors on a given die area than a 4nm one (roughly).

4nm -> 3nm no longer means size goes down as a result directly ratiometricly. You have to look at what TSMC is claiming for their improvements. They're claiming 5nm -> 3nm is a 70% density improvement (I can't find any 4nm -> 3nm claims)... so 4->3 must be much less.

Also, most folks seem to have gone directly from 5nm to 3nm, and skipped 4nm altogether.

Apple is rumored to be taking 90% of TSMCs 3nm production.

It will be quite the showdown, then. The M1 struggled to compete with current-gen Nvidia cards at release, we'll have to see if the same holds true for M3.