Even if you think this particular team cheated, the idea that nobody will find ways of making training more efficient seems silly - these huge datacenter investments for purely AI will IMHO seem very short sighted in 10 years
More like three years. Even in the best case the retained value curve of GPUs is absolutely terrible. Most of these huge investments in GPUs are going to be massive losses.
I actually wonder if this is true in the long term regardless of any AI uses. I mean, GPUs are general-purpose parallel compute, and there are so many things you can throw at them that can be of interest, whether economic or otherwise. For example, you can use them to model nuclear reactions...
Do we have any idea how long a cloud provider needs to rent them out for to make back their investment? I’d be surprised if it was more than a year, but that is just a wild guess.
Operating costs are usually a pretty significant factor in total costs for a data center. Unless power efficiency stops improving much and/or demand so far outstrips supply that they can't be replaced, a bunch of 10 year old GPUs probably aren't going to be worth running regardless.
There is a big balloon full of AI hype going up right now, and regrettably it may need those data-centers. But I'm hoping that if the worst (the best) comes to happen, we will find worthy things to do with all of that depreciated compute. Drug discovery comes to mind.
The "pure AI" data center investment is generically a GPU supercomputer cluster that can be used for any supercomputing needs. If AI didn't exist, the flops can be used for any other high performance computing purpose. weather prediction models perhaps?
But we're in the test time compute paradigm now, and we've only just gotten started in terms of applications. I really don't have high confidence that there's going to be a glut of compute.
