The news that there's an oversupply of Nvidia GPUs convinced me that the AI bubble is bursting and it'll burst *hard*. So here's a thread where I go over the financial, technological and hardware implications of this. Strap in because this is going to be long and probably a bit rambling too. 🧵 1/
5 Boosts
0 Likes
If Nvidia GPUs are in significant oversupply as was reported then it means the prices will come down, as there's nobody willing to buy that stuff anymore. And this poses an immediate and very tangible problem for those who bought them: they have to write down the loss of value for the stuff they already own. That's hundreds of billions of dollars of value evaporating, because that value only existed if these things were really scarce, which they aren't. Oops. 3/
@gabrielesvelto it’s like DeBeers manufacturing diamond scarcity by hoarding, when there is an actual abundance.
Which brings to another very thorny topic: mark-to-market accounting. Let's say you buy something worth 1 billion $, say DRAM, that's 1 billion $ of assets on your balance. But let's say that the next year the price of that stuff goes up by 100%, what are you going to do? The prudent choice is to leave that asset at the value you paid it for, in case it comes down later. And then there's the choice of changing the value of that asset to account for the increase in price, that's mark-to-market accounting. Enron became famous for this, I guess you can see why. 4/
I don't know if hyperscalers did it, but given everybody involved in the AI bubble has been reckless and dishonest my guess is that it's highly probable they did. It's also considered a generally accepted accounting practice (GAAP) these days. Anyway, if they did it it means that when the prices of this mountain of hardware come down the companies owning it will have to do gigantic write downs on their balance sheets, booking huge losses, much larger than if they hadn't. 5/
But wait, it's worse than this because none of these companies had the money needed to buy this hardware. They either used venture capitals or made loans. Hundreds of billions of loans. And what does a company use as the collateral for a loan? Its own assets, right? Like GPUs. And memory. And the datacenters they're in. Which all of a sudden are losing value. And what do you call a loan whose collateral is insufficient to cover it? An underwater loan. And what does it cause? A margin call: the debtor must either deposit cash or provide a different collateral to compensate. I guess you already know where this is going. 6/
Every single company in the tech space has negative cash flow. That is they're spending more money than they're making. A lot of it. So how they'll answer the margin call? If this is starting to sound a lot like the subprime mortgage crisis it's because it very much is, but with datacenters instead of houses. 7/
This is where we have to thank Ed Zitron for its excellent reporting on the finances of these companies and his remarkable work in connecting the dots whenever some CEO talked bullshit to investors while the rest of the press was looking the other way. This is also why the Gamers Nexus investigations about GPU smuggling where so important: the demand could not slow down, by all means necessary, so a little illegal smuggling on the side is OK, say a few tens of billions of it. 8/
And this is when we start talking about the tech itself and why it is unsustainable. Every single company in this space is losing money, every AI startup, every AI division in an existing company, everybody. They're losing money because the costs of running an LLM - inference - are large, and it's unclear if anybody has been profitable there. But even if they are nobody has been able to cover the escalating costs of training LLMs, which is of course a prerequisite to running them. 9/
Training LLMs requires a gigantic amount of hardware, brute force calculations, storage, memory and power. Enormous amounts of data whose scraping is not being paid by the companies involved, it's a cost they've spread over the entire internet. As models get larger training gets more expensive. But even as companies sink more money on training LLMs have already reached a plateau in term of functionality and in some cases are degrading in a process called model collapse. 10/
Let's talk statistics for a minute to understand model collapse. Say you have a bunch of measurements of something and you want to build a mathematical function that will tell you what values appear in the measurements and how often they appear, this is what is called a distribution. A distribution can be used to reproduce the material it was created from, but the output will not be the same because something was lost in the process of making it. The less frequent samples, the stuff that lives at the fringes of your data set, the details. 11/
Now let's say you use your distribution to generate a new bunch of data: it will look a lot like the inputs you used to make it, but it won't be the same. And if you try to make a new distribution out of that data it will be distorted. You'll lose more information and for every iteration the outputs will look different, they will be more uniform, they'll contain less details, like when you recompress a JPEG file over and over and over. 12/
And this is exactly what happens when you train an LLM on its outputs, because an LLM is at its core a distribution of human language. It's also the reason why LLM vendors started putting invisible watermarks in LLM outputs. They need to filter out the slop they make from their training set. But even then some slop will make its way in, and even the slop that gets filtered out is still an additional cost: you scraped it, you processed it to check for your watermark, and then you threw it away. 13/
Let's start with the oversupply and what that means for everybody involved. Whether it's traditional players like Google or new entries like CoreWeave every single company in the tech sector has been buying an increasing amount of hardware at increasingly steep prices. GPUs, memory, servers, power supplies, and the prices went up and up and up. Now when a company does that, it looks neutral on their balance sheet, because the money they spend is compensated by now having an asset of the same value. Then over the following years they write down that value in what is called depreciation, this is visible as an expense on their balance. 2/