Discussion
(Disclaimer. I work at Red Hat, which is owned by IBM. But my opinions here are my own. So don't quote me, dear journalists)
On-Prem "AI" works and is Good Enough™ so more companies are running Open Weight LLM inference on their own bare metal. Says IBM CEO Arvind Krishna. I am not surprised. If you can save 90% of the costs and, more importantly, keep your data and prompts local, err, sovereign —it not only makes business sense, it also keeps the lawyers happy.
@hwa The focus is often on coding LLMs where the calculation between Good Enough™ and good in-house developers does get you to 90% cost savings. It of course depends on how you calculate the cost of on-prem inference. If you stick to tokens or to TCO divided by result metrics, which can be quite complicated. The main point is that on-prem inference results in fixed TCO whereas renting models is a very chaotic calculation. And compliance is growing fast in importance for (big) companies in the EU
@hwa But LLM use is far broader. Call center support, dealing with internal processes etc are also use cases where the cost calculation depends mo0re on internal training on effective use. It's a growing field of practice and mistakes have been made. But the bean counters are now involved ;)
@jwildeboer how much #VRAM or #RAM on average per #server ?
@jwildeboer Who makes those open weight models and how long will they still do it? :)
@Rainer_Rehak Ultimately verticals will come together and train/share them amongst themselves. This is already happening, BTW. It makes a lot of sense for insurances, banks etc. to have their own, shared models that are trained on vetted data and can be made to care about local regulations/laws through RL (Reinforcement Learning). A shared base model that gets specialised with internal data by the companies on top of that.
@jwildeboer @Rainer_Rehak it would make sense for the three Berlin universities that use SAP to co-develop their sap-apps together instead of reinventing the wheel 3 times, and yet, here we are
@pkraus The faculty kingdoms in German academics are a very special fortress that refuses cooperation almost always by default. Le sigh. @Rainer_Rehak
And also to be very clear: This is about companies, not the basement GPUs you run at home. These systems are not cheap, but here is an interesting development I hear more and more about in the various rumour mills. Racks and racks of NVIDIA systems are sitting in storage as data centres can't be spun up fast enough and a market of selling that surplus to companies that have empty racks in their own datacenter is real. And growing. But don't talk about it or else ;)
Remember that most of these new data centres are built by shell companies, tasked or created by bigger ones like Microsoft, Oracle or Google and these shell companies are growing more and more desperate for cashflow to serve the promises given to investors. It's a weird world out there, once you dig deeper ;)
If you are a journalist and want the official version from me as a Red Hat Spokesperson, feel free to contact me via DM or on my Red Hat email address. I'll add even more :)
Install Fedizen
Get the full app experience
@jwildeboer
I wouldn't say you can save 90% of the cost, as unlocking performance with self hosted requires some upfront investment and at modest scale, people too. But I will say that if you have the resources and skill to self host open weights models, you can easily get factor 5-15 of what you'd pay for a subscription (that's factor 5-15 below cost covering to boot).