Discussion
In order to use all the RAM I decided to see what running a local LLM for coding was like. It does work, this particular model uses about 50GB of RAM and causes the base M5 Ultra Studio to suck up about 200-250W of power while running. Performance is noticeably worse than the cloud models, but I’d say bordering usable.
Quality reminds me of what the big 2 models were like at the start of the year, so ranging between OK and terrible.
@paul I’m on an M5 Max MacBook Pro with 128GB RAM and definitely am finding that 130B parameter models are quite usable for local AI usage. I’ve only done reverse engineering work so far but I’ve already finished one project and am on to another one that should be wrapping up in the next day or so.
@narles how much ram does that suck up?
Anyone at Apple want to send me a 512GB review unit for a couple weeks? 😘
Would love to actually see what one of the big open source models could actually do. I’d probably be crazy enough to spend $10k on a machine to avoid sending $20-100 a month to the AI bros.
@paul been legit contemplating this for months. Was going to build out a stupid v100 rig but decided against it in favor of Macs and then just haven’t pulled the trigger yet. Would heavily follow your posts if you were given a review unit 🙌❤️
@paul I’ve been using some of the big open weight models through greenpt.com and berget.ai. Bigger models than can fit on a personal Mac, running on renewable energy, and on EU servers with good data privacy. GLM 5.3 and Kimi K3 give near Opus level quality with coding in my experience.
@paul But when the AI companies start charging enough to cover their costs…
@paul Qwen 3.8 next is pretty noticeable in terms of usability jump (if you can fit it). Unsloth quants use dynamic shenanigans so you can fit more with less vram
@b3ll half of that was
@paul qwen 3.8 next or whatever is called is beeg
Unsloth makes it smol (https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-IQ3_XXS) by doing maths and making the important bits not compress too much and the less important bits compress lots
So sometimes you can use a compressed version of a beeg model and that will outperform the full version of a smol model (in your case Q8).
Similarly you can use the unsloth version of the model you have (UD 4 XL) and it should go even faster with very minimal loss in quality
@b3ll I told Sol to just give me a command to download what you previously suggested and now I’m downloading 80GB+.
@paul qwen3.8 seemed workable