Discussion
In order to use all the RAM I decided to see what running a local LLM for coding was like. It does work, this particular model uses about 50GB of RAM and causes the base M5 Ultra Studio to suck up about 200-250W of power while running. Performance is noticeably worse than the cloud models, but I’d say bordering usable.
Quality reminds me of what the big 2 models were like at the start of the year, so ranging between OK and terrible.
@narles how much ram does that suck up?
Anyone at Apple want to send me a 512GB review unit for a couple weeks? 😘
Would love to actually see what one of the big open source models could actually do. I’d probably be crazy enough to spend $10k on a machine to avoid sending $20-100 a month to the AI bros.
@paul But when the AI companies start charging enough to cover their costs…
@paul Qwen 3.8 next is pretty noticeable in terms of usability jump (if you can fit it). Unsloth quants use dynamic shenanigans so you can fit more with less vram
@b3ll half of that was
@paul qwen 3.8 next or whatever is called is beeg
Unsloth makes it smol (https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-IQ3_XXS) by doing maths and making the important bits not compress too much and the less important bits compress lots
So sometimes you can use a compressed version of a beeg model and that will outperform the full version of a smol model (in your case Q8).
Similarly you can use the unsloth version of the model you have (UD 4 XL) and it should go even faster with very minimal loss in quality
@paul I’m on an M5 Max MacBook Pro with 128GB RAM and definitely am finding that 130B parameter models are quite usable for local AI usage. I’ve only done reverse engineering work so far but I’ve already finished one project and am on to another one that should be wrapping up in the next day or so.