Discussion
quick summary on where i stand on transformers (LLMs, gen AI), for future reference:
- SaaS sucks, from eavesdropping to datacenters. just no.
+ personal transformers are useful.
- using gen AI (example based) to produce art is boring and a sign of incompetence. this includes demoscene productions & fictional writing.
+ agents for software development are useful, provided you are competent enough to steer them.
- the AI bubble must pop. don't believe the hype.
+ keep your enemies closer.
@drq now this is post scarcity
@lritter Fuck yeah. If this thing shakes out, we can finally put all that "Permissive vs Copyleft" debate to rest, as it wouldn't matter anymore: everything is open anyway, physically, no matter what the legal department has to say.
The war is over. We won. We can go home now.
@drq the war will never be over ;)
@lritter @boggo a few years ago, i built this PC with an RTX 4090 and 96 GB RAM, specifically because i knew local LLMs were going to be a thing and i wanted to try them out, so i figured i'd need loads of RAM and VRAM. the punchline is that the prices of everything have gone insane, and i haven't even used the PC for LLMs at all, and my interest in doing so has basically evaporated
@lritter @boggo i'd like to give a reason, but all the things i can think of are moral post-justifications. i don't know why. i just stopped finding them as interesting as i did in 2022. if i squint, i can see how someone could incorporate an LLM into a workflow that was useful for them. for me, though, it's like i'm tending my vegetable garden and someone is trying to sell me a combine harvester
@lritter Finally, somebody sane.
we do on top have an availability problem. how good your agent is depends on what you can afford.
however, if you can provide:
* 32 to 64GB of RAM
* a fistful of CPU cores
* 8-12GB of VRAM
* a dedicated GPU (e.g. RTX 3060)
then you can run a 2-bit quantized version of Qwen 3.8 Flash Next via Niko1221/Strata at a fluid token rate with a context spanning about 2 hours of work.
that's the very best r/n in the PC tier.
the next lower tier:
@lritter
I think this is going to pop the bubble. "Cheap" AI boxes that are good enough at coding, making SaaS LLMs obsolete.
Something like Opus level is good enough.
@n3wjack Yeah, I figured that all I really, personally want out of AI, for the most part, is coding and refactoring assistance, decompilation and RevEng tooling and long text summarization.
So I'm building a rig just for that. From what I've heard, Qwen or Kimi are plenty for that, if set up correctly. Bought a couple PCI-E V100s the other day.
Pictures and videos? I dunno. It's mega-rare, but I've seen actually good shit done this way. So, maybe. Maybe not. The word on the street is, V100s are not good for that. Not a priority anyway.
The wider AI industry? I'll pass. At least, until the hype peak dies down.
if you have:
* 8 GB of VRAM
* a dedicated GPU
you can run prism-ml/Ternary Bonsai 2 on a llama.cpp fork, that's Qwen 3.8 27b quantized to a novel ternary format with very little loss compared to the full FP16 model.
great token rate, same context length.
the issue with Qwen 2.8 27b is that it is prone to overthinking on xhigh reasoning, and on medium quality is insufficient. there is a finetune called Swift 1.5 that reduces this (also for TB2), but introduces more clumsiness
next lower tier:
> the issue with Qwen 2.8 27b is that it is prone to overthinking on xhigh reasoning
Oh, yeah. I've tried it with Roo Code. It reminded me of myself when I'm having a hard executive paralysis.
if you have:
* 32 GB of RAM
* a iGPU or similar with UMA (e.g. 860M)
you can run a 2-4 bit quantized Qwen 3.6 35b a3b on stock llama.cpp at a good token rate and about the same to half context length.
this model is capable but needs to be told more explicity what to do and how to do it. it forgets to write tests and update docs. it has less experience overall. it needs more micromanagement. it needs skill files written by better models (e.g. claude). but it can do work.
that's it.
@lritter i think I tend to agree with your overall philosophy that if I have to use AI I'd rather it was very explicit prompts to local models. In my experience trying a lot of this stuff though, I feel stuck between:
- I don't want to use any of this SaaS stuff, they all look like hell on earth, and they tempt you with temporarily low costs and convenience
- Qwen but it kind of sucks and instead of faffing with it for ages trying to get it to not get confused I could just code the thing.
Install Fedizen
Get the full app experience
@lritter Also:
+ The veritable deluge of reverse engineering projects and open-source versions of expensive proprietary products that have always been a major roadblock for switching to FOSS operating systems is REALLY cool, and I'm happy to see proprietary software as a concept finally on its death bed.
You can't spell REVerse ENGinEering without REVENGE. Oh yeah.