> Qwen3.8 27B (Q4_K_M, 17GB) generates at ~14 tokens/s on my Mac Studio M3 Ultra
>~14 tokens/s
For anyone reading that has never ran local llms, please understand that anything under 100 tok/sec is worthless. You are faster typing stuff into Gemini free version that you get with a google account and copy/pasting it in (and you can easily build browser automation with playwright or any other js runtime to have this available in a chat window)
I get 80 tok/s on the same model, and it's pretty usable. I'm not chatting with it; it's either given a bag of tokens to generate an answer or it's doing some agentic coding.
The standard is not what you can do with it, the standard is what is the free alternative. Local LLMs need to be able to beat the rate limits on all the free models like Gemini to be useful. The only way to do that is to have high enough tok/sec, especially for agentic loops.
I agree that 14t/s is pretty tedious for interactive use, yes. But 50-60tk/s is faster than I can read. 100tk/s is outright fast. Don't forget there is a limit entering content into meatspace.
Also, Gemini may be free but what if I don't want to give all my data to Google? This is precisely why I have a lot of stuff locally.
And will it remain free? How are they going to make back all those trillions of investment?
But yeah I would kinda balk at 14tk/s too that's why I use old datacenter/workstation-class GPUs.
I wouldn't say worthless but once you get used to 100+ tokens / s, it's really a visible slowdown, especially when running multiple agents acting on something more than basic prompt processing.
Still, as you said - for day to day, 50-60 tokens / s is a good baseline.
30 is worth it on a meaningfully better model, though annoying. 50 is pretty much unnoticeable (good old 60 FPS). 80-100 is heaven (aka 120 FPS - once you get used to it, it does suck to go back).
I would still rather have 30 tokens per second of GLM 5.3 Flash than 100 tokens per second of Qwen 3.6 35B-A3B.
In agentic scenarios, an LLM has to read far more tokens than it outputs. I think focusing only on the decode speed is somewhat misleading. 14 tok/sec for decode is actually okayish. 93 tok/sec prefill is what's abysmal, my RTX 5090 goes above 2000 tok/sec with 5 bit quants.
>~14 tokens/s
For anyone reading that has never ran local llms, please understand that anything under 100 tok/sec is worthless. You are faster typing stuff into Gemini free version that you get with a google account and copy/pasting it in (and you can easily build browser automation with playwright or any other js runtime to have this available in a chat window)