Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385

When that article was first published , I started looking into llama.cpp. With the help of an LLM I researched the knobs to turn that made most sense.

Things went from "local models are useless unless you have a 512GB GPU I guess" to "oh.. I can do a lot of stuff locally now!"

Had a similar experience. Llama.cpp compiled natively; parameter sweep to find best options fitting my use case for the qwen models with 16GB VRAM. The whole thing packaged into a portable container.

From the Ollama docs for this new model:

Ollama's MLX engine provides state-of-the-art performance on Apple Silicon, with support for DFlash and image input:

ollama run muse-glimmer:30b-mlx

What do you use instead?

If you need a GUI, Unsloth Studio and LM Studio are both great frontends for llama.cpp. If you don't need a GUI, llama.cpp is the business for single-user deployments. Easy to use, always gets new model support very quickly, built-in Hugging Face client/cache support, works on probably everything (Mac, ROCm, CUDA, Vulkan, etc.).

I use llama.cpp w/ llama-swap

https://github.com/ggml-org/llama.cpp https://github.com/mostlygeek/llama-swap

FYI, llama-server can now be run in router mode so llama-swap is probably only needed for more exotic scenarios.

I'm running it in router mode, but people on Reddit were recommending people use llama-swap instead. Am I missing something by using router mode?

I believe it's useful for running multiple llama.cpp forks at the same time (e.g. a model you want requires special patching)

I’m using llama-swap because it can manage arbitrary backends, not just llama-server instances. I have llama.cpp chat and embedding models running alongside whisper-server all behind a single endpoint with per-model TTLs so they don't fight over the limited vram I have available on this box. Native routing could replace the llama.cpp part but not whisper so I guess I'm exotic ;)

Thanks for the link to llama-swap. Didn’t know about it and will definitely install it.

try oMLX or vMLX - both great projects that offer some amazing performance optimizations for Apple Silicon that utilize UMA and NVME caching efficiently.

https://omlx.ai https://vmlx.net

That said, it's been a few weeks since I've looked so maybe llama.cpp has those features now... they really do move that quickly.

The link in that discussion has a section about that

Jan is so much better.

lm studio

seems to underperform on Terminal Bench compared with qwen3.6-27b: 51.7 vs 60.7

To be honest, I never give benchmarks a look. I just use the models for whatever I need to work on, so I can't really make comparisons that are useful for other people.

Newb question but I’m curious what would help it to run faster? Would it need more vRAM or just system memory?

The biggest gain you'll get is faster memory, provided you have enough capacity to load all the weight into vram. The DGX sparks and Apple silicon memory bandwidth (and also memory access latency) drag down the decode speed quite a bit.

I have two GPU rigs both with 2x RTX Pro 6000, can get ~250 tk/s decode with deepseek-v4-flash in native mixed precision. For context, in antirez's dwarfstar project he only gets ~20-40 tk/s on the same model @ 2bpw on M5 Max.

The latter is for sure usable if it's your only option, but it's really hard for me to personally go back to speeds like that when I've experienced the former.

(Also worth noting dwarfstar only has experimental support for dspark spec dec, when that lands it will definitely give a big boost at higher acceptance rates)

It runs very quickly on my RTX 5090 fwiw. Whole thing is loading entirely into vRAM with a ~130k context size (the max) fitting as well.

That's a $5k 32GB card for anyone who doesn't know all these off the top of their heads (like myself).