llama.cpp v0.7.0 puts a 2.2x MoE speedup behind one flag

llama.cpp v0.7.0 shipped on October 15, and the headline feature is a memory trick that makes mixture-of-experts models behave like something you can actually run at home. The release adds a GPU cache for MoE experts that live in host memory, exposed as --moe-cache-mib. Experts get pulled onto the GPU from an LRU cache instead of being re-read from system RAM on every token, which the release notes measure at up to roughly 2.2x faster decode when experts are offloaded. The same cache is now shared across multiple GPUs, per pull request #30112, so a two-card box does not maintain two divergent copies of the same expert weights.

That number deserves scrutiny, and the release notes give it to you. The 2.2x applies “when experts are offloaded,” meaning the baseline is the pathological case where every expert lookup stalls on PCIe. If your model already fits in VRAM, this flag does close to nothing. If you are running a large MoE on a single 24GB card with the rest of the weights in DDR5, it is the difference between unusable and interactive. That is the honest framing, and it is a narrower claim than the release title suggests.

The port change is the real breaking change

Buried under the model additions is a line that will break more setups than any kernel work in this release: the server now listens on port 9931 by default instead of 8080, per PR #30159. Docker images keep 8080 inside the container, which means the people most likely to be surprised are the ones running llama.cpp directly on a host and pointing a reverse proxy, a health check, or a client library at the old default. This is a small thing that costs people an afternoon. It is also the kind of thing a project does when it decides it is infrastructure rather than a demo binary, because 8080 collides with everything.

The rest of the server work supports that reading. The chat layer is refactored around a common_chat_session object with feed() and finish() methods (#30210), and formatted output becomes an array of parts instead of a single string (#30389). Slot save and restore now preserves context checkpoints and draft model state (#26004, #30574), and the prompt cache stops saving non-restorable states so rerank and embedding requests no longer pollute it (#30498). These are not features. They are the plumbing you need before you can run llama.cpp as a shared service instead of a one-user toy.

LiliCorr, GLM5-Next, and the speculative decoding pile-up

Speculative decoding gets two additions. LiliCorr (#30506) uses a small draft transformer on top of DFlash for candidate selection, and GLM5-Next gets MTP speculative decoding with an optimized sparse attention path (#29928). There is also a new --spec-draft-n-ubatch flag to size the draft context batch separately, which saves VRAM (#30459). The pattern across all three is the same: speculative decoding is being tuned per-architecture rather than treated as a generic wrapper. That is more work for the maintainers and better results for everyone running a model someone else already profiled.

The MoE cache has a subtler consequence worth naming. If experts can live in host memory and stream to the GPU efficiently, the economics of local inference shift toward bigger, sparser models on cheaper hardware. A 70B-class dense model still needs the VRAM it needs. An MoE with a large total parameter count but a small active set can now lean on system RAM in a way that was previously a trap. This is the same direction the industry has been moving at the datacenter level, where MoE routing is standard, arriving on consumer hardware through a cache flag.

If you are running a large MoE on one 24GB card with the rest of the weights in DDR5, this flag is the difference between unusable and interactive.

A model list that says who is actually shipping

The new model support is a useful census. K2 Horizon arrives in dense and MoVA variants with conversion, tokenizer, chat template, and -sm tensor splitting (#29535). Prism Bonsai 2 27B brings ternary weight support (#29600). EmbeddingGemma-2 handles text, vision, and audio embeddings in one model (#30054). LiquidAI’s d1-3B and d1-omni-600M land alongside vLLM decision 2, vLLM d3, and pplx-decider as “decision models” wired into the /v1/systemone endpoint (#30110, #30114, #30378, #30383, #30044). MiniCPM-V 4.7 (#29416), Nemotron Super 3.5 VL, and Nemotron Bolt 74B A9B (#30520) round out the multimodal additions, with maion-coder (#29778) and Mellum (#30578) for code and MoE gating respectively.

The decision-model category is the interesting one. Five separate small models from three different organizations, all routed through a single server endpoint, suggests a class of task where a 600M-parameter model is the right tool and a frontier model is waste. llama.cpp is positioning itself as the place those get served, which is a different ambition than being the place a chat model gets quantized.

What to watch

Two things. First, whether the MoE cache holds up on real hardware rather than benchmark configurations, because LRU hit rates depend entirely on routing patterns and those vary by model. Second, whether the port change and the common_chat_session refactor push downstream projects to pin versions, which would fragment the ecosystem that llama.cpp’s compatibility has been its main asset. The ggml bump to v0.27.0 also raises the RPC protocol to v8 for tensor-parallel servers (#26610), so mixed-version clusters will not talk to each other. For anyone running llama.cpp in production, this release is worth reading the changelog for before upgrading, not after.