llama.cpp 0.4.1 ships ternary MoE support and a JSON schema rewrite

The most interesting thing in llama.cpp v0.4.1, released September 14, is not the model list. It is a one-line bug fix buried in the core changes: GDN normalization was being computed from max instead of rsqrt for affected Qwen, Kimi, and GLM models (#28068). That is a correctness fix, not a performance tweak. Anyone who has been running those architectures through llama.cpp on a laptop or a workstation has, until now, been getting subtly wrong numbers out of the normalization stage.

The headline additions are real but more conventional. The release adds Maple 20B-A1B, a ternary mixture-of-experts architecture that runs on CPU (#27000); Tencent Hy 4 preview support under the hy_v4 tag (#28127); and Spark2.5 (#27868). It also bumps ggml to v0.24.0, which the release notes describe as focused on backend coverage: a new precision-control API plus work across Vulkan, SYCL, Hexagon, and OpenCL.

Ternary MoE on CPU is the tell

Ternary weights are the part worth pausing on. Maple 20B-A1B joins the small but growing set of models that quantize activations or weights to three values, roughly {-1, 0, +1}, which collapses multiply-accumulate into additions and sign flips. The architecture is a mixture of experts, so only a slice of the 20B parameters is active per token, and the release explicitly notes CPU support. That combination matters: ternary plus MoE plus CPU is a bet that useful inference does not require a GPU at all.

llama.cpp has been making that bet for years, but the model side has only recently started meeting it. The project’s 128k GitHub stars and 23.2k forks are a rough proxy for how many people want to run models on hardware they already own. Every new architecture merged here is a signal that a lab somewhere decided the GGUF path was worth the conversion effort. Tencent shipping an hy_v4 preview into llama.cpp before or alongside its own serving stack is the kind of detail that used to be unusual and now is not.

The unglamorous work is where the value is

Scan the changelog and most of it is plumbing. A --fuse-qkv flag fuses Q, K, and V tensors during Hugging Face to GGUF conversion (#22780). JSON schema handling gets a common_schema internal representation (#28736). Structured logging arrives via --log-jsonl and a LOG_JSON macro (#28437, #28586). The deprecated --mmap, --mlock, and --direct-io flags are removed in favor of --load-mode (#28334).

That last one will break scripts. So will the API change to llama_sampler_chain_n(), which now returns int32_t instead of int (#28631). These are the costs of a project that has stopped treating itself as a demo. The server gets a refactored child-process monitor thread (#28555), an LRU hang fix for repeated requests on the same model (#28539), and a fix for context checkpoint eviction on short prompts (#28302). None of this is exciting. All of it is the difference between a tool you tolerate and a tool you deploy.

The MTP context KV cache allocation fix for DeepSeek2 and GLM-MoE (#28630) belongs in the same bucket. So does the Kimi-K3 recurrent-state rollback support (#28466). Recurrent-state rollback is what you need when a model keeps state across a sequence and you want to rewind without recomputing from scratch. It is the kind of feature that only makes sense once people are actually running these models in loops, not in one-shot demos.

Ternary weights plus mixture of experts plus CPU support is a bet that useful inference does not require a GPU at all.

What the bug fix implies about the ecosystem

The GDN normalization error is the most consequential item here, and the release notes give it one line. A max versus rsqrt swap in normalization is the sort of thing that produces plausible-looking output that is quietly off. It affects Qwen, Kimi, and GLM, three of the most-downloaded open-weight families. If you have been benchmarking any of them through llama.cpp and getting numbers that did not match a reference implementation, this is a candidate explanation.

That llama.cpp is where the fix landed first, rather than in the upstream model repositories, says something about where the de facto reference implementation now lives. The project is not just a runner. It is increasingly the place where architecture details get pinned down, because it is the common denominator across so many backends: CUDA, Metal, Vulkan, SYCL, Hexagon, OpenCL, and plain CPU. The ggml 0.24.0 bump brings a CUDA fallback to F32 on devices without BF16 hardware acceleration (#28846) and a Vulkan workaround for an NV queuesubmit driver bug (#28830). The project absorbs hardware quirks so its users do not have to.

What to watch

Two things. First, whether Maple 20B-A1B’s ternary MoE approach shows up in other models, because if it does, the CPU inference story gets materially better and the argument for local serving gets stronger. Second, whether the --load-mode migration and the int32_t return-type change cause enough downstream breakage to matter. The project has a long history of deprecating things slowly, and this release is more aggressive than usual.

For builders, the practical read is simple. If you ship anything on top of llama.cpp, pin your version before upgrading, and re-run your evals against Qwen, Kimi, or GLM if you depend on them. The normalization fix alone could move your numbers.