llama.cpp shipped v1.0.0 on November 13, and the version number is the least interesting part of it. The release deletes 35 deprecated functions from llama.h, promotes the NextN speculative-decoding API out of llama-ext.h, adds OCP FP8 KV caches, and introduces a host-memory MoE expert cache with a configurable CPU share. ggml bumps to 1.0.0 alongside it. For a project that has spent years as the de facto reference implementation for running quantized models on anything with a memory bus, crossing 1.0 is a statement about API stability. The actual substance is in what the maintainers chose to freeze.

The pruning is the point

Removing llama_set_warmup, defrag_thold, and 35 other deprecated entry points is a breaking change, and that is the correct call. llama.cpp has accumulated a decade’s worth of compatibility shims because every downstream project, from Ollama to LM Studio to a thousand hobbyist scripts, links against its C surface. Freezing that surface at 1.0 means the next round of changes has a stable target. The NextN API promotion matters more than it looks: speculative decoding was buried in an extension header, which signaled “experimental.” Moving it into llama.h tells every downstream integrator that draft-model acceleration is now a first-class feature rather than a patch they have to track by hand.

The FP8 KV cache work (PR #28392) is the change most likely to matter at scale. Storing keys and values in f8_e4m3 with optional static per-tensor scales roughly halves KV memory versus FP16, which translates directly into longer contexts or more concurrent requests on the same GPU. OCP FP8 is not exotic anymore; it is the format Hopper and Blackwell parts accelerate natively. Putting it in the KV path is a quiet admission that the memory wall, not compute, is what limits local inference.

MoE caching is the real story

The host-memory MoE expert cache (PRs #30966, #31233, #31888) deserves more attention than it will get. Sparse mixture-of-experts models have a brutal property: only a fraction of parameters activate per token, but the full weight set has to live somewhere. llama.cpp now keeps experts in host memory with a configurable CPU share via moe_cache_size and moe_cache_cpu_share, uses LRFU and ClassLRU replacement policies, and crucially does not block on cached experts. That last detail is the whole game. A cache that stalls the GPU while it fetches from system RAM is worse than no cache at all, because it adds latency without adding throughput. Non-blocking retrieval means the scheduler can overlap the fetch with other work.

This is the kind of optimization that lets a 65B-class sparse model run on a consumer box with 24GB of VRAM and 64GB of system RAM. It is unglamorous. It is also how the local-inference crowd keeps pace with cloud pricing.

The memory wall, not compute, is what limits local inference. FP8 KV caches and a non-blocking MoE expert cache are the two levers that actually move it.

New architectures land fast

Three text architectures arrive in one release: Kimi-K3.1-Pro (K3 with full-attention MLA layers swapped for DSA, PR #31779), Ripple-65B-Preview from ricursive (a dense 65B architecture with no new ops, #31692), and xing4_0 with mHC, MLA, MoE and MTP (#31589). Vision support extends to MageVL-Cowork and Kimi-K3.1, and drama-3 TTS from fishaudio joins the multimodal path. The pace is the message. When a single point release absorbs three new model families, the maintainers are running a treadmill against every lab’s architecture churn, and they are winning enough to keep up.

The mtmd refactor (PR #31865) removes clip.h and consolidates model and subcontext infrastructure, then shares the compute buffer across subcontexts (#31945). For models that combine vision, audio, and generation, that is real memory saved per request. Computing log-mel in the graph instead of spawning threads per request (#31808) is the same instinct: fewer moving parts, less per-request overhead.

The server hardening is overdue

The security fixes in llama-server are the part that should worry anyone who has exposed an instance to a network. The release patches SSRF in the CORS proxy and media URL handling (#32115), DNS rebinding (#32098), and child-instance access from the browser (#32120). It adds opt-in flags --allow-unauthenticated-tools and --media-allow-private-urls (#32113, #32115), which means the safe default is now closed rather than open. HF_TOKEN is restricted to huggingface.co hosts (#31730). MCP tool calls get cancelled on client disconnect or timeout, with a notification sent back to the MCP server (#32025).

Read that list again. SSRF, DNS rebinding, token leakage, orphaned tool calls. These are the failure modes of a project that grew from a local demo into a network service without a security review at each step. The 1.0 label is partly a promise that the server is no longer a toy. The fact that these fixes all landed in the same release as the version bump suggests the maintainers agree.

The OpenAI-compatible /v1/audio/speech endpoint (#31763) is the commercial tell. Every local stack now wants to be a drop-in for the OpenAI API so that client libraries written against the cloud work against a laptop. Speech synthesis closes one of the last gaps.

What it means for builders

If you maintain a downstream integration, v1.0.0 is a migration, not a drop-in. The 35 removed functions and the defrag_thold removal will break builds. The NextN API moving into llama.h means speculative decoding is stable enough to depend on. The MoE cache parameters are new knobs worth tuning if you serve sparse models.

The bigger read is about direction. llama.cpp is no longer optimizing for “can it run.” It is optimizing for “can it serve, safely, at scale, on hardware you own.” FP8 KV caches, non-blocking expert caches, and a server that defaults to closed are the features of a production runtime. The 0.x era was about proving the concept. The 1.0 era is about the operational details that decide whether local inference is a hobby or an alternative to the API bill.

Watch the downstream projects in the next few weeks. Ollama, LM Studio, and the various Python bindings will either pin to 0.9.x or do the migration work. Which ones move fast tells you who is treating local inference as infrastructure rather than a demo.