llama.cpp v0.8.0 ships realtime voice and 15 new model families

The most interesting thing in llama.cpp v0.8.0, released October 28, is not the model count. It is that the project now speaks audio in real time, in both directions, over an OpenAI-compatible wire protocol. The release wires a new /v1/realtime endpoint for conversation and transcription and a POST /tts endpoint for text-to-speech, backed by a new experimental mtmd_audio_stream API that turns PCM samples into decoder-ready embedding rows as they arrive. For a project that began as a C/C++ port of Meta’s LLaMA and grew into the default way people run quantized models on their own hardware, this is a category shift: llama.cpp is no longer just a chat-completions server. It is now a voice server.

That matters because the realtime protocol is the one OpenAI defined for its own speech products. Implementing it in llama.cpp means a developer can point a client built for OpenAI’s realtime API at a laptop running LFM3-Omni-2B and get streaming voice chat without touching a hosted endpoint. The release notes are explicit that the endpoint handles both conversation and transcription, with response.done sent after committed audio drains. That last detail is the unglamorous plumbing that makes a voice loop feel responsive rather than laggy, and it is the kind of thing that only gets fixed when someone is actually trying to ship a voice product.

The model list is the tell

Count the architectures added in this single release. LiquidAI’s LFM3 family in four sizes (LFM3-1.7B text, LFM3-VL-2B vision, LFM3-Audio-2B, LFM3-Omni-2B), plus LiquidAI d1-2B and d1-omni-2B decision models. DeepSeek V4.1 Flash. SenseNova U1. Beam from Reflection. Kolibri-1 from Aleph-Alpha. Flint. Muse Spark Open in text, vision and audio. Granite Switch Decision, including a 256-way decision head. Nemotron-3.5-Embed with non-standard GQA and BPE tokenizer support. H2O-Lightning-4B. MiniCPM-OCR 1.0 and MiniCPM-OCR-Jev. Soprano TTS.

That is a dozen-plus families in one minor version bump, and the composition is the story. The frontier labs that dominate hosted inference (OpenAI, Anthropic, Google) are largely absent from this list. What is here is the second tier: LiquidAI, DeepSeek, IBM’s Granite line, NVIDIA’s Nemotron, Baidu’s SenseNova, and a cluster of smaller European and Chinese shops. The pattern is that the models willing to publish weights, or at least publish architectures that can be converted, are the ones that show up in llama.cpp within weeks of release. DeepSeek V4.1 Flash landing in the same release as the LFM3 family is not a coincidence; it is the cadence of an ecosystem where open-weight releases are the input and llama.cpp is the distribution channel.

The LFM3 entries are worth dwelling on. The release says LFM3-1.7B and LFM3-VL-2B were “verified against the reference HF implementation,” which is the project’s way of saying the port was checked against LiquidAI’s own PyTorch code rather than eyeballed. LFM3-Omni-2B is the one to watch: ASR, TTS and voice chat from a single 2B model. If a 2B omni model runs acceptably on a laptop, the economics of voice interfaces change. You no longer need a hosted speech-to-text vendor, a hosted LLM, and a hosted text-to-speech vendor stitched together with three API keys and three latency budgets. You need one GGUF file.

Sparse attention gets a fused op

The core changes are less visible but arguably more consequential for anyone running large mixture-of-experts models on consumer hardware. Sparse attention (QSA/DSA) was reworked onto a fused masking op, GGML_OP_TOP_K_MASK, added to ggml v0.28.0 with a Metal implementation in the same release. The selection is now built without kq_mask, and k-pool views are shared between layers. In plain terms: the sparse-attention path got cheaper and less memory-hungry, which is exactly what you want when the model is too big to fit comfortably.

The MoE memory work is the other half. Experts kept in host memory now use a 50/50 SLRU cache instead of a plain LRU cache. SLRU, segmented LRU, protects frequently reused entries from being evicted by a burst of one-off accesses. For MoE inference, where a router sends each token to a small subset of experts, that protection matters: the hot experts stay resident and the cold ones churn. Whether 50/50 is the right split is an empirical question the project does not answer in the notes, but the direction is correct. The release also adds -1 as a valid expert index in MUL_MAT_ID and ADD_ID, meaning the slot is skipped and the output row zeroed. That is a small semantic change with outsized implications: it lets a model express “no expert here” rather than forcing a routing decision, which is the kind of flexibility sparse models need.

The frontier labs are largely absent from this release’s model list. What is here is the second tier, and that is the point.

The backend grind

ggml v0.28.0 is where the hardware story lives, and it is a story of grinding down latency on every accelerator the project touches. CUDA gets fused MoE top-k and gate/up+GLU with chunked graph launches, plus a top-k fusion for 384 experts. SYCL gets graph record/replay. OpenCL gets Q2_K/Q3_K weights and a lightning indexer. Vulkan gets subgroup reductions and src1 precision. Hexagon HTP was restructured. KleidiAI SVE GEMV/GEMM kernels landed for Arm. WebGPU gets Q2_0 and ZDNN gets flash attention.

Read that list as a map of where local inference is actually happening. It is not just NVIDIA GPUs. It is Qualcomm’s Hexagon DSP, Arm server chips via KleidiAI, AMD via HIP, Intel via SYCL, and a long tail of Vulkan-capable devices. The project is hedging against a future where the cheapest place to run a small model is not a discrete GPU but the accelerator already sitting in a phone or a laptop. The Hexagon HTP rework and the KleidiAI SVE kernels are the clearest signals of that bet.

There is also a security-flavored fix worth noting: grammar recursion depth is now limited and enforced, and deep JSON is rejected before walking it. That closes a stack-overflow vector in a component that parses untrusted model output. It is the sort of fix that never makes a release highlight but prevents a class of crashes in production.

What it means for builders

The practical upshot is that the gap between “hosted API” and “local server” keeps narrowing, and it is narrowing fastest on the multimodal and realtime edges. A year ago, running voice chat locally meant gluing Whisper, an LLM, and a TTS model together yourself. Now /v1/realtime and /tts are endpoints in the same binary, and the Omni model is a single file.

The caveats are real. The streaming audio API is explicitly experimental. The realtime endpoint implements OpenAI’s protocol, which means it inherits that protocol’s assumptions and will drift as OpenAI changes them. And the model list, for all its breadth, is a snapshot of what the open-weight world shipped recently; the frontier closed models are not here and will not be.

The thing to watch is whether the Omni path holds up under real latency budgets. A 2B model doing ASR, reasoning, and TTS in one pass is elegant on paper. Whether it feels like a conversation on a mid-range laptop, rather than a walkie-talkie with a delay, is the question the next few months of user reports will answer. The release notes say the plumbing is in place. The benchmarks are not.