llama.cpp v0.5.0 ships Qwen4 sparse attention and a multi-address server

Georgi Gerganov’s llama.cpp cut v0.5.0 on September 23, and the version number is the least interesting thing in it. The project has been shipping nightly builds for years under a b11146-style tag; jumping the semantic version to 0.5.0, off a 0.4.1 predecessor, signals that the maintainers now treat this as a stable interface rather than a rolling experiment. The release itself is three commits past the previous tag, but the changelog behind it is enormous: implicit-GEMM CUDA conv2d (#29135), Metal MoE and SSM_CONV fusion (#28948), multi-address HTTP binding (#28690), and a bump of the underlying ggml library to 0.25.0.

Here is the take. llama.cpp stopped being a hobbyist’s way to run a quantized model on a laptop some time ago. v0.5.0 is the release where it starts behaving like infrastructure, and the evidence is less in the flashy kernel work than in the boring server fixes nobody will tweet about.

The kernels are the headline, but not the story

The performance work is real. CUDA conv2d now runs through implicit GEMM, which matters for vision encoders and the multimodal path, not for text generation. Metal picks up MoE and SSM_CONV fusion on Apple silicon, which is where a meaningful share of llama.cpp’s user base actually lives. ggml 0.25.0 expands hyper-connection, flash-attention, and fused MoE/SSM support across backends, and the RPC protocol moves to major version 7.

The model coverage list is where the AI-research read sits. The release adds HRM-Text / DFM Mimir 1B support (#27625), MiMo-V2.6 conversion (#29257), DFlash for HunyuanOCR (#28890), extended Nemotron MTP and Nemotron-H handling (#29018, #28989), and, notably, Qwen4Exp hyper-connection ops plus sparse flash attention (#28901, #28770). Sparse attention in a local runtime is not a small thing. Sparse and hyper-connected architectures are the direction several labs have pushed to cut the quadratic cost of long context, and llama.cpp now has kernels for them on CUDA and Metal (metal : support qwen4exp hc ops, #29000). When a research architecture lands in a weekend-runnable runtime within weeks, the gap between a paper and a laptop closes again.

That is the pattern worth naming: llama.cpp has become the fastest path from “a lab published weights” to “someone is running them on a 4090.” The release also adds Ling 3.0 (Bailing V3), DeepSeek V3.2/V4, qwen3-coder, Muse Glimmer, and Gemma 4 parser fixes (#28682, #29008, #28869, #29242, #29115). The chat-template surface is now a maintenance burden in its own right.

The server is quietly becoming a serving stack

The most consequential changes are in the server and router directories, and they are defensive.

The router now fixes eviction races with its existing queue (#29217) and stops forwarding the log file or API key file to router-spawned child instances (#29212, #28938). That second one is a security fix dressed as housekeeping. If you run llama.cpp’s router in front of multiple models and it was handing its own API key file path to children, you had a credential-propagation problem. The release also dedups the draft HF model via dedup-cache-models (#27934), improves startup and model-source logging (#29125), and vendors cpp-httplib 0.57.1 (#29239).

The --host flag now accepts comma-separated TCP addresses and UNIX sockets (#28690). Multi-address binding is the kind of feature you add when people are deploying this behind reverse proxies and on dual-stack networks, not when they are demoing on localhost. Combine that with input_image support in server function-call outputs (#22575), OpenAI video_url content types and data: video URIs (#27921), and WEBM uploads in the UI (#28622), and the picture is a runtime that wants to be a multimodal API server, not a CLI.

When a research architecture lands in a weekend-runnable runtime within weeks, the gap between a paper and a laptop closes again.

What the API changes tell you about the users

The API surface moved in ways that only matter to people embedding llama.cpp in something else. llama_adapter_lora_init_from_file_ptr() loads a LoRA from an open FILE (#28993), and the docs now spell out that llama_model_load_from_file_ptr() reads from the current position and requires aligned mmap. LLAMA_VOCAB_TYPE_TEST adds a dummy tokenizer for testing (#29084). Environment variables now cover temperature, top-p, min-p, and penalties (#27380), so you can tune sampling without rewriting a command line. The sampler backend probe shrank (#29285).

None of this is glamorous. All of it is the difference between a project you fork and a library you depend on. The LoRA-from-FILE change in particular reads like someone needed to load adapters from a stream they did not control, which is a production constraint, not a tinkering one.

There is also a correctness layer that suggests the maintainers are now chasing silent bugs rather than crashes. The release fixes tensor-parallel split state and granularity for fused QKV models (#28965), a Mamba time-step projection contiguity bug (#28832), a SigLIP bucket buffer overrun for tall or wide images (#29276), a CUB argsort corruption from in-place keys (#28389), and a ggml_permute dimension and stride truncation bug (#29227). It adds sanity checks for mtmd layer indices, SAM layer counts, resize targets, and graph allocation (#29276, #28149), plus allocation-failure checks to prevent crashes (#28149, #26070). A project that adds buffer-overrun checks for extreme aspect ratios has users feeding it real images at real sizes.

The hardware story underneath

The backend spread is the tell for where compute is going. This release touches CUDA, Metal, SYCL, Vulkan, OpenCL, WebGPU, Hexagon, and the CPU path. Hexagon gets an HMX-optimized GATED_DELTA_NET (#29199), 64-bit DMA mappings (#29197), and flash-attention head-dim padding for DK=DV=72 (#26539). Vulkan adds Intel Xe flash-attention kernels for Xe-LPG Plus, Xe2, and Xe3 (#24406), plus IQ4_XS and IQ3_S matmul kernels (#28415, #28822). SYCL extends MMVQ GLU fusion and adds rms_norm+scale and ssm_conv+silu fusions (#28931). OpenCL adds a pile of dp4a binary kernels for Q4_0, Q6_K, and Q8_1.

The message: llama.cpp is no longer an NVIDIA-and-Apple project. Intel, Qualcomm, and AMD paths are getting first-class kernel work, and the fused-op pattern (rms_norm+scale, ssm_conv+silu, gated delta nets) is the same one the big serving stacks use. The gap between “runs on my GPU” and “runs everywhere” is closing on the low end of the hardware spectrum, which is exactly where the cost-sensitive inference happens.

What to watch

The version bump to 0.5.0 is a promise the project has not made before: that the API is stable enough to build against. The ggml RPC protocol moving to major v7 undercuts that slightly, since a major protocol bump is a breaking change by definition. Watch whether the next few releases keep the server and API surface steady or keep churning it. The other thing to watch is Qwen4Exp sparse attention. If the kernels hold up under long-context load, llama.cpp becomes a legitimate testbed for sparse-attention research outside a cluster, and that changes who gets to experiment with the architectures that matter.