llama.cpp 0.3.0 ships tensor-split for DeepSeek 4, dots3-note multimodal

llama.cpp, the open-source project that made running large language models on a laptop a normal thing, hit version 0.3.0 on August 25. The release notes, published by maintainer Georgi Gerganov under the ggml-org umbrella, read like a checklist of things that used to be impossible on local hardware. Tensor-split for DeepSeek 4. A new multimodal model called dots3-note with a novel KV cache. Multi-token prediction for GLM-4.5-Air. Underneath it all, a significant rework of the ggml backend that touches every GPU family the project supports.

The headline feature is tensor-split for DeepSeek 4, enabled with -sm tensor (pull request #26490). This is not the same as the layer-split approach that has been llama.cpp’s default for multi-GPU setups. Tensor splitting carves individual model tensors across devices, so a single attention head or feed-forward layer can span two GPUs. That matters for models like DeepSeek 4 whose largest variants are too big for any single consumer card. Layer splitting has a hard constraint: each layer must fit entirely on one device. Tensor splitting relaxes that, which is why llama.cpp also fixed meta tensor split state propagation for tensor parallelism in #27574. The two fixes together mean a user with two mid-range cards can now run a model that previously required a single expensive one.

The release also fixes a nasty bug in DeepSeek 4’s rollback with multiple sequences (#26756). Rollback is what happens when a generated sequence is rejected and the model has to rewind its KV cache to a previous state. With multiple concurrent sequences, the bookkeeping got corrupted. That is the kind of bug that only shows up under real server load, and its presence in the changelog says something about where llama.cpp is being used: not just by hobbyists on a single prompt, but as an inference backend for concurrent traffic.

The dots3-note addition and a new KV cache

The most interesting new model in this release is dots3-note, added via #27060. It ships with a new KV cache type called DSA-ISWA, and the mtmd multimodal module now supports its vision and audio inputs (#27524). The release notes do not specify what DSA-ISWA stands for, which is typical for this project’s terse style. What is clear is that llama.cpp is now carrying a second multimodal architecture alongside the existing CLIP-based models, and that the project’s maintainers judged the cache design worth a dedicated implementation rather than a workaround.

The mtmd changes go beyond dots3-note. WebP image decoding via ffmpeg (#27520), a fix for videos with the moov atom at the end of the file (#27596), and a Pillow-accurate resize algorithm (#27594). That last one matters more than it sounds. Image preprocessing is a silent compatibility killer in multimodal models. If your resize algorithm differs from the one used at training time, the model sees slightly distorted inputs and quality degrades. Matching Pillow’s behavior, the reference Python image library, is the pragmatic way to ensure local inference matches what the model authors benchmarked.

ggml v0.22.0: the backend underneath

The ggml library, which is the compute engine under llama.cpp, bumps to v0.22.0 in this release. The changelog lists three structural changes. First, tensor-split support in the multi-backend (meta) backend with improved split-state propagation. Second, Metal kernels reworked into per-op sources with parallel compilation. Third, a fix making ggml_clamp a proper non-in-place operation.

The Metal change is worth pausing on. Apple’s GPU compiler is slow, and compiling a monolithic kernel file for every new model variant has been a recurring pain point for Mac users. Splitting kernels into per-operation sources with parallel compilation directly attacks that latency. The release notes also mention per-device tuned parameters for flash-attention vectors on Metal (#26570), which suggests the project is still squeezing performance out of Apple Silicon rather than treating it as a secondary platform.

Elsewhere in the backend: new POOL_1D and PAD_REFLECT_1D ops, Q2_K SYCL kernels, MoE bias fusion on OpenCL, and fixes across CUDA, Metal, SYCL, Vulkan, and WebGPU. The breadth of backend coverage is itself a statement. llama.cpp remains one of the few serious inference stacks that treats AMD, Intel, Apple, and NVIDIA as first-class citizens.

What this release signals

Read the changelog as a whole and a pattern emerges. This is not a release about new model capabilities. It is a release about distributing compute across whatever hardware you happen to own. Tensor-split for DeepSeek 4, split-state propagation fixes, per-op Metal kernels, MoE bias fusion on OpenCL. Every significant change is about making inference faster or more memory-efficient on hardware that is already in the field.

That is the quiet argument at the center of llama.cpp’s continued relevance. The frontier labs are building data centers with tens of thousands of accelerators. llama.cpp is building the opposite: software that lets one person with two GPUs run a model that would have been server-only a year ago. The dots3-note addition, the GLM-4.5-Air MTP support, the DeepSeek 4 fixes. These are all recent frontier models, and they are all landing in a project whose stated goal is running them locally.

There is also a server-side story here. The new LLAMA_SERVER_SLOTS_N_DIFF environment variable (#27600) widens a debug window for slot state, and the slot fitting logic moved into the common library where it now accounts for n_streams (#27496). These are operational details for people running llama.cpp as a production inference server. The web UI gained tabbed chat navigation (#27263), which is a small thing but signals that the project is not abandoning its casual users.

The gap that remains

For all the progress, the release notes are honest about limits. One test for the dots3-note architecture is disabled on WebGPU (#27654). The SYCL backend marks the tq2_0 quantization type as unsupported (#27660). WebGPU still has issues with infinity values during ARGSORT and TOP_K (#27538). The project is shipping across more backends than any commercial inference stack, and each one has its own edge cases.

The deeper limitation is not in the code. It is in the model supply. llama.cpp can now run DeepSeek 4 with tensor-split, but it depends on the model’s weights being released in a usable form. The project’s velocity tracks the open-weight ecosystem, and that ecosystem is the real constraint on what local inference can do. When a lab releases weights, llama.cpp tends to support them within weeks. The release cadence of this project, now at version 0.3.0 with nightly build b10621, is a reliable barometer for how fast open models are moving.

The takeaway for AI builders is straightforward. The cost curve for running serious models locally keeps dropping, and this release is a concrete step on that curve. Tensor-split means the multi-GPU setups that were once the domain of layer-splitting workarounds now have a proper path. The Metal kernel rework means Mac users get faster compile times. The multimodal additions mean local vision and audio are not just a CLIP-only story anymore.

What to watch next is whether the tensor-split path generalizes. DeepSeek 4 got it first. If the pattern holds, the next few releases will extend -sm tensor to other large models, and the line between “runs on a workstation” and “requires a cluster” will move again. That line has been llama.cpp’s territory since 2023, and version 0.3.0 keeps pushing it.