llama.cpp b9946: Qualcomm's Hexagon backend gets a serious upgrade

llama.cpp shipped b9946 on July 9, and the headline is not a new model or a flashy feature. It is a deep, architectural overhaul of the Hexagon DSP backend — the code path that runs inference on Qualcomm’s embedded digital signal processors. The release, tagged b9946, is a reminder that the real frontier in AI software is not just making models smarter. It is making them run on the hardware that already exists in phones, cars, and IoT devices.

The core change is a set of optimizations for unary operations — element-wise functions like activation layers (ReLU, sigmoid, tanh) that are simple in theory but expensive at scale. On Hexagon, these operations hit a specific bottleneck: the Vector Tensor Coprocessor Memory (VTCM), a fast on-chip scratchpad that is large but not infinite. When a unary op processes a wide row of tensor data, it can overflow VTCM, forcing expensive spills to slower memory.

The b9946 fix is tiling. The commit message — “hexagon: tile wide rows in pointwise unary ops to avoid VTCM overflow” — describes a strategy that breaks wide rows into smaller chunks that fit inside VTCM. This is not new in principle. Tiling is a standard technique in GPU compute kernels. But applying it to the Hexagon DSP, with its specific memory hierarchy and thread model, required significant engineering.

The release includes a suite of related changes. The team replaced division operations in the unary kernel with fastdiv, a technique that converts divides into multiplies and shifts. They added a VTCM layout and host-computed kernel parameters, moving initialization logic out of the kernel hot path. They introduced specialized thread functions to improve generated code quality. And they added tracing instrumentation specifically for unary ops, giving developers visibility into kernel execution on the DSP.

The commit also factors out HVX (Hexagon Vector eXtensions) kernels into a separate code path, reducing duplication and streamlining the backend. This is the kind of cleanup that makes future optimizations easier. A separate fix addresses a std::min collision with the Windows min macro — a reminder that cross-platform C++ still has sharp edges.

What is not in the release is almost as interesting. The macOS Apple Silicon build with KleidiAI enabled is listed as “DISABLED”, with a link to pull request #23780. KleidiAI is Arm’s open-source library for efficient ML kernels on Arm CPUs. The fact that it is disabled suggests either a compatibility issue or a performance regression that the team is still debugging. For a project that prides itself on supporting every plausible target, a disabled build is a signal worth watching.

The broader picture is this. llama.cpp has become the default runtime for local inference across an absurd range of hardware. The b9946 release ships binaries for macOS (arm64 and x64), Linux (x64, arm64, s390x), Windows (x64, arm64), Android (arm64), and multiple GPU backends including Vulkan, ROCm 7.2, CUDA 12 and 13, OpenVINO, SYCL, and HIP. The Hexagon backend is one piece of a much larger machine.

But it is an increasingly important piece. Qualcomm’s Hexagon DSP is embedded in Snapdragon chips that power hundreds of millions of Android phones. Running inference on the DSP, rather than the CPU or GPU, can dramatically reduce power consumption and thermal load. For on-device AI applications — voice assistants, real-time translation, camera processing — that efficiency advantage is decisive.

The b9946 release is not going to make headlines in the AI press. It does not announce a new model with a higher benchmark score. It does not claim to run a 70-billion-parameter model on a phone. What it does is make the existing hardware work better for the operations that actually get used in production. Unary ops are everywhere. Every activation function, every normalization layer, every element-wise transform runs through them. Optimizing them on the DSP is a direct improvement to the user experience of any on-device AI application built on llama.cpp.

The commit history shows the work was co-authored by Max Krasnyansky at Qualcomm’s QTI division. That is significant. It means Qualcomm is not just providing documentation and hoping developers figure out the hardware. It is actively contributing code to the open-source project that runs on its chips. That is a vote of confidence in llama.cpp as the runtime of choice for on-device inference, and a signal that Qualcomm sees the DSP as a first-class AI compute target.

The Hexagon backend is still not the main path for most users. The CUDA and Vulkan backends handle the bulk of inference on desktop and server hardware. But the mobile and embedded market is where the growth is. As models get smaller and more efficient — think Phi-3, Gemma 2, or the upcoming Llama 4 Nano — the ability to run them on DSPs becomes a competitive advantage. The phone that can run a local model without draining the battery or heating up wins.

There is a lesson here for AI builders. The easy wins in inference optimization are gone. The low-hanging fruit — quantizing weights, swapping in a faster attention kernel, using a smaller model — has been picked. The next generation of improvements will come from deep, hardware-specific work: tiling for a DSP, fastdiv for a vector unit, tracing for a custom memory hierarchy. That work is hard, slow, and invisible to most users. But it is what makes the difference between a model that runs and a model that ships.

b9946 is a release about engineering discipline. It is about making the right thing easy on a specific piece of hardware. For anyone building on-device AI applications, that is the kind of progress that matters.