llama.cpp b9968 brings int8 MoE prefill to Adreno GPUs

llama.cpp tagged b9968 on July 12, and the headline change is a quiet earthquake for on-device AI. The release adds int8 dp4 dense and mixture-of-experts (MoE) prefill optimizations for Adreno GPUs, co-authored by Li He of Qualcomm. This is not a routine backend tweak. It is a direct assault on the memory and compute bottleneck that has kept serious MoE models off mobile hardware.

MoE architectures route each token through a subset of expert sub-networks, not the full model. That sparsity saves flops at inference time but creates a memory-access pattern that punishes conventional GPU kernels. The prefill phase, where the model processes the entire input prompt in parallel, is especially brutal on mobile GPUs because it demands high-bandwidth random access to expert weights. Most mobile inference runtimes punt on MoE prefill entirely, falling back to slower CPU paths or simply refusing to load models with more than a handful of experts.

b9968 changes that. The new OpenCL kernel uses int8 dot-product-accumulate (dp4a) instructions, a hardware feature of Qualcomm’s Adreno 7xx and 8xx series GPUs. Dp4a packs four int8 values into a 32-bit register and computes four multiply-accumulates in a single instruction. Combined with a GEMM (general matrix multiply) kernel written specifically for MoE routing, the result is a prefill path that keeps the GPU busy rather than stalling on memory fetches.

The release notes list builds for Windows arm64 with OpenCL Adreno support, alongside the usual macOS, Linux, and Android variants. Notably, the macOS Apple Silicon build with KleidiAI enabled is marked as DISABLED, pointing to a separate pull request (#23780) that appears to have been reverted or paused. The Adreno optimization, by contrast, ships live.

This is the second time in recent months that Qualcomm has contributed directly to llama.cpp. Li He’s co-authorship on the b9968 commit signals a deeper engagement from the chipmaker’s engineering team. Qualcomm has been positioning the Snapdragon platform for on-device AI since the Snapdragon 8 Gen 2 introduced dedicated AI acceleration, but most of that effort has gone into Qualcomm’s own AI Engine and SNPE runtime. Contributing to an open-source project that runs on competing hardware too is a shift in strategy.

The practical effect is measurable. MoE models like Mixtral 8x7B and Qwen2.5-MoE have been usable on desktop GPUs for months, but mobile inference has lagged. A phone with an Adreno 740 or 750 GPU can now run a Mixtral-class MoE prefill at usable speeds. The exact throughput depends on model size and prompt length, but the bottleneck has moved from “will it run” to “how fast.”

For the broader AI ecosystem, this matters because MoE is the dominant scaling strategy for frontier labs. OpenAI, Google DeepMind, and Mistral all use MoE in their largest models. The architecture allows them to grow total parameter count without proportionally increasing inference cost per token. But that advantage has been locked to datacenter hardware. If MoE inference becomes practical on devices, the economics of on-device AI change.

Consider what happens when a phone can run a 7-billion-parameter MoE model at interactive speeds. Cloud inference costs vanish for that session. Latency drops to zero network round trips. Privacy improves because the data never leaves the device. The trade-off has always been quality: smaller dense models underperform larger MoE models on reasoning and knowledge tasks. Closing that gap on hardware that fits in a pocket is the prize.

b9968 does not close the gap entirely, but it narrows it. The int8 dp4a kernel is a low-level optimization that any MoE model can exploit without modification. It does not require model re-quantization or architecture changes. That is the power of llama.cpp’s design philosophy: optimize the runtime, not the model.

The release also ships builds for ROCm 7.2, OpenVINO 2026.2.1, SYCL (FP32 and FP16), and s390x Linux. The breadth of backends is itself a statement. llama.cpp now targets AMD GPUs via ROCm, Intel GPUs via SYCL, Qualcomm GPUs via OpenCL, and Apple Silicon via Metal, plus Vulkan as a cross-vendor fallback. No other inference runtime covers this much hardware with a single codebase.

What is missing from b9968 is equally instructive. There is no CUDA-specific MoE prefill optimization in this release. NVIDIA GPUs already handle MoE well through Tensor Cores and its own kernel libraries. The gap is on non-NVIDIA hardware, and that is where the open-source community and hardware vendors are now investing.

Qualcomm’s contribution to b9968 is a signal to developers: the Snapdragon GPU is a first-class target for open-source AI inference, not an afterthought. For AI builders shipping on-device features, that means one fewer reason to default to Apple Silicon or NVIDIA Jetson for edge deployment.

The last word belongs to the disabled KleidiAI build. KleidiAI is Arm’s kernel library for AI acceleration, optimized for Cortex-X and Cortex-A CPUs. Its absence in b9968 suggests ongoing integration work or a performance regression. When it returns, it will complement the Adreno GPU path with a CPU fallback that is itself hardware-optimized. That combination, GPU for prefill and CPU for token generation, is the standard pattern for on-device inference. b9968 delivers half of it today.