llama.cpp b10099 ships NVFP4: 4-bit inference gets a CUDA rewrite

The latest release of llama.cpp, tagged b10099 and published on GitHub on July 23, is not a model dump or a new architecture. It is a surgical rewrite of how the most popular open-source inference engine handles 4-bit floating-point quantization on NVIDIA GPUs. The headline change: a fused kernel for NVFP4 W4A4 activation quantization that uses the nvfp4x4 intrinsic when available, cuts memory traffic on each matmul, and makes the 4-bit path viable on Blackwell and Hopper hardware.

The commit history, compressed into a single squashed pull request numbered 25730, reads like a low-level optimization diary. “Add 32-byte loads, restore per-block amax.” “Fuse per-channel amax and quantization kernels.” “Add back scale-search, but optimize it with intrinsics.” Each line is a decision about how to move bytes from DRAM to registers to tensor cores with the least waste. The result is a quantized inference path that does not just reduce model size but also reduces the time spent on the quantization step itself.

What NVFP4 changes

NVFP4 is NVIDIA’s 4-bit floating-point format, introduced with the Blackwell architecture and backported to Hopper through microcode updates. Unlike the integer quantization (INT4) that most open-source inference engines default to, NVFP4 preserves a dynamic range that matters for activations. Activations, the intermediate values that flow between layers, tend to have outliers that integer quantization clips. FP4 keeps those outliers alive at the cost of a slightly larger memory footprint per value.

The b10099 release targets the W4A4 configuration: weights and activations both quantized to 4 bits. This is the aggressive regime. Most production deployments run W8A8 or W4A16, where activations stay at higher precision. Going to W4A4 doubles the theoretical compute throughput of the matmul relative to W8A8, but it also risks accuracy collapse if the quantization scheme is not tight enough.

llama.cpp’s approach, detailed in the commit messages, fuses two operations that were previously separate: the per-channel amax (absolute maximum) calculation and the actual quantization. Fusing them means the data passes through the GPU’s memory hierarchy once instead of twice. The commit also introduces a dedicated scale-search loop, optimized with intrinsics, that finds the best per-block scaling factor rather than using a fixed heuristic.

The intrinsic that makes it work

The critical hardware feature is the nvfp4x4 intrinsic. This is a Blackwell-era tensor core instruction that operates on 4x4 tiles of NVFP4 data. When available, llama.cpp routes the quantized matmul through this path, which keeps the data in registers and avoids the shared-memory round trip that generic fallbacks require.

The commit guards the __builtin_align__(32) struct to NVIDIA only, noting that HIP does not have it. This is a practical constraint: AMD GPUs, which use the HIP compiler, cannot yet run this optimized path. The release ships ROCm 7.2 binaries for Linux, but those use a different quantization backend. The NVFP4 rewrite is NVIDIA-specific.

This matters for the broader ecosystem. The open-source inference stack has historically been GPU-agnostic, with llama.cpp supporting CUDA, ROCm, Vulkan, SYCL, OpenVINO, and CPU backends. But the most advanced quantization techniques are increasingly tied to specific hardware intrinsics. A fused NVFP4 kernel that uses nvfp4x4 cannot run on an AMD MI300X or an Intel Gaudi 3. The gap between what is possible on NVIDIA and what is possible on everything else is widening, and b10099 is a concrete example.

What got disabled

The release notes also include a notable absence. The macOS Apple Silicon (arm64) build with KleidiAI enabled is listed as “DISABLED” with a link to pull request 23780. KleidiAI is Arm’s library for optimized ML kernels on its CPUs. Its disablement suggests either a compatibility issue with the new quantization path or a performance regression that the maintainers chose to gate.

This is a reminder that llama.cpp ships for over a dozen platform targets, from Ubuntu s390x mainframes to Windows ARM64 with OpenCL Adreno. Each target requires its own kernel selection and build configuration. The NVFP4 rewrite touches only CUDA, but it adds complexity to the build matrix that the project’s CI system, running on GitHub Actions, must validate.

What this means for AI builders

For anyone running local inference on an NVIDIA GPU, b10099 is a drop-in upgrade that improves both speed and memory efficiency for models that support NVFP4 quantization. The fused kernel reduces the overhead of on-the-fly quantization, which matters for interactive use cases like chat or code completion where latency per token is the metric that matters.

The release also signals that the open-source inference stack is tracking NVIDIA’s hardware roadmap closely. Blackwell’s nvfp4x4 intrinsic is not a gimmick. It is a concrete performance lever that llama.cpp now pulls. Builders who deploy on Blackwell GPUs will see the benefit immediately. Those on older architectures, like Ampere or Ada Lovelace, will fall back to the generic path.

But the hardware lock-in is real. The same commit that adds the nvfp4x4 intrinsic also guards it behind a compiler flag that only NVIDIA supports. AMD’s ROCm stack has no equivalent. Intel’s SYCL backend does not either. The open-source community now faces a choice: maintain a unified codebase that runs everywhere at reduced performance, or fork into hardware-specific optimization branches.

The open-source inference engine as a moving target

llama.cpp b10099 is a small release in terms of user-facing features. No new model format. No new API endpoints. No new UI. But it is a meaningful release in terms of engineering. It takes a quantization path that existed in theory and makes it fast enough to use in practice.

The commit history, with its squashed WIP commits and compiler massaging to avoid unnecessary LDCs, is the kind of work that does not make headlines. It is the kind of work that makes inference 20 percent faster on the same hardware, and that is the edge that open-source engines need to stay relevant against proprietary APIs.

The outstanding question is whether the rest of the hardware ecosystem can catch up. NVIDIA is moving fast on quantization formats, and the open-source stack is following. AMD and Intel have their own formats, but they lack the library support and the developer mindshare to get the same level of optimization. For now, the fastest path to 4-bit inference on local hardware runs through CUDA and the nvfp4x4 intrinsic. b10099 proves that the path exists.