llama.cpp b9982: The Bug That Silently Discarded Per-Request Reasoning Budgets

A bug in llama.cpp’s OpenAI-compatible server silently discarded any per-request override of the reasoning_budget_tokens parameter. The fix landed in release b9982, pushed by a contributor on July 13. The root cause: a copy-loop ordering error in the oaicompat_chat_params_parse function that caused the server to prefer its own default over the value the caller actually sent.

The bug is small, mechanical, and easy to miss. The implications are not.

llama.cpp is the most widely used local inference engine for large language models. It powers everything from single-user chat UIs to production-grade API servers running on consumer hardware. Its OpenAI-compatible server endpoint is the primary way developers integrate llama.cpp into tools, workflows, and multi-model pipelines. When that endpoint silently ignores a parameter, the developer’s intent is lost without any error message.

Here is what happened. The oaicompat_chat_params_parse function read the server-level default (opt.reasoning_budget, typically -1) and the Anthropic-style alias thinking_budget_tokens, but never the canonical reasoning_budget_tokens field from the request body. Because the function wrote the default into llama_params before the generic body-copy loop ran, that loop found the key already present and skipped the caller-supplied value. Any per-request override, including setting the budget to 0 to suppress thinking entirely, was discarded.

The same bug affected reasoning_budget_message, the string injected before the end tag when the budget is exhausted. The server wrote the default message into llama_params before the copy loop, so any per-request override of that message was also silently dropped. The fix reads both fields from the request body first, falling back to the server default only when no caller-supplied value exists.

The fix includes unit tests in test-chat.cpp that exercise the path via oaicompat_chat_params_parse with a Qwen3 template, which the autoparser detects as a thinking-capable model. The test asserts that the returned llama_params carries reasoning_budget_tokens == 0 when the caller requests it.

This is not a dramatic bug. It is a two-line fix, a copy-loop ordering error, a missed fallback. But it is exactly the kind of bug that erodes trust in local AI tooling.

The reasoning budget parameter is central to the emerging class of “thinking” models: DeepSeek R1, Qwen3, and the many fine-tunes that expose a chain-of-thought reasoning phase before generating a final answer. Developers use reasoning_budget_tokens to control how much compute a model spends on reasoning, to cap latency, or to suppress thinking entirely for simple queries. When the server ignores that parameter, the model may spend thousands of tokens reasoning about a question that needs none, or skip reasoning when the application requires it.

The bug also highlights a broader pattern in open-source AI infrastructure. The OpenAI-compatible API surface is sprawling, with dozens of parameters, aliases, and optional fields. The reasoning_budget_tokens field is the canonical OpenAI parameter. The thinking_budget_tokens alias is Anthropic-style. The server tried to support both, but the implementation had a subtle ordering dependency that broke the canonical path. This is the kind of compatibility debt that accumulates when a project grows fast and the test coverage lags behind the feature surface.

llama.cpp’s maintainers have been aggressive about adding OpenAI-compatible endpoints. The server now supports streaming, function calling, structured outputs, vision, and reasoning budgets. Each new feature adds code paths that interact in unexpected ways. The b9982 fix is a reminder that compatibility is not just about parsing the right fields. It is about ensuring the values propagate correctly through every layer of the stack, from the HTTP request parser to the sampling layer.

The fix also reveals something about the state of local AI. The reasoning budget parameter is not a niche feature. It is a core control surface for models that are increasingly central to the local AI ecosystem. DeepSeek R1 and Qwen3 are among the most popular models on Hugging Face. Developers building on top of llama.cpp need those parameters to work correctly, every time, without silent failures.

The release notes for b9982 list the usual matrix of build targets: macOS Apple Silicon, Linux x64 with Vulkan, Windows with CUDA 12 and 13, Android arm64, and several others. The fix itself is architecture-agnostic. It is a pure logic bug in the request parsing layer. But the breadth of the build matrix is a reminder that llama.cpp runs on everything from a Raspberry Pi to a multi-GPU workstation. A bug that breaks per-request reasoning budgets affects every user on every platform.

The unit test added in b9982 is a small but important step. It uses a Qwen3 template, which the autoparser detects as a thinking-capable model, and asserts that the returned llama_params carries the per-request reasoning_budget_message rather than the server default. That test will catch regressions in the future, but only for this specific path. The broader question is how many other parameters have similar ordering bugs.

The fix was co-authored by Xuan Son Nguyen at Hugging Face, which suggests that the bug was caught in a production or integration context where the silent discard of per-request parameters caused observable failures. That is the pattern: these bugs are invisible in unit tests that only check the happy path, but they surface immediately when a real application sends a non-default value and gets the wrong behavior.

For developers building on llama.cpp, the takeaway is straightforward. Update to b9982 if you use per-request reasoning budget overrides. The fix is small, but the behavior change is meaningful. For anyone running a llama.cpp server that serves thinking models, the bug means that every request that tried to set reasoning_budget_tokens to 0 to suppress thinking was silently ignored. That is a lot of wasted tokens and a lot of confused users.

The broader lesson is about the fragility of API compatibility in open-source inference engines. The OpenAI-compatible surface is a moving target. Each new model family introduces new parameters, new aliases, new behaviors. The code that parses those parameters must be correct, but it must also be structured so that ordering bugs like this one are hard to introduce. The b9982 fix is a good example of the right approach: read the request body first, fall back to defaults, and test with a real model template.

llama.cpp remains the most important piece of local AI infrastructure. Its maintainers ship fixes quickly, and the community catches bugs that would otherwise persist for months. The b9982 release is a small patch, but it fixes a bug that silently broke developer intent. That is worth updating for.