llama.cpp b9908 enforces strict prompt cache RAM limits

The latest release of llama.cpp fixes a bug that quietly undermined one of the most important features for local AI deployment: predictable memory usage.

Release b9908, tagged on July 8, enforces the --cache-ram flag as a hard limit for the first time. Before this commit, the prompt cache was allowed to exceed the user-specified RAM budget in at least two ways. The cache always kept at least one entry, even if that entry alone exceeded the limit. And old entries were only evicted after saving the new one, which meant the cache could temporarily balloon past the configured ceiling.

The fix is straightforward. The server now skips saving a state to cache if the state itself exceeds the RAM limit. It evicts old entries before saving a new one to ensure the total fits. And the token-limit cleanup path can now remove the last remaining cache entry instead of always preserving one.

This is a small change in code terms. But it fixes a class of reliability bugs that have plagued local AI deployments since the earliest days of prompt caching.

Why prompt caching matters

Prompt caching is the mechanism that lets llama.cpp avoid recomputing the key-value cache for repeated prefixes in a prompt. When a user sends a request with a long system prompt or a lengthy conversation history, the server can reuse the cached internal state from the previous request instead of processing the entire prefix again.

The performance gain is substantial. For a long system prompt, caching can cut time-to-first-token by 50 percent or more. For multi-turn conversations, the savings compound with each turn.

But caching has a cost. The key-value cache for a large model takes significant RAM. A 70-billion-parameter model with a 32,000-token context window might need several gigabytes of cache per active sequence. On a machine with 64 GB of RAM, a few cached sequences can consume a quarter of the available memory.

Before b9908, users who set --cache-ram 8096 to reserve 8 GB for caching could find the server using 10 GB or more. The cache would save a new entry, then evict old ones. During that window, the server could run out of memory and crash, or the operating system would start swapping.

The practical impact

For developers running llama.cpp in production, this fix removes a class of unpredictable failures. A server that respects its memory budget is a server that stays up. A server that occasionally exceeds its memory budget is a server that needs manual monitoring and restart scripts.

The change also matters for the growing ecosystem of applications built on llama.cpp. Desktop apps, mobile apps, and embedded systems all depend on predictable resource usage. A cache that respects its limit makes it easier to write reliable software on top of the inference engine.

The release notes also show that the macOS Apple Silicon build with KleidiAI enabled is disabled in this release, linked to pull request #23780. The reason is not specified in the release notes, but the pattern of disabling a specific optimization build suggests a compatibility or stability issue that the maintainers are working through.

What this says about the project

llama.cpp has grown from a single-file demo to a full-featured inference server with GPU acceleration, quantization, and a plugin architecture. The project now ships binaries for macOS, Linux, Windows, Android, and iOS, covering CPU, Vulkan, ROCm, CUDA, OpenVINO, SYCL, and HIP backends.

The prompt cache fix is the kind of work that happens when a project matures past the initial capability rush. The first versions of llama.cpp were about making models run at all. Later versions were about making them run fast. This release is about making them run reliably.

The change also reflects the project’s open-source development model. The fix in pull request #25070 was submitted by a contributor, reviewed by the maintainers, and merged into the main branch. The release notes credit the specific commit hash (6c487e2) and include a GPG signature verification from the GitHub Actions bot.

The broader context

Prompt caching is not unique to llama.cpp. OpenAI, Anthropic, and Google all offer prompt caching as a paid feature in their API services. The economics are straightforward: caching reduces compute cost for the provider, and they pass some of that savings to the user.

But the implementation details matter. A cloud API provider can afford to overallocate memory because they have large pools of RAM and can spin up new instances. A local deployment on a laptop or a Raspberry Pi does not have that luxury.

The llama.cpp fix shows that local AI is converging on the same reliability standards as cloud infrastructure. The difference is that local deployments have harder constraints. A cloud cache that exceeds its budget by 10 percent causes a minor billing surprise. A local cache that exceeds its budget by 10 percent causes an out-of-memory kill.

What to watch

The next area where llama.cpp needs similar hardening is the GPU memory path. The --cache-ram fix applies to CPU memory. But many users run llama.cpp with GPU offloading, where the key-value cache lives in GPU VRAM. GPU memory is typically smaller and more expensive than system RAM, and an over-budget cache there can cause GPU driver crashes that are harder to recover from.

The project’s maintainers have shown a willingness to tackle these reliability issues. The prompt cache fix is the latest example. The disabled KleidiAI build is another sign that the team prioritizes correctness over feature count.

For builders, the takeaway is concrete. If you are deploying llama.cpp in production, upgrade to b9908 and set your --cache-ram flag with confidence that the server will respect it. If you are building an application on top of llama.cpp, you can now assume that the cache will not exceed its budget, which simplifies your error handling and monitoring code.

That is the kind of progress that matters more than a new model or a faster kernel. It makes the infrastructure invisible, which is exactly what production software needs.