llama.cpp b9957: A quiet refactor that changes how models use tools

The latest release of llama.cpp, b9957, landed on July 10 with a commit message that reads like a routine cleanup: “server: improve tools, remove apply_diff.” Under the hood, it is something rarer. A mature open-source project is making a bet on how local AI agents should work.

The headline change is the introduction of a tools_io abstraction and a tools_io_basic implementation. The project also removed apply_diff, a tool that let the model apply a diff to a file. The edit tool was improved. These are not flashy additions. They are architectural decisions about how llama.cpp’s server handles tool calls from large language models.

Tool calling is the mechanism by which a language model can request an action be taken outside of generating text. It is the foundation of agents. A model says “call function X with arguments Y.” The server executes it, returns the result, and the model continues. Every major frontier lab ships this capability. What llama.cpp does differently is run it on a laptop, a phone, or a single GPU.

The tools_io abstraction matters because it separates the logic of tool execution from the logic of I/O. The basic implementation handles file reads and writes. But the abstraction layer means developers can plug in their own I/O handlers. A tool call that reads a file on a desktop could, with a different backend, read from a cloud bucket or a database. The server does not need to know the difference.

This is infrastructure thinking. It is the same pattern that made operating systems and web servers extensible: define the interface, let the ecosystem fill in the implementations. The commit adds tools_io as a class member of the server, making it a first-class concern rather than a side effect of request handling.

The removal of apply_diff is equally telling. That tool allowed the model to directly modify source files by applying a diff. It was powerful and dangerous. A model that can rewrite your code is a model that can introduce bugs, security holes, or worse. The llama.cpp maintainers chose to remove it in favor of a more general edit tool. The edit tool likely requires more explicit user interaction or safer boundaries.

The release ships 27 assets across macOS, Linux, Windows, Android, and openEuler. Build targets include CPU, Vulkan, ROCm 7.2, OpenVINO, SYCL FP32 and FP16, CUDA 12 and 13, HIP, and OpenCL Adreno. The KleidiAI-enabled build for Apple Silicon remains disabled, linked to pull request 23780. That is a lot of targets for a project that started as a single-file C++ implementation of LLaMA inference.

What is notable is what is absent. No new quantization format. No new model architecture. No kernel rewrite. This release is about the server, the part of llama.cpp that turns a model into a service. The server is where agents live. It is where the model interacts with the world.

The implications for AI builders are concrete. If you are running a local agent based on llama.cpp, b9957 gives you a cleaner path to custom tool implementations. You can write a tools_io backend that integrates with your local file system, your note-taking app, your code editor, or your database. The abstraction means you do not need to fork the server or patch the source.

For the broader AI ecosystem, this release signals that local inference is not just about chat. It is about agency. The frontier labs are racing to build cloud-based agents that can browse the web, book flights, and write code. llama.cpp is building the on-device equivalent. The difference is that local agents run on hardware you own, with data that never leaves your machine.

The tradeoff is capability. Cloud agents can call thousands of tools per second, orchestrate complex workflows, and use models with hundreds of billions of parameters. Local agents are constrained by memory, compute, and latency. But they offer privacy, offline operation, and zero marginal cost per call.

The tools_io abstraction does not solve the capability gap. It does something more important. It makes the local agent architecture extensible enough that the gap can narrow over time. As hardware improves and models shrink, the abstraction will still be there.

The removal of apply_diff is a safety signal. The maintainers are thinking about what happens when a model has write access to your files. The edit tool is safer because it likely requires confirmation or operates on a copy. This is the kind of design decision that prevents a local agent from accidentally deleting your work.

The release also ships an iOS XCFramework and Android arm64 builds. Mobile inference is the frontier of edge AI. A phone can run a 7B or 8B model at usable speeds. With tool calling, that phone can act on your behalf. The tools_io abstraction means a mobile app can define its own I/O handlers for accessing contacts, calendar, or local storage.

The KleidiAI build being disabled is a reminder that not every optimization ships on schedule. KleidiAI is Arm’s matrix multiplication library for mobile and edge. It promises significant speedups on Apple Silicon. The fact that it is disabled suggests the integration is not yet stable. The maintainers chose reliability over performance.

llama.cpp has 120,000 stars on GitHub and 20,400 forks. It is one of the most popular open-source AI projects. Its release notes are read by researchers, hobbyists, startups, and enterprise engineers. A refactor like b9957 does not make headlines. It makes the next generation of local agents possible.

The project is now at build 9957. The version number is approaching five digits. That is a lot of releases for a project that started in early 2023. The pace of change is slowing, which is a sign of maturity. The architecture is stabilizing. The abstractions are hardening.

What b9957 says to AI builders is this: if you are building a local agent, the foundation is ready. The server can handle tools. The I/O is pluggable. The safety boundaries are being drawn. The rest is up to you.