BaseRT claims fastest Apple Silicon inference, beating llama.cpp by up to 56%

A small team called Base Compute has released BaseRT, an inference runtime for Apple Silicon written entirely against Metal’s API. No MLX, no PyTorch, no CoreML, no cross-platform CPU-first codebase. The numbers are the kind that make you double-check the benchmark methodology: up to 1.56x faster decode than llama.cpp on an M4 Pro, up to 1.35x faster than MLX, and on mixture-of-experts prefill, a lead of 1.81x over llama.cpp and 1.78x over MLX.

The team published a detailed blog post and paper on Hugging Face on July 1, 2026, walking through exactly where those gains come from. The answer is not a new algorithm. It is the systematic removal of every abstraction that existing runtimes carry.

What BaseRT does differently

The core insight is that llama.cpp and MLX both pay a tax for portability. llama.cpp runs on CPU, CUDA, Vulkan, and Metal from a single codebase. MLX is a lazy-evaluated array framework designed for research flexibility. BaseRT throws both approaches out and targets Metal exclusively, with hand-written kernels per quantization format from Q2 through Q16.

The runtime expresses model architecture as a compact descriptor rather than branching on model identity in the hot path. The decode loop allocates zero bytes after load time. Residual buffers, attention scratch space, feed-forward intermediates, even the KV cache are all pre-sized and reused. Kernel fusion combines operations that other runtimes dispatch separately, removing a launch and a global-memory round trip per fusion.

The matmul kernels are specialized per operation: dedicated GEMV kernels for decode where M=1, GEMM kernels for prefill where M>1. Dequantization happens inline in the compute loop rather than materializing full-precision weights to memory. Memory traffic scales with the compression ratio, not the uncompressed size.

Where the wins land, and where they don’t

The benchmark results on an M4 Pro (16 GPU cores, 24GB unified memory) show a clear pattern. BaseRT’s advantage is largest on small dense models. Qwen3-0.6B at Q4 delivers 464.5 tokens per second against llama.cpp’s 297.4, a 56% lead. On Llama 3.2 1B the gap is 28%. On Llama 3.2 3B it narrows to 15%. On the large MoE models, Qwen3-30B-A3B and Gemma 4 26B-A4B, the decode advantage shrinks to 4% and 7% respectively.

This is exactly what you would expect. Small models are dispatch-bound. The per-token overhead of kernel launches, buffer management, and CPU-side scheduling is a larger share of total latency. Eliminate that overhead and the gains are dramatic. Large models are memory-bandwidth-bound. The bottleneck moves from software to hardware, and no amount of kernel fusion changes the fact that you are waiting on the memory controller.

The prefill results are more surprising. On small dense models, all three runtimes land within a few percent of each other. GEMM-bound prefill saturates the GPU’s matmul units regardless of runtime. On MoE models, however, BaseRT leads by up to 1.81x on a 128-token prompt. The team attributes this to the architecture descriptor approach, which avoids branching on expert routing and keeps the GPU pipeline full.

The tradeoffs matter

BaseRT is not a drop-in replacement for everyone. It requires macOS 14 or later and Apple Silicon M1 or newer. It ships its own .base model format, meaning every model must be converted from GGUF, Hugging Face, or MLX checkpoints. The engine binary itself is proprietary software distributed under a separate license, even though the CLI, format, and bindings are Apache-2.0.

The comparison with uzu, another Metal-native runtime, is instructive. BaseRT leads uzu on decode by up to 1.19x on small models but trails on prefill by 4-13% on Qwen3-0.6B. The reason: uzu routes part of its workload through MPSGraph, which can dispatch to the Apple Neural Engine for additional GEMM throughput. BaseRT’s GPU-only path does not use the ANE yet. The team acknowledges this in the blog post.

What this means for the local AI market

Apple Silicon’s unified memory is a structural advantage for local inference. A Mac Studio with 192GB of unified memory can run models that require an NVIDIA A100 with 80GB of VRAM to match, at a fraction of the power budget. The bottleneck has always been software. MLX is flexible but carries the overhead of a lazy-evaluation framework. llama.cpp is portable but optimized for CUDA first.

BaseRT is the first serious attempt to build for the hardware as it actually exists, not as a subset of a cross-platform target. The gains are real and measurable. The question is whether the team can sustain the engineering effort required to keep up with new model architectures and Apple’s chip generations, or whether this becomes another showcase project that the incumbents absorb.

For now, anyone running local inference on a Mac has a new best option for throughput-critical workloads. The gap is wide enough that it changes the economics of running small models locally for coding agents, chatbots, and embedding services. That is the kind of concrete improvement that shifts developer behavior.

BaseRT is not a revolution. It is a reminder that the largest performance gains in AI infrastructure right now come not from new model architectures but from removing the abstractions that made the previous generation work.