TokenSpeed's C++ control plane is the real bet, not its 685 tokens/s

The most interesting number in the TokenSpeed repository is not 685 tokens per second. It is the one nobody puts in a headline: the control plane is a finite-state machine written in C++, and the type system enforces safe KV cache reuse at compile time rather than runtime. That is the architectural claim, and it is the one worth arguing about.

TokenSpeed is an open-source LLM inference engine from the LightSeek Foundation, a non-profit. The project describes itself as the first inference engine to separate the control plane from the execution plane. The control plane handles request lifecycle, KV cache ownership, and overlap timing, encoded as an FSM in C++. The execution plane is Python, which the project says lowers cognitive load for researchers and engineers. Kernels sit in a separate, pluggable subsystem with a portable public API and a centralized registry.

Why the split is the story

Every serious inference engine today, vLLM included, carries a scheduling core that grew organically. Request state, KV block allocation, and prefill/decode overlap live in the same process, often in the same language, and correctness depends on runtime checks. TokenSpeed’s bet is that moving the scheduling state into a typed C++ FSM turns a class of bugs into compile errors. If a request can’t legally reach a decode step without owning a KV block, the compiler should refuse to build the path that lets it.

That is a real claim, and it is falsifiable. The question is whether the velocity researchers want from Python survives contact with a C++ core that owns the state machine. The project says yes, because the Python execution plane iterates fast while the FSM stays fixed. The honest answer is that this works until someone needs to change the FSM, at which point the C++ layer becomes the bottleneck it was meant to prevent. TokenSpeed has not published data on how often that happens.

The benchmarks are real but narrow

The repository lists a string of Day 0 enablements: Qwen3.8 Flash Next and GLM 5.3 Flash in August 2026, Kimi K3 in July, TML Inkling with FP4 inference on both NVIDIA and AMD in July. The headline number is 685 tokens per second per user serving Qwen3.8-Flash-Next on NVIDIA GB300. In May 2026 the project reported 580 TPS on Qwen3.5-397B-A17B for agentic workloads. In October it shipped MLA 2 for NVIDIA Vera Rubin and published an AMD blog on advancing Kimi K3 inference on Instinct MI355X GPUs.

Read those numbers carefully. They are per-user, on agentic workloads, on specific hardware. Per-user throughput is the metric that matters for agents because a single agent session is latency-bound, not batch-bound. But it is also the metric most easily gamed by workload shape. The project’s own comparison is a Pareto curve against TensorRT-LLM on Kimi K2.5 and B200, which is the right frame: not “we beat TensorRT-LLM” but “here is the frontier.” Whether TokenSpeed sits on that frontier for workloads other than the ones LightSeek chose to publish is unverified.

The Switzerland play

The structural fact that matters most for the AI economy is the governance. TokenSpeed is backed by the LightSeek Foundation, a non-profit, not a commercial vendor. The project calls itself “the Switzerland of open-source LLM inference engines” and cites serious hardware and engineering support from NVIDIA and AMD. That is an unusual position. vLLM is community-governed but grew out of UC Berkeley’s LMSYS. TensorRT-LLM is NVIDIA’s. SGLang has its own institutional home. A neutral engine with first-class support from both major accelerator vendors is a different kind of asset.

It is also a different kind of risk. Neutrality is only as durable as the foundation’s funding. The repository lists sponsors and partners but does not disclose amounts or commitments. If NVIDIA’s support outpaces AMD’s, or vice versa, the “Switzerland” framing gets tested. The project joined the PyTorch Ecosystem in August 2026, which gives it a governance anchor, but PyTorch does not pay for kernel engineers.

What the kernel layer signals

The kernel subsystem is the part builders should watch. TokenSpeed treats kernels as a first-class, modular subsystem with a public API, a centralized registry, and a plugin mechanism for heterogeneous accelerators. The project claims one of the fastest MLA implementations on Blackwell for agentic workloads. MLA, multi-head latent attention, is the DeepSeek-origin attention variant that has become standard for memory-efficient long-context serving. Having a fast MLA kernel on Blackwell is table stakes for serving frontier models cheaply.

The plugin mechanism is the more interesting bet. If a kernel registry with a portable API actually works, it decouples engine performance from any single vendor’s compiler stack. That is the thing that would make TokenSpeed genuinely neutral rather than nominally neutral. It is also the thing most likely to break, because kernel performance is inseparable from the specific hardware and the specific compiler, and a portable API that abstracts both tends to cost you the last 20 percent.

What to watch

The next signal is not another TPS number. It is whether TokenSpeed publishes the FSM change rate, the frequency with which the C++ control plane has to be modified to support a new model or workload. If that number is low, the architecture holds and the project has a real moat. If it is high, the Python execution plane is doing the work and the C++ core is a correctness tax.

The second signal is the AMD path. The October blog on Kimi K3 inference on Instinct MI355X is the first real test of whether the kernel registry delivers on heterogeneous accelerators. If MI355X performance tracks GB300 performance on the same model, the neutrality claim has teeth. If it trails, the Switzerland framing is a press release.

For AI builders, the practical read is narrower. TokenSpeed is worth a benchmark run if you serve agentic workloads on Blackwell or MI355X and you care about per-user latency more than batch throughput. It is not yet worth a migration off vLLM for general serving, because the ecosystem, the tooling, and the operational track record are all younger. The architecture is the reason to watch. The tokens per second are the reason to test.