Janas streams 80B experts from SSD, hits 24 tok/s on a 32GB laptop

Maurizio “camauri” Cammalleri has published [Janas-LLM](https://github.com/prabanta-dev/janas), a C inference engine with no dependencies beyond the C library that runs mixture-of-experts language models larger than a machine’s RAM by keeping the weights on disk and streaming the experts each token needs from an NVMe SSD. On a laptop with 32 GB of memory, a 48.4 GB Qwen3-Next-80B-A3B model writes at 24 tok/s and reads a 1,024-token prompt at 62 tok/s. The same model under llama.cpp, the incumbent for local inference, manages 2.8 tok/s writing and 16 reading. That is roughly an 8.5x gap on generation, on the same hardware, with the same weights.

The machine is unremarkable: an Intel Core Ultra 9 185H, its integrated Arc GPU, 32 GB of DDR5, a 1 TB NVMe SSD reading at about 5.7 GB/s, Debian 13 with kernel 6.12.107. The benchmark ran 9–11 October 2026. Janas was faster than llama.cpp on every model compared, across twelve models from Qwen3-0.6B up to Qwen3-Coder-Next at 48.4 GB. On the smallest model, Qwen3-0.6B, it writes at 130 tok/s against llama.cpp’s 45. On Qwen3-4B, 27 against 18.

The trick is the cache, not the streaming

Streaming weights from disk is not new. llama.cpp has done it through mmap and the page cache for years. What Janas does differently is decide, at model-open time, how much free memory it can claim, then use that memory as an expert cache. The experts that keep coming back stay resident. The rest are read with O_DIRECT, bypassing the page cache entirely, so the engine controls what is in RAM rather than leaving it to the kernel’s eviction policy.

For Qwen3-Next-80B-A3B, that cache came to about 19 GB. The remaining 29 GB of weights are read from disk while the model answers. This is why it works at all: a mixture-of-experts model only activates a fraction of its parameters per token, so the working set is far smaller than the file. The 80B model has 3B active parameters. Janas is betting that the experts a given conversation touches cluster tightly enough that a 19 GB cache catches most of them.

The benchmark numbers suggest the bet pays off. llama.cpp’s own attempt at the same model required three non-default flags to run at all: GGML_CPU_REPACK=OFF, --no-host 1, -lm mmap. Its CPU backend copies weights into its own memory to repack them, and its Vulkan backend copies the experts it offloads into buffers the system cannot reclaim. On a 32 GB machine with a 48 GB model, that is fatal. Janas ran the same file with no flags.

What Cammalleri is honest about

The repository leads with a warning: “This is an experiment, not a product.” That is not false modesty. The public API and command line “change without notice, and have changed several times a week.” Stability is listed as “Not verified.” The engine fills free memory with its cache and reads the disk at full speed, which on a machine with less headroom than it estimates “can be pushed into swap and become slow or unresponsive.” Using the GPU has once caused a driver reset that took the desktop with it.

Everything was developed and measured on one laptop. Other CPUs, especially AMD or anything without AVX-VNNI, discrete GPUs, and slower disks are untested. Cammalleri asks for janas-bench reports as the most useful thing a new user can send back. That is the right ask, and it also tells you where the project is: a single-machine proof that the architecture works, not a distribution.

The gap is not a small optimization. It is the difference between a 48 GB model being unusable and being usable on hardware most people already own.

There is a caveat worth flagging on the headline numbers. The two tools do not measure identically: llama-bench writes without a prompt before it, while janas-bench measures after the 1,024-token prompt. The Janas figures for the largest models are one round each, not the three-round mean used elsewhere in the table. And it is one machine, one SSD, one CPU. The 8.5x gap may narrow on different hardware. It is unlikely to close.

Why this matters beyond one repo

The interesting claim here is not that Janas is fast. It is that the memory wall for large models is softer than the tooling has assumed. If a 48 GB mixture-of-experts model runs at 24 tok/s on a 32 GB laptop, then the practical ceiling for local inference is set by disk bandwidth and cache policy, not by RAM capacity. That reframes a hardware question as a software one.

That has consequences for the AI economy. The pitch for cloud inference rests partly on the premise that capable models need data-center memory. Janas does not break that premise for frontier dense models, which have no expert sparsity to exploit. It does erode it for the open mixture-of-experts class, which is exactly where Qwen, Gemma, and most of the recent open releases live. A developer with a gaming laptop and a fast SSD can now run a model that previously required a rented A100.

It also puts pressure on llama.cpp. The project is the default for local inference and has an enormous contributor base, but its memory handling was designed for a world where models fit. Janas’s O_DIRECT cache is a pointed critique of that design, delivered with a benchmark table. Whether llama.cpp adopts the approach or Janas grows a community around it, the mechanism is now public and the numbers are reproducible.

What to watch

Two things. First, whether anyone reproduces the benchmark on AMD hardware or a discrete GPU. Cammalleri has not tested either, and the engine’s choices are all made from measurements at runtime, so the behavior could differ sharply. Second, whether the .jns format on Hugging Face attracts converters for more models. The format is versioned and stable, which is the one part of the project Cammalleri promises not to break. If conversions accumulate, Janas becomes a real option rather than a curiosity. If they do not, it stays a very good benchmark and a very good argument.

The repository is at github.com/prabanta-dev/janas. The install is a tarball and a shell script on Linux, a zip on Windows 10 and 11. On the same laptop under Windows 11, tested 12 October 2026, Janas wrote Qwen3-4B at 25.6 tok/s and Qwen3-30B-A3B at 34.2, with dense-model prompt reading 1.8x faster than on Linux through Intel’s own driver. The engine travels. The rest is untested, and Cammalleri says so.