Soup CLI fine-tunes Llama-3.1-8B on a 4 GB laptop GPU, and that changes who owns AI

Soup CLI is a free, Apache-2.0 post-training tool that claims it can fine-tune Llama-3.1-8B on a 4 GB laptop GPU, and the product page backs that claim with measured numbers, not marketing. The tool, built and maintained on a single 4 GB laptop, reports 119.6 tok/s for Llama-3.1-8B in 3.32 GB of VRAM on an RTX 3050 Laptop, with the GPU at 100% utilization. That is not a rounding error or a clever benchmark hack. It is a structural shift in who gets to train models, and it deserves more attention than a Product Hunt launch thread.

The mechanism is layer streaming, a technique that never loads the frozen base model into VRAM at all. Soup keeps the base in CPU RAM or on NVMe and copies it into VRAM one decoder layer at a time on a dedicated CUDA stream. Peak VRAM is bounded by one layer, not by the model. Quantizing that streamed base to NF4 shrinks it about fourfold. The docs are explicit about the tradeoffs: streaming is slower than resident training (1.52x the layer reads for the reference model), and the logits tensor still stays resident, which at batch 8 on a 152k-token vocabulary reached 8.71 GB on its own. This is not magic. It is careful engineering of the memory hierarchy, and it is open source.

The economics here are the story. Fine-tuning an 8B model has historically required either a rented A100 or H100 instance or a desktop with a 24 GB card. That puts post-training in the hands of labs with cloud budgets or individuals with expensive hardware. Soup’s layer streaming collapses that barrier. A 4 GB RTX 3050 Laptop GPU, the kind of card that ships in a $700 consumer laptop, can now run DPO, ORPO, SimPO and KTO over a streamed base. The reference model for DPO is the same streamed base with its adapters switched off, costing no extra weights: 0.914x the supervised peak on a 365M synthetic fixture, where forcing a real second instance cost 730.44 MB. The tool supports nine architectures, including Llama, Qwen, Mistral, Gemma and Phi, and a pre-flight check refuses a run that will not fit.

The honest parts of the docs are as notable as the capabilities. The site lists two kinds of runs that completed successfully and were wrong anyway: a v0.72.0 streamed adapter that loads as a no-op outside the streaming path, and a 32B or larger NF4 streamed run before v0.73.0 that had wrong gradients while its loss looked healthy. Neither announced itself. The maintainers fixed both and published the full re-run list. They also note that the 119.6 tok/s figure predates the v0.73.0 correctness repair and has not been re-measured on that card. Three borrowed days on rented H100s were enough to find a silent correctness bug nobody could have found on the 4 GB laptop. That transparency is rare in a field where benchmark numbers are often aspirational.

The broader context matters. Soup arrives at a moment when the frontier labs are consolidating around massive training runs, while a parallel ecosystem of small, local models is growing fast. Qwen 3.5, DeepSeek-V4, GLM-5.1, Kimi K2.6 and Llama 4 Scout are all listed as supported. These are not toy models. They are capable open-weight models that run on consumer hardware. Soup is betting that the value in AI is shifting from the training run to the post-training loop: data curation, preference tuning, evaluation, and the iterative ship/don’t-ship decision. That is where the tool’s 23 training methods and 142 recipes come in. The CLI auto-detects the GPU, sets the batch size, configures LoRA, and writes the config from rules rather than a hyperparameter search. It even gates every save on a SHIP or DON’T-SHIP verdict that can be committed next to the weights.

The governance angle is worth noting too. Soup ships with soup init --template hipaa, soc2, eu-ai-act or sr-11-7, which starts from a regulation-shaped config, and soup ci init blocks the merge on a DON’T-SHIP verdict. It emits signed ML-BOMs and SLSA-3 provenance. For enterprises that need to demonstrate where a model came from and how it was trained, this is a meaningful step beyond the usual “we fine-tuned it in a notebook” story. The compliance pack is a differentiator against LLaMA-Factory, Axolotl and Unsloth, which the site explicitly lists as tools that do not do this.

There are real limits. The docs are clear that 8B is the largest size measured on the 4 GB card, and that streaming bounds the weights, not the logits. The 8.71 GB logits figure at batch 8 on a 152k-token vocabulary shows that long-context or large-vocabulary runs will still need more memory. Multi-GPU, Apple Silicon and a card the maintainers “do not have to give back” are listed as blocked on hardware. The tool is BETA for layer streaming, with limits stated: transformers, text, plain LoRA. No tok/s is claimed for a preference loss because none was measured. The maintainers are upfront that streaming is slower and that the reference model costs time, not memory.

What does this mean for AI builders? The practical take is that the marginal cost of fine-tuning a capable model is collapsing toward the cost of the electricity to run a laptop. That changes the calculus for startups that currently rent GPU time by the hour for every experiment. It also changes the risk profile: if post-training runs locally, data never leaves the building, which matters for regulated industries and for anyone training on proprietary data. The compliance templates and provenance tooling suggest the maintainers are thinking about exactly that buyer.

The strategic take is sharper. The frontier labs are spending billions on pre-training, but the long tail of AI value is in adaptation: taking a capable base model and making it do a specific job well. Soup is an aggressive bet that this adaptation layer will be commoditized, open source, and run on hardware people already own. The tool’s own origin story supports the thesis. It was built on a single 4 GB laptop, which is why every performance number on the site is measured rather than claimed. The maintainers ask for donations to buy GPU time for what a 4 GB card still cannot reach, and note that three borrowed days on rented cards found a bug that the laptop could not.

The open question is whether layer streaming scales beyond 8B models and beyond the BETA label. The maintainers are candid that a 32B NF4 streamed run had wrong gradients while the loss looked healthy, and that bug was only found on rented H100s. That is a warning about the limits of testing on consumer hardware. The next version of the tool, Soup Zero, is a full desktop workbench with playground, data, evals, fine-tuning, deploy and monitoring, all on local hardware. If it ships with the same measured honesty, it will be a serious piece of infrastructure.

For now, the number to remember is 3.32 GB. That is the VRAM footprint for fine-tuning Llama-3.1-8B on a 4 GB card, measured, not claimed, with the caveats published next to it. The AI economy has spent the last two years consolidating compute upward. Soup is pulling a meaningful slice of it back down to the laptop, and it is doing so in the open, with the failure modes documented. That is the kind of progress that does not need a press release.