Privatemode published a [method for turning a general-purpose LLM into a typed decision model](https://www.privatemode.ai/blog/system-one-from-glm-flash) without fine-tuning it. The trick is to prefill the assistant turn with choice_index: so the model’s very first output token is the answer, then read the logits at that single position instead of generating a JSON object. On a benchmark of 28 text datasets, GLM-5.3-Flash running this way lands on par with TypeSafe’s Jev, the specialized “System One” decision model: each wins 10 datasets, 8 fall within a percentage point, and the median gap is 0.7 points in Jev’s favor at p = 0.64.

That is the interesting part. Not that an LLM can classify. It is that a decision-shaped model, long assumed to require dedicated training, can be approximated by prompt construction and one forward pass through vLLM.

What the method actually does

The writeup, by Johannes Hötter and Marko Rosenmüller, describes three steps. Number the options in the prompt as JSON with an index on each. End the prompt inside the assistant’s answer with choice_index:. Then read the probability distribution the model assigns to each option index at that one position, renormalize over the options, and pick the highest.

The implementation details matter more than the concept. Privatemode uses vLLM’s /chat/completions endpoint with continue_final_message and add_generation_prompt: false so the model continues the prefilled turn rather than starting a new one. It sets allowed_token_ids to mask the vocabulary down to the option indexes, dropping everything else to -inf. It uses logprob_token_ids rather than top_logprobs, because top_logprobs reports the distribution before the restriction applies, letting formatting tokens like a leading space crowd out real options. And it fetches token ids from the server via /completions with echo, because digits are not always single tokens: GLM-5.3-Flash tokenizes 12 as one token.

The code sits in the edgelesssys/privatemode-decisions repository, with a separate benchmark repo so every number in the post can be recomputed. The playground runs in the browser and is end-to-end encrypted.

The benchmark is the real story

Privatemode compared three systems on 29 public labeled datasets with 2 to 151 options each, covering intent routing, sentiment, topic classification, moderation, entailment, QA, legal text, and scanned documents, in English and German. All three got the same state, the same option names in the same order, and the same instruction. Jev and Laya ran at default settings, and the GLM prompt was not tuned on these datasets.

Two findings stand out. First, the option count matters more than the model choice. On TREC, going from 6 to 42 question types drops Jev from 92.1% to 85.6%, GLM-5.3-Flash from 91.2% to 79.6%, and Laya from 88.4% to 51.2%. Second, the run-to-run noise is larger than the headline gap: even at temperature 0, GLM-5.3-Flash and Jev changed up to 3.5% of their answers between identical runs, because batching and floating-point arithmetic keep a busy server from being bit-reproducible.

Laya, Convai’s 421-million-parameter model run locally, loses to both by a median 13 to 15 points, significant at p < 0.001. That is a real result for anyone weighing a small local model against a hosted one.

Where Jev still wins, and where it does not

Privatemode is honest about the tradeoff. Cost per decision still favors Jev by several times. The company does not publish the exact multiple in the excerpt we read, but the direction is clear: a dedicated small model doing one job is cheaper per call than a frontier-class LLM doing the same job through a prompt trick.

What the GLM setup buys instead is vision. Because the prompt can carry images next to text, the same single-pass method produces typed decisions over scanned invoices and documents, which Jev cannot do. The playground ships a scanned-invoice example and a question that depends on the caller’s local time.

The accuracy is a tie. The cost is not. The capability set is not. That is the whole decision.

Why this matters for builders

The practical implication is that the boundary between “decision model” and “general LLM” is thinner than the specialized-model vendors would like. If a prompt template plus a logit read gets you within noise of Jev on 28 datasets, then the moat for a System One model is not accuracy. It is cost, latency, and the operational simplicity of not running a large model for a yes/no call.

That reframes a few things. Teams that already run vLLM for other workloads can add typed decisions without a second model in the stack. Teams that need vision plus structured output now have a path that did not exist before. Teams optimizing purely on cost per decision should still look hard at Jev or a small local model.

There is also a confidence-value angle. The method returns a probability for every option, not just the argmax, which is exactly what you need to route low-confidence decisions to a human. Privatemode notes the model solves most classic trick questions but not all of them, which is the right caveat: a distribution is more useful than a single label, and also more honest about what the model does not know.

What to watch

Two open questions. First, whether the 0.7-point gap holds at larger option counts, where GLM-5.3-Flash dropped harder than Jev on TREC. The benchmark’s own data hints that the gap widens as the decision space grows, and that is where routing systems actually live.

Second, whether the prompt-prefill approach generalizes beyond GLM-5.3-Flash and vLLM. The library is written against any vLLM-backed endpoint, and the token-id lookup is server-driven, so the design is portable in principle. Whether other models hold Jev-level accuracy under the same template is untested here.

For now, the takeaway is narrow and useful: a single forward pass, a prefilled answer, and a renormalized logit vector get you a typed decision with a confidence value, on a model you did not train. Jev is still cheaper per call. GLM-5.3-Flash can read the invoice.