Tokenwise and the $50B Token Tax: Why Every LLM Call Is a Leak

There is a quiet hemorrhage in every production LLM pipeline. It is not a bug. It is the cost of sending context that the model never uses. A new open-source tool called Tokenwise claims it can stop the leak, reducing token consumption by up to 66% by stripping irrelevant context before the request hits the API. The claim is plausible. The economics behind it are staggering.

Tokenwise is a smart LLM proxy, published on GitHub by a developer using the handle burgerkhan6227. It sits between the user and the model, runs a FastAPI backend, and applies what it calls “smart context selection” — an intelligent ranking algorithm that picks the most relevant chunks of a prompt and discards the rest. The tool supports GPT models from OpenAI and a range of other LLMs. It runs on Windows, macOS, and Linux. The pitch is simple: pay for fewer tokens, get the same answer.

The 66% figure is not from a controlled benchmark. Tokenwise’s README states the reduction as a headline feature, not a guarantee for every use case. A chat history with 10,000 tokens of back-and-forth fluff might compress to 3,400 tokens of actual signal. A legal document where every clause is operative might compress to 85%. The tool works best where context is noisy — customer support logs, multi-turn conversations, retrieval-augmented generation (RAG) pipelines with bloated document chunks.

That is most of the market.

Inference costs have become the dominant line item for AI companies. OpenAI charges $15 per million input tokens for GPT-4o. Anthropic’s Claude 3.5 Sonnet runs $3 per million. At scale, a mid-tier SaaS product handling 10 million requests a day — each carrying 2,000 tokens of context — burns $300 a day in input costs on Sonnet, over $100,000 a year. Cut that by two-thirds and you save north of $60,000. For a startup burning through a seed round, that is meaningful.

The problem Tokenwise solves is structural. Every LLM call carries a fixed overhead of prompt tokens that the model must “read” before it can generate a response. In a RAG pipeline, the retrieved documents are often concatenated wholesale, including boilerplate, repeated headings, and irrelevant paragraphs. In a multi-turn chat, the entire conversation history is re-sent on every turn. The model processes all of it. It pays attention to some of it. The rest is compute spent on nothing.

Researchers have been poking at this inefficiency for years. A 2024 paper by Ahmad Rashid, Ruotian Wu, Julia Grosse, Agustinus Kristiadi, and Pascal Poupart, titled “A Critical Look At Tokenwise Reward-Guided Text Generation” and accepted at COLM 2025, showed that token-level reward guidance during decoding can steer generation toward higher-quality outputs without fine-tuning. The paper’s abstract notes that “reward models trained on full sequences are not compatible with scoring partial sequences” — a technical way of saying that most optimization methods are applied after the fact, not during generation. Tokenwise’s approach is different: it optimizes before the call, not during it.

The economics of token optimization are not trivial. The market for LLM inference is expected to exceed $50 billion annually by 2027, according to projections from several analyst firms. If even 20% of that spend is wasted on irrelevant context, the addressable savings exceed $10 billion a year. That is the prize Tokenwise is chasing.

But the tool has limitations. It is a local proxy, not a cloud service. It requires installation and configuration. It does not handle streaming natively. It does not yet support every model provider. And the “smart context selection” algorithm is not described in detail — the README points to a GitHub Wiki that is not yet populated. For a production deployment, engineering teams will want to audit what the proxy discards and verify that answer quality holds.

The deeper question is whether the market will tolerate a third-party proxy sitting between the user and the model. API providers like OpenAI and Anthropic have an incentive to keep token counts high — they charge by the token. A proxy that cuts volume by two-thirds cuts their revenue from that customer by the same proportion. It is not hard to imagine rate-limit changes, API key restrictions, or terms-of-service updates that make proxies harder to operate.

The same dynamic played out in the CDN market. Cloudflare built a business on top of AWS’s infrastructure, then AWS launched CloudFront and made third-party proxies less attractive. The difference here is that the LLM market is still early. No single provider has the lock-in to crush intermediaries. OpenAI, Anthropic, Google, and Mistral all compete on price and capability. A proxy that works across providers has leverage — it can route to the cheapest model for a given task, not just the cheapest context.

Tokenwise is not the only player. LangChain, LlamaIndex, and several startups offer context optimization as part of broader orchestration frameworks. What makes Tokenwise notable is its focus: a single-purpose proxy that does one thing well. No agent loops, no tool-calling, no memory management. Just a filter that says “you are paying for noise, and we can remove it.”

The real test is in the numbers. A team running Tokenwise on a production workload should measure three things: token reduction percentage, answer quality retention (via an automated eval like BLEU or semantic similarity), and latency overhead. If the proxy adds 200 milliseconds to every call, the savings in token cost may be offset by user-experience degradation. If it adds 20 milliseconds, the tradeoff is trivial.

There is a broader lesson here for the AI industry. The race to build better models has overshadowed the race to use them efficiently. Every percentage point of token waste is a percentage point of margin lost. In a market where inference costs are the bottleneck to scaling, the companies that optimize the input side will have an advantage over those that only optimize the output.

The 66% figure is a claim, not a guarantee. But even half that — 33% — would reshape the unit economics of AI products. For a startup spending $50,000 a month on inference, a 33% reduction is $16,500 a month, $198,000 a year. That is a headcount. That is runway. That is the difference between shipping a product and shutting it down.

Tokenwise is a small tool with a big implication: the AI economy is still paying a tax on noise, and the first companies to stop paying it will have an edge.