Zro's private inference pitch: coding agents without the cloud

A startup called Zro is [pitching private inference for coding agents](https://www.producthunt.com/products/zro) on Product Hunt, and the product raises a question the AI industry has mostly avoided: what happens when developers decide they do not want their code passing through someone else’s GPU?

Zro runs large language models entirely on the user’s own hardware. No API calls to OpenAI, Anthropic, or Google. No data leaving the machine. The pitch is aimed at coding agents, the class of AI tools that generate, review, and refactor source code autonomously. Most of those agents today depend on cloud inference providers. Zro says that dependency is both a privacy risk and a cost liability.

The company is not naming which models it supports or what hardware it targets. The Product Hunt listing is light on technical specifics. But the direction is clear: on-device inference for code generation, with the promise that your source code never touches a third-party server.

This is not a new technical idea. Apple, Google, and Qualcomm have all demonstrated on-device LLM inference for consumer use cases. What is new is applying it to coding agents, a market that has grown fast enough that the infrastructure question is now urgent.

The privacy argument is real. Source code is among the most sensitive data a company produces. It contains business logic, API keys, database schemas, and often credentials. Sending that data to a cloud inference provider means trusting that provider’s data handling, their security posture, and their terms of service. Most developers have accepted this tradeoff because cloud models are better. But the gap is narrowing.

Zro is betting that for a meaningful subset of coding tasks, on-device models are good enough. Code completion, simple refactors, test generation, boilerplate creation. These do not require a 1-trillion-parameter model. A smaller model running locally can handle them, and it can do so without the latency of a round trip to a data center.

The cost argument is more fragile. Cloud inference pricing has been dropping steadily. OpenAI, Anthropic, and Google have all cut prices multiple times in the past 18 months. At current rates, a developer generating a few hundred code completions per day costs pennies. The economics of buying a local GPU or a high-end laptop to run inference at home are not obviously better for individual developers.

For teams, the math changes. A team of 50 developers making heavy use of coding agents could spend thousands per month on API calls. That same team could buy a single workstation with a high-end GPU and run inference locally for everyone on the network. Zro is selling to that second group.

There is a deeper structural bet here. The coding agent market is currently dominated by companies that bundle the model and the inference together. GitHub Copilot uses OpenAI. Amazon CodeWhisperer uses Bedrock. Google’s Duet uses its own models. These are vertically integrated products. Zro is unbundling the inference layer, offering it as a standalone service that developers can plug into any agent framework.

That is a harder sell. It requires developers to manage their own model deployment, handle updates, and troubleshoot performance. The cloud-based products handle all of that. But it also gives developers control. They can choose which model to run, how to update it, and when to turn it off.

The timing is interesting. On-device inference hardware is improving fast. Apple’s M-series chips include a Neural Engine that can run small models efficiently. Qualcomm’s Snapdragon X Elite targets on-device AI. Intel’s Lunar Lake chips include an NPU. The hardware is arriving, but the software stack to use it for coding agents is immature. Zro is trying to fill that gap.

The company is entering a market that does not yet exist. There is no established category of “private inference for coding agents.” There is no standard benchmark for comparing on-device code generation quality against cloud models. There is no clear pricing model. Zro is building the category as it builds the product.

The risks are clear. On-device models are smaller and less capable. A coding agent running on a local LLM will produce worse results for complex tasks than one running on GPT-5 or Claude 4.7. Developers who try Zro and find the quality insufficient will not come back. The company needs to be honest about where its models excel and where they fall short.

There is also the question of model updates. Cloud models improve constantly. A local model is static until the user downloads a new version. Zro needs a distribution mechanism that keeps models current without requiring manual intervention.

What this means for AI builders. The coding agent market is heading toward commoditization of the inference layer. If Zro succeeds, it will accelerate that trend. Developers will have a choice: pay for cloud inference with convenience and privacy tradeoffs, or run their own inference with more control and lower recurring costs. That choice will pressure cloud providers to offer better privacy guarantees and lower prices.

The bigger question is whether on-device inference can keep pace with cloud models in quality. The gap is closing, but it has not closed. Zro is betting that for a large class of coding tasks, good enough is good enough. That bet is plausible. It is not proven.

For now, Zro is a small startup with a Product Hunt listing and a thesis. The thesis is that developers want their code to stay on their machines. The industry has assumed otherwise. Zro is testing that assumption.