The efficient frontier of LLM inference is real, and it is jagged

Baseten, the model-serving company, [published a taxonomy of LLM inference techniques](https://www.baseten.co/blog/the-efficient-frontier-of-llm-inference/) that reframes how engineers should think about serving costs. The core distinction: some techniques trade one outcome for another along a fixed curve, while others push the entire curve outward. Most production teams spend their time on the first kind, and that is a strategic mistake.

The post borrows the efficient-frontier concept from economics and applies it to inference. A model is “frontier” if it offers the most intelligence at a given cost or size. The same logic applies to serving: latency versus throughput, quality versus speed via quantization, intelligence versus response time via reasoning level. Baseten splits the toolbox into two buckets: techniques that move a deployment along the frontier, and techniques that shift the frontier itself.

The distinction matters because the second bucket compounds. Baseten’s example: doubling performance from hardware and doubling again from software yields a four times overall improvement, allocable across latency and throughput. The first bucket, by contrast, only reallocates a fixed pool. Knowing which bucket a technique falls into changes where engineering hours go.

The article assumes a concrete deployment: an LLM like GLM-5.3 or Kimi K3 running agentic coding workloads, with KV cache reuse enabled and optimal KV-aware routing. That assumption is worth pausing on. Agentic coding is the highest-value inference workload in the industry right now, and it is also the one where output sequences are long and predictable. That makes it the ideal test bed for the techniques Baseten describes.

The jagged frontier is the real lesson

The most useful observation in the post is that the efficient frontier is not smooth. Baseten calls it “very jagged”: small configuration changes can produce large swings in outcomes, and the cutoff points are unintuitive. They must be discovered empirically through sweeps, not derived from first principles.

This is the part that most model-serving guides miss. The tidy two-axis chart of latency versus throughput implies a continuous tradeoff curve, where any point is reachable with the right dial. In practice, the dials have thresholds. A batch size of 32 might give excellent per-user latency, while 33 collapses it. A quantization level that looks fine on paper might destroy quality on a specific workload, while a slightly lower one is nearly lossless.

Baseten names the mechanisms behind the jaggedness. Batch sizing is the most obvious latency-throughput dial: small batches give great per-user latency but high cost per token; large batches flip that. Parallelism strategy adds another layer: Tensor Parallelism (TP) lowers latency over high-bandwidth NVLink interconnects, Expert Parallelism (EP) can boost either latency or throughput depending on width, and Attention Data Parallelism (ADP) replicates attention layers to raise system throughput at the cost of per-request speed.

Each of these has a cliff somewhere. The post does not map the cliffs, and it does not need to, because the cliffs are workload-specific. The point is that they exist, and that finding them requires measurement.

Quantization is a frontier-pusher with a quality tax

Baseten places quantization in an interesting middle position. Quantized models improve both latency and throughput, which means quantization pushes out the serving frontier. But it introduces a new tradeoff between quality and serving efficiency, and that frontier is especially jagged.

The post cites microscaling floating-point formats like MXFP4 and NVFP4 as the current sweet spot: large serving-efficiency gains with little-to-no quality loss. That is a specific, testable claim. The implication for builders is that quantization is no longer a blunt instrument. The old choice was FP16 versus INT8, with visible quality degradation. The new choice is a spectrum of sub-4-bit formats where the quality cliff is sharp but located far to the left of where most teams assume.

This matters for the economics of agentic coding. If a 4-bit quantized GLM-5.3-class model serves at near-full quality, the cost per token drops enough to change deployment decisions. Teams that skip quantization sweeps are leaving money on the table, and the money is not small.

Speculative decoding has grown up

The post’s treatment of speculative decoding is the most encouraging section. When speculative decoding was new, it posed a genuine latency-throughput tradeoff: speculation was expensive, sequence lengths were short, and acceptance rates were low. It only made sense at small batch sizes.

Baseten says that has changed. Techniques like EAGLE-3, DSpark, and DFlash still compete with the main model loop for resources, which caps maximum batch sizes. But on code generation, where output tokens are relatively predictable, they deliver efficiency gains from skipped forward passes on top of raw latency reduction. More tokens per second per user, at scale.

That is a frontier-pushing technique that has crossed the threshold from research curiosity to production tool. The post’s framing suggests the acceptance-rate problem that killed early speculative decoding has been largely solved for structured outputs like code. The remaining constraint is resource contention with the main model loop, which is a solvable engineering problem rather than a fundamental barrier.

Disaggregation is the quiet throughput winner

P/D disaggregation, separating prefill and decode onto dedicated workers, gets a relatively short section, but it is the technique most likely to matter for high-volume deployments. Baseten notes that running prefill and decode independently lets workers be optimized for each phase, and the ratio between worker types can be adjusted to match input and output sequence lengths and cache hit rates from incoming traffic.

The practical effect: higher throughput with the same or slightly better latency. That is the definition of pushing out the frontier. For agentic coding workloads, where the ratio of prefill to decode is wildly different from chat traffic, disaggregation is not optional. A coding agent that reads a large repository and then generates a long patch has an extreme prefill spike followed by extended decode. Fixed workers for both phases waste either compute or time.

The post frames disaggregation as most useful for throughput, which undersells it. In agentic contexts, disaggregation is also a latency technique, because it prevents prefill bursts from stalling decode workers.

The takeaway for builders

The distinction between tradeoff techniques and frontier-pushing techniques is not academic. It determines where to spend engineering time. Tuning batch size, parallelism, and routing is table stakes; it moves a deployment along a curve that everyone else is also on. Kernel optimization, speculative decoding, and disaggregation are the techniques that separate serving stacks.

The compounding point is the one to internalize. Baseten’s example of hardware doubling and software doubling producing a four times gain is the argument for investing in both. Too many teams treat inference optimization as a hardware problem or a software problem. The efficient frontier moves only when both move.

The jaggedness also has a policy implication. If the frontier is empirically discovered through sweeps, then small teams with access to good benchmark harnesses can find configurations that large teams miss. The cost advantage is not locked behind proprietary hardware. It is locked behind measurement discipline.

Baseten’s post is a taxonomy, not a benchmark. It does not claim that MXFP4 beats NVFP4, or that EAGLE-3 beats DSpark. It does something more useful: it gives engineers a map of where the levers are and which ones move the whole curve. The rest is sweeps.