Migrating to GPT-5.6: 2.2x faster, 27% cheaper, and a masterclass in hidden model debt

OpenAI released GPT-5.6 Sol this morning. By lunchtime, Ploy — a startup whose AI agent builds and edits real marketing websites — had already switched its default model from Claude Opus 4.8. The numbers are striking: 2.2x faster wall-clock time, 27% lower cost, and a higher visual score on completed builds.

Ploy published the full migration post on Monday. It is the most instructive model-switch autopsy since the first Claude-to-GPT migrations of 2024. The headline numbers are real. But the story the post tells is about something deeper: the invisible infrastructure debt every production agent accumulates around its incumbent model.

For four months, Ploy tested every frontier release against Claude Opus. Nothing beat it. GPT-5.6 Sol is the first. The initial eval run showed immediate promise: builds finishing in 3 minutes 42 seconds versus Opus’s 8 minutes, at $2.22 per completed build versus $3.06. Output tokens nearly halved, from 33K to 17.1K. GPT-5.6 writes leaner code. On one matched pair, Opus produced a 17,957-character CSS file with 174 mostly unused CSS variables; GPT-5.6 wrote 2,508 characters and 45 variables for a comparable rendered page.

But the first cross-model eval was a disaster. Roughly a third of the raw failures traced back to harness assumptions, not model behavior. Ploy’s tool-call budgets were sized for Opus’s sequential style; GPT-5.6 fans out parallel calls and blew through them on cases it was solving correctly. The eval executor didn’t support batched file reads, which Opus rarely used and GPT-5.6 uses constantly. A silent minScore threshold defaulted to 1.0, causing GPT-5.6 to “fail” a hero it scored 0.98 on.

The lesson is blunt: if you are evaluating a challenger model against an incumbent, triage the traces before you trust the pass rate. Otherwise you are grading the new model on how well it imitates the old one.

The real engineering work began after the harness was fixed. Ploy discovered three provider-specific behaviors that had quietly become part of “how the agent works.”

First, tool-call verbosity. Claude Opus sends only the parameters it uses. GPT-5.6 sends all 25 parameters every time, inventing plausible values for the ones it does not need: offset: 0, timeout: 120000, siteId: "00000000-0000-0000-0000-000000000000". Over three days of production traces, 6,635 GPT-5.6 calls carried all 25 properties. Only 4 of 2,898 Claude Opus calls did the same. The invented values were indistinguishable from intended ones. Ploy’s file-read implementation treated offset: 0 as a real argument, and 52% to 64% of GPT-5.6’s file reads came back empty. The tool returned success both ways, so the model had no way to know it was reading blank files.

Prompting does not fix this. A tool-description directive to “omit unused parameters” still produced 25/25. Per-property “OPTIONAL, omit if unused” hints still produced 25/25. OpenAI’s strict mode produced identical behavior. Ploy’s fix was a schema transform at the provider boundary: rewrite every optional property to be required but nullable using anyOf: [T, null], then strip the nulls before validation. Empty file reads went from 52% to 0%. The agent needed roughly 30% fewer tool calls.

Second, prompt caching. This was the most instructive engineering difference. On the surface, both providers offer “prompt caching.” Under the hood, the designs are entirely different. Ploy’s agent has a static prefix of roughly 29K tokens (tool schemas plus core system prompt) identical for every conversation. On Claude, cache breakpoints make that prefix cache across the entire organization: any conversation, any workspace, one shared entry. Cache hit rates run 92% to 96%.

GPT-5.6 changed OpenAI’s caching model. Earlier GPT models cached implicitly on partial prefix matches. GPT-5.6 dropped partial-prefix matching. Implicit caching now only creates whole-prompt entries keyed on the latest message. A new conversation sharing the 29K static prefix cached 0% of it. Every conversation re-billed the full prefix at the uncached rate, plus a 1.25x cache-write surcharge.

The intended mechanism is explicit: prompt_cache_breakpoint markers plus a mandatory prompt_cache_key. The key is part of cache identity. Identical prompt, different key: zero cache hits. Each key maps to a cache node that sustains roughly 15 requests per minute before OpenAI fans traffic to other nodes with independent, cold caches.

Ploy found the sweet spot was a per-workspace key. All conversations in a customer workspace share entries; per-key traffic stays low. First-call cache hits went from roughly 0% to 83.7%. Total uncached input tokens dropped 28%. GPT-5.6’s per-suite cost landed below Opus’s. Every dollar of the gap Ploy had been staring at was cache misconfiguration, not model pricing.

One consequence has no workaround: cross-workspace sharing of the static prefix is structurally impossible on OpenAI. Anthropic can share it because its cache is org-scoped without key partitioning. On GPT-5.6, every workspace pays one 29K cold write per idle window, about $0.18. A real cost, but bounded and predictable.

Third, reasoning replay. GPT-5.6’s Responses API replays prior-turn reasoning as server-side item references by default. Ploy’s agent started failing mid-conversation with Item 'rs_...' not found. The fix is store: false, which makes the SDK request encrypted reasoning content and replay self-contained blobs instead of pointers to server state.

Ploy’s post is a case study in what the industry should stop calling “model switching.” It is not switching. It is rewriting the stack around a new provider’s assumptions about tool schemas, caching identity, and state management. The model is the smallest part of the migration.

For AI builders, the takeaway is not that GPT-5.6 is better than Claude Opus. It is that every production agent accumulates invisible dependencies on provider-specific behaviors, and those dependencies are the real cost of switching. Ploy used Vercel’s AI SDK, a universal LLM SDK, and still had to rebuild tool schemas, caching strategy, and reasoning replay. The abstraction layer helps with API compatibility. It does not help with the behavioral differences that matter in production.

The numbers are real. Ploy’s agent is now 2.2x faster and 27% cheaper. But the migration cost — the engineering time, the debugging, the schema transforms, the cache redesign — is the part that does not make it into the benchmark tables. Every frontier model switch is a stack rewrite. The question is whether the performance gain justifies the surgery. For Ploy, it did. For most teams, the answer will depend on how much of their agent’s behavior is actually the model, and how much is the invisible infrastructure they built around the last one.