The Four Horsemen Behind Postgres Outages: A Database Autopsy for the AI Age

Postgres is the database of choice for most AI startups. It stores model metadata, user sessions, pipeline state, and feature stores. And it keeps falling over. A new post by Malte Schwarzkopf, the creator of pgrust, names four root causes that account for the majority of production outages. They are not exotic. They are VACUUM, connection limits, bad query plans, and JSON.

The post is a technical autopsy. But it is also a warning for anyone building AI systems on Postgres. The database is not designed for the workload patterns AI teams throw at it. The fixes are architectural. And one of them is being built in Rust.

VACUUM and the wraparound trap

The number one cause of Postgres outages is VACUUM. When Postgres deletes a row, it does not actually remove it. It marks it as deleted and keeps it around. A background process called the vacuum eventually reads all the data and physically removes dead rows. That read competes with production queries for I/O. If the vacuum falls behind, deleted rows accumulate. If it runs too aggressively, it starves queries.

The deeper problem is transaction ID wraparound. Postgres uses a 32-bit transaction ID, which means it can handle at most four billion transactions before IDs must be reused. The vacuum is responsible for reclaiming old IDs. If it falls too far behind, Postgres shuts down the database to prevent data corruption. Schwarzkopf estimates this has caused “thousands or tens of thousands of outages.”

For AI teams running high-throughput pipelines, this is a ticking clock. A model training loop that writes thousands of rows per second can exhaust the transaction ID space in weeks. The fix is either 64-bit transaction IDs, which add overhead to every row, or an alternative architecture that eliminates VACUUM entirely. pgrust is exploring both.

Connection limits and the process model

Postgres has a hard limit on connections. Exceeding it means the database refuses new connections. Changing the limit requires a restart. The reason is architectural: Postgres spawns a new operating system process for every connection. Processes are expensive to create and consume significant CPU.

For AI applications that need many concurrent connections, this is a bottleneck. Tools like PgBouncer help, but they add complexity. The deeper issue is that Postgres cannot efficiently parallelize queries because spawning processes for parallel query execution is only worthwhile for long-running queries.

pgrust uses a thread-based model from day one. Rust’s compile-time safety guarantees make threads as safe as processes, without the overhead. This is not a minor refactor. It is a fundamental architectural choice that Postgres itself has been unable to make for years.

Bad query plans and the JSON problem

Postgres’s query planner is smart but brittle. When it chooses a bad plan, a query that normally takes 10 milliseconds can take 10 minutes. The planner has no meaningful controls. Other databases like MySQL support “planner hints” that let developers influence execution. Postgres does not.

The problem is worse with JSON. Postgres collects detailed statistics for most data types: histograms, most common values, and more. For JSON, it collects nothing. When filtering on a JSON column, Postgres estimates that the filter will match 0.1% of rows. That is a magic number, not a calculation. The real match rate could be 80% or 0.0001%.

AI teams love JSON. They store model outputs, feature vectors, and unstructured metadata in JSON columns. They get bad query plans. They get outages.

Schwarzkopf’s proposed fix is an adaptive query planner that detects regressions automatically. If a query that normally takes 10 milliseconds suddenly takes 10 seconds, pgrust would check if the plan changed, check if statistics drifted, and make a correction. No DBA required.

What this means for AI builders

The four horsemen are not new. They are well-known Postgres pathologies. What is new is the workload. AI pipelines generate high write throughput, many concurrent connections, and heavy JSON usage. Postgres was not designed for this.

The pgrust project is at 96% compatibility with the Postgres regression suite. It runs in a browser via WASM. It is not production-ready yet. But it represents a bet that the database layer needs to change, not just the application layer.

For AI teams, the lesson is uncomfortable. The database is the bottleneck. And the fixes are architectural, not configurational. No amount of connection pooling or VACUUM tuning will solve the wraparound problem. No planner hint will fix JSON statistics. The database itself needs to be different.

Schwarzkopf is building that difference in Rust. Whether pgrust succeeds or not, the problems it targets will not go away. They will get worse as AI workloads grow. The four horsemen are riding.