Why most AI demos never make it into production

The demo was great. The room was impressed. Six months later the tab is closed and the workflow is manual again. I’ve been called in behind enough dead AI pilots to see the pattern, and it’s rarely the model’s fault.

Demos are built on the happy path

A demo processes ten hand-picked examples; production meets the full distribution — the scanned invoice at an angle, the support ticket in two languages, the lead named “asdf.” Without an eval suite built from real historical data, nobody can even say how often the system fails, which means nobody can responsibly turn it on.

This is the first thing I build now, before any agent logic: a few hundred real cases with known-correct answers. It converts “seems good” into a number, and every subsequent change gets measured against it. Boring, decisive.

Nobody priced the failure modes

The demo question is “can it do the task?” The production questions are: what happens when it’s wrong, who catches it, and how much does catching cost? Systems survive when verification is cheap — a human approving drafts in seconds — and die when every output needs expert review that takes longer than doing the work.

Cost curves kill quietly too: inference that’s trivial at demo volume can be a five-figure monthly line item at production volume. Budgets belong in the architecture phase, not the invoice-surprise phase.

No one owned it

A production AI system is software: it drifts as the business changes, breaks as APIs update, and degrades as models get swapped underneath. A demo has a builder; a production system needs an owner — someone accountable for evals, monitoring, and iteration. The internal coding agent that hit 3.2× PR velocity survived precisely because its eval suite runs on every change and a named person watches it.

The uncomfortable summary: the model is maybe 30% of the work. Evals, guardrails, integration, and ownership are the other 70% — and they’re the 70% that separates the systems that ship from the demos that die.