Shipping AI features without falling into the demo trap

Every AI feature has two lives: the one in the demo and the one in production. We've watched dozens of features die in the gap between them. The demo runs in a clean room — curated inputs, the founder driving, the model's best day. Production is messy audio, weird edge cases, users who type one word and expect magic, and a 99th-percentile latency budget the model doesn't care about.

The fix is not better prompting. It's the workflow itself. Here's the one we use.

Start with the evals

Before we ship any AI feature, we collect 200–500 real inputs from the actual use case, label what a good output looks like, and write an eval that scores model outputs against them. The eval is a real piece of software with its own CI, dashboards, and on-call runbook.

Every prompt change, model upgrade, retrieval tweak, or guardrail addition runs through that eval. Numbers go up or down. We don't ship if they go down.

Treat the model like an unreliable dependency

• Set a strict per-request timeout and a retry budget. Some prompts time out — that's fine, fall back.
• Route through a gateway (Vercel AI Gateway in most cases) so you can swap models without redeploying.
• Cache aggressively when the input space is narrow.
• Log the inputs and outputs — minus PII — so you can debug what users actually sent.

Design the disagreement flow first

The single biggest UX investment in any AI feature is what happens when the model is wrong. If your interface assumes the model is right, you've shipped a demo. Users need to spot errors fast, fix them in place, and trust that their corrections matter.

“The model gets to participate in the work. It doesn't get to own it.”

Ship behind a flag, ramp slowly

Real users do things your evals didn't predict. Roll out to 1%, then 10%, then 50%, then the long tail. Watch the eval numbers, the latency, and the user feedback. Pull the flag if anything drifts.

None of this is glamorous. None of it goes in the demo video. But this is the difference between a feature that looks impressive in a board meeting and a feature that makes your users' lives better six months in.