Free Resource

AI Production Reliability Checklist

47 checkpoints across evals, observability, prompt design, failure handling, and versioning.

Use this before shipping any AI feature. Run through it after incidents. Revisit when quality degrades for no obvious reason.

Evaluation & Testing

9 items

Defined measurable reliability contract (e.g., "95% task completion rate")Eval set exists and is tied to user outcomes, not model outputsEval set drawn from real production data (not only synthetic)Evals run automatically in CI on every prompt or system changeEdge case examples explicitly included in eval setFailure mode examples included in eval setEval scoring is automated (LLM-as-judge or rule-based)Eval results tracked and versioned alongside codeBaseline eval score established before first deployment

Observability & Logging

10 items

Failure Handling

9 items

Every failure mode has a defined user-facing behaviorConfidence thresholds set — low confidence routes to fallbackFallback defined for every critical path (human, rule-based, or explicit refusal)Input validation strips or rejects inputs that produce systematic failuresOutput validation checks format/schema before downstream useRetry logic is bounded (max 3 retries, with variation on retry 2+)Rate limiting implemented to prevent runaway inference costsCircuit breaker pattern for external tool/API dependenciesNo silent failures — errors always surface to user or ops

Prompt Design

8 items

Each prompt has one clear, bounded taskExplicit constraints list defines what model should NOT doOutput format enforced (JSON schema, output template, or structured output mode)Few-shot examples cover core use case AND edge casesComplex prompts decomposed into chains of simple promptsPrompts do not rely on conversation history to determine current statePrompt length is minimized — every instruction earns its placePersona instructions kept minimal; behavioral constraints are primary

Prompt Versioning & Change Management

6 items

All prompts stored in version control (not env vars or DB configs)Every prompt version tagged with eval results that validated itNo production prompt updated without running evals firstRollback path exists and has been testedCommit messages explain what changed and why (not just what)Prompt changes require review like code changes

Feedback Loops & Improvement

5 items

Production logs reviewed on a regular cadence (weekly minimum)New edge cases from production converted to eval cases promptlyUser feedback incorporated into prompt or retrieval changesImprovement cycle is documented (input → eval → change → eval → deploy)Quality metrics tracked over time (not just at launch)

Get it in your inbox

Save this checklist for your next AI feature

Subscribe and I'll send you this checklist as a structured document, plus occasional notes on building reliable AI products and shipping better software.

No spam. Unsubscribe anytime.