Free resource
AI Feature Shipping Checklist
Use this before shipping any RAG, copilot, or agentic AI feature. It is designed to catch the parts that demos usually hide: trust, workflow control, evals, traces, fallback behavior, and production risk.
1. Product and user workflow
- The AI feature solves a named user workflow, not a vague productivity problem.
- The user knows when AI is confident, uncertain, or waiting for human input.
- There is a clear path for correction, retry, escalation, or manual override.
2. RAG and data trust
- Retrieval quality is tested with realistic questions, not only happy-path prompts.
- Citations, source freshness, permissions, and metadata filters are visible in the design.
- The system has a fallback when retrieval is weak, missing, stale, or contradictory.
3. Agent and tool safety
- Each tool has a clear contract: input schema, allowed actions, error behavior, and limits.
- High-impact actions require approval gates before execution.
- Agent traces show planning, tool calls, failures, retries, and final user-facing output.
4. Evaluation and release readiness
- Golden test cases cover common, edge, and failure-mode workflows.
- Cost, latency, hallucination risk, fallback rate, and user correction signals are logged.
- There is a rollback plan if model, prompt, retrieval, or tool behavior regresses.
How to use it
Turn every item into a design review question. If your answer is vague, the AI feature is not ready. If your logs cannot prove behavior, you cannot safely improve it. If users cannot correct the system, the product will fail after the demo.