You've shipped an AI agent. Now it's doing something unexpected in production, and you don't have a clean way to reproduce the failure, measure whether a fix actually helped, or prevent the same regression from slipping through next release. This playbook gives you the methodology layer that's missing from every tool's documentation. It covers how to define what your agent must actually do before you can evaluate it, how to build test suites that catch the failures that matter, how to design scoring rubrics that don't lie to you, and how to connect your observability traces to a diagnosis process that produces decisions – not just dashboards. It also addresses the harder problems: structuring regression gates so releases don't silently degrade behavior, naming and classifying the failure modes specific to agent systems, and running post-mortems that actually change your process. Vendor-neutral throughout. Bring your own stack.
What's included
- A structured framework for defining agent behavioral requirements before writing a single eval – so your test suite measures the right thing from the start
- A test set construction methodology covering how to source, label, and balance cases that reflect real failure conditions rather than idealized inputs
- A scoring rubric design guide that walks through how to choose a judgment method, calibrate it against human baselines, and detect when your scorer is drifting from what you actually care about
- A regression suite architecture with release-gating logic – including how to set behavioral baselines, decide what constitutes a regression, and structure your suite so it scales without becoming unmaintainable
- A failure taxonomy for agent systems that gives you a shared vocabulary for classifying what goes wrong – covering reasoning failures, tool misuse, context handling errors, and compounding multi-step breakdowns
- An observability-to-eval pipeline that shows how to move from raw traces to structured diagnostic signals you can act on
- A Goodhart's Law defense framework for distinguishing genuine score improvements from metric gaming – plus a full incident post-mortem protocol for turning production failures into durable process changes
