Testing AI: Engineering Confidence in Non-Deterministic Systems

AI makes software faster to create and harder to trust.

Testing AI is a practical operating manual for shipping AI systems with confidence. It is written for developers, testers, QA leaders, AI builders, product engineers, architects, technical leaders, and executives who need evidence before putting AI-generated code, agents, chatbots, search systems, RAG workflows, or model-backed products into production.

This book moves beyond one-off demos and simple pass/fail testing. It shows how to measure behavior under uncertainty, design useful evals, use human and LLM judges responsibly, reason about sampling and confidence intervals, test generated code and tool-using agents, monitor production behavior, and decide when to ship, hold, canary, shadow, or roll back.

Inside, you will learn how to:

Evaluate non-deterministic systems across repeated runs, slices, risks, and real user behavior. Build evals, rubrics, release gates, and production monitors that support actual shipping decisions. Use statistics such as confidence intervals, p-values, NDCG, precision, recall, and F-scores without fooling yourself. Test RAG systems, AI agents, prompt injection defenses, generated code, guardrails, cost, latency, and rollback plans. Work with human raters and LLM judges while accounting for calibration, disagreement, bias, and fallibility. Understand the emerging discipline of confidence engineering: the practical work of deciding whether AI behavior is trustworthy enough to release Along the way, Jason Arbon shares hard-earned lessons from Microsoft, Bing, Google, Chrome, test.ai, and testers.ai, including production failures, misleading metrics, flaky automation, biased labels, broken rollbacks, and the strange new problems created when AI starts helping test AI.

Testing is the doorway. Confidence engineering is the destination.