11. Phase 9 — Evaluation, Testing & Observability
Estimated time: 2 weeks initially, then continuous.
11.1 Why Evals Are Mandatory
LLM systems are probabilistic and may regress when you change prompts, retrieval, model versions, tool schemas or application logic. A production team needs repeatable evidence that a new version is better or at least not worse. This directly extends the ML concept of train/validation/test discipline from Chapter 4 into the world of prompts, retrieval, and agents — evals are simply the testing methodology adapted to non-deterministic systems.
💡 Interview Tip
If asked "How do you know a prompt change actually improved things?": "I wouldn't trust a handful of manual spot-checks — I'd run the change against a versioned regression eval set with both old and new outputs scored on the same metrics, and only ship if the new version is measurably better or equal, with no regressions on previously-passing cases."
11.2 Build an Eval Dataset
- Collect representative user tasks, not only happy-path examples.
- Include expected behavior and, where possible, reference answers/evidence.
- Add edge cases, ambiguity, unsupported questions, adversarial inputs and permission tests.
- Version the dataset and keep a fixed regression subset.
- Record real production failures as new eval cases.
📌 Sticky Note — Remember
Every real production failure should become a permanent regression test case. This is the single most reliable way an eval suite improves over time — it grows directly from what actually went wrong, not just from what the team imagined might go wrong.
11.3 What to Evaluate
| System Type | Core Measures |
|---|
| Structured extraction | Field accuracy, schema validity, missing/extra fields, latency, cost |
| RAG | Retrieval recall, relevance, faithfulness, citation correctness, answer correctness |
| Tool calling | Tool choice, argument accuracy, permission compliance, side-effect safety |
| Agents | Task success, steps, loops, error recovery, tool efficiency, budget adherence |
| Classification/routing | Precision/recall/F1 by class; confusion matrix |
| User experience | Task completion, acceptance rate, thumbs-up/down, escalation rate |
11.4 Evaluation Approaches
- Deterministic checks: exact values, JSON schema, regex, database state.
- Reference-based metrics when a gold answer exists.
- Human review for nuanced quality.
- LLM-as-judge only with calibrated rubrics and spot-checked agreement.
- Pairwise comparison for model/prompt A vs B.
- Online A/B tests where risk is acceptable.
Prerequisite Concept — LLM-as-Judge
Definition. LLM-as-judge uses a (typically stronger or differently-configured) model to score or compare outputs against a rubric, rather than requiring a human to review every case. It scales far better than pure human review, but is itself an unreliable, non-deterministic evaluator unless calibrated: its scores should be periodically compared (spot-checked) against human judgments to confirm agreement, and the rubric should be explicit and detailed rather than a vague "is this good?" prompt.
Candidate answer ─┐
Reference/rubric ─┼──▶ Judge model ──▶ Score + justification
Retrieved evidence ─┘
Fig 11.1 — An LLM judge scores a candidate output against explicit criteria, not just its own unaided opinion of quality.
11.5 Observability
- Trace each request end-to-end.
- Capture model, prompt version, tools, retrieved documents, latency and errors.
- Track token usage and cost by feature/user/tenant.
- Track tool latency/failure rate.
- Track retrieval quality signals.
- Redact sensitive data from logs.
- Correlate AI traces with backend/infrastructure logs.
Prerequisite Concept — Tracing
Definition. A trace is the recorded, structured path of everything that happened while handling a single request — every model call, tool call, retrieval, and their inputs/outputs/timings/errors — typically visualized as a timeline or nested tree. Tracing is what makes it possible to answer "why did this specific answer come out wrong" after the fact, rather than only observing aggregate metrics.
Trace: request_id=8823
├─ retrieve() 142ms 12 chunks returned
├─ rerank() 38ms top 5 selected
├─ generate() 910ms model=gpt-x, tokens_in=1204, tokens_out=187
│ └─ tool_call: get_customer_balance 61ms success
└─ total 1091ms
Fig 11.2 — A single trace exposes exactly where time and tokens were spent, and which step (if any) failed.
⚠️ Common Mistake
Logging full prompts and responses without redacting sensitive data (PII, secrets, proprietary content). Observability tooling is a common, under-scrutinized source of data leakage — treat logs with the same access-control discipline as the production database itself.
11.6 Release Gate
🚧 Release Gate
No prompt/model/retrieval change goes to production merely because five manual tests looked good. Run the regression suite and compare quality, latency and cost.
Common Questions
Q: Why isn't manual spot-checking sufficient before shipping a prompt change?
A handful of examples cannot reliably represent the full input distribution or catch regressions on previously-passing edge cases; a versioned regression suite provides repeatable, comparable evidence instead of anecdote.
Q: When is LLM-as-judge inappropriate?
When the judge model's own biases or blind spots are uncalibrated against human judgment, or when the task requires domain expertise/legal-financial correctness the judge can't reliably assess — human review remains necessary there.
Q: What would you put on a production AI dashboard?
Task success/quality metrics, latency percentiles, token usage and cost by feature/tenant, tool failure rates, retrieval quality signals, and user feedback (thumbs-up/down, escalation rate).
Interview / Viva Questions
Q: Why should real production failures be added to the eval set?
They represent actual, observed failure modes rather than hypothetical ones, and adding them as permanent regression cases ensures the same bug can't silently reappear in a future change.
Q: What's the difference between deterministic checks and LLM-as-judge evaluation, and when would you use each?
Deterministic checks (schema validity, exact match, regex) are used when correctness is objectively verifiable; LLM-as-judge is used for nuanced quality (tone, helpfulness, faithfulness) that resists exact-match rules, but needs calibration against human agreement.
Q: What does "release gate" mean in the context of LLMOps?
A required check — running the full regression eval suite and comparing quality/latency/cost — that must pass before a prompt, model, or retrieval change is deployed to production.
Q: Why must sensitive data be redacted from observability logs?
Traces often capture full prompts and responses, which can contain PII or proprietary content; unredacted logs become a significant, easily-overlooked data-leakage surface.
Chapter Summary
- Evals adapt the ML testing discipline (Chapter 4) to non-deterministic LLM systems, providing repeatable evidence of quality before shipping changes.
- Eval datasets should include edge cases, adversarial inputs, and — critically — real production failures as permanent regression cases.
- Different system types (extraction, RAG, tools, agents) need different core metrics.
- LLM-as-judge scales evaluation but requires a calibrated rubric and periodic human spot-checks.
- Observability requires end-to-end tracing, cost/latency tracking, and careful redaction of sensitive data from logs.
- No change ships to production without passing the regression eval suite — a hard release gate.