Build representative evaluation systems, diagnose cross-layer failures, compare releases, and turn incidents and feedback into reproducible improvements.
Define datasets, rubrics, deterministic and model-based metrics, human review, baselines, thresholds, experiments, and regression gates.
Isolate failures across prompts, context, retrieval, model invocation, tools, permissions, quotas, data, deployment, and observability.
GenAI testing must evaluate distributions of behavior, not one deterministic output. Define what must remain invariant—authorization, schema, prohibited actions, privacy, critical factual support—and what may vary within a rubric. Use representative, adversarial, boundary, and regression cases. Keep release decisions reproducible by versioning the dataset, rubric, model, prompt, retrieval, tools, guardrails, code, and scoring configuration.
Troubleshoot from observable symptoms through layers. Do not change the model first merely because the user sees a bad answer.
| Objective | Required judgment | Proof |
|---|---|---|
| 5.1 Build evaluation systems | Select datasets, deterministic checks, rubrics, model-based metrics, human review, thresholds, baselines, and experiments | The same versioned inputs reproduce a release decision and expose critical failures |
| 5.2 Troubleshoot GenAI applications | Isolate prompt, context, retrieval, model, tool, identity, quota, data, deployment, or telemetry failures | A layer-level hypothesis is tested before configuration changes are promoted |
Start with the real task and harm model. Build cases from normal workflows, difficult examples, rare but costly failures, known incidents, user feedback, supported languages and modalities, authorization boundaries, malformed input, prompt injection, stale data, missing evidence, tool failure, and required abstention. Use synthetic data where production data is sensitive, but verify that the synthetic set still represents the important decisions.
Separate development, tuning, and release sets. If prompt authors repeatedly inspect the release answers, the gate becomes a tuning set and overstates generalization. Version cases and expected properties. Preserve a rationale for why each case exists and which requirement or risk it covers.
Use deterministic checks where possible: JSON schema, required fields, allowed values, exact identifiers, citation presence, permission decision, tool parameters, latency, cost, and prohibited strings. Use rubrics and human review for context-dependent quality, relevance, completeness, tone, or harm. Model-based judges can scale review but may share model biases, be sensitive to prompt wording, or prefer verbose answers. Calibrate them against human-reviewed examples and do not let a judge override a hard deterministic failure.
Metrics must match the layer. Retrieval evaluation can measure whether required evidence is found and ranked. Generation evaluation can measure support, completeness, correctness, and citation quality. Safety evaluation measures defined harmful or prohibited behavior and false positives. Operational evaluation measures latency, throughput, errors, quotas, and cost. Business evaluation measures task completion and escalation. A single aggregate score can hide a catastrophic failure, so define hard thresholds and slice results by critical scenario, language, tenant, or user group where relevant.
Compare candidate and baseline on the same versioned set. Record confidence intervals or sample limitations when the set is small. Require improvement or non-regression according to the change goal. A candidate with higher average quality must still fail if it violates authorization, leaks data, performs a prohibited action, or crosses another hard safety boundary.
Use offline evaluation before deployment, integration tests against real service boundaries, and limited online evaluation with approved traffic and monitoring. Shadow testing can compare behavior without returning candidate output, but it still processes data and needs privacy approval. A/B testing is appropriate only when exposure and metrics are ethically and operationally acceptable. Keep rollback inputs ready.
Reproduce with the smallest safe case. Compare the failed trace to a successful baseline. Change one controlled variable. Record hypothesis, evidence, result, and remediation. Do not repair production by making an unversioned prompt edit.
| Symptom | Likely layers | High-value test |
|---|---|---|
| Fluent unsupported answer | Retrieval, prompt, grounding, abstention | Inspect retrieved evidence and run a no-support case |
| Correct source not retrieved | Chunking, embedding, filter, index, query | Check gold-passage recall before generation |
| Cross-tenant result | Authorization, metadata, cache, memory | Attempt known adversarial tenant queries |
| Intermittent malformed JSON | Prompt, model behavior, streaming, parser | Capture raw bounded output and enforce schema retry policy |
| Tool action duplicated | Timeout, retry, idempotency, state | Replay with the same idempotency key |
| Quality drops after release | Model, prompt, corpus, router, user drift | Compare exact release inputs by dataset slice |
| Latency spikes | Retrieval, model queue, tool, concurrency, quota | Trace time by layer and correlate saturation |
A candidate release improves average answer relevance from the current benchmark, but occasionally cites a document the user cannot access and doubles p95 latency.
