Build representative evaluation systems, diagnose cross-layer failures, compare releases, and turn incidents and feedback into reproducible improvements.
Define datasets, rubrics, deterministic and model-based metrics, human review, baselines, thresholds, experiments, and regression gates.
Isolate failures across prompts, context, retrieval, model invocation, tools, permissions, quotas, data, deployment, and observability.
Build representative, adversarial, boundary, refusal, safety, privacy, latency, and cost cases from real task requirements without exposing protected data. Version the dataset, expected behavior, rubric, evaluator, model, prompt, retrieval configuration, tools, and thresholds.
Use deterministic checks for structure, exact constraints, citations, permissions, and known facts where possible. Use human review for contextual judgment. Model-based evaluators can scale review but need calibration, disagreement analysis, bias controls, and protected evaluation inputs.
Compare candidates against a baseline. Use hard gates for safety, authorization, and critical task failures; averages can hide severe regressions. Sample production behavior through privacy-safe feedback and convert confirmed failures into reproducible evaluation cases.
Reproduce the input and release version. Isolate data, retrieval, context, prompt, model invocation, guardrail, tool, identity, quota, network, application, and output-processing layers. Change one variable at a time. A model swap is not a diagnosis.
Given an unsupported answer, determine what evidence would distinguish missing source data, failed ingestion, poor chunking, retrieval miss, prompt failure, model behavior, citation rendering, and stale cache.
