Evaluation, testing and optimisation

Episode Nine turns integration into measurable system quality. By the end, you should be able to construct representative datasets, isolate retrieval and generation metrics, diagnose failure before changing the prompt, and optimise cost or latency without hiding downstream harm. Now define what success means for the whole system and each component. An architect should define success before optimising prompts, models, or infrastructure. Otherwise the team is tuning toward whichever demonstration looks impressive that day. What dimensions might Harbour Resolution evaluate? Task accuracy, evidence faithfulness, safety, security, latency, cost, consistency, escalation quality, and human effort. Each dimension needs an operational definition. Accuracy for structured extraction may be measured per field with different weights for consequential fields.

Text description

• Single average: Summarises overall performance but can conceal important differences and trade-offs.
• Stratified results: Breaks results down by case type, document quality, language, complexity, risk, or user cohort.

Faithfulness may measure whether every material claim is supported by an applicable retrieved source. Safety may measure prohibited actions attempted, unsafe advice, or failures to escalate specified cases. Latency may include time to first useful output and time to completed approved workflow. Cost may include model tokens, retrieval, reranking, tools, review, rework, and operational support. A single average score hides important trade-offs. For example, a system can improve average accuracy while becoming worse on vulnerable-customer cases. Use stratified reporting across case type, document quality, language, complexity, risk, and user cohort where relevant. How should an evaluation dataset be built? Sample the real task distribution, include difficult edge cases, preserve a held-out set, and maintain trusted labels with clear provenance.

Include negative cases where the correct behaviour is no action, no finding, or escalation. Teams often overfill datasets with obvious positive examples because they are easy to collect. That creates systems that appear sensitive but generate expensive false positives in production. Separate development, validation, and final test sets to reduce tuning leakage. Version the dataset, source content, prompt, model, tools, and scoring logic so results are reproducible. When production failures occur, add representative cases to a regression suite without endlessly fitting to individual anecdotes. Now separate component evaluation from end-to-end evaluation. Retrieval can be measured using recall of relevant sources, ranking quality, freshness, permission correctness, and metadata filtering.

Text description

• Retrieval: Relevant-source recall, ranking, freshness, permissions, and metadata filtering.
• Generation: Correctness, groundedness, completeness, style, and uncertainty handling with controlled context.
• Tool use: Tool selection, valid arguments, unnecessary calls, error recovery, and prohibited calls.
• Whole workflow: Resolution quality, human acceptance, handling time, customer outcome, and operational failure.

Generation can be measured using correctness, groundedness, completeness, style, and uncertainty handling given controlled context. Tool use can be measured using tool selection, argument validity, unnecessary calls, error recovery, and prohibited-call rate. The whole workflow can be measured using resolution quality, human acceptance, handling time, customer outcome, and operational failure. Why is this separation valuable? It identifies whether a failure comes from retrieval, prompting, model fit, tool design, validation, or workflow policy. Replacing the model will not fix a stale index or broken authorisation filter. Use trace replay where possible to compare components under controlled conditions. Now choose graders. Code-based graders are fast and reliable for exact structure, deterministic rules, executable tests, and known references.

Text description

• Code grader: Fast and reliable for exact structure, deterministic rules, executable tests, and known references.
• Human grader: Authoritative for nuanced domain judgement, but slower, costly, and subject to disagreement.
• Model grader: Scales rubric-based review, but requires evaluation for bias, consistency, and expert agreement.

Human graders are flexible and authoritative for nuanced domain judgement, but they are slower, expensive, and can disagree. Language-model graders scale nuanced rubrics, but they must themselves be evaluated for bias, consistency, and agreement with experts. Use the fastest reliable grader that fits each criterion. A code grader can confirm that every citation identifier exists. A domain expert may judge whether the cited policy actually applies. A calibrated model grader may screen large volumes and send uncertain or high-risk cases to experts. Should the same model configuration grade its own outputs? It can support iteration, but independent grading and periodic human calibration reduce correlated blind spots.

A grader prompt needs explicit rubrics and constrained output just like any other component. Measure inter-rater agreement and investigate systematic disagreement rather than averaging it away. Now discuss experiments. Change one major variable at a time when you want causal understanding. Run paired comparisons on the same examples to reduce noise. Use confidence intervals or repeated runs where stochastic variation matters. A B testing in production can measure user and business outcomes, but protect users through staged exposure and rollback criteria. Canary releases expose a small controlled cohort before wider rollout. Shadow mode lets the new system run without controlling outcomes, which is useful for consequential workflows.

Text description

• Expected answer and source: Confirm the trusted expected answer and the authority of its source.
• Input and routing: Inspect input quality and whether the task reached the correct route.
• Evidence and permissions: Check retrieved evidence, freshness, relevance, and access controls.
• Tools, prompt, and context: Review tool observations, prompt instructions, and context assembly.
• Model and validation: Inspect the generated output and the checks applied to it.
• Downstream actions: Verify rendering, approval, and state updates after generation.

Human override and correction rates are valuable signals, but interpret them carefully because reviewers may accept polished errors or resist unfamiliar correct answers. Qualitative review remains important for discovering failure modes the metric set did not anticipate. Let us build a diagnosis sequence. When an answer is wrong, first confirm the expected answer and source authority. Then inspect input quality and task routing. Then inspect retrieved evidence and permissions. Then inspect tool calls and observations. Then inspect prompt and context assembly. Then inspect model output and validation. Then inspect downstream rendering, approval, and state updates. This order follows the data rather than assuming the model caused every failure.

What does model mismatch look like? The task may require reasoning, context, modality, or tool reliability beyond the chosen configuration even when inputs and retrieval are sound. However, demonstrate that with comparative evaluation rather than intuition. What does prompt failure look like? The model has adequate evidence and capability but receives ambiguous criteria, conflicting instructions, poor examples, or an unclear output contract. What does retrieval failure look like? The model is supplied with missing, irrelevant, stale, unauthorised, or badly chunked evidence. Now optimise cost and latency without abandoning quality. Reduce irrelevant context and verbose tool results. Use prompt caching when a substantial stable prefix repeats across requests.

Place cacheable stable material before dynamic content according to current platform guidance. Use batch processing for asynchronous volume. Select the smallest model that passes the workload's evaluation thresholds. Route only genuinely complex cases to stronger configurations. Run independent tools in parallel where safe. Stream interactive answers to improve perceived responsiveness. Set sensible output limits and ask for concise machine-readable intermediate results. Reuse retrieved or computed artefacts when their freshness and authorisation remain valid. Every optimisation introduces assumptions that must be tested. Caching creates invalidation and privacy questions. Routing creates misclassification risk. Shorter context can remove needed evidence. Parallelism can amplify load and complicate partial failure.

Text description

• Cost per model call: May fall with a cheaper configuration while overlooking downstream effort and failures.
• Cost per successful outcome: Accounts for model use together with retrieval, tools, review, rework, and operational support.

A cheaper model can increase rework or human review. Optimise cost per successful outcome, not cost per model call. Production monitoring continues the evaluation loop. Track quality proxies, user feedback, override rates, retrieval drift, source freshness, tool errors, prohibited-action attempts, token use, latency percentiles, and spend. Detect changes by cohort and version. A stable average can hide a failure affecting one product or policy version. Set alert thresholds and define who investigates, who can disable the feature, and what fallback users receive. Episode Nine is complete when every proposed change names the metric it should improve, the cohort it affects, and the regression it could introduce.

For source navigation, use the success-criteria and evaluation guidance, latency material, usage-and-cost interfaces, analytics references, and the Anthropic Cookbook. In the next episode we will place governance and security controls around the system, including the threats carried through retrieved content and tools.