Evaluation & Feedback

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

Layer Guide — X7: Evaluation & Feedback
What this layer is responsible for
Knowing whether the system is good, whether a change made it better or worse, and catching degradation before users do.
The defining constraint: you cannot unit test a probabilistic system. There is no assertion that the model returns exactly this string. The same input produces different outputs, both potentially correct. A change that improves ninety cases and breaks ten is a net improvement that a traditional test suite would report as ten failures.
So testing becomes measurement over a set. Not "does this produce the right answer" but "across a representative sample, how often is the output acceptable, and did that rate move?"
Three framings carry through.
Localize before you optimize. An end-to-end quality score tells you the system is worse and not where. Retrieval, generation, and safety fail differently and are fixed in different layers, so component metrics are what make quality actionable.
Offline and online evaluation answer different questions. Offline asks "would this change have been better on cases we've curated." Online asks "is it better for real users right now." Neither substitutes for the other, and offline-only evaluation is how a change that scores well ships and degrades production.
Evaluation infrastructure is the thing that makes every other layer tunable. Chunk size, embedding dimension, reranking, prompt versions, model selection, guardrail thresholds — every one of those decisions is guesswork without a way to measure the effect. This layer is the prerequisite for the rest of the guide being actionable rather than theoretical.

Pattern class 1 — Evaluation dataset construction
The problem in full
Measurement requires cases to measure against: representative inputs with some notion of what a good output looks like. This dataset is the foundation of everything else in the layer, and building it is the unglamorous work that gets deferred indefinitely.
The variables:
What "correct" means for your task — a specific answer, a set of required facts, a quality bar, a preference between candidates.
Coverage — common cases, edge cases, adversarial cases, and the long tail.
Size, which trades statistical confidence against construction cost.
How it stays current as the corpus and requirements change.
Who can author it — general annotators or domain experts.
Subtypes
Golden set with reference answers. Curated questions with known-good responses. Highest value, highest cost, and the basis for reference-based metrics.
Question with required facts. Rather than a full reference answer, a list of facts the response must contain. Much cheaper to author and more robust to legitimate phrasing variation.
Question with source passage. For retrieval evaluation: the query plus the chunk that should be retrieved. This is what makes recall measurable.
Production-sampled. Real queries from invocation logs, labeled after the fact. Most representative, and requires logging to have been on.
Adversarial set. Jailbreaks, injections, out-of-scope questions, and known failure modes.
Regression set. Cases that previously failed and were fixed, retained permanently so they don't regress.
Decision variables
What is the smallest useful set? Fewer cases than people assume. A hundred well-chosen cases detect meaningful regressions. A thousand gives tighter confidence intervals. Starting with fifty is far better than waiting to build a thousand, and the fifty will teach you what the thousand should contain.
Does it match production distribution? A set of clean, well-formed questions evaluating a system whose real traffic is messy and elliptical produces scores that don't predict production quality. Sampling from logs is the corrective.
Is it held out? Cases used to tune prompts must not also be the cases used to judge them, or you're measuring memorization. Keep a genuinely untouched evaluation set separate from a development set.
How does it stay current? The corpus changes, so reference answers go stale. A question whose correct answer changed when a policy was updated will fail for the right reason and look like a regression. Dataset maintenance is ongoing work.
AWS mechanisms
Model invocation logging to S3, queried with Athena, as the source of production-sampled cases. This is why logging matters beyond debugging.
SageMaker Ground Truth for labeling at volume, with private or vendor workforces where content is confidential and Mechanical Turk disqualified for anything sensitive.
A2I producing labels as a byproduct of production review.
Bedrock batch inference for generating candidate answers or synthetic questions at scale cheaply.
S3 with versioning as the dataset store, so evaluations can reference a dataset version.
DynamoDB or Glue Data Catalog for dataset metadata and lineage.
The details that decide success
Start small and grow from failures. Every production failure becomes a permanent case. Over a year this accretes into a set that reflects your actual risk surface far better than anything designed up front.
Required-facts beats reference-answer for most tasks. Authoring "the response must mention the 30-day window and the exception for damaged goods" is faster than writing an ideal paragraph, and it doesn't penalize legitimate phrasing differences.
Include the cases you're afraid of. Ambiguous questions, questions with no answer in the corpus, questions that look adjacent to a prohibited topic. These are where systems fail and where evaluation sets are thinnest.
Version the dataset alongside results. A score is meaningless without knowing which dataset version produced it.
Record why each case exists. A case added because of a specific incident should carry that context, or someone will eventually delete it as redundant.
Cost model
Human authoring time, usually the dominant cost and invisible in any AWS bill. Ground Truth per object plus workforce cost. Model calls for synthetic generation, which batch inference makes cheap. Storage is negligible.
Failure modes and diagnostics
No dataset. Every tuning decision is guesswork. The most common state.
Distribution mismatch. Good offline scores, unchanged production quality.
Contaminated holdout. Tuning against the evaluation set, producing inflated confidence.
Stale references. Correct answers marked wrong after a corpus update.
Only easy cases. A set that everything passes and that detects nothing.

Pattern class 2 — Retrieval evaluation
The problem in full
In a RAG system, if retrieval doesn't return the right passages, nothing downstream can compensate. Evaluating retrieval separately is what lets you localize a quality problem to a layer instead of blaming the model.
The metrics
Recall@k — of the passages that should have been retrieved, what fraction appear in the top k. The primary retrieval metric. If the right passage isn't in the context, the answer cannot be right.
Precision@k — of the retrieved passages, what fraction are relevant. Low precision wastes context budget and distracts the model.
MRR (mean reciprocal rank) — how high the first relevant result appears. Sensitive to ordering, which is what reranking improves.
NDCG — rank-sensitive relevance accounting for graded relevance rather than binary.
Hit rate — the fraction of queries where at least one relevant passage was retrieved. Coarse and useful as a headline.
Context precision and context recall — of the context assembled, how much was actually needed, and did it contain everything needed. These bridge retrieval and generation.
Decision variables
Which metric matters for your failure mode? Low recall means the passage isn't being found — fix with hybrid search, query transformation, chunking, or index parameters. Good recall with poor MRR means it's found and ranked badly — fix with reranking. These are different problems with different fixes, and the metrics distinguish them.
What is k for measurement? Measure at the k you actually use in production, and also at a larger k — if recall@50 is high but recall@5 is low, reranking will help. If recall@50 is also low, reranking cannot help and the problem is upstream.
How is ground truth established? Someone must say which passages are relevant to which queries. This is the expensive part, and it's why production-sampled queries with annotated results are valuable.
AWS mechanisms
Bedrock Knowledge Base evaluation — evaluates retrieval and generation, with retrieval metrics including context relevance and coverage, and generation metrics assessed against the retrieved context. This is the managed path and it evaluates the two stages separately, which is exactly the localization the layer needs.
Retrieve called directly in an evaluation harness, returning ranked chunks with scores for comparison against expected passages.
Exact brute-force search on a sample as ground truth for ANN recall specifically — if exact search finds the passage and ANN doesn't, the problem is index parameters rather than embeddings or chunking. This diagnostic is cheap and rarely run.
Athena over logged retrieval results for production distribution analysis: zero-result rates, score distributions, and result count patterns.
Custom evaluation harnesses in Lambda, Step Functions, or SageMaker Processing jobs.
The details that decide success
Log retrieval results in production. Chunk identifiers and scores, correlated to the request. Without this, offline evaluation is the only signal and production retrieval quality is invisible.
Separate the ANN recall question. Comparing approximate against exact search isolates index tuning from every other variable.
Measure zero-result rate continuously. It's a direct measure of corpus coverage and a prioritized list of gaps.
Re-evaluate after every build-time change. Chunking, embedding model, dimension, augmentation, and index parameters all change retrieval, and the evaluation set makes those changes assessable rather than faith-based.
Watch precision as well as recall. Optimizing recall by raising k inflates cost and can degrade answers by burying the relevant passage in noise.
Cost model
Retrieval calls per evaluation run — cheap. Ground truth annotation is the real cost. Managed evaluation jobs are priced per evaluation.
Failure modes and diagnostics
Only end-to-end evaluation. Cannot tell whether retrieval or generation failed.
No retrieval logging in production.
Measuring at the wrong k.
Recall optimized at the expense of precision.

Pattern class 3 — Generation quality evaluation
The problem in full
Given the right context, is the response good? Correct, complete, grounded, relevant, appropriately toned, and free of fabrication.
This is harder to measure than retrieval because "good" resists a single definition and reference-based comparison works poorly for open-ended generation.
The metric families
Reference-based lexical metrics — exact match, F1, ROUGE, BLEU. They compare the response to a reference by word overlap. Weak for generative output, because a correct answer phrased differently scores badly. Useful for constrained tasks with narrow correct answers — extraction, classification, short factual lookup — and misleading for anything open-ended.
Semantic similarity — comparing embeddings of response and reference. Better than lexical overlap, still blunt: a response can be semantically similar and factually wrong in a critical detail.
Fact-based checking — does the response contain the required facts, and does it avoid the prohibited ones. Robust to phrasing, cheap to author, and the most practical approach for most systems.
Faithfulness and grounding — is every assertion supported by the provided context. This is the hallucination metric and the one that matters most in RAG.
Answer relevance — does the response address the question, independent of correctness.
Task-specific rubrics — a structured scoring guide for the qualities you care about, applied by a model or a human.
Decision variables
Is the task constrained or open-ended? Constrained tasks tolerate reference-based metrics. Open-ended ones need rubrics or fact-based checks.
What failure worries you most? Hallucination, incompleteness, tone, refusal, or verbosity. Measure that specifically rather than a generic quality score.
How correlated are automated metrics with actual quality? This must be validated. Compute the metric on a sample that humans have also judged, and check they agree. An automated metric that doesn't correlate with human judgment is worse than no metric, because it produces confident wrong conclusions.
AWS mechanisms
Bedrock model evaluation jobs in three modes:
Automatic — built-in metrics computed programmatically over a dataset.
LLM-as-a-judge — a model scores responses against criteria (covered next).
Human — a workforce scores responses, using your own team or an AWS-managed workforce.
Available metric dimensions typically include correctness, completeness, faithfulness, helpfulness, coherence, relevance, harmfulness, refusal, and stylistic qualities — and for RAG evaluations, citation precision and coverage.
Bedrock Knowledge Base evaluation assessing retrieval and generation together, which is the RAG-specific form.
Guardrail contextual grounding scores as a continuous production signal for faithfulness — not just a block/allow decision but a distribution you can track.
SageMaker Clarify for foundation model evaluation including accuracy, robustness, toxicity, and bias dimensions.
Custom harnesses running the metric computation you need, orchestrated by Step Functions or SageMaker Processing.
Athena and Quick Sight for storing and presenting results over time.
The details that decide success
Validate your metrics against human judgment. On a sample, at least once, and again when the task changes. This is the step that separates measurement from theatre.
Measure faithfulness separately from correctness. A response can be faithful to retrieved context that was itself wrong, or correct despite unsupported reasoning. Conflating them hides both failure modes.
Track distributions, not just averages. A mean quality score of 0.85 could be uniformly decent or mostly excellent with a bad tail. The tail is what users complain about.
Evaluate refusals. A system that refuses too often looks safe on harm metrics and is useless. Refusal rate is a quality metric.
Keep the evaluation harness fast enough to run often. An evaluation taking four hours will not be run on every change.
Cost model
Model calls for the responses being evaluated, plus judge calls or human time. For a 500-case set evaluated on every change, this recurs — batch inference for generating the responses reduces it substantially.
Failure modes and diagnostics
Lexical metrics on open-ended tasks. Penalizing correct answers.
Unvalidated metrics. Optimizing something that doesn't correlate with quality.
Averages hiding tails.
Refusal rate unmeasured.
Evaluation too slow to run routinely.

Pattern class 4 — LLM-as-judge
The problem in full
Human evaluation is accurate and doesn't scale. Automated metrics scale and are crude for open-ended output. Using a strong model as the judge sits between: it scales, handles nuance, and applies rubrics consistently — with characteristic biases that must be managed.
The known biases
Position bias. When comparing two responses, judges favor one position systematically. Mitigate by randomizing order and evaluating both orderings.
Verbosity bias. Longer responses score higher regardless of quality. Mitigate with rubrics that penalize unnecessary length explicitly.
Self-preference. A judge favors output from the same model family. Mitigate by using a different model as judge than the one being evaluated where practical.
Leniency. Judges tend to accept, particularly when asked "is this acceptable" rather than scored against explicit criteria.
Formatting susceptibility. Well-formatted but shallow responses score above dense correct ones.
Decision variables
Pointwise or pairwise? Pointwise scoring against a rubric gives an absolute score. Pairwise comparison — which of these two is better — is more reliable because relative judgment is easier than absolute, and it's the natural fit for comparing a change against a baseline.
Which judge model? A strong one, since judging is a harder task than it looks. Also a different one from the system under evaluation where possible.
How explicit is the rubric? Detailed criteria with examples produce far more consistent judgments than "rate this 1-5." The rubric is prompt engineering and deserves the same care.
How is it calibrated? Run the judge on cases humans have scored and measure agreement. A judge that doesn't agree with humans is measuring something else, and you need to know that before trusting it.
AWS mechanisms
Bedrock evaluation jobs in LLM-as-a-judge mode, with built-in metric dimensions and the ability to evaluate model outputs or RAG outputs.
Custom judge implementations invoking a model with a rubric prompt, using structured output via tool-calling so scores come back as clean data rather than prose to parse.
Batch inference for judging at scale, since evaluation runs have no latency requirement — this is a clear batch use case.
Prompt Management versioning judge prompts, since a rubric change invalidates comparability with prior results.
Ground Truth collecting the human judgments used for calibration.
The details that decide success
Calibrate against humans before trusting it. Agreement rate on a sample. Without this you have a number of unknown meaning.
Randomize position in pairwise comparisons, and ideally evaluate both orderings and require consistency.
Version the judge prompt and the judge model. Changing either changes the scale, so historical comparisons become invalid — this catches people out when scores shift after a judge upgrade rather than a system change.
Ask for reasoning before the score. Judges produce more reliable scores when they articulate justification first, and the reasoning is diagnostically useful.
Use pairwise for change decisions. "Is B better than A" is more reliable than comparing two absolute scores.
Don't judge with the system's own model on its own output where avoidable.
Cost model
One or more judge calls per evaluated case. For a 500-case set with pairwise comparison in both orderings, that's a thousand judge calls per evaluation run — real money at frontier model rates, and a strong argument for batch inference and for a capable-but-not-maximal judge model.
Failure modes and diagnostics
Uncalibrated judge. Scores of unknown validity.
Position and verbosity bias unmitigated.
Judge prompt changed mid-programme. Historical results incomparable.
Self-preference inflating scores for the incumbent model.
Judge cost exceeding value at large scale.

Pattern class 5 — Human evaluation
The problem in full
For final quality judgments, subtle domain distinctions, and calibrating everything else, humans remain the reference. The problem is throughput, consistency, and cost.
Decision variables
Who can judge? General annotators for clarity and coherence. Domain experts for correctness in specialized fields — and expert time is scarce and expensive, which bounds the scale.
What is the task format? Absolute scoring, pairwise preference, error annotation, or free-form critique. Pairwise is easiest and most consistent; absolute scoring drifts between annotators and over time.
How is consistency measured? Inter-annotator agreement on overlapping cases. Low agreement means the rubric is ambiguous, and no amount of additional annotation fixes an ambiguous rubric.
Is the content sensitive? This determines the workforce and eliminates public crowdsourcing.
AWS mechanisms
Bedrock evaluation jobs in human mode, using your own workforce or an AWS-managed one, with configurable task instructions and rating dimensions.
SageMaker Ground Truth with custom task templates for side-by-side comparison, error annotation, or rubric scoring. Annotation consolidation across multiple annotators per case, which both improves quality and yields the agreement measurement.
A2I for production-integrated review, which produces evaluation data as a byproduct of quality control rather than as a separate exercise.
Private and vendor workforces for confidential content; Mechanical Turk disqualified there.
The details that decide success
Write the rubric before annotating, calibrate annotators against a gold set, and measure agreement before scaling. Skipping this produces a dataset that encodes disagreement as noise.
Overlap a fraction of cases across annotators permanently, so agreement is monitored rather than measured once.
Prefer pairwise. It's faster, more consistent, and directly answers the question you usually have.
Capture the reasoning, not just the score. Error annotations are far more actionable than a number, and they feed directly into fixes.
Human evaluation is the calibration source for automated and judge-based metrics. Budget for a recurring sample, not just an initial one.
Cost model
Annotator time, dominant and usually invisible in AWS billing. Ground Truth per object. Expert annotation can be very expensive per case, which is what makes automated metrics worth calibrating.
Failure modes and diagnostics
No agreement measurement. Noisy labels treated as truth.
Absolute scoring drift. Scores shifting over time as annotators recalibrate.
Experts unavailable. Evaluation blocked on scarce time.
Confidential content in a public workforce.

Pattern class 6 — Feedback capture
The problem in full
Production users are continuously evaluating the system, and most of that signal is discarded. Capturing it turns every interaction into evaluation data at near-zero marginal cost.
Subtypes
Explicit rating. Thumbs up/down, star ratings. Low participation, clear signal.
Explicit correction. The user edits or rewrites the response. Highest-value signal — it's a labeled example.
Implicit behavioral. Regeneration, abandonment, copy actions, follow-up rephrasing, session length. Free, high volume, noisy.
Downstream outcome. Did the ticket get resolved, did the recommendation get accepted, did the document get approved. The most meaningful signal and the hardest to instrument.
Escalation. The user asked for a human. An unambiguous quality signal.
Decision variables
What can you capture without asking? Regeneration and abandonment are strong negative signals requiring no user effort. Copy or export is a positive one. These have far higher volume than explicit ratings and should be instrumented first.
How do you handle response bias? Explicit feedback over-represents strong reactions, particularly negative ones. Treat rates as directional rather than absolute.
What context is captured with the feedback? This is the decisive detail. A thumbs-down alone is nearly useless. A thumbs-down joined to the prompt, retrieved context, model version, prompt version, and trace is a diagnosable case.
Does downstream outcome exist? For contact centers, ticket resolution. For sales tools, conversion. If an outcome exists, connecting to it is worth more than any proxy.
AWS mechanisms
Application-captured events written to DynamoDB for operational access or streamed via Kinesis Data Firehose to S3 in Parquet for analysis.
Correlation identifiers threading the feedback to the invocation log, trace, retrieval results, and conversation record. Without this the feedback is unattributable and worthless, and it cannot be added retroactively.
Athena joining feedback against invocation logs for analysis.
Quick Sight for trend dashboards.
A2I for structured review of flagged responses.
EventBridge routing negative feedback to a triage queue or alerting on rate changes.
The details that decide success
Capture context, always. The single most common feedback mistake is storing a rating with no reference to what was rated.
Instrument implicit signals first. Higher volume, no user burden, and often better predictors than explicit ratings.
Close the loop visibly. Users who see feedback act on things provide more of it.
Feed corrections into the evaluation set. A user correction is a labeled case — the highest-value artifact this pattern produces.
Watch feedback rate as a metric. A sudden change in thumbs-down rate is a quality alarm, and it's faster than any offline evaluation cycle.
Design it in early. Retrofitting correlation identifiers means historical feedback stays unattributable.
Cost model
Negligible — event storage and processing. The value is disproportionate to the cost, which is why the main failure is not doing it rather than doing it expensively.
Failure modes and diagnostics
Feedback without context. Unusable.
Explicit only. Missing the high-volume implicit signal.
Never analyzed. Collected and ignored.
Response bias treated as absolute.
Corrections discarded rather than becoming evaluation cases.

Pattern class 7 — Online experimentation
The problem in full
Offline evaluation predicts; production decides. A change that improves offline scores can degrade real usage — different distribution, different user behavior, unmodeled interactions. Online experimentation measures the actual effect.
Subtypes
A/B testing. Traffic split between variants, outcomes compared.
Canary release. A small traffic fraction on the new version, monitored, then ramped.
Shadow testing. The new version processes real traffic in parallel without serving results. Zero user risk, no user-outcome signal.
Interleaving. For ranking, results from both variants merged and clicks attributed. Higher statistical power for retrieval changes.
Holdback. A fraction permanently on the old version, measuring cumulative effect over time.
Decision variables
What is the success metric? It must be measurable online and meaningful. Thumbs-up rate, regeneration rate, escalation rate, task completion, or a business outcome. Offline quality scores are not available in production.
How long to run? Long enough for statistical significance given traffic and effect size. Generative quality effects are often subtle, and small effects need substantial traffic.
How is assignment done? Per user, per session, or per request. Per-user is usually right for conversational systems — inconsistent behavior within a session is confusing and contaminates the measurement.
What is the risk? Shadow testing carries none and gives no user-outcome data. Canary limits blast radius. Full A/B exposes half your users to a possibly worse experience.
AWS mechanisms
AppConfig feature flags with deployment strategies — gradual rollout with configurable growth, bake time, and automatic rollback on CloudWatch alarms. This is the primary mechanism for controlled generative rollout, since the change is usually a prompt, model, or retrieval setting rather than code.
SageMaker production variants with traffic weights for endpoint-level splits, and shadow variants which receive a copy of production traffic without returning responses — purpose-built for the shadow pattern.
Application-level assignment with the variant recorded in logs and feedback, which is what makes per-variant analysis possible.
Route 53 weighted routing for coarse splits between separate deployments.
CodeDeploy canary and linear deployment configurations with alarm-based rollback for code changes.
Athena and Quick Sight for analysis by variant.
The details that decide success
Record the variant on every request. In logs and in feedback. Without it, no analysis is possible.
Define the metric and duration before starting. Otherwise you'll stop when the numbers look good, which is how noise becomes a conclusion.
Assign per user for conversational systems. Switching mid-conversation confuses users and pollutes the data.
Automate rollback. An alarm-triggered revert limits damage from a bad variant far better than a human noticing.
Shadow first for risky changes. Compare outputs offline on real traffic before exposing anyone.
Watch cost as an experiment metric. A variant that improves quality and doubles token consumption may not be a win, and cost is easy to omit from the success criteria.
Cost model
Running two variants means serving both — shadow testing doubles inference cost for the shadowed fraction while producing no user value directly. Budget for it and scope the fraction accordingly.
Failure modes and diagnostics
Variant not recorded. Experiment unanalyzable.
Stopping early on noise.
Per-request assignment in conversational systems.
No rollback automation.
Cost ignored in the success criteria.

Pattern class 8 — Regression gating
The problem in full
Prompts, models, retrieval settings, and guardrails change. Each change can degrade quality in ways no conventional test detects. Gating deployment on an evaluation run is how you prevent regressions from shipping.
This is what makes prompts and model configuration deployable artifacts rather than things people edit in a console.
Decision variables
What triggers evaluation? Prompt changes, model version changes, retrieval configuration changes, guardrail changes, and corpus rebuilds all warrant it. Code changes that don't touch the generative path usually don't.
What is the pass criterion? An absolute threshold ("faithfulness above 0.9") or a relative one ("no worse than baseline by more than 2%"). Relative is generally better, because absolute thresholds either block everything or nothing depending on where you set them.
How long can the gate take? An evaluation run that takes two hours will be bypassed. Keeping the gate set small and fast matters more than making it comprehensive — a comprehensive nightly run plus a fast gate is the usual compromise.
Blocking or advisory? Blocking prevents regressions and creates pressure to bypass. Advisory informs and gets ignored. Blocking on a small high-confidence set, advisory on the broader one, is a reasonable split.
AWS mechanisms
CodePipeline with a stage invoking a Bedrock evaluation job and failing the deployment on regression. This is the canonical shape.
CodeBuild running a custom evaluation harness.
Step Functions orchestrating a multi-stage evaluation with parallel case execution.
Lambda for lightweight gates on small sets.
Bedrock Prompt Management versions as the artifact being gated, promoted only after passing.
SageMaker Model Registry with approval status gating model deployment, and the approval transition triggering the pipeline via EventBridge.
AppConfig validators and deployment strategies as the runtime gate for configuration changes, with alarm-based rollback catching what pre-deployment evaluation missed.
The details that decide success
Gate prompt changes like code changes. A prompt edit can degrade quality as severely as a code bug, and treating prompts as configuration edited in a console is how silent regressions happen.
Keep the gate fast. Under ten minutes, or it gets bypassed.
Use relative thresholds against a stored baseline, and update the baseline deliberately when quality genuinely improves.
Include adversarial and regression cases. Previously-fixed failures should be permanently guarded.
Report the diff, not just pass/fail. Which cases changed, and how, is what makes the gate useful rather than obstructive.
Combine with runtime rollback. Pre-deployment evaluation catches what your set covers; alarm-based rollback catches what it doesn't.
Cost model
Model calls per gated case per pipeline run. A hundred-case gate on a frequently-deploying pipeline accumulates — batch inference is impractical here because the gate needs a fast answer, so this is one of the few places real-time evaluation is justified.
Failure modes and diagnostics
No gate on prompt changes. The most common gap.
Gate too slow. Bypassed under pressure.
Absolute thresholds that block everything or nothing.
Pass/fail with no diff.
Stale baseline.

Pattern class 9 — Adversarial evaluation
The problem in full
Ordinary evaluation measures behavior on ordinary inputs. Adversarial evaluation measures behavior under attack: jailbreaks, injections, out-of-scope probing, and attempts to extract data or induce harmful output.
This is testing the safety layer, and it needs its own dataset and cadence.
Subtypes
Direct injection. Attempts to override instructions.
Indirect injection. Instructions planted in retrievable content or tool results — testing that the system screens the assembled context, not just user input.
Data extraction. Attempts to elicit content from other tenants, the system prompt, or the training data.
Scope probing. Requests just outside intended boundaries.
Excessive agency. Attempts to induce an agent into unintended actions — the highest-stakes category.
Denial of wallet. Inputs designed to maximize cost.
Decision variables
What is the threat model? A public consumer product faces adversarial pressure an internal tool doesn't. Test proportionally.
Automated or human red-teaming? Automated suites cover known techniques cheaply and repeatedly. Human red-teaming finds novel ones and doesn't scale. Both, at different cadences.
What does failure mean? For content filtering, an unsafe response. For agents, an unintended action — and this is the more consequential test, because it exercises permissions rather than filters.
How often? Every deployment for the automated suite; periodically for human red-teaming; after any change to guardrails, tools, or permissions.
AWS mechanisms
A curated adversarial dataset run through the standard evaluation harness, with pass criteria being appropriate refusal or safe handling.
ApplyGuardrail for testing guardrail behavior directly, without a model call, which makes iterating on filter configuration fast and cheap.
Guardrail trace output showing which policy fired on each adversarial case — the diagnostic for tuning.
Bedrock evaluation jobs with harmfulness and refusal metric dimensions.
Agent trajectory logs for verifying that injection attempts did not produce unintended tool calls.
IAM Access Analyzer policy checks verifying the agent's permissions haven't widened — which is testing the containment boundary rather than the filter.
CloudWatch guardrail intervention metrics as the production signal for attack rate.
The details that decide success
Test indirect injection specifically. Plant instructions in a document, index it, and query in a way that retrieves it. This is the vector most systems don't test and the one most likely to succeed.
Test the blast radius, not just the filter. For agents, the question is what a successful injection could actually do — which is answered by reviewing permissions, not by counting blocked prompts.
Maintain the suite. New techniques appear. A suite from a year ago tests last year's attacks.
Measure over-refusal too. A system that refuses everything passes every harm test and is useless. Include legitimate edge cases that should succeed.
Alarm on production intervention rate. A rising rate means either an attack or an over-tuned filter.
Cost model
Model and guardrail calls per adversarial case. ApplyGuardrail testing without model invocation is substantially cheaper for filter tuning. Human red-teaming is expert time.
Failure modes and diagnostics
Only direct injection tested.
Filters tested, permissions not.
Stale suite.
Over-refusal unmeasured.

Pattern class 10 — Responsible AI evaluation
The problem in full
Beyond correctness and safety: is the system fair across groups, is its behavior explainable, and does it meet the obligations that apply to consequential decisions.
Decision variables
Is this a consequential decision? Lending, hiring, insurance, healthcare, and housing carry legal obligations around fairness and explainability. Informational assistants generally don't, though reputational considerations still apply.
What groups matter? Fairness is measured with respect to specified attributes, which must be named. You cannot detect bias along a dimension you didn't identify.
What kind of system is it? Fairness and explainability tooling is mature for predictive models — bias metrics, SHAP attributions. For generative output the question is different: does the model produce systematically different quality or tone for different groups.
AWS mechanisms
SageMaker Clarify for predictive models: pre-training bias metrics on the dataset (class imbalance, difference in positive proportions), post-training bias metrics on predictions (disparate impact, accuracy differences across groups), and SHAP-based feature attribution for global and per-prediction explanation. Also supports foundation model evaluation dimensions including bias and toxicity.
Bedrock evaluation with stereotyping and harmfulness metric dimensions.
SageMaker Model Monitor for bias drift and feature attribution drift in deployed predictive models — requiring data capture enabled on the endpoint, which is another default-off switch.
Citations from RetrieveAndGenerate as the generative form of explainability — showing the sources an answer drew on.
Guardrail trace as the record of what policy applied.
AWS Audit Manager and Config for evidence collection where compliance obligations require it.
The details that decide success
Clarify is for predictive models; guardrails are for generative content. Both are "responsible AI" and they answer different questions — fairness and explainability versus content safety. Substituting one for the other is a category error.
Model Monitor is not a hallucination detector. It measures statistical drift against a training baseline. Generative faithfulness is contextual grounding's job.
Name the sensitive attributes. Bias measurement requires them.
Citations are the practical explainability mechanism for RAG, and they raise the stakes on retrieval being right — a cited wrong answer is more damaging than an uncited one.
Test generative quality across groups. Does the assistant answer questions about some populations less well? This requires a deliberately constructed evaluation set and is rarely done.
Cost model
Clarify processing jobs, Model Monitor scheduled jobs, and evaluation runs. Modest relative to the regulatory cost of getting it wrong.
Failure modes and diagnostics
Clarify proposed for generative content safety.
Model Monitor proposed for hallucination.
Data capture not enabled, so monitoring cannot run.
Sensitive attributes unnamed.
Fairness assessed once at launch and never again.

Putting the layer together
A working evaluation practice has four loops running at different cadences:
Per change. A fast regression gate in the pipeline — small case set, relative threshold, blocking.
Per release. A fuller offline evaluation including adversarial cases, plus a canary or shadow run before full rollout.
Continuously in production. Quality signals from X6 — grounding scores, intervention rates, zero-result rates, refusal rates — plus feedback capture and synthetic canaries asserting on content.
Periodically. Human evaluation for calibration, red-teaming, fairness assessment, and evaluation set maintenance.
Most organizations have none of these and diagnose quality by anecdote. Having even the first and third — a fast gate and production signals — puts you ahead of most.

Cross-cutting decisions in this layer
Does an evaluation set exist, is it held out, and does it match production distribution?
Can you localize a failure to retrieval versus generation? Only with component metrics and logged retrieval results.
Are automated metrics calibrated against human judgment?
Is feedback captured with enough context to be diagnosable?
Is there a regression gate on prompt and model changes?
Is there a production quality signal that doesn't require asking users?
Is adversarial testing covering indirect injection and agent permissions?
Interfaces to adjacent layers
From X6 (Observability). Invocation logs are the source of production-sampled evaluation cases; correlation identifiers make feedback attributable; quality signals are the continuous production layer of this practice. Everything here depends on logging having been enabled.
To B2a/B2b and C1. The evaluation set is what makes chunking, embedding, hybrid weighting, reranking, and index parameters tunable rather than guessed. It is the single artifact that unlocks the build-time layers.
To B3 (Customization). The held-out set is what determines whether a fine-tune helped, and whether it caused catastrophic forgetting.
To S1 (Safety). Adversarial evaluation tests this layer's controls, and over-refusal measurement keeps tuning honest.
To X9 (Change management). Regression gating is where evaluation becomes a deployment control, and alarm-based rollback is the runtime complement to pre-deployment evaluation.
To X8 (Cost). Evaluation has real cost — judge calls, shadow traffic, gate runs — and quality changes have cost consequences that belong in the success criteria.

Unstarred

Dunk (exclude from study)

Evaluation & Feedback

Review Questions

Todos

Sensemaker