Module 5: Agent Evaluation

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

Module 5 — Agent Evaluation
Tier: Standard. Covers Skill 5.1.7 (evaluating agentic systems — trajectories, tool use, and multi-step behaviour).
5.0 Why this module exists
Module 4 evaluated a pipeline whose output is a response. This module evaluates a system whose output is a trajectory — and the difference is not a matter of degree.

The unit of evaluation is a trace or session, not a single input-output pair.
Three properties make agent evaluation genuinely different:
The final answer can be right while everything that produced it was wrong. An agent that calls the wrong tool, gets a useless result, ignores it, and answers from parametric knowledge produces a correct response and a broken system. Scoring the answer alone ratifies the failure.
Correct behaviour has many valid surface forms. String-match assertions cannot test an agent, because there is rarely one right trajectory — and reference-trajectory matching penalizes valid alternative paths.
The failure signal is not an error. The agent responds fluently either way. Latency, error rate, and throughput all stay green while answer quality drops — which is Domain 4's observation, arriving here with a sharper consequence, because an agent's failure can also mean it did something wrong rather than merely said something wrong.
5.1 The four dimensions
  +------------------------------------------------------------------+
  |  1. TRAJECTORY                                                    |
  |     The PATH of steps taken.                                      |
  |     -> was it efficient? did it loop? did it take a               |
  |        reasonable route?                                          |
  +------------------------------------------------------------------+
  |  2. TOOL USE                                                      |
  |     Correct tool SELECTION and correct ARGUMENTS.                 |
  |     -> two separate things, and both fail independently           |
  +------------------------------------------------------------------+
  |  3. TASK COMPLETION                                               |
  |     Was the user's GOAL met?                                      |
  |     -> a semantic judgment over the whole session                 |
  +------------------------------------------------------------------+
  |  4. MULTI-TURN QUALITY                                            |
  |     Does performance HOLD across a conversation?                  |
  |     -> constraints stated at turn 2 respected at turn 20?         |
  +------------------------------------------------------------------+
5.1.1 Evaluators have scopes
A structural point that shapes the whole design: agent evaluators operate at different levels of the trace, and matching the metric to the right scope is part of the work.
  TOOL_CALL level    -> tool selection accuracy, parameter
                        accuracy
                        (evaluated per call)

RESPONSE level     -> correctness, faithfulness,
                        helpfulness
                        (evaluated per turn)

SESSION level      -> goal success rate, resolution
                        (evaluated over the whole
                         interaction)
Scoring a session-level property per response, or a per-call property per session, produces a number that means nothing. "Did the agent achieve the user's goal" is not answerable from one turn; "did it pick the right tool" is not answerable from the session as a whole.

5.1.2 The minimal viable metric set
Start with one metric per dimension, and resist the urge to instrument everything:
  TASK COMPLETION   a BOOLEAN judgment on the root of each
                    run
  TOOL USE          a tool-selection correctness check on
                    tool-call observations
  TRAJECTORY        a numeric STEP COUNT with a budget
                    threshold
  MULTI-TURN        a session-level resolution score

PLUS: cost and latency per run -- which tracing gives you
        with no extra evaluator
Expand only when a metric has caught a real regression. Unused metrics are noise.
That discipline matters more here than elsewhere, because agent evaluation is expensive — every metric is a judge call over a multi-step trace — and a sprawling metric set becomes too costly to run and too noisy to read.
5.2 Trajectory evaluation
5.2.1 The three matching modes
Comparing an observed tool sequence against an expected one:
  EXACT MATCH      the same tools, in the same order, with
                   nothing extra
                   -> strictest; almost always too strict

IN-ORDER MATCH   the expected tools appear in the
                   expected order, extra steps permitted
                   -> tolerates exploration

ANY-ORDER MATCH  the expected tools all appear, order
                   irrelevant
                   -> for genuinely independent steps
5.2.2 The false-negative problem
Reference-trajectory matching penalizes valid alternative paths, and this is its central weakness.
  Expected trajectory:  search_docs -> get_policy
  Observed trajectory:  search_faq  -> get_policy

Exact match: FAIL.
  But if search_faq found the answer, the agent did the
  right thing by a different route.
The same problem in a general form: if your reference trajectory uses one tool and the agent reaches for an equivalent one, strict matching reports a failure that is not a failure.
Three responses:

Assert on required steps, not full sequences. "The agent must call verify_identity before issue_refund" is a meaningful assertion; "the agent must call exactly these five tools" usually is not.
Use in-order or any-order matching rather than exact.
Score trajectory quality with a judge where the space of valid paths is large — asking whether the path taken was reasonable, rather than whether it matched.
The practical rule: use strict matching for required and forbidden steps, and judged assessment for everything else.

5.2.3 Step efficiency
  step_efficiency = steps taken / minimum steps required

ratio ~1.0   efficient
  ratio 2-3    exploration, or planning inefficiency
  ratio > 3    likely a TOOL-USE LOOP or oscillation
High ratios indicate planning inefficiency or tool-use loops — Domain 2, Module 1's oscillation failure, measured.
And a step budget is the cheapest agent metric available. It is a numeric count with a threshold, it requires no judge, and it catches the runaway-loop class directly.

5.2.4 The reasoning–output mismatch
A failure mode specific to agents and worth knowing because it defeats the obvious diagnostic.

Sometimes the chain of thought leads to the right answer and the final output contradicts it. The agent "knows" the correct response and outputs something else — driven by generation pressure, where once output starts moving toward a wrong answer it becomes increasingly likely to continue in that direction even while the model's internal state still holds the correct signal.
What makes it dangerous: it is invisible to trace analysis of reasoning alone. An engineer reading the trajectory sees correct reasoning and concludes the agent worked — and the user received something else.
The detection requires comparing the reasoning to the final output, not inspecting either in isolation. A judge asked "is the final response consistent with the reasoning that preceded it?" catches it; reading the trace does not.
5.3 Tool use evaluation
5.3.1 Two separate things
  TOOL SELECTION ACCURACY
    did the agent choose an appropriate tool for this step?

TOOL PARAMETER ACCURACY
    did it extract the correct parameters from the user's
    query?
These fail independently and have different fixes. Wrong tool selection is a tool-description problem (Domain 2, Module 2, §2.2.1); wrong parameters with the right tool is a schema or prompt problem.

5.3.2 "Appropriate" is rarely singular
A nuance that affects how you score it:

In production toolsets where multiple tools have overlapping capabilities, tool selection rarely has a single right answer.
So a strict "did it call the expected tool" check produces false failures wherever two tools could reasonably serve. Score appropriateness rather than identity — a judge asked whether the chosen tool was appropriate for the request handles overlap; an equality assertion does not.
Where a strict assertion is right: required and forbidden tools. "Given this input, the agent must call search" is a good regression test, and "the agent must never call delete_record for a read request" is a safety assertion.

5.3.3 The cheap production signal
A boolean tool-call score is a good alerting signal because it is free to compute on every sampled trace. No judge, no ground truth — did the agent call the tool it should have, or call one it should not have.
This is the deterministic tier applied to agents (Module 3, §3.2), and it is available on every trace rather than a sampled golden set.

5.3.4 Judging tool use semantically
Where the question is semantic — was this tool appropriate — a judge evaluator takes the tool-call observations as input.
A practical detail worth knowing: you can pass only tool names to the judge, or the full array including arguments, and the choice changes what is being assessed. Names alone evaluate selection; names with arguments evaluate selection and parameter extraction, at higher token cost.
5.4 The diagnostic patterns
Agent metrics read together produce specific diagnoses, in the same way Module 4's four RAG metrics do.

5.4.1 The primary pattern
High tool selection accuracy with low goal success rate: the agent selects appropriate tools and fails to complete user objectives.
  LIKELY CAUSES
    - the toolset is INCOMPLETE -- some user requests
      cannot be satisfied with the tools available
    - the agent STRUGGLES WITH MULTI-STEP SEQUENCES --
      each individual choice is right and the plan is
      wrong

AND CHECK HELPFULNESS: the agent may use tools correctly
  and explain the results poorly, which fails the user
  without any tool-level failure.
That third possibility is the one teams miss. Tool metrics are green, goal success is low, and the failure is entirely in how results were communicated.

5.4.2 The wider pattern table
Pattern / Diagnosis

High tool selection, low goal success
- Diagnosis: Incomplete toolset, multi-step planning weakness, or poor explanation — check helpfulness

Low tool selection, high goal success
- Diagnosis: The agent is recovering from bad choices — fragile, and it will not always

High step efficiency ratio
- Diagnosis: Loops or planning inefficiency (§5.2.3)

Correct final answer, wrong trajectory
- Diagnosis: Possible parametric-knowledge answer — the agent did not use its tools

Good per-turn correctness, low session resolution
- Diagnosis: Multi-turn degradation — constraints or context lost across turns

High variance across identical runs
- Diagnosis: Non-determinism is the problem, not capability (§5.5)

Reasoning correct, output wrong
- Diagnosis: The reasoning–output mismatch (§5.2.4)

The second row is worth dwelling on. An agent achieving goals despite poor tool selection is succeeding by luck or by recovery, and the metric that looks bad is telling you the truth about a system whose success metric looks fine.

5.5 Non-determinism and multi-trial evaluation
5.5.1 Why single runs mislead
Agents are non-deterministic across the whole trajectory, not just the final token. A different tool choice at step 2 produces an entirely different path — so the same task can succeed and fail on consecutive runs.
A single-run evaluation of an agent is close to meaningless, and this is more true here than for a single-turn model where non-determinism affects only phrasing.

5.5.2 Variance as a first-class metric
Variance across runs — the standard deviation of success rates across repeated trials on identical tasks — is essential for non-deterministic agents.
  Agent A: 85% success, low variance
  Agent B: 85% success, high variance

Same mean. Agent A is reliable; Agent B succeeds
  brilliantly sometimes and fails badly other times on the
  SAME INPUT.

For most production purposes, A is substantially better.
Report variance alongside the mean, or two very different systems will look identical.

5.5.3 Multi-trial gating
For major releases, run each task several times and gate on the stricter criterion.
  num_trials = 5, gating on pass^k
    -> the task must succeed on ALL k trials, not on
       average
    -> catches NON-DETERMINISTIC FAILURES that a single
       run or an average would hide
The pass^k criterion is much stricter than a mean, and deliberately so: an agent that completes a task four times in five has a 20% failure rate on that task, which an 80% average across many tasks would conceal.
The cost is linear in trials, so this is a release gate rather than a per-commit check.
5.6 Multi-turn evaluation
5.6.1 What degrades across turns
  CONSTRAINTS STATED EARLY are forgotten
    -> the sliding-window trap (Domain 4, Module 2, 2.3.2)

CONTEXT ACCUMULATES and dilutes
    -> attention spreads; early content is used less
       reliably

ERRORS COMPOUND
    -> a wrong conclusion at turn 3 conditions everything
       after it

THE AGENT LOSES THE THREAD
    -> a topic switch at turn 8 and a return at turn 15
Single-turn evaluation cannot see any of this, which is why session-level metrics are a distinct requirement rather than an elaboration.

5.6.2 How multi-turn evaluation works
Pass a session identifier, and the evaluation fetches all spans for that session and evaluates the complete dialogue. The scoping from §5.1.1 then applies:
  TRAJECTORY evaluators   verify tool usage ACROSS ALL TURNS
  GOAL SUCCESS assertions apply to THE SESSION
  CORRECTNESS evaluators  score EACH INDIVIDUAL RESPONSE
                          against its corresponding
                          expected answer
Note the mixed granularity: correctness is per-response even within a session-level evaluation, because "was this answer right" is a per-turn question while "did the user achieve their goal" is not.

5.6.3 Simulated conversations
Evaluating multi-turn behaviour requires multi-turn inputs, and scripted sequences do not adapt to what the agent says.
A simulator — a model playing the user — enables realistic multi-turn evaluation: it responds to the agent's actual output, follows a persona and goal, and can be instructed to be difficult in specific ways.
The caution: a simulated user is a model, so it has the same biases and blind spots. It produces plausible conversations rather than real ones, and simulated results should be validated against real multi-turn traffic once available.
5.7 The AWS surface
5.7.1 AgentCore Evaluations
A fully managed service for assessing agent performance, and — per Module 1, §1.8.6 — the product boundary is that Bedrock Evaluations handles models and RAG while AgentCore Evaluations handles agents.
Roughly 13 built-in evaluators span response quality, safety, task completion, and tool usage, requiring no configuration — they are pre-built with optimized prompt templates and standardized scoring criteria, and apply to any agent producing observable traces.
The named ones worth knowing, with their scopes:
  Builtin.Correctness          RESPONSE -- factual accuracy
  Builtin.Faithfulness         RESPONSE -- grounded in
                               available context rather
                               than fabricated
  Builtin.Helpfulness          RESPONSE -- useful and
                               actionable from the user's
                               perspective
  Builtin.ToolSelectionAccuracy  TOOL_CALL -- did it choose
                               the right tool for THIS STEP
  Builtin.GoalSuccessRate      SESSION -- did the agent
                               ultimately achieve the
                               user's goal ACROSS THE FULL
                               INTERACTION
  plus tool parameter accuracy, safety, and context
  relevance
The scope annotations are the important part. ToolSelectionAccuracy at tool-call level and GoalSuccessRate at session level is §5.1.1 made concrete — and it is why the two can diverge in the §5.4.1 diagnostic.

5.7.2 Two operating modes
  CONTINUOUS PRODUCTION EVALUATION
    the service SAMPLES live agent interactions and scores
    them continuously
    -> configure a SAMPLING RATE and optional FILTERS
    -> results publish to CLOUDWATCH in the AgentCore
       Observability dashboard
    -> so you can build ALARMS AND AUTOMATIONS on quality,
       which is Domain 4, Module 8's requirement satisfied
       natively

ON-DEMAND DATASET RUNNER
    evaluate systematically across a CURATED DATASET by
    INVOKING THE AGENT for every scenario
    -> the regression-gate mode
The distinction maps onto Module 1's offline/online split (§1.6.2): the dataset runner gates the change; continuous sampling detects what the dataset could not anticipate.

5.7.3 Custom evaluators
Business-specific quality metrics with your choice of prompt and model. Everything from Module 3 applies — rubric design, the biases, and validation — and more urgently, because a custom evaluator has had no design review at all.

5.7.4 Cost and throttling
Evaluation cost scales with sampling rate × evaluator count × traffic, and agent traces are long.
The documented remediations when evaluation is slow or throttling:
  LOWER THE SAMPLING RATE -- evaluate a smaller percentage
    of sessions
  REDUCE THE EVALUATOR COUNT
  REQUEST A QUOTA INCREASE for the evaluator model
  SWITCH to a model with higher default quotas
A production sampling rate of 1–5% is a reasonable starting point — begin low and increase once evaluator costs are confirmed acceptable.
Note that this is the same reasoning as Module 3, §3.11: fewer, better evaluators cost less and mean more, and §5.1.2's "expand only when a metric catches a real regression" is the discipline that keeps the bill viable.
5.8 The rollout and gating pattern
A production blueprint worth knowing as a shape.
  PHASE 1 -- DEVELOPMENT
    the ON-DEMAND DATASET RUNNER against a curated
    scenario set
    -> fast iteration, full control

PHASE 2 -- STAGING GATE
    thresholds that BLOCK DEPLOYMENT, for example:
      TOOL SELECTION ACCURACY below 95%   -> blocked
      TASK COMPLETION below 80%           -> blocked
    -> and for major releases, MULTI-TRIAL with
       num_trials = 5, gating on pass^k (5.5.3)

PHASE 3 -- SHADOW
    run the candidate agent against REAL QUERIES WITHOUT
    AFFECTING USERS
    -> the real distribution, no user risk
    -> this is where offline-curated overestimates get
       corrected (Module 1, 1.6.1)

PHASE 4 -- PRODUCTION
    100% traffic with CONTINUOUS ONLINE EVALUATION at a
    1-5% sampling rate
    -> alarms on CloudWatch

PHASE 5 -- CLOSE THE LOOP
    TURN PRODUCTION FAILURES INTO TEST CASES
    -> the regression suite that compounds (Module 2,
       2.5.2)
The threshold asymmetry is worth noting: tool selection at 95% and task completion at 80%. Tool selection is held to a higher bar because it is upstream — a wrong tool choice propagates, and a 5% tool error rate is not a 5% task failure rate, it is worse, because errors compound across steps.
A calibrating datapoint: one production agent operated at 87% tool selection accuracy — meaning roughly one in eight queries returned wrong results. Stated as a percentage it sounds tolerable; stated as one in eight it does not, and converting rates into user-facing frequencies is how you make a threshold argument land.
5.9 Consolidated decision reference
  +-----------------------------------------------------------------+
  |  THE UNIT OF EVALUATION IS A TRACE OR SESSION, NOT AN            |
  |  INPUT-OUTPUT PAIR.                                              |
  |    The final answer can be RIGHT while everything that           |
  |    produced it was WRONG -- scoring the answer alone ratifies    |
  |    the failure.                                                  |
  |    Correct behaviour has MANY VALID SURFACE FORMS, so string     |
  |    matching cannot test an agent.                                |
  |    The failure signal is NOT AN ERROR -- latency, error rate,    |
  |    and throughput stay green.                                    |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  FOUR DIMENSIONS, AND EVALUATORS HAVE SCOPES                     |
  |    TRAJECTORY     the path of steps                              |
  |    TOOL USE       selection AND parameters -- two things         |
  |    TASK COMPLETION the goal                                      |
  |    MULTI-TURN     does it hold across a conversation             |
  |                                                                  |
  |    TOOL_CALL level -> tool selection, parameter accuracy         |
  |    RESPONSE level  -> correctness, faithfulness, helpfulness     |
  |    SESSION level   -> goal success, resolution                   |
  |    -> scoring a session property per response produces a         |
  |       number that means nothing                                  |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  THE MINIMAL METRIC SET                                          |
  |    boolean TASK COMPLETION on the run root                       |
  |    TOOL-SELECTION correctness on tool-call observations          |
  |    numeric STEP COUNT with a budget threshold                    |
  |    session-level RESOLUTION score                                |
  |    + cost and latency, free from tracing                         |
  |                                                                  |
  |    EXPAND ONLY WHEN A METRIC HAS CAUGHT A REAL REGRESSION.       |
  |    Unused metrics are noise -- and expensive noise, since        |
  |    every one is a judge call over a long trace.                  |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  TOOL USE                                                        |
  |    SELECTION and PARAMETERS fail independently and have          |
  |      different fixes (tool descriptions vs schema/prompt)        |
  |    "APPROPRIATE" IS RARELY SINGULAR where tools overlap          |
  |      -> judge appropriateness, do not assert identity            |
  |      -> reserve strict assertions for required and forbidden     |
  |    A BOOLEAN TOOL-CALL SCORE IS FREE on every sampled trace      |
  |      -> the best production alerting signal available            |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  DIAGNOSTIC PATTERNS                                             |
  |    high tool selection + LOW GOAL SUCCESS                        |
  |      -> incomplete toolset, multi-step planning weakness, OR     |
  |         poor explanation -- CHECK HELPFULNESS                    |
  |    LOW tool selection + high goal success                        |
  |      -> succeeding by recovery or luck. Fragile.                 |
  |    correct answer + WRONG TRAJECTORY                             |
  |      -> possibly answered from PARAMETRIC KNOWLEDGE              |
  |    per-turn correctness fine, SESSION resolution low             |
  |      -> multi-turn degradation                                   |
  |    REASONING correct, OUTPUT wrong                               |
  |      -> generation pressure; INVISIBLE to trace analysis of      |
  |         reasoning alone. Compare reasoning TO output.            |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  NON-DETERMINISM                                                 |
  |    A SINGLE RUN IS CLOSE TO MEANINGLESS -- a different choice    |
  |    at step 2 produces an entirely different path.                |
  |    REPORT VARIANCE ALONGSIDE THE MEAN. Two agents at 85% with    |
  |      different variance are very different systems.              |
  |    MAJOR RELEASES: num_trials = 5, gate on PASS^K -- must        |
  |      succeed on ALL trials, not on average.                      |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  THE AWS SURFACE                                                 |
  |    AGENTCORE EVALUATIONS for agents; BEDROCK EVALUATIONS for     |
  |      models and RAG.                                             |
  |    ~13 BUILT-IN EVALUATORS, no configuration required.           |
  |      Builtin.ToolSelectionAccuracy  -> TOOL_CALL level           |
  |      Builtin.GoalSuccessRate        -> SESSION level             |
  |      Builtin.Correctness / Faithfulness / Helpfulness            |
  |                                     -> RESPONSE level            |
  |    TWO MODES: continuous production SAMPLING (1-5%, results to   |
  |      CLOUDWATCH -> alarms and automations) and the ON-DEMAND     |
  |      DATASET RUNNER (the regression gate).                       |
  |    THROTTLING? lower the sampling rate, reduce evaluator count,  |
  |      raise the evaluator model's quota, or switch models.        |
  +-----------------------------------------------------------------+
5.10 Checkpoint
1. Why does scoring an agent's final answer ratify failure? (5.0)

Because the final answer can be right while everything that produced it was wrong. An agent that calls the wrong tool, gets a useless result, ignores it, and answers from parametric knowledge produces a correct response and a broken system.
2. Why can string-match assertions not test an agent? (5.0)

Correct behaviour has many valid surface forms — there is rarely one right trajectory, and reference matching penalizes valid alternative paths.
3. Name the four dimensions of agent evaluation. (5.1)

Trajectory (the path of steps), tool use (selection and arguments), task completion (was the goal met), and multi-turn quality (does performance hold across a conversation).
4. What are the three evaluator scopes, and why do they matter? (5.1.1)

Tool-call level (tool selection, parameter accuracy), response level (correctness, faithfulness, helpfulness), and session level (goal success, resolution). Scoring a session-level property per response — or a per-call property per session — produces a number that means nothing.
5. State the minimal viable agent metric set and the expansion rule. (5.1.2)

A boolean task-completion judgment on the run root, a tool-selection correctness check on tool-call observations, a numeric step count with a budget threshold, and a session-level resolution score — plus cost and latency, which tracing gives free. Expand only when a metric has caught a real regression; unused metrics are noise.
6. What is the weakness of reference-trajectory matching, and how do you handle it? (5.2.2)

It penalizes valid alternative paths — an equivalent tool reached by a different route reports a failure that is not one. Use strict assertions for required and forbidden steps, in-order or any-order matching rather than exact, and judged assessment where the space of valid paths is large.
7. What does a step-efficiency ratio above 3 suggest? (5.2.3)

A tool-use loop or oscillation. And step efficiency is the cheapest agent metric available — a count with a threshold, no judge required.
8. Describe the reasoning–output mismatch and why it defeats trace inspection. (5.2.4)

The chain of thought reaches the right answer and the final output contradicts it, driven by generation pressure. It is invisible to trace analysis of reasoning alone — an engineer reading the trajectory sees correct reasoning and concludes the agent worked. Detection requires comparing the reasoning to the output.
9. Why is strict tool-selection matching often wrong? (5.3.2)

Because in production toolsets with overlapping capabilities, "appropriate" rarely has a single right answer. Judge appropriateness rather than asserting identity — and reserve strict assertions for required tools and forbidden ones.
10. What is the best free production quality signal for an agent? (5.3.3)

A boolean tool-call score — did it call the tool it should have, or one it should not have. No judge, no ground truth, and computable on every sampled trace.
11. Tool selection accuracy is high and goal success is low. Give three possible causes. (5.4.1)

The toolset is incomplete and some requests cannot be satisfied; the agent struggles with tasks requiring multiple sequential tool calls; or — the one teams miss — the agent uses tools correctly and explains results poorly, so check helpfulness.
12. What does low tool selection accuracy with high goal success indicate? (5.4.2)

The agent is succeeding by recovery or luck despite poor choices. It is fragile, and the metric that looks bad is telling the truth about a system whose success metric looks fine.
13. Why is a single evaluation run of an agent close to meaningless? (5.5.1)

Agents are non-deterministic across the whole trajectory, not just the final token — a different tool choice at step 2 produces an entirely different path, so the same task can succeed and fail on consecutive runs.
14. Why report variance alongside mean success rate? (5.5.2)

Two agents both at 85% with different variance are very different systems — one reliable, the other succeeding brilliantly sometimes and failing badly other times on the same input. For most production purposes the low-variance one is substantially better, and the mean alone cannot distinguish them.
15. What is pass^k gating and what does it catch? (5.5.3)

Running each task several times — for example five trials — and requiring success on all of them rather than on average. It catches non-deterministic failures that a single run or a mean would hide: four successes in five is a 20% failure rate on that task.
16. How does multi-turn evaluation handle mixed granularity? (5.6.2)

Passing a session ID fetches all spans and evaluates the complete dialogue: trajectory evaluators verify tool usage across all turns, goal success applies to the session, and correctness scores each individual response against its corresponding expected answer.
17. Which product evaluates agents, and at what scopes do its two headline evaluators operate? (5.7.1)

AgentCore Evaluations — Bedrock Evaluations handles models and RAG. Builtin.ToolSelectionAccuracy operates at the tool-call level; Builtin.GoalSuccessRate is a session-level evaluator measuring whether the agent achieved the user's goal across the full interaction.
18. Evaluation is throttling. What are the remediations, in order? (5.7.4)

Lower the sampling rate to evaluate fewer sessions; reduce the evaluator count; request a quota increase for the evaluator model; or switch to a model with higher default quotas.
5.11 Practice cases
Case 5.11 — The agent that passed
Scenario. A customer service agent has these evaluation results:

MetricScoreFinal answer correctness0.91Helpfulness0.88Tool selection accuracy0.62Goal success rate0.94Mean step count6.8 (minimum required: 2.4)
The team's position: correctness and goal success are strong, so the agent is working and the tool metrics are a secondary concern.
Produce: an assessment.
Discussion.
The team is reading the metrics that look good and dismissing the two that are telling them something.
Tool selection at 0.62 with goal success at 0.94 is §5.4.2's pattern: the agent is succeeding by recovery, not by competence. Roughly four in ten tool choices are inappropriate, and it is reaching the goal anyway — by retrying, by calling something else afterward, or because the wrong tool happened to return usable information.
That is fragile in a specific way. Recovery depends on the agent noticing the bad result and trying again, which works when the wrong tool returns an obvious error and fails silently when the wrong tool returns something plausible. The 6% of sessions that miss the goal are unlikely to be random.
The step count corroborates it. 6.8 steps against a minimum of 2.4 is a ratio of 2.8 — squarely in §5.2.3's "planning inefficiency or tool-use loops" band, and consistent with an agent making bad choices and recovering from them. The two bad metrics are the same story told twice.
And the cost consequence is direct: nearly three times the necessary steps means nearly three times the model calls, with context accumulating quadratically (Domain 4, Module 2). This agent is roughly three times more expensive than it needs to be, which is an argument the team's own framing does not surface.
What to check next, in order:
  1. IS TOOL SELECTION ACCURACY MEASURED FAIRLY? (5.3.2)
     If several tools overlap, a strict identity check
     produces false failures. Is 0.62 real, or is it
     penalizing valid alternatives?
     -> re-score with a judged appropriateness criterion
     -> this is the FIRST check, because if the metric is
        wrong the rest of the analysis is wrong

2. If 0.62 is real: WHY?
     -> Domain 2, Module 2: ambiguous or overlapping tool
        DESCRIPTIONS are the most common cause, not model
        capability
     -> build a confusion matrix: which tools are mistaken
        for which?

3. VARIANCE (5.5.2). A 0.94 mean tells you nothing about
     consistency. Run the same tasks 5 times and measure
     the spread.
     -> and gate a release on PASS^K rather than the mean

4. THE 6% THAT FAIL. Sample them. Do they correlate with
     particular tool confusions? That is the actionable
     link between the two metrics.
What I would say to the team:

The system reaches the goal 94% of the time by making poor tool choices and recovering. That works until it does not — and when a wrong tool returns something plausible rather than an error, there is nothing to recover from. It also costs about three times what it should.
The correct priority is fixing tool selection, and per Domain 2, Module 2 the fix is almost certainly tool descriptions rather than a better model: sharper names, explicit negative bounds, and consolidating overlapping tools.
One thing worth crediting: they measured all five, including step count. They have the data to see this — the failure is in the reading, which is a cheaper problem than an instrumentation gap.
What a strong answer includes: identifying high-goal-success-with-low-tool-selection as succeeding by recovery; connecting the step count to the same cause rather than treating it separately; questioning whether the tool metric is measured fairly before acting on it; naming the cost consequence; requiring variance and pass^k; and locating the likely fix in tool descriptions rather than model capability.
The reasoning being tested: can you read two poor metrics as one coherent story, and do you recognize that a strong success rate can conceal a fragile mechanism?
Case 5.12 — Designing agent evaluation
Scenario. You are designing evaluation for an IT support agent about to enter production.
The agent: answers questions from an internal knowledge base, checks ticket status, resets passwords (with approval), checks device inventory, and creates tickets. Five tools. Conversations average four turns. Roughly 900 sessions per day expected.
Constraints: the team wants a release gate and production monitoring; evaluation budget is modest; there is no labelled dataset yet.
Produce: the evaluation design.
Discussion.
Start with the minimal set (§5.1.2), because the budget is modest and an over-instrumented agent evaluation is expensive and noisy.
  TASK COMPLETION   boolean goal success, SESSION level
  TOOL USE          tool selection accuracy, TOOL_CALL level
                    + tool parameter accuracy
  TRAJECTORY        step count with a budget threshold
  MULTI-TURN        session resolution
  + cost and latency, free from tracing
Then add the deterministic assertions that this specific toolset demands — and these come from the architecture rather than from a generic list:
  SAFETY ASSERTIONS (free, no judge, every trace):
    - reset_password is NEVER called without the approval
      step preceding it
      -> a required-order assertion (5.2.2), and this is
         exactly where strict matching is right
    - create_ticket is not called more than once per
      session without a new user request
      -> catches duplicate-ticket loops
    - no write tool is called on a read-only query

These are FORBIDDEN and REQUIRED-ORDER assertions, which
  is the case where strict trajectory matching is correct
  rather than brittle.
Password reset with approval is the one that must be an assertion rather than a score. It is a control (Domain 3, Module 1), so it gates at 100% rather than averaging into a quality metric.
The dataset, with no labels yet:
  WEEK 1  Build ~80-120 scenarios from IT support ticket
          history -- real questions, real phrasing
          (Module 2, 2.2)

Stratify by:
            - single-tool (status check, inventory lookup)
            - multi-tool (diagnose then create ticket)
            - APPROVAL-REQUIRING (password reset)
            - OUT OF SCOPE (should decline or escalate)
            - AMBIGUOUS (should ask a clarifying question)

Ground truth is CHEAP HERE:
            - goal success is largely BINARY and
              observable ("was the ticket created with the
              right fields?")
            - the required tool for most scenarios is
              obvious to a support engineer
          -> unlike Module 4's RAG case, this dataset does
             not need expensive relevance labelling
That last point is worth making explicit. Agent evaluation datasets are often cheaper to build than RAG ones, because task completion is frequently observable rather than judged — a ticket either exists with the right fields or it does not.
The gate:
  ON-DEMAND DATASET RUNNER against the scenario set

BLOCKING THRESHOLDS:
    tool selection accuracy   >= 95%
    task completion           >= 80%
    safety assertions         100%, no exceptions
    step budget               p95 below threshold

MAJOR RELEASES: num_trials = 5, gate on PASS^K (5.5.3)
    -> a four-turn agent with five tools has substantial
       trajectory non-determinism; a single run will not
       catch it
The threshold asymmetry follows §5.8: tool selection at 95% because errors propagate across steps; task completion at 80% because it is downstream and partly recoverable. And safety assertions at 100% because they are controls, not quality dimensions.
Production monitoring:
  CONTINUOUS EVALUATION at a 1-5% SAMPLING RATE
    -> at 900 sessions/day, 3% is ~27 sessions/day
       evaluated -- meaningful and affordable
    -> START LOW, raise once evaluator cost is confirmed

Results to CLOUDWATCH -> alarm on:
    goal success rate deviation from baseline
    tool selection accuracy deviation
    step count p95 rising
    ANY safety assertion failure -> page

PLUS the free signals on EVERY trace (5.3.3):
    boolean tool-call assertions
    step count
    termination reason
    -> no judge, no sampling limit
The split between sampled judged evaluation and free deterministic checks on every trace is the design decision that makes the budget work. The expensive layer runs on 3%; the cheap layer runs on 100% and catches the safety and loop classes.
Closing the loop (§5.8, phase 5): every production failure becomes a scenario in the dataset. This is what makes the suite compound (Module 2, §2.5.2), and it is free.
Two things I would flag:
Multi-turn is a real requirement here. Four-turn averages mean session-level resolution matters, and a per-turn correctness score will look fine while sessions fail. Evaluate at session scope (§5.6.2).
Watch the tool-selection metric for false failures (§5.3.2). check_ticket_status and check_device_inventory are distinct, but a query like "what's happening with my laptop request" could reasonably route to either. Score appropriateness, not identity, or the metric will report failures that are not.
What a strong answer includes: starting from the minimal set rather than instrumenting everything; deriving safety assertions from this specific toolset and gating them at 100%; recognizing that agent ground truth is cheaper here than RAG ground truth; the threshold asymmetry with a reason; splitting sampled judged evaluation from free deterministic checks on every trace; pass^k for major releases; and anticipating false failures on overlapping tools.
The reasoning being tested: can you design agent evaluation to a modest budget by pushing work into free deterministic assertions, and do you derive the assertions from the architecture in front of you?
Case 5.13 — The agent that got worse quietly
Scenario. An internal research agent has been in production five months. Reported state:

Goal success rate: stable at 0.89 for five months
Tool selection accuracy: stable at 0.93
Mean step count: risen from 5.1 to 8.4 over the period
Cost per session: risen 71%
Latency p95: risen from 22 s to 41 s
No deployments in the last two months
Users have not complained
The team's view is that quality is stable so there is nothing to investigate.
Produce: an assessment.
Discussion.
The quality metrics are stable and the system is degrading, which is why step count is in the minimal metric set.
The step count is the finding. 5.1 to 8.4 is a 65% rise, and the cost and latency figures corroborate it — 71% cost and 86% latency growth are what you expect when steps rise 65% with context accumulating (Domain 4, Module 2, §2.6).
Step efficiency is now poor. If the minimum required is around 3, the ratio has moved from roughly 1.7 to 2.8 — from "efficient with some exploration" into §5.2.3's "planning inefficiency or tool-use loops" band.
"No deployments in two months" does not mean nothing changed. Candidates:
  - A PROVIDER-SIDE MODEL UPDATE (Domain 4, Module 8) --
    produces a behaviour shift with zero deployments and
    zero errors. The leading hypothesis.
  - THE KNOWLEDGE BASE GREW, so retrieval returns more
    and the agent explores more
  - TOOLS WERE ADDED by another team, expanding the
    decision space
  - USER QUESTIONS GOT HARDER as people learned what the
    agent can do -- normal, and worth confirming rather
    than assuming
Why goal success stayed flat while everything else degraded: the agent is compensating with more steps. It reaches the goal by trying more things — which holds the success metric constant and burns cost, latency, and its own step budget.
That is a system running closer to its limits. If step counts continue rising, sessions will start hitting iteration caps (Domain 2, Module 4), and goal success will then fall suddenly rather than gradually. The stable metric is a lagging indicator.
"Users have not complained" is consistent with this and not reassuring. Latency p95 at 41 seconds is bad and users of an internal research tool may simply be tolerating it — Domain 4's proxy signals (abandonment, regeneration) would tell you whether they are.
The investigation:
  1. TERMINATION REASON distribution over time.
     Are sessions increasingly hitting caps rather than
     completing? -> the leading indicator that goal
     success is about to fall

2. STEP COUNT DISTRIBUTION, not the mean.
     A mean of 8.4 could be uniform drift or a bimodal
     split -- most sessions unchanged and a growing tail
     of runaways. VERY different problems.

3. TOOL-CALL SEQUENCES on high-step sessions.
     Repeated identical calls -> a loop.
     Alternating pairs -> oscillation (Domain 2, Module 1).

4. MODEL VERSION. Compare against two months ago, and
     replay archived sessions against the current model.

5. TOOL INVENTORY. Has the toolset grown? More tools
     means a larger decision space and typically lower
     selection accuracy (Domain 2, Module 2).

6. VARIANCE (5.5.2). Has it risen? Rising variance with
     a stable mean means the system is becoming
     inconsistent, which precedes becoming worse.
Step 2 is the one that changes the response. A uniform rise is a gradual degradation to tune; a growing tail of runaway sessions is a stopping-condition problem with a different and more urgent fix.
The immediate protective action, regardless of cause: a per-session cost ceiling and a step budget (Domain 2, Module 4, §4.2), so the tail is bounded while the investigation proceeds. That is hours of work and it caps the exposure.
The process finding: the team is monitoring quality and treating efficiency as incidental. Step count, cost, and latency are agent quality metrics, not operational ones — a system that reaches the goal in 8.4 steps is worse than one that reaches it in 5.1, even at identical success rates. Alarm on step-count deviation from baseline, and the next occurrence is caught in days.
What a strong answer includes: reading step count as the primary signal and cost and latency as corroboration; explaining that stable goal success is caused by the extra steps rather than independent of them; identifying a provider-side model update as the leading hypothesis given "no deployments"; distinguishing uniform drift from a growing tail before choosing a fix; bounding the tail immediately with a cost ceiling; and treating efficiency as a quality dimension.
The reasoning being tested: do you recognize that a stable success metric can be masking degradation, and can you identify which distributional question determines the fix?
5.12 Skill coverage audit
Skill statementWhere discharged5.1.7 — Evaluating agentic systems: trajectories, tool use, and multi-step behaviourWhy the unit of evaluation is a trace or session, and the three properties that make agent evaluation structurally different: §5.0. The four dimensions, evaluator scopes at tool-call, response, and session level, and the minimal viable metric set with its expansion rule: §5.1. Trajectory evaluation — the three matching modes, the false-negative problem and its three responses, step efficiency, and the reasoning–output mismatch: §5.2. Tool use — selection versus parameters, why "appropriate" is rarely singular, the free boolean production signal, and semantic judging: §5.3. The diagnostic patterns, including high-tool-selection-with-low-goal-success and its three causes: §5.4. Non-determinism, variance as a first-class metric, and pass^k multi-trial gating: §5.5. Multi-turn degradation, session-scoped evaluation with mixed granularity, and simulated conversations: §5.6. The AgentCore Evaluations surface — the product boundary, built-in evaluators with their scopes, the two operating modes, custom evaluators, and throttling remediation: §5.7. The five-phase rollout and gating pattern with threshold asymmetry: §5.8. Applied in Cases 5.11, 5.12, and 5.13.5.13 Module summary
The unit:

A trace or session, not an input-output pair.
The final answer can be right while everything that produced it was wrong — scoring the answer alone ratifies the failure.
Correct behaviour has many valid surface forms, so string matching cannot test an agent.
The failure signal is not an error. Latency, error rate, and throughput stay green.
Four dimensions, three scopes:

Trajectory, tool use (selection and parameters), task completion, multi-turn quality.
Evaluators operate at tool-call, response, and session scope. Scoring a session property per response produces a number that means nothing.
Minimal set: boolean task completion, tool-selection correctness, a step count with a budget, and a session resolution score — plus cost and latency, free from tracing. Expand only when a metric has caught a real regression.
Trajectory:

Exact, in-order, and any-order matching. Reference matching penalizes valid alternative paths, so use strict assertions only for required and forbidden steps and judged assessment elsewhere.
Step efficiency — steps taken over minimum required — is the cheapest agent metric: a count and a threshold, no judge. Above 3 suggests loops.
The reasoning–output mismatch is invisible to trace analysis of reasoning alone; detecting it requires comparing reasoning to output.
Tool use:

Selection and parameter accuracy fail independently, with different fixes.
"Appropriate" is rarely singular where tools overlap — judge appropriateness rather than asserting identity.
A boolean tool-call score is free on every sampled trace and is the best production alerting signal available.
Diagnostics:

High tool selection with low goal success → incomplete toolset, multi-step planning weakness, or poor explanation — check helpfulness.
Low tool selection with high goal success → succeeding by recovery. Fragile.
Correct answer with a wrong trajectory → possibly parametric knowledge.
Per-turn correctness fine with low session resolution → multi-turn degradation.
Non-determinism:

A single run is close to meaningless — a different choice at step 2 produces an entirely different path.
Report variance alongside the mean. Two agents at 85% with different variance are very different systems.
Major releases: several trials, gated on pass^k — success on all trials, not on average.
The AWS surface:

AgentCore Evaluations for agents; Bedrock Evaluations for models and RAG.
Around 13 built-in evaluators requiring no configuration. Builtin.ToolSelectionAccuracy is tool-call scoped; Builtin.GoalSuccessRate is session scoped; correctness, faithfulness, and helpfulness are response scoped.
Two modes: continuous production sampling at 1–5%, publishing to CloudWatch so you can alarm and automate; and the on-demand dataset runner as the regression gate.
Throttling: lower the sampling rate, reduce evaluator count, raise the evaluator model's quota, or switch models.
Gating:

Tool selection at a higher bar than task completion — around 95% versus 80% — because tool errors propagate across steps.
Safety assertions gate at 100%; they are controls, not quality dimensions.
Convert rates into user-facing frequencies when arguing thresholds: 87% tool selection sounds tolerable, and "one in eight queries returns a wrong result" does not.
Carry into Module 6: Modules 3 through 5 automated the measurement. Module 6 addresses the layer that calibrates all of it — human feedback and annotation, where the judges get their ground truth and where the quality of the whole programme is ultimately set.
End of Module 5. Module 6 — Human Feedback and Annotation — follows.

Unstarred

Dunk (exclude from study)

Module 5: Agent Evaluation

Review Questions

Todos

Sensemaker