Module 1: Evaluation Foundations

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

Module 1 — Evaluation Foundations
Tier: Standard. Covers Skill 5.1.1 (evaluation frameworks and metrics for GenAI systems).
1.0 Why this module comes first
Most evaluation failures are not measurement failures. They are definition failures — the team measured something real and it was not the thing that mattered.
Three specific mistakes account for most of them, and all three are conceptual rather than technical:
Expecting a single quality number. Quality is multi-dimensional and the dimensions conflict. A composite score is a policy decision, not a measurement, and treating it as a measurement conceals the tradeoff being made.
Using a metric that cannot answer the question. BLEU on an open-ended assistant, exact match on a summarization task, perplexity as a proxy for correctness — each is a real metric applied where it establishes nothing.
Drawing conclusions the sample cannot support. A three-point difference on fifty examples is noise, and a surprising number of published GenAI "improvements" sit inside their own error bars.
This module fixes the definitions. The rest of the book builds the machinery.
1.1 Why traditional ML metrics do not transfer
1.1.1 The structural difference
  TRADITIONAL ML
    input -> model -> prediction
                        |
                     compare to a LABEL
                        |
                     correct or not
    -> accuracy, precision, recall, F1, AUC
    -> a DISCRETE label space; correctness is a comparison

GENERATIVE
    input -> model -> a STRING
                        |
                     compare to... what?
    -> an UNBOUNDED output space
    -> MANY valid answers to the same question
    -> quality dimensions that TRADE AGAINST EACH OTHER
All the familiar metrics presume a discrete label space. Accuracy requires a set of correct answers to be a member of. F1 requires true and false positives to be countable. None of that survives contact with an output space of arbitrary strings.

1.1.2 The three consequences
There is no single correct answer. Two summaries can differ in every word and both be excellent. An evaluation that penalizes lexical divergence is measuring similarity to a reference, not quality.
Correctness is not sufficient. A response can be factually accurate, well-sourced, appropriately hedged — and useless, because it answered a question the user did not ask.
The dimensions conflict. This is §1.3, and it is the point most often missed.
What survives from traditional ML is the statistical discipline: sample sizes, confidence intervals, held-out sets, and the recognition that a difference within error bars is not a difference. §1.7.
1.2 The quality dimensions
Know each one, know how to measure it, and know what it does not tell you.
  +------------------------------------------------------------------+
  |  RELEVANCE                                                        |
  |  Does it address what was asked?                                  |
  |  -> a perfectly accurate answer to a different question           |
  |     scores zero here and high on correctness                      |
  +------------------------------------------------------------------+
  |  FACTUAL ACCURACY / CORRECTNESS                                   |
  |  Is it true?                                                      |
  |  -> requires ground truth, or a verifiable claim                  |
  +------------------------------------------------------------------+
  |  FAITHFULNESS / GROUNDEDNESS                                      |
  |  Is it supported by the PROVIDED SOURCE?                          |
  |  -> DISTINCT from correctness. A response can be faithful to      |
  |     a WRONG source, or correct but UNGROUNDED.                    |
  +------------------------------------------------------------------+
  |  COMPLETENESS                                                     |
  |  Does it cover what it should?                                    |
  |  -> the dimension that conflicts most sharply with concision      |
  +------------------------------------------------------------------+
  |  CONSISTENCY                                                      |
  |  Same input, same output? And is it internally coherent?          |
  |  -> two different questions sharing one word                      |
  +------------------------------------------------------------------+
  |  FLUENCY / COHERENCE                                              |
  |  Is it well-formed?                                               |
  |  -> NEARLY SATURATED on modern models. Rarely the binding         |
  |     constraint, and measuring it is usually wasted effort.        |
  +------------------------------------------------------------------+
  |  SAFETY / HARMLESSNESS                                            |
  |  Domain 3's concerns, measured.                                   |
  +------------------------------------------------------------------+
  |  FORMAT COMPLIANCE                                                |
  |  Does it match the required schema?                               |
  |  -> DETERMINISTIC and cheap. Measure it first.                    |
  +------------------------------------------------------------------+
  |  TONE / STYLE ADHERENCE                                           |
  |  Does it match brand voice or register?                           |
  +------------------------------------------------------------------+
1.2.1 The distinction that matters most
Faithfulness and correctness are not the same thing, and conflating them is the single most common conceptual error in RAG evaluation.
  FAITHFUL BUT INCORRECT
    The source document says the policy waiting period is
    60 days. It was changed to 30 days last month and the
    document is stale.
    The response says 60 days.
    -> perfectly faithful. Wrong.
    -> a DATA FRESHNESS problem, invisible to any
       faithfulness metric

CORRECT BUT UNFAITHFUL
    Retrieval returned nothing useful. The model answered
    from parametric knowledge and happened to be right.
    -> correct. Ungrounded.
    -> a DEFECT, not a success -- it will fail identically
       on anything outside training

COMPLETE BUT IRRELEVANT
    A thorough, accurate summary of a document that does
    not answer the user's question.

FLUENT BUT USELESS
    Well-formed, confident, and empty.
Each of these passes some metrics and fails others, which is precisely why a single score cannot represent quality. Domain 3, Module 4 developed the hallucination taxonomy that these map onto.

1.2.2 Fluency is a solved problem
Worth saying plainly because it saves effort: fluency and coherence are near-saturated on modern models. They were meaningful metrics when generation was unreliable. Today, a fluency score of 0.97 tells you nothing you did not already know.
Measure it only if you have evidence it is your constraint — which for most production systems it is not.
1.3 The dimension conflict
The idea that makes this module foundational.
  MAXIMIZE COMPLETENESS   -> responses get longer
                          -> CONCISION suffers
                          -> and cost and latency rise

MAXIMIZE FAITHFULNESS   -> the model refuses more often
                          -> HELPFULNESS suffers
                          -> "I don't have information about
                             that" is maximally faithful

MAXIMIZE HELPFULNESS    -> the model attempts more
                          -> SAFETY and faithfulness suffer

MAXIMIZE CONSISTENCY    -> temperature to zero
                          -> creativity and variety suffer
1.3.1 There is no single quality score
A weighted composite is a policy decision requiring a documented owner — not a measurement.
This is the same shape as the fairness-definition problem in Domain 3, Module 8: several defensible criteria that cannot be jointly maximized, so someone must choose and own the choice.
What follows practically:

Report dimensions separately. A dashboard showing "quality: 0.86" hides which dimension moved.
If you must produce a composite, document the weights and who set them, and report the components alongside it.
When a change improves one dimension, check the others. An improvement in faithfulness that raised the refusal rate by fifteen points is not obviously an improvement.
1.3.2 The weighting is task-dependent
  MEDICAL INFORMATION
    faithfulness >> helpfulness
    -> a refusal is far better than a confident error

CREATIVE ASSISTANCE
    helpfulness and variety >> consistency
    -> identical outputs would be a defect

STRUCTURED EXTRACTION
    format compliance and correctness >> everything
    -> fluency is irrelevant

CUSTOMER SUPPORT
    relevance and helpfulness, with a safety floor
    -> completeness matters less than answering the actual
       question
Deriving the weighting from the task is the architect's job, and doing it explicitly at design time is what prevents the argument happening later during an incident.
1.4 Reference-based and reference-free evaluation
The taxonomy that determines what you can measure at all.
  +------------------------------------------------------------------+
  |  REFERENCE-BASED                                                  |
  |  Compare the output to a known-correct answer.                    |
  |    exact match, F1, BERTScore, ROUGE, BLEU                        |
  |                                                                   |
  |  + DETERMINISTIC and reproducible                                 |
  |  + cheap -- no model call to score                                |
  |  + no judge to validate                                           |
  |  - REQUIRES GROUND TRUTH, which is expensive to produce           |
  |  - RIGID: penalizes valid answers that differ from the            |
  |    reference                                                      |
  |  -> best for CLOSED tasks: classification, extraction,            |
  |     short factual answers                                         |
  +------------------------------------------------------------------+
  |  REFERENCE-FREE                                                   |
  |  Score the output on its own terms, or against a source.          |
  |    LLM-as-a-judge, heuristics, grounding checks                   |
  |                                                                   |
  |  + handles OPEN-ENDED output                                      |
  |  + no ground truth required                                       |
  |  + can score dimensions references cannot capture                 |
  |  - NOISIER and non-deterministic                                  |
  |  - the judge ITSELF NEEDS VALIDATING (Module 3)                   |
  |  - costs a model call per evaluation                              |
  |  -> best for OPEN tasks: summarization, assistance,               |
  |     generation                                                    |
  +------------------------------------------------------------------+
1.4.1 The selection rule
The question is not which is better. It is which one your task admits.
  Does a SINGLE correct answer exist?
    YES -> reference-based is available, cheaper, and
           deterministic. Use it.
    NO  -> reference-based will penalize valid answers.
           You need reference-free, or a hybrid.

Is there a SOURCE the answer must be grounded in?
    YES -> faithfulness can be measured reference-free
           against the source.

Is the property DETERMINISTICALLY CHECKABLE?
    YES -> do that FIRST, before any model-based scoring.
           Schema validity, citation integrity, required
           elements, forbidden patterns.
The third question is the one teams skip. Domain 3, Module 3 established that a deterministic check answers with certainty at no model cost. Any evaluation programme should exhaust the deterministic tier before reaching for a judge.

1.4.2 The hybrid, which is usually right
  TIER 1  DETERMINISTIC     schema, citation integrity,
                            required elements, forbidden
                            terms
                            -> free, certain, run on
                               everything

TIER 2  REFERENCE-BASED   where ground truth exists
                            -> cheap, reproducible

TIER 3  REFERENCE-FREE    LLM-as-judge for the open
                            dimensions
                            -> expensive, needs validation,
                               run on a sample or on the
                               golden set

TIER 4  HUMAN             the calibration layer for
                            tiers 2 and 3
                            -> expensive, run on a small
                               sample
Each tier calibrates the one above it, and each is cheaper than the one below. Domain 1, Module 6 argued the same structure for prompt QA; it generalizes.
1.5 Legacy metrics and their limits
Know these by name and by limitation, because they appear in scenarios as plausible-looking wrong answers.

MetricWhat it measuresWhy it misleadsBLEUN-gram overlap with a reference, precision-orientedBuilt for machine translation. Correlates poorly with quality for open-ended generation — a better answer that shares fewer n-grams scores worse.ROUGEN-gram overlap, recall-orientedBuilt for summarization against reference summaries. Same problem: rewards lexical similarity, not quality.BERTScoreEmbedding similarity between output and referenceBetter than n-gram overlap because it captures paraphrase — and still reference-bound, so it cannot evaluate an answer with no reference.PerplexityThe model's confidence in a sequenceMeasures confidence, not correctness. A confidently wrong output has low perplexity. Useful for model training diagnostics, misleading as a quality proxy.Exact matchString identity with a referenceOnly meaningful for closed short answers. Brittle to formatting.F1 (token overlap)Harmonic mean of token precision and recallReasonable for short extractive answers; poor for anything generative.
The pattern: every one of these was designed for a task with references, and most were designed before open-ended generation was the norm. They remain useful for closed tasks and are actively misleading for open ones.
The exam-relevant version: a scenario proposing BLEU or ROUGE to evaluate an open-ended assistant is proposing a metric that will not measure what the team cares about.
1.6 Offline and online evaluation
  OFFLINE
    Fixed dataset, run pre-deployment or on a schedule.
    + reproducible, controlled, safe, comparable across
      runs
    - the dataset is CURATED, and curated distributions
      differ from real ones

ONLINE
    Live traffic, measured in production.
    + the real distribution, real users, real stakes
    - noisy, confounded, no ground truth, and failures
      reach users
1.6.1 Why offline results fail to predict online performance
This is a structural problem, not a sampling accident.
  Curated datasets are built by people who understand the
  system. They contain:
    - well-formed questions
    - questions the system was designed to answer
    - questions whose answers the curator knew

Real traffic contains:
    - ambiguous and underspecified questions
    - questions outside the intended scope
    - multi-part questions
    - typos, fragments, and pasted content
    - questions that reveal gaps nobody anticipated
So offline evaluation systematically overstates performance, and the gap is largest exactly where it matters — on the inputs nobody thought of.
The remedy is not to abandon offline evaluation but to build the dataset from real traffic (Module 2) and to run both.

1.6.2 The complementary roles
  OFFLINE answers:  "did this change help, holding the
                     input distribution constant?"
                     -> the REGRESSION and COMPARISON question

ONLINE answers:   "is the system working for real users
                     right now?"
                     -> the MONITORING question

You need both. Offline gates the change; online detects
  what offline could not anticipate.
Domain 4, Module 8's proxy quality signals are the online layer — regeneration rate, escalation, abandonment. They require no ground truth and they reflect the real distribution.
1.7 Statistical literacy
The section that prevents most false conclusions in this domain.

1.7.1 The core discipline
A three-point improvement on fifty examples is noise.
  With n = 50 and a score around 0.85, the confidence
  interval is roughly +/- 10 percentage points.

So 0.85 vs 0.88 on 50 examples is INDISTINGUISHABLE.

To detect a genuine 3-point difference with confidence,
  you need HUNDREDS of examples -- and the smaller the
  effect, the more you need.
A surprising number of GenAI improvements sit inside their own error bars, and shipping them wastes effort and creates false institutional knowledge about what works.

1.7.2 What to do about it
  1. REPORT CONFIDENCE INTERVALS, not point estimates.
     "0.87 +/- 0.04" is a claim. "0.87" is a number.

2. SIZE THE DATASET FOR THE EFFECT YOU NEED TO DETECT.
     Deciding in advance what difference matters tells you
     how many examples you need.

3. WHEN TWO OPTIONS ARE STATISTICALLY TIED, SAY SO --
     and choose on cost or latency.
     This is a CORRECT and defensible answer, and it is
     frequently the right one.

4. USE PAIRED COMPARISON where possible.
     Scoring both variants on the SAME examples removes
     between-sample variance and detects smaller effects
     with fewer examples.

5. BEWARE MULTIPLE COMPARISONS.
     Testing twenty prompt variants and taking the best
     will find an apparent winner by chance. Validate the
     winner on a HELD-OUT set.
Point 4 is the cheapest improvement available to most evaluation programmes. Paired comparison on the same examples is strictly more powerful than independent samples, and it costs nothing extra.
Point 5 is the one that produces false institutional knowledge. A team that tests many variants and ships the highest scorer without held-out validation has selected for noise, and will believe something about their system that is not true.

1.7.3 Non-determinism compounds it
Generative output varies run to run at temperature above zero, which adds variance on top of sampling variance.
  REMEDIES
    - evaluate at temperature 0 where the task permits,
      removing one source of variance
    - or run N samples per example and average, accepting
      N times the cost
    - report the VARIANCE, not only the mean -- a model
      with the same mean and higher variance is worse for
      most production purposes
1.8 The AWS surface
1.8.1 One API, two families
The organizing structure is clean and worth knowing precisely.

Everything in Amazon Bedrock Evaluations is created through one API operation — CreateEvaluationJob — and a single top-level field decides which family you are in. applicationType takes exactly two values: ModelEvaluation or RagEvaluation.
  applicationType = ModelEvaluation
    -> "is this MODEL good for my task?"

applicationType = RagEvaluation
    -> "is my RETRIEVAL, or my end-to-end RAG pipeline,
        good?"
Within each family, job types are distinguished by how scoring is performed.

1.8.2 Model evaluation — three job types
Job typeMechanismCharacterProgrammatic (automatic)Traditional NLP algorithms — BERTScore, F1, exact-match variants — computed against built-in curated datasets or your own. No judge model involved.Deterministic, reference-basedLLM-as-a-judgeA second foundation model — the evaluator — scores the responses of the model under test (the generator), assigning a score and an explanation.Reference-free, probabilisticHumanYour own workforce, or an AWS-managed one, rating against built-in or custom metrics.The calibration layer
"Which is better, programmatic or LLM-as-a-judge?" is the wrong question — they answer different questions, and §1.4 gives the selection rule.

1.8.3 RAG evaluation — two types
TypeMetricsRetrieval onlyContext relevance, context coverage — is the knowledge base retrieving relevant and complete context?Retrieve and generate (end-to-end)Quality: correctness, completeness, faithfulness (hallucination detection). Responsible AI: harmfulness, answer refusal, stereotyping.
The split maps directly onto Domain 1's diagnostic: evaluate retrieval separately from generation, because they are separate failure modes. Module 4 develops this.
A practical sequence worth knowing: start with retrieval-only evaluation to settle chunking strategy and retrieval settings, then move to end-to-end once retrieval is solid. Optimizing generation over broken retrieval is wasted effort.

1.8.4 Metric categories for LLM-as-a-judge
Four areas, and knowing the grouping helps you select:
  QUALITY               correctness, completeness, faithfulness
  USER EXPERIENCE       helpfulness, coherence, relevance
  INSTRUCTION COMPLIANCE following instructions,
                        professional style and tone
  SAFETY                harmfulness, stereotyping, answer
                        refusal
Scores are normalized 0 to 1, and the reported figure for each metric is an average across all queries in your dataset — which means a single low-scoring subset is diluted by the aggregate. Disaggregate (Domain 4, Module 8).

1.8.5 Bring your own inference
The capability that makes the service general, and it is frequently the answer to a scenario constraint.

If you provide only prompts, Bedrock invokes the model for you. If you provide a modelResponses array (for model-as-judge) or an output block (for RAG), Bedrock skips the invoke step and scores your bring-your-own-inference data.
And critically: this is how you evaluate a non-Bedrock model. A SageMaker-hosted open-weights model, a third-party API, an on-premises deployment — generate the responses yourself and submit them for scoring.
It also lets you evaluate a custom RAG pipeline that is not built on Bedrock Knowledge Bases.

1.8.6 The agent boundary
A scoping fact worth knowing precisely:

AgentCore Evaluations is a separate product for evaluating agents — trajectories, tool use, and multi-step behaviour. Use Bedrock Evaluations for models and RAG; use AgentCore Evaluations for agents.
Module 5 covers agent evaluation, and the reason it needs a different product is developed there: an agent's output is a trajectory, not a response, and scoring the final answer misses most of what went wrong.

1.8.7 Cost
RAG evaluation is not cheap, because it runs multiple LLM calls per test case — a judge call per metric, over every query in the dataset.
The consequence for design: evaluation cost scales with dataset size × metric count × frequency. Run the deterministic tier on everything, the judge tier on a golden set, and human evaluation on a sample (§1.4.2). Running every metric on every example on every commit is how an evaluation programme becomes too expensive to keep.
1.9 Consolidated decision reference
  +-----------------------------------------------------------------+
  |  THERE IS NO ACCURACY                                            |
  |    Generative output has an UNBOUNDED output space and MANY      |
  |    valid answers. Traditional ML metrics presume a discrete      |
  |    label space and do not transfer.                              |
  |    Evaluation is MULTI-DIMENSIONAL SCORING AGAINST CRITERIA      |
  |    YOU DEFINE -- and defining them is the engineering work.      |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  THE DISTINCTION THAT MATTERS MOST                               |
  |    FAITHFUL != CORRECT                                           |
  |      faithful but incorrect -> the SOURCE is wrong (a data       |
  |        freshness problem no faithfulness metric sees)            |
  |      correct but unfaithful -> answered from parametric          |
  |        knowledge. A DEFECT, not a success.                       |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  WHICH EVALUATION APPROACH?                                      |
  |    deterministically checkable -> DETERMINISTIC CHECK, first,    |
  |      (schema, citations,          free, certain, on everything   |
  |       required elements)                                         |
  |    a single correct answer     -> REFERENCE-BASED                |
  |      exists                       (cheap, reproducible, rigid)   |
  |    open-ended output           -> REFERENCE-FREE / judge         |
  |                                   (needs validating; costs a     |
  |                                    call per evaluation)          |
  |    calibrating the above       -> HUMAN, on a small sample       |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  LEGACY METRICS -- KNOW THE LIMITS                               |
  |    BLEU / ROUGE  n-gram overlap; POOR correlation with quality   |
  |                  for open-ended generation                       |
  |    BERTScore     embedding similarity; better, still             |
  |                  REFERENCE-BOUND                                 |
  |    PERPLEXITY    model CONFIDENCE, not correctness -- a          |
  |                  confidently wrong output has low perplexity     |
  |    exact match   closed short answers only; brittle              |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  STATISTICS                                                      |
  |    A 3-POINT IMPROVEMENT ON 50 EXAMPLES IS NOISE.                |
  |    Report CONFIDENCE INTERVALS. Size the dataset for the         |
  |      effect you need to detect.                                  |
  |    Statistically tied -> SAY SO, and choose on cost or latency.  |
  |    USE PAIRED COMPARISON -- same examples, both variants --      |
  |      strictly more powerful, no extra cost.                      |
  |    Testing 20 variants and taking the best SELECTS FOR NOISE.    |
  |      Validate on a HELD-OUT set.                                 |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  THE AWS SURFACE                                                 |
  |    ONE API: CreateEvaluationJob                                  |
  |    applicationType = ModelEvaluation | RagEvaluation             |
  |                                                                  |
  |    MODEL EVAL job types: PROGRAMMATIC (BERTScore, F1, exact      |
  |      match -- deterministic, no judge), LLM-AS-A-JUDGE           |
  |      (evaluator scores the generator, with an explanation),      |
  |      HUMAN                                                       |
  |    RAG EVAL types: RETRIEVAL ONLY (context relevance,            |
  |      context coverage) and RETRIEVE-AND-GENERATE (correctness,   |
  |      completeness, faithfulness + harmfulness, answer refusal,   |
  |      stereotyping)                                               |
  |                                                                  |
  |    BYOI: supply modelResponses or an output block and Bedrock    |
  |      SKIPS the invoke -- this is how you evaluate a              |
  |      NON-BEDROCK model or a custom RAG pipeline                  |
  |    AGENTS: use AGENTCORE EVALUATIONS, a separate product        |
  |    Scores are 0-1, AVERAGED across the dataset -> DISAGGREGATE   |
  |    RAG eval runs MULTIPLE LLM CALLS PER TEST CASE -> not cheap   |
  +-----------------------------------------------------------------+
1.10 Checkpoint
1. Why do traditional ML metrics not transfer to generative output? (1.1.1)

They presume a discrete label space where correctness is a comparison to a label. Generative output has an unbounded output space, many valid answers to the same question, and quality dimensions that trade against each other.
2. Distinguish faithfulness from correctness with an example of each failure. (1.2.1)

Faithful but incorrect: the source says 60 days, the policy changed to 30, the document is stale — perfectly faithful and wrong, and no faithfulness metric sees it. Correct but unfaithful: retrieval returned nothing useful and the model answered from parametric knowledge and happened to be right — a defect, because it will fail identically on anything outside training.
3. Which quality dimension is rarely worth measuring, and why? (1.2.2)

Fluency and coherence. They are near-saturated on modern models, so a score of 0.97 tells you nothing you did not already know. Measure it only with evidence it is your binding constraint.
4. Give three pairs of conflicting dimensions. (1.3)

Completeness against concision; faithfulness against helpfulness, because maximizing faithfulness raises refusals; helpfulness against safety; and consistency against variety.
5. What is a weighted composite quality score, properly understood? (1.3.1)

A policy decision requiring a documented owner, not a measurement. It is the same shape as the fairness-definition problem — several defensible criteria that cannot be jointly maximized, so someone must choose and own the choice. Report the components alongside it.
6. State the selection rule between reference-based and reference-free. (1.4.1)

Does a single correct answer exist? If yes, reference-based is available, cheaper, and deterministic. If no, reference-based penalizes valid answers and you need reference-free. And before either: is the property deterministically checkable? If so, do that first — it is free and certain.
7. Describe the four-tier evaluation stack. (1.4.2)

Deterministic checks on everything (schema, citation integrity, required elements); reference-based where ground truth exists; reference-free judging for open dimensions on a golden set; human evaluation on a small sample. Each tier calibrates the one above it, and each is cheaper than the one below.
8. Why do BLEU and ROUGE mislead for open-ended generation? (1.5)

They measure n-gram overlap with a reference, so a better answer sharing fewer n-grams scores worse. They were designed for translation and summarization against reference texts and correlate poorly with quality when many valid answers exist.
9. What does perplexity actually measure? (1.5)

The model's confidence in a sequence, not correctness. A confidently wrong output has low perplexity. It is useful for training diagnostics and misleading as a quality proxy.
10. Why do offline results systematically overstate online performance? (1.6.1)

Curated datasets are built by people who understand the system, so they contain well-formed questions the system was designed to answer. Real traffic contains ambiguous, underspecified, out-of-scope, and multi-part questions. The gap is largest exactly where it matters — on inputs nobody anticipated.
11. What question does each of offline and online evaluation answer? (1.6.2)

Offline: did this change help, holding the input distribution constant — the regression and comparison question. Online: is the system working for real users now — the monitoring question. Offline gates the change; online detects what offline could not anticipate.
12. Two prompt variants score 0.85 and 0.88 on 50 examples. What do you conclude? (1.7.1)

Nothing. At that sample size the confidence interval is roughly ±10 points, so the difference is indistinguishable from noise. Either gather more examples or declare them tied and choose on cost or latency — which is a correct and defensible answer.
13. Which statistical technique is the cheapest improvement to most evaluation programmes? (1.7.2)

Paired comparison — scoring both variants on the same examples. It removes between-sample variance, detects smaller effects with fewer examples, and costs nothing extra.
14. What goes wrong when you test twenty variants and ship the best? (1.7.2)

You have selected for noise. Some variant will score highest by chance, and shipping it creates false institutional knowledge about what works. Validate the winner on a held-out set.
15. What are the two values of applicationType, and what does each answer? (1.8.1)

ModelEvaluation — is this model good for my task? — and RagEvaluation — is my retrieval, or my end-to-end RAG pipeline, good? Everything is created through the single CreateEvaluationJob operation.
16. Name the three model evaluation job types. (1.8.2)

Programmatic, using traditional algorithms like BERTScore, F1, and exact match with no judge model — deterministic and reference-based. LLM-as-a-judge, where an evaluator model scores the generator's responses with a score and an explanation. And human evaluation with your own or an AWS-managed workforce.
17. How do you evaluate a model that is not hosted on Bedrock? (1.8.5)

Bring your own inference — supply a modelResponses array (or an output block for RAG) and Bedrock skips the invoke step and scores the data you provide. The same mechanism evaluates a custom RAG pipeline not built on Knowledge Bases.
18. Which product evaluates agents, and why is it separate? (1.8.6)

AgentCore Evaluations. An agent's output is a trajectory — tool use and multi-step behaviour — not a single response, so scoring the final answer misses most of what went wrong.
1.11 Practice cases
Case 5.1 — Four failures, four dimensions
Scenario. An insurance customer assistant produces the following four responses. For each, identify which quality dimensions pass and which fail, and say what kind of problem it indicates.
A. User asks: "What's the waiting period for dental coverage?" Retrieved document (last updated 14 months ago) states a 12-month waiting period. The policy was changed to 6 months last quarter; the document was never updated. Response: "The waiting period for dental coverage is 12 months, as stated in the policy document [POL-3320]."
B. User asks: "Does my plan cover physiotherapy?" Retrieval returns nothing relevant — the physiotherapy schedule was never indexed. Response: "Yes, physiotherapy is covered under most standard plans, typically with a limit of 10 sessions per year." This happens to be correct for their plan.
C. User asks: "How do I add my newborn to my policy?" Response: a 900-word, accurate, well-organized summary of the company's family coverage options, eligibility rules, and premium structures. It does not state how to add a dependent.
D. User asks: "Is my claim approved?" Response: "I understand you're asking about your claim status. Claim decisions depend on a number of factors including policy terms, submitted documentation, and assessment timelines. I'd encourage you to review your policy documents or contact our claims team for the most accurate information."
Produce: the dimension analysis for each, and the underlying problem.
Discussion.
A — Faithful, incorrect. A data freshness problem.

DimensionResultRelevancePass — it answers the question askedFaithfulnessPass, perfectly — it accurately reports the source and cites itCorrectnessFail — the answer is wrongFormat, fluencyPass
The critical point: no faithfulness metric will catch this, and a grounding check would score it highly. The response is a faithful rendering of a stale document.
The underlying problem is not in the model or the prompt — it is corpus freshness (Domain 1, Module 3). This is why faithfulness and correctness must be measured separately, and why a system with excellent faithfulness scores can still be giving wrong answers.
Detection: correctness requires ground truth or a source-of-truth reconciliation. A document-age monitor and a reconciliation between the corpus and the authoritative policy system would catch the class.
B — Correct, unfaithful. A defect that looks like a success.

DimensionResultRelevancePassCorrectnessPass — it happens to be rightFaithfulnessFail — nothing supported itCompletenessPass
This is the failure that looks like success (Domain 3, Module 4, §4.2.2). The model answered from parametric knowledge, bypassing grounding entirely.
Why it is a defect despite being correct: it will behave identically on the next question — including one about a plan-specific limit, a recent product, or an internal term the model has never seen — and be confidently wrong. The correctness here is luck, and it is not repeatable.
And note the hedging language — "most standard plans," "typically" — which is the tell. A grounded answer would cite the schedule.
Detection: citation integrity (Domain 3, Module 4, §4.4.2) catches it deterministically. No citation, or a fabricated one, means the answer was not grounded. A sufficient_context field would have let the model abstain instead.
C — Complete, accurate, irrelevant.

DimensionResultCorrectnessPassFaithfulnessPassCompletenessPass — thoroughlyRelevanceFail — it does not answer the questionConcisionFail
This is the case that shows why relevance is a distinct dimension. The response would score well on correctness, faithfulness, and completeness — three of the four quality metrics — and it is useless.
The underlying problem is likely retrieval: a query about how to add a dependent retrieved documents about coverage options, probably because both are semantically close in embedding space. This is a retrieval precision problem (Module 4), not a generation problem.
And note the completeness/relevance interaction: optimizing for completeness would make this response worse, because it would encourage more coverage of the wrong topic.
D — Fluent, safe, faithful, and useless.

DimensionResultFluencyPassSafetyPassFaithfulnessPass — it asserts nothing unsupportedHelpfulnessFailRelevanceMarginal — on-topic, and it does not answer
This is the faithfulness-versus-helpfulness conflict in its purest form (§1.3). The response is maximally safe and faithful precisely because it says nothing.
The interesting question is whether this is a failure at all, and the answer is task-dependent:

If claim status is genuinely unavailable to the assistant, this is correct behaviour — an honest abstention with a route to resolution.
If the assistant could look up claim status via a tool and did not, it is a tool-selection failure.
If it could and was instructed not to, the design is wrong.
The design lesson: a system optimized only for faithfulness converges on this response for everything. It is why helpfulness must be measured alongside faithfulness, and why a refusal rate is a quality metric rather than a safety metric alone.
The overall pattern: four responses, four different dimensions failing, and each one passes the metrics the others fail. A single composite score would rank these as roughly equivalent mediocre answers and tell you nothing about what to fix — which is the argument for dimensional reporting made concrete.
The reasoning being tested: can you separate the quality dimensions cleanly, and do you recognize that the underlying problem for each sits in a different part of the system?
Case 5.2 — The improvement that was not
Scenario. A team reports a prompt improvement.

They tested 20 prompt variants
Each was scored by an LLM judge on a 60-example evaluation set
The baseline scored 0.78
The best variant scored 0.86
They propose shipping it
Additional facts: the evaluation set was written by the team from their understanding of user needs; the judge is the same model family as the generator; scoring was done once per example at temperature 0.7.
Produce: an assessment.
Discussion.
Four independent problems, any one of which undermines the conclusion.
1. Testing 20 variants and taking the best selects for noise (§1.7.2).
With 20 variants and a 60-example set, some variant will score highest by chance. An eight-point apparent improvement is well within what random variation produces across 20 attempts at this sample size.
The fix: validate the winner on a held-out set the variants were not selected against. If 0.86 holds on fresh examples, it is real. If it regresses toward 0.78, it was noise — and this is the most likely outcome.
2. Sixty examples cannot support an eight-point claim — or rather, it is borderline, and the team has not said.
At n=60 around 0.8, the confidence interval is roughly ±10 points. An eight-point difference may or may not be significant, and reporting a point estimate without an interval makes the claim unassessable.
The fix: report confidence intervals, and use paired comparison — score baseline and variant on the same examples — which removes between-sample variance and would make this comparison substantially more powerful at no extra cost.
3. The judge is the same model family as the generator — self-preference bias (Domain 3, Module 8, §8.8.1).
Judges systematically favour output from their own family. This does not invalidate the comparison between two variants of the same generator, since the bias applies to both — but it does mean the absolute scores are inflated, and it would invalidate any cross-model comparison.
More seriously: the judge has not been validated against human labels (Module 3). An unvalidated judge is an opinion generator, and 0.78 versus 0.86 on an unvalidated scale is two numbers rather than a measurement.
4. The evaluation set was written by the team, not drawn from real traffic (§1.6.1).
A curated set built by people who understand the system contains well-formed questions the system was designed to answer. It systematically overstates performance and misses the inputs that actually cause problems.
The fix: build the dataset from production traffic (Module 2), stratified to include the ambiguous, out-of-scope, and multi-part queries that curation omits.
One further issue: scoring at temperature 0.7, once per example.
Non-determinism adds variance on top of sampling variance (§1.7.3). Either evaluate at temperature 0, removing a source of noise, or sample N times and average — and report the variance either way.
What I would tell the team:
  The result is not yet a result. To make it one:

1. Validate the winner on a HELD-OUT set. This is the
     single most important step -- it distinguishes a real
     improvement from a selected fluke.
  2. Use PAIRED comparison and report CONFIDENCE INTERVALS.
  3. VALIDATE THE JUDGE against ~50 human-labelled
     examples. If agreement is poor, fix the rubric before
     trusting any score.
  4. Rebuild the evaluation set FROM PRODUCTION TRAFFIC.
  5. Evaluate at temperature 0, or average over N samples.
  6. Check the OTHER dimensions -- did faithfulness improve
     while helpfulness or refusal rate got worse? (1.3)
Point 6 deserves emphasis and the team has not mentioned it. An eight-point gain on a composite judge score is compatible with a large improvement on one dimension and a regression on another. A single number cannot tell you which, and if the variant improved faithfulness by refusing more often, that may not be an improvement at all.
What is genuinely good about their process: they tested variants systematically rather than changing the prompt once and declaring victory, and they used a fixed evaluation set. The methodology is close to sound and needs the statistical discipline added, which is a much better position than having no evaluation at all.
The reasoning being tested: can you recognize multiple-comparison selection, and do you check the statistical, judge-validity, and dataset-representativeness dimensions rather than accepting a number at face value?
1.12 Skill coverage audit
Skill statementWhere discharged5.1.1 — Evaluation frameworks and metrics for GenAI systemsWhy traditional ML metrics do not transfer, and the three structural consequences: §1.1. The nine quality dimensions, the faithfulness-versus-correctness distinction with worked failure modes, and why fluency is near-saturated: §1.2. The dimension conflict, why a composite is a policy decision rather than a measurement, and task-dependent weighting: §1.3. Reference-based versus reference-free evaluation, the selection rule, and the four-tier hybrid stack: §1.4. Legacy metrics — BLEU, ROUGE, BERTScore, perplexity, exact match — with their specific limitations: §1.5. Offline versus online evaluation and why offline systematically overstates: §1.6. Statistical literacy — sample sizing, confidence intervals, paired comparison, multiple comparisons, and non-determinism: §1.7. The Bedrock Evaluations surface — one API, two families, three model-evaluation job types, two RAG types, metric categories, bring-your-own-inference, the AgentCore boundary, and cost: §1.8. Applied in Cases 5.1 and 5.2.1.13 Module summary
There is no accuracy:

Traditional ML metrics presume a discrete label space. Generative output has an unbounded space, many valid answers, and conflicting quality dimensions.
Evaluation is multi-dimensional scoring against criteria you define, and defining them is the engineering work.
The dimensions:

Relevance, correctness, faithfulness, completeness, consistency, fluency, safety, format compliance, tone.
Faithful is not correct. A response can be faithful to a stale source (a data freshness problem no faithfulness metric sees), or correct while ungrounded (a defect that looks like success).
Fluency is near-saturated — measuring it is usually wasted effort.
The conflict:

Completeness against concision; faithfulness against helpfulness; helpfulness against safety; consistency against variety.
A weighted composite is a policy decision with an owner, not a measurement. Report dimensions separately, and when one improves, check the others.
The weighting is task-dependent, and deriving it explicitly at design time prevents the argument during an incident.
Choosing an approach:

Deterministic checks first — free, certain, run on everything.
Reference-based where a single correct answer exists — cheap, reproducible, rigid.
Reference-free for open-ended output — needs the judge validated, costs a call per evaluation.
Human as the calibration layer on a small sample.
BLEU and ROUGE mislead for open-ended generation; BERTScore is better and still reference-bound; perplexity measures confidence, not correctness.
Offline and online:

Offline systematically overstates, because curated sets contain well-formed questions the system was built to answer.
Offline gates the change; online detects what offline could not anticipate. You need both.
Statistics:

A three-point improvement on fifty examples is noise. Report confidence intervals; size the dataset for the effect you need to detect.
Paired comparison is the cheapest available improvement — same examples, both variants, strictly more powerful.
Testing many variants and shipping the best selects for noise. Validate on a held-out set.
Non-determinism compounds sampling variance — evaluate at temperature 0, or average over N and report the variance.
The AWS surface:

One operation, CreateEvaluationJob; applicationType is ModelEvaluation or RagEvaluation.
Model evaluation: programmatic (BERTScore, F1, exact match — deterministic, no judge), LLM-as-a-judge (evaluator scores generator, with an explanation), and human.
RAG evaluation: retrieval only (context relevance, context coverage) and retrieve-and-generate (correctness, completeness, faithfulness, plus harmfulness, answer refusal, stereotyping).
Bring your own inference — supply responses and Bedrock skips the invoke, which is how you evaluate a non-Bedrock model or a custom RAG pipeline.
Agents use AgentCore Evaluations, a separate product, because an agent's output is a trajectory.
Scores are 0–1 averaged across the dataset — disaggregate. And RAG evaluation runs multiple LLM calls per test case, so it is not cheap.
Carry into Module 2: this module defined what to measure. Module 2 addresses what to measure it on — dataset construction, ground truth, stratification, and why an evaluation set built by the team is the most common reason a good evaluation programme produces misleading results.
End of Module 1. Module 2 — Evaluation Datasets and Ground Truth — follows.

Unstarred

Dunk (exclude from study)

Module 1: Evaluation Foundations

Review Questions

Todos

Sensemaker