Cost Optimization, Performance, and Observability

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

Cost Optimization, Performance, and Observability
A Complete Architect's Reference — Domain 4
AWS Certified Generative AI Developer – Professional
Front Matter
What this book is
This is a self-contained textbook for Domain 4. It assumes you will not read anything else. Where a concept matters, it is explained from first principles. Where a tradeoff exists, both sides are argued rather than summarized.
It follows the same structure as the previous three books. One thing distinguishes this domain from them: vague answers die faster here than anywhere else in the certification.
"Use caching" is not an answer. "A semantic cache at a 0.95 similarity threshold, which our query distribution suggests will hit on roughly 30% of traffic, saving X against a measured wrong-answer rate of Y" is an answer. Every question in this domain has a number behind it, and the arithmetic is usually simple enough to do in your head — which means being unable to do it is conspicuous.
Every module follows the same shape:

SectionPurposeConceptsThe mental model, service-agnostic.MechanicsHow it works underneath. Enough depth to reason about failure.AWS surfaceSpecific services, their limits, their failure modes.Decision frameworksExplicit procedures for making the choice.CheckpointShort recall questions to verify retention.Practice casesRealistic architect scenarios with worked discussion.Skill coverageAn audit table mapping exam skill statements to sections.
The organizing question
Domain 1 asked whether you could get the right information in front of a model. Domain 2 asked whether you could let it act safely. Domain 3 asked whether you could prove it.
Domain 4 asks:

Can you run it at scale without going broke or going slow — and would you know if you were?
Two framings that unlock the domain
1. Tokens are the unit of everything.
Cost is tokens. Latency is tokens. Quotas are tokens. Capacity is tokens. If you find yourself reasoning in "requests," translate to tokens first — a system serving 100 requests per second of 50-token completions and one serving 100 requests per second of 4,000-token completions have almost nothing in common operationally, and any conclusion drawn about one will be wrong about the other.
2. Throttling and latency are different problems with different controls.
Throttling is a capacity problem. It appears as a 429 and is governed by token-per-minute quotas. Latency is a timing problem, measured in time-to-first-token and output tokens per second. A request can be fast and still be throttled; it can have abundant quota and still be slow. Neither set of remedies helps the other.
The exam conflates these deliberately in its distractors. Diagnosing which one a scenario describes is the first move, and it is frequently the whole answer.
  +------------------------------------------------------------------+
  |                         DOMAIN 4 ARC                              |
  +------------------------------------------------------------------+

Task 4.1                    Task 4.2
     Cost and resource     +     Application
     efficiency                  performance
         |                            |
         +--------------+-------------+
                        v
                   Task 4.3
                   Monitoring systems for GenAI
                   (the layer that tells you whether
                    any of the above is working)
Module map and depth tiers
ModuleTitleTierSkills1The token economyStandardfoundation — supports all2Token efficiency and context optimizationStandard4.1.13Model selection, tiering, and distillationStandard4.1.24CachingDeep4.1.45Throughput, capacity, and scalingDeep4.1.3, 4.2.3, 4.2.56Latency optimization and inference tuningStandard4.2.1, 4.2.4, 4.2.67Retrieval and vector store performanceStandard4.2.2, 4.3.58Observability, monitoring, and troubleshootingDeep4.3.1–4.3.4, 4.3.6
Deep ≈ 10–12k words. Standard ≈ 6–8k.

Recommended reading order
Straight through works. Two notes:
Module 1 is short and load-bearing. Do the arithmetic in it rather than reading past it. Every later module assumes you can compute a token cost and a utilization ratio without hesitation.
Modules 4, 5, and 8 are where the exam concentrates. Caching type selection, the throttling-versus-latency diagnosis, and GenAI-specific failure detection produce the scenario questions with four plausible-looking answers. Module 3 is conceptually straightforward and arithmetic-heavy — practise the cascade calculation until it is automatic.

A note on volatility
Prices, quota values, model unit throughput, and discount rates change. This book teaches relationships and shapes — that output tokens cost several times more than input tokens, that batch inference carries a substantial discount, that committed capacity has a utilization crossover — rather than specific figures. Where a number is needed for a decision, the book gives the arithmetic structure and tells you to look up the input.
What does not change: that prefill is compute-bound and decode is memory-bandwidth-bound, that context grows monotonically in an agent loop, that caching's hardest problem is invalidation, and that a system can be fully available and completely wrong.

What you should have from Domains 1–3
Prefill and decode as distinct phases — prefill parallel and compute-bound, decode sequential and memory-bandwidth-bound.
The KV cache grows with sequence length and batch size, and is why prompt caching works.
Tokenization varies by model and content type.
Context grows monotonically through an agent loop, making cost roughly quadratic in step count.
A GenAI system can be fully available, fast, error-free, and completely wrong.

Domain 4 — Closing
What this book covered
  Module 1   The token economy. Cost and latency equations,
             the composite request, tokenization economics,
             the optimization hierarchy, and the
             throttling-versus-latency distinction.

Module 2   Token efficiency. The six components, estimation
             and tracking, context budgets, pruning,
             compression, the cacheable prefix, and response
             limits.

Module 3   Model selection and tiering. The Pareto frontier,
             right-sizing, four tiering patterns, the cascade
             crossover arithmetic, managed routing, and
             distillation.

Module 4   Caching. Four types, sizing the opportunity, the
             threshold problem, the asymmetric principle,
             verified caching, invalidation, and prompt
             caching operationally.

Module 5   Throughput and capacity. The throttling taxonomy,
             service tiers, cross-region inference, reserved
             capacity, three batchings, concurrency, and the
             remediation ladder.

Module 6   Latency. The metric vocabulary, the OTPS identity
             and its two-way diagnosis, perceived versus
             actual, latency-optimized inference, and
             parallelization.

Module 7   Retrieval performance. The triangle, query-time
             versus build-time parameters, segments,
             quantization and rescoring, filtering cliffs,
             and index health.

Module 8   Observability. The four dimensions, cost
             attribution, baselines, quality monitoring,
             correlation, dashboards, and the incident
             runbook.
The ideas that recur
1. Tokens are the unit of everything. Cost, latency, quotas, and capacity are all token quantities. Any statement in requests per second is meaningless without a token distribution attached — and output tokens count more than they appear to, in price and in quota burndown. (Modules 1, 5.)
2. Throttling and latency are different problems with different controls. Capacity remedies do not fix timing, and timing remedies do not fix capacity. Diagnosing which one a scenario describes is frequently the whole answer. (Modules 1, 5, 6.)
3. Reduce demand before buying supply. The optimization hierarchy runs from don't call the model to optimize the serving layer, and teams reliably start at the wrong end. Buying capacity to serve waste is the most expensive possible outcome. (Modules 1, 2, 5.)
4. Decompose before optimizing. An aggregate says "expensive" or "slow"; the decomposition names the intervention. Input tokens by component, latency by stage, errors by type, InvocationLatency into TTFT and OTPS. (Modules 2, 6, 7, 8.)
5. Measure before committing, and compute rather than recall. The cascade crossover, the reservation utilization, the cache hit-rate ceiling, the recall knee — each is simple arithmetic that inverts a plausible-sounding recommendation. (Modules 3, 4, 5, 7.)
6. The cheapest fix is usually configuration, not architecture. Query-time parameters over reindexes, tier changes over reservations, ef_search over m, maxTokens over anything. Ask what class the proposed fix belongs to. (Modules 5, 6, 7.)
7. Features fail silently. Prompt caching below a threshold, latency optimization falling back to standard, a guardrail that never applied, a cache invalidating on every request. Instrument whether a feature is applying, not merely whether it is configured. (Modules 2, 4, 6.)
8. Never report a cost or latency win without the quality delta. The characteristic failure of this domain is an optimization that improves two dimensions and quietly degrades a third — and it looks like success on every dashboard. (Modules 2, 4, 8.)

How to use this book for the exam
Scenario questions in this domain almost always contain the numbers you need. Do the arithmetic. The cascade crossover, the peak-to-average ratio, the reservation utilization, the token-equivalent comparison — these take thirty seconds and they frequently eliminate three of four options.
The distractor patterns are consistent: a latency remedy offered for a capacity problem, a build-time change offered where a query-time parameter would serve, and a serving-layer optimization offered where demand reduction is available. Ask which class the proposed fix belongs to before evaluating whether it would work.
Several practice cases turn on recognizing that a stated conclusion is wrong on the numbers — Case 4.7's cascade, Case 4.13's quota increase, Case 4.14's reservation. Verify the premise before designing against it.
The consolidated decision references are the densest revision material, and the arithmetic in Modules 3 and 5 is the most likely to be needed under time pressure.
End of Domain 4.

Unstarred

Dunk (exclude from study)

Cost Optimization, Performance, and Observability

Review Questions

Todos

Sensemaker