Module 4: Caching

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

Module 4 — Caching
Tier: Deep. Covers Skill 4.1.4 (intelligent caching systems to reduce costs and improve response times by avoiding unnecessary FM invocations — semantic caching, result fingerprinting, edge caching, deterministic request hashing, prompt caching).
4.0 Why this module exists
Caching is step 1 of the optimization hierarchy — the only step that eliminates a model call entirely rather than making it cheaper. It is frequently the single highest-leverage item in this domain.
It is also the one most often built wrong, in two specific ways.
Built before the opportunity is sized. Teams implement semantic caching, deploy it, and discover their traffic is 90% unique context-dependent queries with a hit-rate ceiling around 12%. Two weeks of work for a rounding error. Section 4.2 gives the hour-long measurement that would have told them.
Built with the wrong success metric. A cache that returns a plausible-but-wrong answer is not a partial success. It is strictly worse than no cache at all, because users receive confidently-presented incorrect data with a latency improvement that makes the system look healthier. Section 4.6 develops the design principle that follows.
The module's organizing insight is that the four cache types are genuinely different mechanisms with different risk profiles — and the safest one is also the one teams skip past on the way to the interesting one.
4.1 The four kinds
  +------------------------------------------------------------------+
  |  1. EXACT / DETERMINISTIC                                         |
  |     Key: a hash of the normalized request                         |
  |     Hit:  byte-identical (post-normalization) request             |
  |     Saves: THE ENTIRE CALL                                        |
  |     False positives: ZERO, by construction                        |
  +------------------------------------------------------------------+
  |  2. SEMANTIC                                                      |
  |     Key: an embedding of the query                                |
  |     Hit:  similarity above a threshold                            |
  |     Saves: THE ENTIRE CALL                                        |
  |     False positives: THE CENTRAL DESIGN PROBLEM                   |
  +------------------------------------------------------------------+
  |  3. PROMPT CACHING                                                |
  |     Key: a stable prompt prefix                                   |
  |     Hit:  the prefix is reused within the TTL                     |
  |     Saves: PREFILL COMPUTE on the cached portion --               |
  |            NOT the call itself                                    |
  |     False positives: none -- it is byte-exact prefix matching     |
  +------------------------------------------------------------------+
  |  4. EDGE / CDN                                                    |
  |     Key: a URL or request signature                               |
  |     Hit:  identical public request                                |
  |     Saves: the call AND the network round trip                    |
  |     False positives: none, but only applies to non-personalized   |
  |                      content                                      |
  +------------------------------------------------------------------+
The distinction that matters most: types 1, 2, and 4 avoid the model call. Type 3 does not. Prompt caching makes the call cheaper and faster by skipping prefill on a reused prefix; the generation still happens. Conflating them produces wrong expectations about savings.
A fifth technique sits alongside these: pre-computation — generating answers offline for predictable questions and serving from a store. It is caching with the population step moved ahead of demand (§4.10.2).
4.2 Size the opportunity first
The most common mistake in this module is building the caching layer before understanding the traffic.

4.2.1 The hour-long measurement
  THE FLOOR -- what exact caching would catch
    1. Log all prompts for a week
    2. Normalize and hash them
    3. Count exact duplicates
    -> typically 15-30% of production traffic

THE CEILING -- what semantic caching could catch
    4. Sample 1,000 requests
    5. Generate embeddings
    6. Cluster them; count how many fall within a
       ~0.92 similarity threshold of another
    -> this is your OPTIMISTIC upper bound

THEN estimate savings against your actual spend.
This costs an hour and it determines whether the project is worth doing. A team whose ceiling is 12% should not build semantic caching; a team whose floor is 28% should build exact caching immediately and think about semantic afterward.

4.2.2 Realistic expectations
Published claims of 90%+ savings do not match production data. Reported production hit rates for semantic caching cluster in the 20–45% range, with the higher end (up to around 70%) reported for agent workflows and FAQ-shaped traffic where repetition is genuinely high.
Exact caching catches 15–30% of production traffic with zero false positives.
The economics at a modest scale, to calibrate:
  Monthly LLM spend: $5,000

20% hit rate  ->  ~$1,000/month saved
    30% hit rate  ->  ~$1,500/month saved
    45% hit rate  ->  ~$2,250/month saved

Cache infrastructure cost (embedding generation +
  vector storage) is typically under 5% of the savings.
The infrastructure being cheap relative to the savings is what makes even a modest hit rate worth capturing — but only if the hit rate is real, which §4.2.1 establishes before you build.
4.3 Exact caching
Skill 4.1.4 names result fingerprinting and deterministic request hashing. This is that, and it is underrated because it is unglamorous.

4.3.1 Why start here
  + ZERO false positives, by construction
  + microsecond lookups; no embedding call on the hit path
  + trivial to implement and to reason about
  + no threshold to tune
  + catches 15-30% of production traffic
There is no correctness risk. That property alone justifies implementing it first, and it means the exact cache can run in front of a semantic cache as a fast, safe first layer.

4.3.2 Normalization is the whole design
A naive hash of the raw request produces a low hit rate, because trivially different inputs hash differently. Normalization is what lifts it.
  NORMALIZE BEFORE HASHING:
    - lowercase
    - collapse whitespace; strip leading/trailing
    - strip terminal punctuation
    - sort any parameter or filter objects into a
      canonical order
    - round or bucket numeric parameters where exactness
      does not matter
    - strip or canonicalize identifiers that do not
      affect the answer

DO NOT NORMALIZE AWAY:
    - anything that changes the answer
    - tenant or user scope (4.8)
    - the model ID and version
    - the prompt version
    - retrieval configuration
The second list is the safety-critical one. A cache key that omits the model ID will serve a response generated by a different model after a migration. A key that omits the prompt version will serve pre-change answers after a prompt update. Both are silent correctness failures.

4.3.3 What belongs in the key
  cache_key = hash(
      normalized_query,
      tenant_id,            <- from the authenticated session
      user_scope_or_role,   <- if answers differ by role
      model_id + version,
      prompt_version,
      retrieval_config_version,
      corpus_version        <- see 4.7
  )
Composing the key from a version tuple is what makes invalidation tractable (§4.7.2): bumping any version component invalidates everything downstream of it without an explicit purge.
4.4 Semantic caching
4.4.1 The mechanism
  query
    |
    v
  [ embed the query ]                     ~5-20ms
    |
    v
  [ vector search the cache for the
    nearest cached query embedding ]      ~1-5ms
    |
    +-- similarity >= threshold  ->  RETURN THE CACHED
    |                                RESPONSE (~3-8ms total)
    |
    +-- below threshold          ->  CALL THE MODEL
                                     (~500-2000ms)
                                     then STORE the query
                                     embedding + response
The value proposition: a cache hit returns in single-digit milliseconds against hundreds or thousands for a generation. That is a latency improvement of two orders of magnitude on the hit path, alongside the cost elimination.

4.4.2 The cache tax on misses
Every miss now costs more than it did before, because you paid for an embedding and a vector search before discovering there was no hit — commonly around 20 ms of added latency.
  The arithmetic at a 67% hit rate:

hits (67%):    ~8ms      instead of ~1,500ms
    misses (33%):  ~1,520ms  instead of ~1,500ms

-> a substantial net latency improvement, because the
     hits dominate

At a 10% hit rate:

hits (10%):    ~8ms
    misses (90%):  ~1,520ms

-> the improvement is marginal, and you have added
     infrastructure and a correctness risk for it
This is a second reason to size the opportunity first. Below a certain hit rate, semantic caching adds latency to most requests, adds a false-positive risk, and saves little.

4.4.3 Infrastructure
Redis with vector similarity search is the standard backing store for semantic caching at production scale, and is sufficient below roughly one million cache entries. Dedicated vector databases are worth the operational cost only above that scale.
Keep the embedding model off the critical path. The embedding call is the dominant latency on the hit path — one reported architecture measures the hit path at around 408 ms precisely because the embedding is a remote API call. Co-locating a small embedding model with the cache reduces this to single-digit milliseconds and changes the economics of the hit path entirely.
4.5 The threshold problem
The critical hyperparameter, and the source of every correctness failure in semantic caching.

4.5.1 The tradeoff
  TOO HIGH (0.99+)
    -> you miss obvious paraphrases
    -> hit rate collapses; the cache does little

TOO LOW (<0.85)
    -> you return the wrong answer for a different question
    -> silent factual errors

TYPICAL PRODUCTION RANGE: 0.92-0.97
    0.92-0.95 works for many English FAQ-shaped applications
But the range is a starting point, not an answer, for three reasons:
Thresholds are embedding-model-dependent. One published system found an optimal threshold of 0.78 for its embedding model, achieving peak F-score at that value. A threshold tuned for one model is meaningless for another.
Optimal thresholds vary by query type within the same application. Factual lookups, comparisons, and multi-part questions tolerate different thresholds.
A single static threshold is a known limitation. Research on adaptive per-embedding thresholds reports substantial improvements over any static value — up to large multiples in hit rate with simultaneous error reduction — precisely because one number cannot serve a heterogeneous query distribution.

4.5.2 The failure modes that survive a high threshold
This is the part that makes semantic caching genuinely dangerous, and it is not solved by raising the threshold.
Queries that differ only in these dimensions routinely exceed 0.95 cosine similarity under strong embedding models:
  TIME PERIOD
    "monthly active users in Q1 2024"
    "monthly active users in Q1 2025"
    -> near-identical embeddings, completely different answers

POLARITY
    "revenue growth by region"
    "revenue decline by region"
    -> negation is poorly represented in embedding space

ANALYTICAL INTENT
    "what is our churn rate"          (lookup)
    "how has our churn rate changed"  (comparison)
    -> similar surface form, different operation

ENTITY SUBSTITUTION
    "cancel my subscription"
    "pause my subscription"
    -> operationally opposite
Raising the threshold to 0.98 does not fix these, because they are genuinely above 0.98. The embedding does not encode the distinction you care about, and no threshold recovers information the representation does not contain.

4.5.3 Tuning requires ground truth
You cannot tune a threshold without knowing which query pairs are actually equivalent.
  1. Sample real query pairs from production logs
  2. Have a human label them: same question or not?
     -> this is the expensive, necessary step
  3. Compute similarity for each pair
  4. Plot the distributions:
       equivalent pairs    -- similarity distribution
       non-equivalent pairs -- similarity distribution
  5. Choose the threshold at the separation point,
     weighted toward avoiding false hits (4.6)
  6. Measure the resulting FALSE POSITIVE RATE explicitly
A reported production system that did this carefully achieved a 0.8% false-positive rate, and noted that those cases occurred primarily at the threshold boundary — similarity just above the cutoff, with subtly different intent. That is the expected shape: errors concentrate at the margin, which is an argument for verification at the margin rather than for a blanket threshold increase.
4.6 The asymmetric principle
4.6.1 The design rule
A cache that returns false hits is strictly worse than no cache, because users receive confidently-presented wrong data.
Therefore: cache misses are tolerable; false hits are not.
This asymmetry should govern every threshold decision, every verification choice, and every measurement. A cache miss costs money and latency. A false hit costs correctness, and it does so invisibly — the response is fast, fluent, and wrong.
The consequence for tuning: when the distributions in §4.5.3 overlap, err toward the higher threshold. A 30% hit rate with no false positives beats a 45% hit rate with a 2% error rate for almost any application where the answer matters.

4.6.2 Verified semantic caching
The architecture that follows from the asymmetric principle: treat embedding similarity as a pre-filter, not as the decision.
  candidate query
       |
       v
  [ EMBEDDING PRE-FILTER ]
    similarity >= threshold?
       |  no -> MISS (call the model)
       |  yes
       v
  [ MULTI-DIMENSIONAL EQUIVALENCE VERIFICATION ]
    a CONJUNCTION of independent checks, each enforcing
    a distinct dimension of query semantics:

- do the time periods match?
      - do the entities match?
      - does the polarity match?
      - does the analytical intent match?
      - do any explicit filters or constraints match?

ALL must pass.
       |  any fails -> MISS
       |  all pass
       v
  RETURN THE CACHED RESPONSE
The reported effect is striking. One enterprise system using this design achieved 100% precision at 87% recall — and noted that an embedding-only cache at the same similarity threshold would have admitted every one of the negative queries as a false hit, since all of them passed the embedding pre-filter.
Critically, every verification dimension contributed rejections that no other dimension caught. That is the argument for a conjunction of independent checks rather than a single better score: the failure modes are orthogonal.
The cost is around 20 ms beyond the embedding computation — negligible against a generation call, and it converts a probabilistic cache into one with a correctness guarantee for the dimensions checked.

4.6.3 Extracting the verification dimensions
The dimensions to check are domain-specific and derive from what distinguishes answers in your application:
  ANY analytics or reporting domain
    -> time period, aggregation level, metric, filters

ANY transactional domain
    -> entity identity, operation type, amount

ANY policy or documentation domain
    -> jurisdiction, product, version, effective date

ANY multi-tenant system
    -> tenant, always (4.8)
A practical way to derive them: take a sample of false hits from a threshold-only cache and ask what dimension each pair differed on. The failure modes name the checks.
4.7 Invalidation
The hardest problem in caching generally, and harder here because correctness depends on a corpus that changes underneath you.

4.7.1 The five staleness sources
  1. THE UNDERLYING FACTS CHANGE
     A policy is updated; a price changes; a product is
     discontinued. Yesterday's correct answer is today's
     wrong one.

2. THE CORPUS CHANGES
     A document is added, edited, or removed. Answers
     derived from it are now stale.

3. THE PROMPT CHANGES
     A new system prompt produces different answers to
     the same question.

4. THE MODEL CHANGES
     A migration, or a provider-side update.

5. THE EMBEDDING MODEL CHANGES
     -> INVALIDATES EVERY CACHED EMBEDDING. Vectors from
        different models are not comparable, so the entire
        semantic cache must be rebuilt.
Source 5 is the one people forget, and it is total rather than partial. An embedding model upgrade does not degrade the cache — it invalidates it completely, because the stored embeddings occupy a different space.

4.7.2 The layered invalidation strategy
Production systems combine three mechanisms, and each covers what the others miss:
  1. SHORT TTL as the baseline safety net
       typically 1-24 hours
       -> bounds the staleness window for everything,
          including changes you did not detect

2. VERSIONED KEYS
       a version tag in every cache key, bumped on:
         - model upgrades
         - system prompt changes
         - embedding model changes
         - corpus reindexes
       -> invalidation becomes a version bump rather than
          a purge operation

3. AN EXPLICIT INVALIDATION ENDPOINT
       manual purge for emergencies:
         - a bad cached response discovered post-hoc
         - a factual error found in a source document
         - a false hit reported by a user
       -> you WILL need this. Build it before you need it.
Versioned keys are the mechanism that scales. Bumping a version invalidates everything derived from that version atomically, without enumerating keys — and it makes the invalidation auditable, since the version tuple records what the cached response was generated under.

4.7.3 Freshness checks
TTL and event-driven invalidation both miss things. A source document edited without emitting an event, a fact that changed outside your system, a response that was wrong when it was cached.
The remedy is a periodic sampled freshness check:
  Daily, sample N cached entries.
  Re-generate the answer.
  Compare against the cached response.

Divergence rate is a MONITORED METRIC.
  -> a rising divergence rate means your invalidation
     is failing before users notice
This is cheap — a few hundred regenerations a day — and it is the only mechanism that catches staleness the other two miss.

4.7.4 Cache poisoning
A failure mode specific to GenAI: the model hallucinates, the wrong answer is cached, and it is then served repeatedly and instantly to many users.
Caching amplifies a single error into a systematic one. Mitigations:

Apply output validation before caching (Domain 3, Module 3): schema conformance, referential integrity on citations, grounding checks
Do not cache low-confidence responses. If the confidence signal (Domain 3, Module 4, §4.5) is below a band, serve it but do not store it.
Do not cache responses that triggered a guardrail intervention
Provide a user-facing way to flag a wrong answer, wired to the invalidation endpoint
"Validate before caching" is the general principle, and it is cheap relative to the cost of serving a hallucination a thousand times.
4.8 Multi-tenancy and cache safety
A cross-tenant cache hit is a data breach, not a bug. This section is short because the rule is absolute.

4.8.1 The rule
The tenant identifier — and any scope that changes what a user may see — must be part of the cache key, derived from the authenticated session.
  cache_key = hash(
      normalized_query,
      TENANT_ID,           <- from the authenticated session,
                              NEVER from the request payload
      USER_SCOPE,          <- if answers differ by role or
                              entitlement
      ... version components ...
  )
This is the same rule that appears in every domain of this material (Domain 3, Modules 5 and 6): identity is derived, never accepted. Here the consequence of getting it wrong is that Tenant B is served Tenant A's answer, quickly and confidently.

4.8.2 Where it goes wrong
Semantic caching makes this easier to get wrong than exact caching, because the natural implementation embeds the query text and searches a single vector index. Two tenants asking the same question produce near-identical embeddings.
  WRONG
    embed(query) -> search the whole cache -> return

RIGHT
    embed(query) -> search WITHIN THE TENANT'S PARTITION
                    (or with a mandatory tenant filter)
                 -> return

BETTER STILL, for high-sensitivity tenancy
    a separate cache namespace or index per tenant, with
    IAM scoping so the credential cannot reach another
    tenant's partition at all
Metadata filtering alone depends on every query path applying the filter correctly. Partitioning makes it structural — the Domain 3, Module 5 §5.5.4 argument applied to a cache.

4.8.3 Role-dependent answers
If two users of the same tenant receive differently-redacted answers to the same question — the role-based redaction pattern from Domain 3, Module 6, §6.5.2 — then role must be in the cache key too. Otherwise an auditor's unredacted cached answer is served to a support agent.
The general test: if two callers could legitimately receive different responses to the identical query, whatever distinguishes them belongs in the key.

4.8.4 The detection
Alarm on any cache hit where the cached entry's tenant differs from the requesting session's tenant. It is cheap to compute and it detects the highest-consequence failure directly — the same argument as Domain 3, Module 5, §5.7.3.
4.9 Prompt caching, operationally
Module 2, §2.6 established the structural requirement and the silent-failure trap. This section covers the operational depth.

4.9.1 What it is and is not
Prompt caching preserves the model's computed internal state after processing a prefix, so subsequent requests with the same prefix skip recomputing it.
  IT SAVES:  prefill compute on the cached portion
             -> lower input cost on that portion
             -> lower TIME TO FIRST TOKEN

IT DOES NOT SAVE:  the model call itself
                     decode -- generation is unaffected
It is therefore complementary to exact and semantic caching, not an alternative. Those avoid the call; this makes the call cheaper.

4.9.2 The mechanics
  CACHE CHECKPOINTS
    markers defining a contiguous prefix to cache.
    The prefix MUST BE STATIC between requests.

MINIMUM TOKEN THRESHOLD
    per-model, commonly 1,024 or 4,096.
    BELOW IT, NOTHING CACHES -- SILENTLY. The request
    succeeds at full price.
    The minimum counts TOOLS + SYSTEM + MESSAGES COMBINED.

MAXIMUM CHECKPOINTS
    per-model, commonly around 4.
    Excess markers are ignored -- again, silently.

TTL
    commonly 5 minutes by default, RESETTING ON EACH HIT.
    Extended TTL (1 hour) is available on a narrower
    model list.

PRICING SHAPE
    cache WRITES cost MORE than standard input tokens
    cache READS cost SUBSTANTIALLY LESS
    -> you pay a premium once and read cheaply thereafter
4.9.3 The TTL insight that changes agent economics
The TTL resets on every successful hit. The consequence:

A workload that issues a request at least once per TTL window keeps its cache warm indefinitely — paying the write premium once and reading cheaply for the remainder of the session.
This makes agent loops an unusually good fit for prompt caching. An agent making a model call every few seconds through a multi-step loop keeps its prefix cached continuously across the whole session. The system prompt and tool definitions — which Module 2 §2.8 identified as a large fixed cost re-sent on every iteration — are paid for once.
Conversely, a workload with sporadic requests spaced beyond the TTL pays the write premium repeatedly and may be worse off than with no caching at all, since writes cost more than standard input. Check the request-interval distribution before enabling it.

4.9.4 The debugging procedure
Module 2 §2.6.3 gave the two-state diagnostic. The full procedure:
  1. EMIT cacheReadInputTokens and cacheWriteInputTokens.
     Without them you are guessing.

2. writes == 0 AND reads == 0
       -> YOU NEVER CACHED
       check: is the prefix above the model's minimum?
              is the checkpoint placed after enough content?
              are you exceeding the maximum checkpoint count?

3. writes > 0 AND reads == 0
       -> YOU CACHE AND THEN INVALIDATE
       DIFF THE RENDERED BYTES of two consecutive requests.
       Usual culprits:
         - a timestamp or UUID early in the prompt
         - tool lists serialized in NONDETERMINISTIC ORDER
         - a model switch between requests
         - any dynamic value in the "stable" section

4. writes > 0 AND reads > 0, but savings are small
       -> check the request interval against the TTL
       -> check what fraction of input is actually in the
          cached prefix
Nondeterministic tool ordering deserves its own mention because it is invisible without a byte diff: every tool is present, the prompt looks identical, and the prefix hashes differently on every request.

4.9.5 Constraints
Batch inference excludes prompt caching. If you moved a workload to batch for the discount (Module 1, §1.7.1), caching is not available on that path.
Tool caching support varies by model family.
Extended TTL covers a narrower model list than the base feature.
Prompt caching is primarily for text.
4.10 Edge caching and pre-computation
4.10.1 Edge caching
Skill 4.1.4 names edge caching. It applies in a narrow but real case: public, non-personalized generated content.
  FITS:
    - generated product descriptions on a public catalogue
    - published FAQ answers
    - generated summaries on public pages
    - anything with a stable URL and no user-specific
      content

DOES NOT FIT:
    - anything personalized
    - anything permission-scoped
    - conversational interfaces
When it fits, it is the strongest cache available: it eliminates the model call, the backend round trip, and most of the network latency simultaneously. CloudFront in front of an API or a static store is the mechanism.
The safety rule is simple and absolute: never edge-cache anything that varies by user. A CDN has no notion of your tenant model, and a cache-control header that permits shared caching of a personalized response is a data leak with wide blast radius.

4.10.2 Pre-computation
Caching with the population step moved ahead of demand.
  Identify PREDICTABLE questions:
    - the top N FAQ items
    - a generated summary per document in a corpus
    - a description per product in a catalogue
    - a daily digest

Generate them OFFLINE, in BATCH (50% discount --
  Module 1, 1.7.1)

Serve from S3 or DynamoDB at single-digit milliseconds.
Two advantages over reactive caching:
The generation cost is at batch rates, not on-demand rates — an immediate halving on top of the avoided repeat calls.
There is no cold start. A reactive cache serves its first requester slowly; a pre-computed store is warm from the first request.
The requirement is predictability. If you can enumerate the questions in advance, pre-compute. If you cannot, cache reactively.
A hybrid worth knowing: pre-compute the head of the distribution — the top few hundred questions covering a large share of traffic — and cache the tail reactively. This is frequently the highest-value configuration, because query distributions in support and FAQ workloads are heavily skewed.
4.11 Measuring caching
4.11.1 The metrics
  PER CACHE LAYER:
    hit rate
    hit latency, miss latency
    entries, memory or storage used
    eviction rate

CORRECTNESS -- the ones that matter most:
    FALSE POSITIVE RATE  (semantic caches)
    freshness divergence rate (4.7.3)
    cross-tenant hit count   -> should be ZERO, alarmed

ECONOMICS:
    calls avoided
    cost saved
    infrastructure cost of the cache itself

PROMPT CACHING:
    cacheReadInputTokens, cacheWriteInputTokens
    read:write ratio
4.11.2 The metric that must never be reported alone
A rising cache hit rate with falling quality is an incident, not a win.
Hit rate improves latency and cost, and both look good on a dashboard. If the improvement came from a lowered threshold that is now producing false hits, the telemetry reports success while users receive wrong answers faster.
Always instrument hit rate alongside a quality signal. The pairing is what makes the metric interpretable, and its absence is why this failure mode persists.

4.11.3 Validating the false positive rate
  Sample cache hits -- weighted toward those near the
  threshold, where errors concentrate (4.5.3).

For each: generate the answer fresh and compare against
  what was served.

Divergent -> a false hit. Record it, and record the
               similarity score.

-> the false positive rate, by similarity band
  -> which directly informs the threshold decision
Sampling near the threshold rather than uniformly is the efficiency point. Errors cluster at the margin, so uniform sampling wastes most of the review budget on hits that were obviously correct.
4.12 Consolidated decision reference
  +-----------------------------------------------------------------+
  |  SIZE THE OPPORTUNITY BEFORE BUILDING -- ONE HOUR                |
  |    FLOOR:   log a week of prompts, normalize, hash, count        |
  |             exact duplicates -> typically 15-30%                 |
  |    CEILING: sample 1,000, embed, cluster, count within ~0.92     |
  |    Then estimate savings against actual spend.                   |
  |                                                                  |
  |    Realistic semantic hit rates: 20-45% (up to ~70% for          |
  |    agent/FAQ traffic). NOT the 90% that gets promised.           |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  WHICH CACHE?                                                    |
  |    identical repeated requests   -> EXACT (zero false            |
  |                                     positives; build FIRST)      |
  |    rephrased equivalent queries  -> SEMANTIC (+ verification)    |
  |    stable prompt PREFIX          -> PROMPT CACHING (saves        |
  |                                     PREFILL, not the call)       |
  |    public, non-personalized      -> EDGE / CDN                   |
  |    enumerable questions          -> PRE-COMPUTE in BATCH         |
  |                                                                  |
  |    They COMPOSE. Exact in front of semantic; prompt caching      |
  |    underneath both.                                              |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  THE ASYMMETRIC PRINCIPLE                                        |
  |    A cache returning FALSE HITS is STRICTLY WORSE than no        |
  |    cache -- users get confidently-presented wrong data.          |
  |    -> cache misses are tolerable; false hits are not             |
  |    -> when distributions overlap, err HIGH                       |
  |    -> 30% hit rate with no false positives beats 45% with 2%     |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  VERIFIED SEMANTIC CACHING                                       |
  |    embedding similarity as a PRE-FILTER, not the decision        |
  |    + a CONJUNCTION of independent equivalence checks:            |
  |      time period, entity, polarity, intent, filters              |
  |    ALL must pass. ~20ms overhead.                                |
  |    Reported: 100% precision at 87% recall, where an              |
  |    embedding-only cache admitted EVERY negative as a false hit.  |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  INVALIDATION -- LAYER ALL THREE                                 |
  |    1. SHORT TTL (1-24h) as the baseline safety net               |
  |    2. VERSIONED KEYS -- bump on model, prompt, embedding         |
  |       model, or corpus change                                    |
  |    3. EXPLICIT PURGE ENDPOINT -- you will need it                |
  |    + SAMPLED FRESHNESS CHECKS daily; divergence rate as a        |
  |      monitored metric                                            |
  |                                                                  |
  |    !! An EMBEDDING MODEL change invalidates the ENTIRE           |
  |       semantic cache -- vectors from different models are        |
  |       not comparable.                                            |
  |    !! VALIDATE BEFORE CACHING -- a cached hallucination is       |
  |       served repeatedly and instantly.                           |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  MULTI-TENANCY                                                   |
  |    TENANT ID in the key, from the AUTHENTICATED SESSION.         |
  |    Role too, if answers differ by role.                          |
  |    Test: if two callers could legitimately get different         |
  |    answers to the same query, what distinguishes them belongs    |
  |    in the key.                                                   |
  |    PARTITION the semantic index; do not rely on a filter alone.  |
  |    ALARM on any cross-tenant hit. It should be ZERO.             |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  PROMPT CACHING                                                  |
  |    Saves PREFILL, not the call. Complementary, not alternative.  |
  |    Below the per-model minimum, NOTHING CACHES, SILENTLY.        |
  |    TTL RESETS ON EACH HIT -> a busy agent loop stays warm        |
  |      indefinitely; sporadic traffic pays the write premium       |
  |      repeatedly and may be WORSE than no caching.                |
  |    Batch inference EXCLUDES prompt caching.                      |
  +-----------------------------------------------------------------+

+-----------------------------------------------------------------+
  |  THE METRIC RULE                                                 |
  |    A RISING HIT RATE WITH FALLING QUALITY IS AN INCIDENT.        |
  |    Never report hit rate without a paired quality signal.        |
  |    Sample hits NEAR THE THRESHOLD to measure false positives --  |
  |    errors cluster at the margin.                                 |
  +-----------------------------------------------------------------+
4.13 Checkpoint
1. Which of the four cache types does not avoid the model call? (4.1)

Prompt caching. It saves prefill compute on a reused prefix, lowering input cost and time-to-first-token, but the generation still happens. Exact, semantic, and edge caching avoid the call entirely.
2. How do you size the caching opportunity in an hour? (4.2.1)

The floor: log a week of prompts, normalize, hash, and count exact duplicates — typically 15–30%. The ceiling: sample 1,000 requests, embed, cluster, and count how many fall within about 0.92 similarity of another. Then estimate savings against actual spend.
3. What are realistic semantic cache hit rates? (4.2.2)

20–45% in production, up to roughly 70% for agent and FAQ-shaped traffic — not the 90%+ that is often claimed. Cache infrastructure typically costs under 5% of the savings, which is why even a modest real hit rate is worth capturing.
4. Why build exact caching first? (4.3.1)

Zero false positives by construction, microsecond lookups, no threshold to tune, trivial to reason about, and it catches 15–30% of traffic. It has no correctness risk, and it can sit in front of a semantic cache as a fast safe first layer.
5. Name three things that must never be normalized away from a cache key. (4.3.2)

Tenant and user scope; the model ID and version; the prompt version; retrieval configuration; and corpus version. Omitting the model ID serves responses from a different model after a migration — a silent correctness failure.
6. What is the cache tax, and when does it make semantic caching a bad trade? (4.4.2)

Every miss now costs an embedding call and a vector search — roughly 20 ms extra — before the model is invoked. At a high hit rate the hits dominate and net latency improves substantially. At a low hit rate you have added latency to most requests plus a false-positive risk, for little saving.
7. Give three reasons the typical 0.92–0.97 threshold range is only a starting point. (4.5.1)

It is embedding-model-dependent — one published system's optimum was 0.78. It varies by query type within the same application. And a single static threshold is a known limitation, with adaptive per-embedding approaches reporting substantially better hit rates and lower error.
8. Name four query differences that exceed 0.95 similarity but have different answers. (4.5.2)

Time period ("Q1 2024" vs "Q1 2025"), polarity ("growth" vs "decline"), analytical intent ("what is" vs "how has it changed"), and entity substitution ("cancel" vs "pause"). Raising the threshold does not fix these — the embedding does not encode the distinction.
9. State the asymmetric principle. (4.6.1)

A cache that returns false hits is strictly worse than no cache, because users receive confidently-presented wrong data. Cache misses are tolerable; false hits are not. When the distributions overlap, err toward the higher threshold.
10. Describe verified semantic caching and its reported effect. (4.6.2)

Treat embedding similarity as a pre-filter, then require a conjunction of independent equivalence checks — time period, entity, polarity, intent, filters — all of which must pass. One reported system achieved 100% precision at 87% recall, where an embedding-only cache at the same threshold would have admitted every negative query as a false hit. Overhead is around 20 ms.
11. Which staleness source invalidates the entire semantic cache? (4.7.1)

An embedding model change. Vectors from different models occupy different spaces and are not comparable, so the cache must be rebuilt entirely rather than degrading gracefully.
12. What three invalidation mechanisms do production systems combine, and what do sampled freshness checks add? (4.7.2, 4.7.3)

Short TTL as a baseline safety net, versioned keys bumped on model/prompt/embedding/corpus changes, and an explicit purge endpoint. Sampled freshness checks catch staleness that TTL and event-driven invalidation both miss — and the divergence rate is a monitored metric that warns before users notice.
13. What is cache poisoning in a GenAI context? (4.7.4)

The model hallucinates, the wrong answer is cached, and it is then served repeatedly and instantly. Caching amplifies a single error into a systematic one. Validate before caching, and do not cache low-confidence responses or ones that triggered a guardrail.
14. What is the test for whether something belongs in the cache key? (4.8.3)

If two callers could legitimately receive different responses to the identical query, whatever distinguishes them belongs in the key — tenant, role, entitlement, and any scope that changes what a user may see.
15. Why does the prompt cache TTL make agent loops a good fit? (4.9.3)

The TTL resets on every successful hit, so a loop issuing calls every few seconds keeps its prefix warm continuously — paying the write premium once and reading cheaply for the whole session. Conversely, sporadic traffic pays the write premium repeatedly and can be worse off than with no caching.
16. Why must hit rate never be reported alone? (4.11.2)

A rising hit rate with falling quality is an incident, not a win. It improves latency and cost on the dashboard while users receive wrong answers faster. Instrument hit rate alongside a quality signal.
17. How should you sample to measure false positives efficiently? (4.11.3)

Weight the sample toward hits near the threshold, where errors concentrate. Uniform sampling spends most of the review budget on hits that were obviously correct.
4.14 Practice cases
Case 4.10 — The threshold sweep
Scenario. A team has built a semantic cache for a customer support assistant. They are choosing a threshold and present measurements:

ThresholdHit rateMeasured wrong-answer rate on hits0.994%0.0%0.9711%0.1%0.9523%0.6%0.9235%2.4%0.8851%8.9%
Monthly spend before caching is $32,000. The assistant answers questions about accounts, billing, and product features.
The team proposes 0.92, reasoning that a 35% hit rate saves roughly $11,000 a month and a 2.4% error rate is "well within tolerance."
Produce: an assessment and a recommendation.
Discussion.
Interrogate "well within tolerance," because nobody set a tolerance. The phrase is doing the work of a decision that was never made, and the number was chosen after seeing the savings — which is the wrong order (Module 2, §2.10.4).
Compute what 2.4% actually means in this context.
  If the cache serves 35% of requests, and 2.4% of those
  are wrong:

0.35 x 0.024 = 0.84% of ALL requests receive a
    confidently-presented wrong answer, quickly.

At, say, 400,000 requests/month:
    ~3,400 wrong answers per month, delivered fast and
    with no indication anything is amiss.
And the domain matters enormously here. This assistant answers questions about accounts and billing. A wrong answer about a product feature is embarrassing. A wrong answer about someone's account balance, billing date, or charge is a customer-trust and potentially a regulatory problem.
Apply the asymmetric principle (§4.6.1): a cache returning false hits is strictly worse than no cache, because users receive confidently-presented wrong data. The design rule is that misses are tolerable and false hits are not, which points sharply away from 0.92.
The recommendation: 0.95, and probably 0.97 for account and billing queries specifically.
But the more important recommendation is that a single threshold is the wrong shape of answer for this application.
Segment by query type, because the risk profile is not uniform:
  PRODUCT FEATURE questions
    - stable answers, low personalization, low harm from error
    - 0.92-0.95 is defensible
    - this is where the hit rate actually lives

BILLING questions
    - time-sensitive ("what was my charge in March")
    - the TIME PERIOD failure mode (4.5.2) applies directly
    - 0.97+, or verification, or do not cache

ACCOUNT-SPECIFIC questions
    - answers depend on the individual account
    - the tenant/user must be in the key (4.8), and even
      then answers change as the account changes
    - SHORT TTL, or DO NOT CACHE
This segmentation likely preserves most of the savings, because product-feature questions are typically the repetitive, cacheable share of support traffic, while account-specific questions are the ones that are unique anyway.
Add verification rather than only tuning the threshold (§4.6.2). The table shows errors rising steeply between 0.95 and 0.92, which is the expected shape — errors concentrate at the margin. Verification checks applied to hits in that band would let them run at a lower threshold safely:
  similarity >= 0.97           -> serve
  0.92 <= similarity < 0.97    -> VERIFY, then serve or miss
  similarity < 0.92            -> miss

Verification dimensions for this domain:
    - do the time references match?
    - do the entities (product, plan, account) match?
    - is the intent the same (lookup vs. change vs. compare)?
    - is the polarity the same?
That structure captures much of the 0.92 hit rate at close to the 0.97 error rate, for around 20 ms of overhead.
Two things missing from the team's analysis:
How was the wrong-answer rate measured, and on what sample? If the sample was uniform rather than weighted toward the threshold boundary (§4.11.3), the true rate at 0.92 may be understated — most uniformly-sampled hits are far above the threshold and obviously correct.
What is the freshness story? The table measures correctness at the moment of caching. It says nothing about answers that were right when cached and became wrong. Billing and account content changes constantly, so TTL and sampled freshness checks (§4.7.3) matter as much as the threshold here.
What a strong answer includes: challenging "within tolerance" as a number chosen after the fact; converting 2.4% into wrong answers per month; weighting the decision by the account/billing domain; applying the asymmetric principle; segmenting the threshold by query type rather than picking one; proposing verification in the marginal band; and questioning both the sampling method and the absence of a freshness analysis.
The reasoning being tested: do you treat a cache error rate as a correctness decision rather than a tuning parameter, and do you recognize that one threshold is the wrong shape of answer for a heterogeneous query mix?
Case 4.11 — The cache that made things worse
Scenario. An engineering documentation assistant added semantic caching eight weeks ago. Reported results:

Cache hit rate: 44%
Average latency improved from 2.1 s to 1.3 s
Monthly cost down 38%
No increase in error rates in monitoring
Then a support escalation: a senior engineer followed cached guidance to configure a service using a parameter that was deprecated four months ago and removed in the current release. The documentation had been updated; the cached answer had not.
Investigation finds:

The cache has a 30-day TTL
No versioned keys
No invalidation on corpus updates
The documentation corpus is reindexed nightly
No freshness checking
61,000 entries currently cached
Produce: an incident analysis and a corrected design.
Discussion.
"No increase in error rates in monitoring" is the finding that should lead, because it explains why this ran for eight weeks. The monitoring could not see this failure class — the cached answers were well-formed, confident, and returned successfully. There is no error to count. This is the Domain 2 Module 8 observation that GenAI failures are frequently invisible to conventional telemetry, appearing here as a caching problem.
The mechanism: the corpus is reindexed nightly, so retrieval has been returning current documentation for four months. The cache, with a 30-day TTL and no invalidation on corpus change, has been serving answers generated before the documentation was updated. The system had the right information and was not using it.
The scope question, and it should be asked immediately: 61,000 cached entries with a 30-day TTL means entries up to a month old are live at any time. How many other cached answers reference content that has changed? This is not one bad answer; it is an unknown number, and the four-month-old deprecation suggests entries have been refreshed and re-cached from a corpus state that was itself stale, or that this entry was re-cached repeatedly without regeneration.
Immediate actions:
  1. PURGE THE CACHE ENTIRELY.
     It contains an unknown quantity of stale guidance.
     Cost is a temporary loss of the 38% saving; the
     alternative is continuing to serve it.

2. REDUCE THE TTL to hours while the fix is designed.

3. Determine the SCOPE -- which documentation changed in
     the last 30 days, and can you identify affected
     entries? If cached entries record their source
     document IDs, yes. If not, that is a finding.

4. Notify users who may have received guidance derived
     from changed documentation, if scope can be
     established.
The corrected design — layer all three invalidation mechanisms (§4.7.2), because each covers what the others miss:
  1. SHORT TTL
     1-24 hours, not 30 days. The TTL is a safety net for
     changes you did not detect, and 30 days is not a net.

2. VERSIONED KEYS -- the structural fix
     Include a CORPUS VERSION in every cache key, bumped
     on each nightly reindex.
     -> the nightly reindex now invalidates the cache
        automatically, atomically, with no purge operation
     -> this alone would have prevented the incident

3. EXPLICIT PURGE ENDPOINT
     For the case where a bad answer is discovered.

PLUS:

4. SAMPLED FRESHNESS CHECKS (4.7.3)
     Daily, regenerate a sample of cached entries and
     compare. Publish the DIVERGENCE RATE as a metric with
     an alarm.
     -> this is the mechanism that would have surfaced the
        problem in days rather than months

5. RECORD SOURCE DOCUMENT IDS on every cache entry
     -> makes scope determination possible next time
     -> enables targeted invalidation on document change
Item 2 is the fix. A corpus version in the key means the nightly reindex invalidates everything derived from the previous corpus state, without anyone remembering to do anything. It converts invalidation from an operational procedure into a property of the key structure.
Item 4 is the detection. Even with correct invalidation, something will eventually go stale. A daily sampled regeneration with a monitored divergence rate is cheap — a few hundred calls — and it is the only mechanism that catches what TTL and event-driven invalidation both miss.
The monitoring finding, which outlives this incident: the system reported a 44% hit rate and a 38% cost reduction with no quality signal attached. A rising hit rate with falling quality looks identical to success (§4.11.2). The metric that should have existed is the freshness divergence rate, and its absence is why the incident was found by a customer rather than by the team.
One design question worth raising: for a documentation assistant over a corpus that changes nightly, is semantic caching the right technique at all? Pre-computation (§4.10.2) may fit better — generate answers to the top questions after each nightly reindex, in batch at a discount, with the whole store rebuilt on a known cadence. That removes the invalidation problem by construction rather than managing it, and it fits the access pattern of a documentation corpus with a heavily skewed query distribution.
What a strong answer includes: identifying that the monitoring could not see this failure class as the reason it ran eight weeks; purging immediately and accepting the temporary cost; corpus version in the key as the structural fix; sampled freshness checks as the detection mechanism; recording source document IDs to enable scope determination; naming the missing quality signal as the finding that outlives the incident; and questioning whether reactive caching is the right technique for a nightly-changing corpus.
The reasoning being tested: do you recognize a staleness failure that no error metric reports, and does your fix make invalidation structural rather than procedural?
Case 4.12 — Designing the cache layers
Scenario. A SaaS company runs a customer-facing assistant across 180 customer organizations.
Traffic characteristics, measured over two weeks:

2.1 million requests per month
31% are exact duplicates of another request within the same tenant (after normalization)
A further 19% fall within 0.93 similarity of another request in the same tenant
24% of requests are about the company's own product documentation, which is identical for all tenants
The remainder are tenant-specific: questions about the tenant's own uploaded documents and account data
The system prompt plus tool definitions total roughly 5,200 tokens
Average session involves 3.4 requests within a few minutes
Product documentation is updated weekly; tenant documents change continuously
Monthly spend: $61,000
Produce: a caching architecture with expected impact per layer.
Discussion.
The measurements have already sized the opportunity (§4.2.1), which is the right starting position. Work each layer against them.
Layer 1 — exact caching. The largest single win, and the safest.
31% exact duplicates within tenant, zero false positives by construction. This should be built first and it requires no threshold decision.
  key = hash(normalized_query, TENANT_ID, model_version,
             prompt_version, corpus_version)

Expected: ~31% of calls eliminated
  Risk: none
Note the tenant in the key (§4.8). The measurement was of duplicates within the same tenant, which is the correct scoping — cross-tenant duplicates must not be served from a shared cache.
Layer 2 — pre-computation for the shared documentation.
24% of requests are about product documentation identical across all 180 tenants. That content is:

Shared — so it is not tenant-scoped, and the cache is one copy serving everyone
Updated on a known weekly cadence — so invalidation is a scheduled rebuild rather than a detection problem
Likely heavily skewed in its query distribution
This is the pre-computation case (§4.10.2), and it is better than reactive caching here:
  After each weekly documentation update:
    - identify the top N product questions from logs
    - generate answers in BATCH (50% discount, Module 1)
    - store in DynamoDB or S3, keyed by normalized question
    - serve at single-digit milliseconds

Reactive semantic caching on the remainder of the
  documentation traffic.
Two advantages: the generation is at batch rates, and the store is warm from the first request after each rebuild rather than cold. And the invalidation problem disappears — the whole store is rebuilt weekly on a known schedule.
Layer 3 — semantic caching, carefully scoped.
19% additional coverage at 0.93 similarity is real, and it carries the false-positive risk. Two decisions:
Scope it by content type. Product documentation questions are stable, non-personalized, and low-harm — semantic caching is appropriate. Tenant-specific questions about uploaded documents and account data are the risky ones: they change continuously and errors are consequential.
  product documentation questions  -> semantic cache, 0.93,
                                      with verification
  tenant document questions        -> semantic cache with a
                                      SHORT TTL and tenant
                                      partitioning
  account data questions           -> DO NOT CACHE
                                      (changes constantly;
                                       errors are costly)
Partition the vector index by tenant (§4.8.2), not merely filter. With 180 tenants, a filter that is omitted on one query path is a cross-tenant leak.
Add verification in the marginal band (§4.6.2) — the entity, time-period, and intent checks — for the range just above the threshold where errors concentrate.
Layer 4 — prompt caching.
5,200 tokens of system prompt plus tool definitions on every request, and an average session of 3.4 requests within a few minutes.
That session pattern is well matched to the TTL behaviour (§4.9.3): requests arriving within minutes of each other keep the prefix warm across the session, so the write premium is paid once per session and the remaining 2.4 requests read cheaply.
Verify the prefix clears the model's minimum threshold — at 5,200 tokens it comfortably does — and that the ordering places all stable content first.
Note this composes with the other layers rather than competing: exact and semantic caching eliminate calls; prompt caching makes the remaining calls cheaper.
The expected shape:
  exact caching          ~31% of calls eliminated
  pre-computation        much of the 24% documentation
                         traffic, at batch rates
  semantic caching       part of the additional 19%,
                         scoped to safe content
  prompt caching         reduces the cost of every
                         remaining call

The layers OVERLAP -- a request eliminated by exact
  caching is not also eliminated by semantic caching --
  so total savings are NOT additive. Model them as a
  cascade of filters, in order.
That last point is the modelling error to avoid. Adding "31% + 19% + 24%" produces a number above 70% that will not materialize, because the categories intersect. Model the layers in sequence: exact catches its share first, semantic catches a portion of what remains, and so on.
What to instrument from day one (§4.11):

Hit rate per layer, so you know which is earning its keep
False positive rate on the semantic layer, sampled near the threshold
Cross-tenant hit count — alarmed at zero
Freshness divergence rate
cacheReadInputTokens and cacheWriteInputTokens
Cost saved, net of cache infrastructure
What a strong answer includes: building exact caching first for its zero-risk 31%; recognizing the shared documentation as a pre-computation case with a known rebuild cadence; scoping semantic caching by content type and excluding account data; partitioning rather than filtering by tenant; matching prompt caching to the session pattern; and explicitly noting that the layer savings are not additive.
The reasoning being tested: can you assemble a multi-layer cache architecture matched to measured traffic characteristics, and do you avoid the arithmetic error of summing overlapping coverage?
4.15 Skill coverage audit
Skill statementWhere discharged4.1.4 — Create intelligent caching systems to reduce costs and improve response times by avoiding unnecessary FM invocations (semantic caching, result fingerprinting, edge caching, deterministic request hashing, prompt caching)The four cache types and the distinction that only three avoid the call: §4.1. Sizing the opportunity before building — the floor/ceiling measurement and realistic hit rates: §4.2. Exact caching (result fingerprinting, deterministic request hashing) — why it comes first, normalization design, and what must never be normalized away: §4.3. Semantic caching — the mechanism, the cache tax on misses, and infrastructure: §4.4. The threshold problem, including embedding-model dependence and the four failure modes that survive a high threshold: §4.5. The asymmetric principle and verified semantic caching with multi-dimensional equivalence verification: §4.6. Invalidation — the five staleness sources, layered TTL/versioned keys/purge endpoint, sampled freshness checks, and cache poisoning: §4.7. Multi-tenancy and cache safety: §4.8. Prompt caching operationally — mechanics, the TTL-reset insight for agent loops, the debugging procedure, and constraints: §4.9. Edge caching and pre-computation: §4.10. Measuring caching, including the hit-rate-with-quality rule and threshold-weighted sampling: §4.11. Applied in Cases 4.10, 4.11, and 4.12.4.16 Module summary
Before building:

Size the opportunity in an hour. Floor: hash a week of prompts and count exact duplicates — typically 15–30%. Ceiling: embed and cluster a sample within ~0.92. Then estimate against actual spend.
Realistic semantic hit rates are 20–45%, up to around 70% for agent and FAQ traffic — not the 90% that gets promised. Infrastructure typically costs under 5% of the savings.
The four types:

Exact — zero false positives, 15–30% of traffic, no threshold to tune. Build it first.
Semantic — catches rephrasings, and introduces the central design problem.
Prompt caching — saves prefill, not the call. Complementary, not alternative.
Edge/CDN — strongest where it applies, and only for public non-personalized content.
Pre-computation — caching with population moved ahead of demand, at batch rates, with no cold start.
Exact caching:

Normalization is the whole design. Never normalize away tenant scope, model ID and version, prompt version, retrieval configuration, or corpus version — each omission is a silent correctness failure.
Composing the key from a version tuple makes invalidation a version bump rather than a purge.
The threshold:

Typical production range is 0.92–0.97, but it is embedding-model-dependent, varies by query type, and a single static value is a known limitation.
Time period, polarity, analytical intent, and entity substitution routinely exceed 0.95 with different answers. Raising the threshold does not fix them — the embedding does not encode the distinction.
Tuning requires human-labelled ground truth on which query pairs are equivalent. Errors concentrate at the threshold boundary.
The asymmetric principle:

A cache returning false hits is strictly worse than no cache. Misses are tolerable; false hits are not.
Verified semantic caching treats similarity as a pre-filter and requires a conjunction of independent equivalence checks. One reported system reached 100% precision at 87% recall, where an embedding-only cache would have admitted every negative as a false hit — for around 20 ms of overhead.
Invalidation:

Five staleness sources, and an embedding model change invalidates the entire semantic cache because vectors from different models are not comparable.
Layer all three: short TTL (1–24h), versioned keys, and an explicit purge endpoint — plus sampled daily freshness checks with divergence rate as a monitored metric, because it is the only mechanism catching what the others miss.
Validate before caching. A cached hallucination is served repeatedly and instantly — caching amplifies one error into a systematic one.
Multi-tenancy:

Tenant and role in the key, from the authenticated session. Partition the semantic index rather than relying on a filter. Alarm on any cross-tenant hit; it should be zero.
The test: if two callers could legitimately receive different answers to the same query, what distinguishes them belongs in the key.
Prompt caching:

Below the per-model minimum, nothing caches, silently.
The TTL resets on each hit, so a busy agent loop stays warm indefinitely — while sporadic traffic pays the write premium repeatedly and can be worse off than not caching.
Batch inference excludes prompt caching.
Diagnose with the usage fields; diff the rendered bytes when writes occur without reads.
Measurement:

A rising hit rate with falling quality is an incident, not a win. Never report hit rate without a paired quality signal.
Sample hits near the threshold to measure false positives — errors cluster at the margin.
Layer savings are not additive; model them as a cascade of filters.
Carry into Module 5: Modules 2–4 reduced demand. Module 5 addresses supply — capacity, throughput, quotas, and scaling, and why a predictable burst is a scheduling problem rather than an autoscaling one.
End of Module 4. Module 5 — Throughput, Capacity, and Scaling — follows.

Unstarred

Dunk (exclude from study)

Module 4: Caching

Review Questions

Todos

Sensemaker