Observability

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

Layer Guide — X6: Observability
What this layer is responsible for
Knowing what the system is doing, why it did it, and whether it is doing it well.
Conventional observability answers whether a system is working: is it up, how fast, what's the error rate. Those questions still matter and they miss the failure that defines generative systems:
A generative system can be perfectly healthy and completely wrong. The endpoint returns 200. Latency is normal. No exceptions are thrown. And the model is confidently hallucinating, answering off-topic questions, citing a retracted policy, or producing subtly worse output than last month because a prompt changed. None of that appears in a conventional dashboard, and none of it generates an error.
This produces a three-layer model:
Infrastructure health — latency, errors, throughput, throttling, utilization. Necessary, insufficient, and the only layer most teams instrument.
Behavioral visibility — what was actually asked, what was actually returned, what was retrieved, which tools were called, which policies fired. This is where generative debugging happens.
Output quality — is it good, grounded, relevant, unbiased. Measured rather than observed, and covered in depth in X7.
Two facts govern the layer.
Almost every generative-specific observability surface is off by default. Model invocation logging, knowledge base logging, guardrail trace, agent trace search, CloudTrail data events, SageMaker data capture. Each is a deliberate switch, and a system that hasn't thrown them has no ability to answer the questions that matter. If a design proposes analyzing model behavior without having enabled logging, it is describing something impossible.
The characteristic failure is silence, not error. A sync that stopped running, a producer that stopped producing, a token that expired and now returns empty results with a 200. Error-rate alarms miss all of these. Alarm on staleness and absence, not only on failures — a theme that recurs in nearly every pattern class below.

Pattern class 1 — Health and performance telemetry
The problem in full
The conventional layer: is the system up, fast, and within capacity. Necessary as a foundation, and in generative systems it carries some metrics that are not conventional at all.
The variables:
Which components — managed services, your compute, self-hosted inference.
Granularity and cardinality, which drive cost.
What thresholds mean, given that latency is inherently seconds.
The metrics that matter, and why
Latency, disaggregated. Total latency conflates prefill, decode, retrieval, and orchestration. For streaming interfaces, time to first token and inter-token latency are the user-visible quantities and total latency hides regressions in either. Instrument them separately.
Token counts. Input and output tokens per invocation are simultaneously a usage metric, a cost metric, and a quality signal — rising input tokens usually means retrieval or history policy drift; rising output tokens often means prompt changes. An alarm on token consumption is the fastest possible detection of runaway cost, firing in minutes where billing data takes a day.
Throttling counts. Rising throttles drive the capacity ladder: backoff, buffering, cross-Region profiles, provisioned throughput.
Error taxonomy. Client errors, server errors, and throttles are different problems. Aggregating them into "error rate" destroys the signal.
Utilization for self-hosted inference — GPU utilization and memory, batch size, queue depth. Without these, the packing arithmetic from M1 is unverifiable.
Queue depth and age for asynchronous paths. Age of the oldest message is the better signal: depth alone doesn't distinguish a burst from a stall.
AWS mechanisms
CloudWatch metrics, published automatically by Bedrock (invocations, latency, input and output token counts, client and server errors, throttles), SageMaker endpoints (invocations, latency, instance utilization), Lambda, API Gateway, SQS, Step Functions, and AgentCore.
Custom metrics via the embedded metric format or PutMetricData for application-level quantities — time to first token, retrieval result counts, cache hit rates.
Metric math and composite alarms for derived conditions.
Anomaly detection on metrics, learning a band from history and alarming on deviation. For token consumption this is materially better than a static threshold, because normal varies by day and season and no fixed number is right.
CloudWatch Application Signals for service-level objectives and application performance, giving SLO tracking rather than raw metrics.
Container Insights and Lambda Insights for those runtimes.
The details that decide success
Disaggregate latency. A single latency number tells you nothing actionable in a multi-stage pipeline.
Watch cardinality. Per-user or per-tenant dimensions on custom metrics get expensive quickly, because custom metrics are priced per unique metric stream. Use logs plus queries for high-cardinality analysis and reserve metrics for low-cardinality aggregates.
Detailed monitoring costs extra. Standard granularity is five minutes; one-minute resolution is a paid option and often worth it for anything you'll alarm on.
Thresholds must reflect generative reality. A five-second p99 is normal here and would be an incident elsewhere. Alarms tuned on conventional intuitions produce noise.
Cost model
Standard metrics from AWS services are included. Custom metrics are priced per metric per month, which is where cardinality bites. Detailed monitoring, dashboards, and alarms each carry small charges that accumulate at scale.
Failure modes and diagnostics
Aggregated latency hiding regressions.
Cardinality explosion. A per-user dimension producing a surprising bill.
Alarms tuned to conventional latency. Constant noise, then ignored.
No token metrics. Cost problems discovered at month end.

Pattern class 2 — Behavioral capture
The problem in full
What was actually asked, and what was actually said? Without this, every question about a generative system's behavior is unanswerable: why did it say that, what are users asking about, did the prompt change help, which requests triggered a policy.
This is the single highest-value switch in the layer, and it is off by default.
The variables:
Volume, since prompts and completions are large.
Sensitivity — logs contain whatever users typed and whatever was retrieved.
Retention obligations, which may be years.
Access patterns — operational search versus analytical scanning.
Decision variables
Where does it go? Model invocation logging can deliver to S3, CloudWatch Logs, or both, and the choice is consequential.
CloudWatch Logs for operational search: recent data, Logs Insights queries, metric filters, and alarms. Ingestion and storage are substantially more expensive per GB than S3.
S3 for retention and analysis: far cheaper, lifecycle tiering, and Athena for aggregate analysis. For any multi-year retention obligation, S3 is the answer — CloudWatch Logs at that duration and volume is expensive for no benefit.
Many systems use both: CloudWatch for a short operational window, S3 for everything.
What about the surrounding context? Invocation logging captures the request and response as sent to the model — which includes the assembled prompt with retrieved context and system instructions. That is richer than an application-level conversation log and is what you need for debugging. It does not capture what retrieval returned before assembly, or which chunks were considered and dropped — those need application logging.
Sensitivity. Prompts contain what users typed; assembled prompts contain retrieved documents. Log data is at least as sensitive as the corpus, and it inherits none of the corpus's access controls automatically. Encryption, access restriction, and retention limits apply.
AWS mechanisms
Bedrock model invocation logging, configured at the account level per Region. Delivers full request and response data including token counts, with large payloads written to S3. Can include guardrail trace information. Off by default.
CloudWatch Logs with log groups, retention settings (which default to never expire — a silent cost problem), metric filters turning log patterns into alarmable metrics, and subscription filters streaming to Lambda, Firehose, or OpenSearch in real time.
S3 with lifecycle policies, Object Lock in compliance mode where immutability is required, and KMS encryption.
Athena over S3 logs, with partitioning and columnar conversion for cost.
Kinesis Data Firehose to transform and land logs in Parquet, which makes Athena queries an order of magnitude cheaper.
Application-level logging for what the model API doesn't see: retrieved chunk identifiers and scores, the transformed query, routing decisions, cache hits, and the correlation identifier tying it together.
The details that decide success
Enable it before you need it. The first serious quality incident is not the time to discover you have no record. This is the most common regret in the layer.
Set log group retention explicitly. Never-expire is the default and it accumulates.
Log retrieval results. Invocation logging shows the assembled prompt; it does not show what retrieval returned and ranked. Distinguishing "retrieval failed" from "generation ignored good context" requires application logging of chunk identifiers and scores, and without it that diagnosis is guesswork.
Log the transformed query. In systems with query rewriting, the searched query differs from the user's, and debugging retrieval without it is debugging a black box.
Thread a correlation identifier through everything. Invocation log, application log, trace, feedback event, and conversation record should join on one key. Without it you have five disconnected datasets.
Treat logs as sensitive data. And note that input-side PII redaction protects the logs too, which is an underappreciated benefit of screening at input.
Cost model
CloudWatch Logs ingestion and storage per GB, meaningfully more than S3. Full prompt and completion logging is high volume — a system doing a million requests a day with 5,000-token prompts is logging a substantial amount daily.
S3 storage with lifecycle tiering is cheap. Athena is per terabyte scanned, reduced dramatically by Parquet and partitioning.
The practical shape: S3 for everything, CloudWatch Logs for a short operational window if at all, Firehose to convert to Parquet, lifecycle rules to tier and expire.
Failure modes and diagnostics
Logging not enabled. No ability to answer any behavioral question.
Retention never set. CloudWatch Logs cost growing indefinitely.
No retrieval logging. Cannot localize failures between retrieval and generation.
No correlation identifier. Datasets that cannot be joined.
PII in logs with corpus-level sensitivity and no corresponding controls.
Analytics cost surprise. Unpartitioned JSON scanned repeatedly.

Pattern class 3 — Build-time pipeline observability
The problem in full
The build-time layers — ingestion, preparation, indexing — fail silently and their failures surface much later as retrieval quality problems. A document that never got indexed presents as "the assistant doesn't know about that," and the investigation starts at the wrong end.
This pattern class exists because pipeline observability is systematically neglected relative to runtime observability.
The signals that matter
Ingestion completeness. Source object count versus indexed document count. Divergence means silent loss.
Ingestion freshness. Time since last successful sync, per source. This is the metric to alarm on, because a stopped pipeline produces no errors.
Per-document processing outcomes. Which documents failed, and why. Aggregated failure counts are not enough; you need the reasons to fix them.
Extraction confidence distribution. A drop in mean OCR confidence for a batch signals a source change or a quality problem before it becomes a retrieval problem.
Quarantine rate. As a rate, not a count — a tripling means something upstream changed.
Duplicate rate. Rising duplication signals a source being re-ingested without change detection.
Chunk count trends. A sudden change in chunks per document means chunking behavior changed, possibly unintentionally.
Embedding call volume. Should track change volume; if it tracks sync frequency instead, change detection is too coarse and you're re-embedding unchanged content.
AWS mechanisms
Bedrock Knowledge Bases logging — a separate configuration from model invocation logging, delivering to CloudWatch Logs, S3, or Firehose. It reports per-document ingestion outcomes, which is how you find documents that failed to process. Off by default, and the specific answer to "how do I troubleshoot knowledge base ingestion."
Query it with CloudWatch Logs Insights for failed documents.
Step Functions execution history where the pipeline is a workflow — per-execution, per-state visibility including which item failed and why.
Distributed Map results and failure manifests in S3, enumerating per-item outcomes for large jobs.
DataSync task reports and S3 Batch Operations completion reports for transfer verification.
S3 Inventory for reconciliation — a daily manifest of what's actually in the bucket, diffable against what's indexed.
Custom metrics from pipeline stages: documents processed, failures by reason, confidence distributions, chunk counts.
CloudWatch alarms with treatMissingData configured appropriately — this setting is what turns "no data points" into an alarm rather than into silence, and it is the mechanism for detecting a pipeline that stopped running entirely.
The details that decide success
Alarm on staleness. Time since last successful ingestion, per source, with a threshold. The most important alarm in the build-time layers, and the one that catches the failure mode error-based alerting cannot see.
Build a reconciliation job. Periodically compare source inventory against indexed documents and alarm on divergence. This is the only mechanism that catches lost events, failed updates, and logic bugs, and it's cheap insurance.
Alarm on rates, not counts. Individual failures are normal; a change in rate is a signal.
Log failure reasons, not just failures. "Extraction confidence 0.42 below threshold 0.80" is actionable; "failed" is not.
Instrument confidence distributions. They're leading indicators of retrieval quality problems and nobody collects them.
Cost model
Logging volume for per-document outcomes is proportional to corpus size and sync frequency — modest for most corpora, worth partitioning and tiering for very large ones. Reconciliation jobs cost inventory generation and comparison compute, both small.
Failure modes and diagnostics
Knowledge base logging not enabled. Ingestion failures invisible.
No staleness alarm. A stopped pipeline discovered by a user.
No reconciliation. Silent index drift.
Failure reasons discarded. Quarantine that nobody can triage.

Pattern class 4 — Distributed tracing
The problem in full
A generative request touches many components: gateway, function, retrieval, model, database, downstream services. When it's slow or fails, the question is where — and without tracing, that's a guess.
Decision variables
Is the pipeline multi-component? For a single Lambda calling Bedrock, logs suffice. For gateway → function → knowledge base → model → database, tracing earns its keep.
What sampling rate? Tracing everything is expensive; sampling means the rare failure you're investigating may not be traced. Higher sampling on error paths is the usual compromise.
What needs to be searchable? Trace annotations are indexed and searchable; metadata is not. Putting the correlation identifier, tenant, and model identifier in annotations is what makes traces findable.
AWS mechanisms
AWS X-Ray with automatic instrumentation for Lambda, API Gateway, and SDK calls, plus manual subsegments for application logic. Produces a service map and per-request traces with timing breakdowns.
Annotations (indexed, searchable, limited in number) versus metadata (arbitrary, not searchable).
Sampling rules configurable by service, path, and rate.
ADOT — the AWS Distro for OpenTelemetry — for standards-based instrumentation, which matters when the estate spans AWS and non-AWS components or when the organization has standardized on OpenTelemetry.
CloudWatch Transaction Search, which ingests spans and makes them searchable at the span level rather than only as sampled traces. This must be enabled once, and it is the prerequisite for agent trace visibility discussed in the next pattern class.
Application Signals building SLOs on top of traced services.
The details that decide success
Instrument the retrieval step explicitly. A trace showing "8 seconds in Lambda" is useless; one showing "1.2s retrieval, 6.1s model, 0.4s database" is actionable.
Annotate with the correlation identifier, so a trace can be found from a log entry and vice versa.
Sample errors more heavily. The traces you want are the ones that failed.
Untraced components appear as gaps, which is misleading — an uninstrumented downstream call looks like unexplained latency in its caller.
Cost model
Priced per trace recorded and per trace scanned or retrieved. Sampling is the primary lever. Transaction Search has its own ingestion pricing based on span volume.
Failure modes and diagnostics
Sampling missing the failure.
Gaps from uninstrumented components.
Traces that can't be correlated to logs.
Annotations exhausted, so filtering isn't possible.

Pattern class 5 — Agent trajectory observability
The problem in full
Debugging an agent is qualitatively different from debugging an API. A failing API produces a stack trace. A failing agent made a sequence of choices — it selected the wrong tool, passed a malformed argument, misread a result, looped, or concluded prematurely. The failure is a trajectory, and you cannot reconstruct it from an error message.
What must be captured
Every reasoning step — what the model produced at each iteration, including the tool-use requests.
Every tool invocation — which tool, with what arguments, returning what, taking how long, succeeding or failing.
The decision context — what was in the context window at each step, or at least its size and composition.
Termination — why the loop ended: natural completion, iteration bound, token bound, or error.
Token consumption per step, since agent cost is variable and per-step accounting is how you understand it.
AWS mechanisms
AgentCore Observability, delivering agent traces to CloudWatch — reasoning steps, tool invocations, model interactions, and session context.
CloudWatch Transaction Search must be enabled once for full trace visibility. This is a genuine gotcha: observability appears configured, traces don't appear, and the cause is a one-time account-level setting rather than anything in the agent. "We enabled observability but see no traces" resolves here.
ADOT instrumentation for custom spans beyond the built-in coverage.
Structured application logging where you built the loop yourself — the trajectory must be logged deliberately, because nothing captures it automatically.
Step Functions execution history as the equivalent for deterministic orchestration, which provides trajectory visibility for free — one of the honest advantages of workflows over agents.
Model invocation logging capturing each iteration's request and response, which reconstructs the trajectory from the model's side.
The details that decide success
Log tool arguments and results, not just tool names. "Called search_customers" tells you nothing; "called search_customers with {region: 'EMEA'} returning 0 results" tells you why the agent then did something odd.
Capture the termination reason. An agent that hit its iteration bound and returned its best guess looks identical in the output to one that finished properly.
Track per-step token consumption. Agent cost variance is the hardest thing to budget, and per-step data is how you understand the distribution.
Correlate the trajectory to the user request. One user request may span dozens of model calls; they must be groupable.
This is where workflows have an advantage. Step Functions execution history is a queryable, complete record by construction. An agent's trajectory exists only if you captured it. That asymmetry belongs in the workflow-versus-agent decision.
Cost model
Trace ingestion and storage proportional to trajectory length and request volume. Agents generate substantially more observability data per user request than single-shot invocations — another respect in which they're more expensive.
Failure modes and diagnostics
Transaction Search not enabled. Traces configured and absent.
Tool arguments not captured. Trajectories that show what happened and not why.
No termination reason. Truncated tasks indistinguishable from completed ones.
Uncorrelated steps. A trajectory scattered across unrelated log entries.

Pattern class 6 — Quality signal instrumentation
The problem in full
Between infrastructure health and formal evaluation sits a set of runtime signals that indicate quality without measuring it directly. They are cheap, continuous, and mostly ignored.
The signals
Guardrail interventions. How often, on which policy, on input or output.
Grounding scores. Where contextual grounding is enabled, the distribution of scores is a continuous hallucination-risk indicator, not just a block/allow decision.
Retrieval result counts and scores. Queries returning zero results, or top scores below a threshold, indicate corpus gaps. Zero-result queries are among the most valuable signals available — they are literally a list of things users want that the system cannot answer.
Stop reasons. A rising rate of max_tokens means truncation is happening and answers are incomplete.
Refusal patterns. The model declining to answer, detectable by metric filters over responses.
Cache hit rates. Prompt cache and semantic cache effectiveness.
Token distributions. Rising input tokens means context drift; rising output tokens means responses are getting longer.
Fallback and error-path rates. How often the system takes a degraded path.
User-side signals. Regenerations, edits, abandonment, thumbs-down — behavioral evidence of quality without asking.
AWS mechanisms
The InvocationsIntervened CloudWatch metric for guardrails, with dimensions that answer different questions:
GuardrailPolicyType — which category fired: content, topic, word, sensitive information, or contextual grounding policy. This is the "which rule" dimension.
GuardrailContentSource — input or output. This is the "where" dimension.
Guardrail trace: enabled returning per-policy assessments in the response, showing which specific rule matched and with what confidence — the difference between knowing a request was blocked and knowing why.
CloudWatch Logs metric filters turning log patterns into metrics: count refusals, count zero-result retrievals, count specific error signatures. This is how you alarm on behavior rather than on infrastructure.
Custom metrics emitted by the application for retrieval counts, scores, cache hits, and routing decisions.
Athena over invocation logs for distributional analysis that metrics can't express.
CloudWatch dashboards or Quick Sight for presenting these, depending on whether the audience is engineers or the business.
The details that decide success
Alarm on intervention rate, not count. A rising rate means either an attack or an over-tuned filter; either is actionable. Individual interventions are normal.
Mine zero-result queries. They are a prioritized list of corpus gaps, and acting on them is one of the highest-return activities available.
Watch token distributions over time. Slow drift in input tokens is how cost grows without anyone deciding it should.
Instrument the degraded paths. How often does the system fall back, retry, or return partial results? These are usually invisible.
Capture user behavioral signals. Regeneration and abandonment are honest quality indicators and require no user effort, unlike explicit feedback.
Cost model
Metric filters and custom metrics are cheap; watch cardinality. Athena analysis is per terabyte scanned.
Failure modes and diagnostics
Only infrastructure instrumented. Quality degradation invisible.
Zero-result queries discarded.
Guardrail dimensions confused. Asking "which rule" and looking at content source.
Token drift unnoticed until the invoice.

Pattern class 7 — Synthetic probing
The problem in full
Passive monitoring only observes traffic that exists. It cannot tell you the system is broken at 3am when nobody is using it, and it cannot detect a quality regression that produces no error.
Synthetic probes generate traffic and assert on the result.
Decision variables
What do you assert? For conventional monitoring, availability and latency. For generative systems the valuable assertion is on content: send a known question and check the response contains an expected fact. That detects a broken retrieval configuration, a failed knowledge base sync, an embedding mismatch, or a prompt regression — none of which produce an error code.
How often, and from where? Frequency trades detection speed against cost, and each probe costs a real inference. Multiple Regions detect regional issues.
How brittle are the assertions? Model outputs vary. Asserting on exact text produces false alarms; asserting on the presence of a key fact or entity is robust.
AWS mechanisms
CloudWatch Synthetics canaries — scripted probes in Node.js or Python, run on a schedule, producing CloudWatch metrics, screenshots, and HAR files. Blueprints cover common patterns; custom scripts cover content assertions.
CloudWatch alarms on canary success rate and duration.
Step Functions or EventBridge Scheduler for more elaborate probing workflows.
A small evaluation set run as a probe — a handful of representative queries with expected content, executed periodically. This is a lightweight continuous version of the evaluation set from X7.
The details that decide success
Assert on content, not just status. Status-only canaries duplicate what health checks already tell you.
Keep assertions robust. "Response mentions the current refund window" rather than an exact string.
Probe through the real path. Testing the model directly misses the whole system — retrieval, prompt assembly, guardrails. The canary should hit the production endpoint.
Budget for the inference cost. A canary running every five minutes across three Regions is thousands of inferences a month.
Maintain the probes. They break when the application changes, and stale canaries produce noise that gets ignored.
Cost model
Per canary run, plus the inference each run triggers, plus CloudWatch metrics and storage. The inference cost usually dominates and should be sized deliberately.
Failure modes and diagnostics
Status-only assertions. Missing the quality regression they were meant to catch.
Brittle assertions. False alarms leading to ignored alerts.
Probing a bypass path. Not exercising the real production route.
Unmaintained canaries.

Pattern class 8 — Audit and governance trail
The problem in full
Who did what, to what, when. Distinct from behavioral capture: that records what the model did; this records what people and services did to the system.
The distinction that gets tested
Four mechanisms answer four different questions and are routinely confused:
   Question Mechanism     Who called the API, when, from where CloudTrail   What was in the prompt and the response Model invocation logging   What happened during document ingestion Knowledge base logging   Which policy blocked this request, and why Guardrail trace   CloudTrail tells you a principal invoked a model. Invocation logging tells you what they asked. These are different mechanisms, separately enabled, answering separate questions.
AWS mechanisms
CloudTrail management events — control-plane operations, logged by default and retained in Event History for 90 days. This covers governance actions: who created or modified a guardrail, who granted model access, who changed an IAM policy, who approved a model version.
CloudTrail data events — high-volume operations including Bedrock model invocation, S3 object access, and Lambda invocations. Opt-in and priced per event. Enabling Bedrock data events tells you which principal invoked which model when — without the content.
Trails to S3 for retention beyond 90 days, optionally organization-wide and multi-Region, with log file validation proving logs weren't tampered with.
CloudTrail Lake as a managed store queryable with SQL, avoiding the Athena setup.
AWS Config for resource configuration history and compliance rules — was invocation logging enabled on this date, does this guardrail meet policy.
IAM Access Analyzer for external access and unused access findings, which is drift detection rather than event auditing.
S3 Object Lock in compliance mode where audit records must be immutable.
The details that decide success
Enable data events deliberately. They cost per event and they're the only way to audit invocation at the principal level. For a regulated deployment this is usually required; for a low-risk internal tool it may not be worth the cost.
Event History is 90 days and management events only. Anything longer needs a trail.
CloudTrail has delivery latency measured in minutes, so it is not a real-time alerting mechanism on its own. Pairing it with EventBridge is how you react quickly to specific events.
Governance actions matter as much as data access. "Who disabled the guardrail" is an audit question CloudTrail answers and invocation logging does not.
Log file validation matters where the audit record itself might be challenged.
Cost model
Management events are free in Event History; trails cost S3 storage. Data events are priced per event and can be significant at high invocation volume — scope them to what actually needs auditing. CloudTrail Lake is priced on ingestion and scanning.
Failure modes and diagnostics
Data events not enabled. No principal-level invocation audit.
Relying on Event History beyond 90 days.
Expecting CloudTrail to show prompt content.
Data event cost surprise at high volume.

Pattern class 9 — Correlation, retention and analysis
The problem in full
The signals exist in separate systems: metrics in CloudWatch, invocation logs in S3, traces in X-Ray, feedback in DynamoDB, conversation records elsewhere. Answering "why did this specific answer go wrong" requires joining them, and joining requires a shared key.
Decision variables
What is the correlation key? A request identifier generated at the edge and threaded through everything: application logs, invocation metadata, trace annotations, feedback records, conversation entries, and job records. This is a design decision that must be made early, because retrofitting it means the historical data can never be joined.
Where does analysis happen? Logs Insights for recent operational search. Athena over S3 for historical and aggregate analysis. OpenSearch where rich search and dashboards over log content are needed.
What is retained, where, and for how long? Different signals plausibly have different retention: metrics for months, operational logs for weeks, invocation logs for years, audit records for longer.
AWS mechanisms
Request identifiers propagated in headers, log fields, trace annotations, and record attributes. Bedrock also returns request metadata that can be correlated.
CloudWatch Logs Insights with its query language for recent analysis, plus metric filters and subscription filters.
Athena over S3 with partitioning by date and Parquet conversion — the two changes that reduce scan cost by an order of magnitude.
Kinesis Data Firehose performing the Parquet conversion on the way in.
OpenSearch fed by CloudWatch subscription filters, where full-text search and dashboards over log content are needed beyond what Logs Insights offers comfortably.
Quick Sight for business-facing dashboards over analysis results; Managed Grafana where the estate is heterogeneous and includes non-AWS sources; CloudWatch dashboards where everything is already in CloudWatch.
S3 lifecycle policies tiering logs to Infrequent Access and Glacier and eventually expiring them, with Object Lock where retention must be immutable.
The details that decide success
Thread the correlation identifier from the first request. It is nearly free at design time and impossible to add retroactively.
Partition and convert. Unpartitioned JSON in S3 scanned by Athena is the most common analytics cost surprise, and Parquet plus date partitioning fixes it.
Set retention explicitly on every store. CloudWatch Logs defaults to never expire.
Match the tool to the audience. Engineers want Logs Insights and Grafana; the business wants Quick Sight; auditors want CloudTrail Lake or Athena over an immutable archive.
Separate operational from analytical stores. Recent data for debugging, historical data for trends, with different costs and different access patterns.
Cost model
CloudWatch Logs ingestion and storage; S3 storage with tiering; Athena per terabyte scanned; OpenSearch resting capacity; dashboard and query costs. The dominant lever is data format and partitioning for analytical stores, and retention settings for operational ones.
Failure modes and diagnostics
No correlation identifier. Disconnected datasets.
Never-expiring log retention.
Unpartitioned analytics.
Duplicate storage of the same data in several systems with no clear ownership.

Pattern class 10 — Alerting and escalation
The problem in full
Signals are only useful if someone acts on them. Alerting decides what wakes a person, and it is where the generative-specific failure mode — silence — must be explicitly handled.
Decision variables
What is worth waking someone for? Availability and severe error rates, certainly. Quality degradation usually is not an overnight page but should be a ticket. Cost anomalies sit in between and can be urgent — a runaway loop can burn a lot of money before morning.
Static threshold or anomaly detection? For metrics with a stable normal, static works. For token consumption and request volume, anomaly detection is materially better, because normal varies by day and season and no fixed number is right.
How do you alarm on absence? This is the crucial one. A pipeline that stops running produces no metrics, and an alarm configured to treat missing data as "not breaching" will never fire. The treatMissingData setting is what turns absence into a signal, and it is the mechanism for detecting that something stopped entirely.
Where do alerts go? Paging for urgent, chat for routine, ticket for quality trends. Alerts to a channel nobody watches are worse than none, because they create false confidence.
AWS mechanisms
CloudWatch alarms — static threshold, anomaly detection, and composite alarms combining conditions to reduce noise. The treatMissingData setting (breaching, notBreaching, ignore, missing) is the control for absence detection.
Metric filters creating alarmable metrics from log patterns — how you alarm on behavior.
SNS as the fan-out target, reaching email, SMS, and other subscribers.
AWS Chatbot delivering into Slack or Teams with the ability to run limited diagnostic commands from the channel. The chain to remember: CloudWatch alarm → SNS → Chatbot → Slack.
EventBridge for routing alarm state changes and service events to automated remediation.
Cost Anomaly Detection with its own monitors and alert subscriptions, learning spending patterns and alerting on deviation with root-cause analysis.
AWS Budgets for planned limits and forecast alerts.
Incident Manager for structured on-call escalation where the operation warrants it.
The alarms worth having
A concrete starting set for a generative system:
   Alarm Why     Error rate above baseline Conventional health   Throttle count rising Capacity pressure   p99 latency above threshold User experience   Token consumption anomaly Fastest cost-runaway detection   Time since last successful ingestion Pipeline stopped — with missing data treated as breaching   Guardrail intervention rate change Attack or over-tuned filter   Queue age above threshold Consumers falling behind   Canary content assertion failure Silent quality regression   DLQ depth above zero Failures accumulating unseen   Reconciliation divergence Index drift   The bolded ones are the generative-specific additions that conventional alerting sets omit.
The details that decide success
Alarm on absence. Configure treatMissingData deliberately for anything where a stopped process is the failure mode.
Alarm on rates, not counts, for anything that has a normal nonzero baseline.
Use composite alarms to reduce noise. A single underlying problem producing eight pages trains people to ignore pages.
Route by urgency. Not everything is a page.
Alarm on the DLQ. Failed messages accumulating unnoticed for months is a common and avoidable failure.
Test the alarms. An alarm that has never fired may be misconfigured, and you'll find out at the worst time.
Cost model
Alarms are priced per alarm per month, with anomaly detection alarms costing more. SNS and Chatbot are negligible. The real cost of bad alerting is human attention.
Failure modes and diagnostics
No absence detection. Stopped pipelines discovered by users.
Alarm fatigue. Too many, too noisy, ignored.
Unwatched channels.
Static thresholds on variable metrics. Constant false positives or permanent silence.
Untested alarms.

The switches to throw
Consolidated, because this is the layer's most actionable content. Every one of these is off by default:
   Switch Enables     Bedrock model invocation logging What was asked and answered   Knowledge base logging Per-document ingestion outcomes   Guardrail trace: enabled Which policy fired and why   CloudWatch Transaction Search Agent trajectory visibility   CloudTrail data events for Bedrock Principal-level invocation audit   SageMaker data capture Endpoint request/response capture for monitoring   X-Ray tracing Cross-component latency attribution   Detailed CloudWatch monitoring One-minute metric resolution   A system with none of these enabled is a system that cannot answer any question about its own behavior.

Cross-cutting decisions in this layer
Which switches are thrown, and were they thrown before launch?
What is the correlation identifier, and does it reach every store?
Where do invocation logs live, and for how long? S3 for retention, CloudWatch for operational search, Firehose to Parquet for analysis cost.
What alarms on absence? Ingestion staleness, canary failure, missing metrics.
Can you distinguish retrieval failure from generation failure? Only if retrieval results are logged.
Who watches what, and has it been tested?
Interfaces to adjacent layers
From every layer. Each has an observability surface: B1 transfer completeness, B2 ingestion outcomes and confidence distributions, M1 utilization and throttling, M2 tokens and stop reasons, C1 retrieval scores and zero-result rates, S1 interventions and traces, O1 execution history and trajectories, E1 identity in logs, E2 client-side latency.
To X7 (Evaluation). Behavioral capture is the raw material for evaluation datasets and quality measurement. Invocation logs plus correlated feedback are what make offline evaluation possible.
To X8 (Cost). Token metrics are the fastest cost signal available, and application inference profiles provide the attribution dimension.
To X9 (Change management). Metrics and alarms are what deployment rollback triggers on. An AppConfig deployment with automatic rollback requires the alarms to exist — a hidden prerequisite that belongs in the observability plan rather than the deployment plan.

Unstarred

Dunk (exclude from study)

Observability

Review Questions

Todos

Sensemaker