Cost & Capacity

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

Layer Guide — X8: Cost & Capacity
What this layer is responsible for
Knowing what the system costs, why, and which decisions change it — and ensuring capacity exists when needed without paying for it when not.
Generative systems have a cost profile unlike anything else you operate, and three properties explain most of the difficulty.
Cost is usage-driven rather than capacity-driven. A conventional system's cost is a function of provisioned resources you can see and count. A generative system's cost is a function of tokens consumed, which varies with user behavior, prompt length, retrieved context size, conversation depth, and how many times an agent loops. You cannot look at the infrastructure and infer the bill.
Most of the cost is decided at design time and paid forever. Chunk size times top-k sets input tokens on every request for the life of the system. History policy sets how conversation cost grows. Model choice sets the per-token rate. These are architectural decisions with permanent recurring consequences, and they are usually made without anyone computing the arithmetic.
Spend can run away in hours. A retry storm, a runaway agent loop, an abusive user, or a batch job accidentally pointed at real-time inference can burn a large amount before anyone notices — and billing data lags by roughly a day. The detection mechanism has to be faster than the billing system.
A fourth property is organizational: without deliberate instrumentation, generative spend is one undifferentiated line item. There is no per-team, per-feature, or per-tenant breakdown unless you created the dimension in advance, and the dimension is not retroactive.

Pattern class 1 — Cost attribution
The problem in full
"Why is the Bedrock bill $40,000, and who is responsible for it?" is unanswerable by default. Invocations from every team, feature, and tenant aggregate into a single figure with no dimension to break down by.
Attribution is a design decision made before the spend happens, not an analysis performed afterward.
The variables:
Granularity needed — per team, per feature, per tenant, per customer.
Whether chargeback is real or attribution is only informational.
How many distinct consumers exist.
Whether historical attribution matters, which it can't retroactively.
Subtypes
Account separation. Each team or environment in its own account. The cleanest boundary and the coarsest.
Tag-based attribution. Resources tagged and cost allocated by tag.
Inference profile attribution. A taggable wrapper around a model, so invocations carry the tag.
Application-level metering. The application records token usage per request with its own dimensions.
Hybrid. Infrastructure by tags, inference by profiles, fine-grained by application metering.
Decision variables
What dimension do you actually need? Per-team is usually enough for budgeting. Per-tenant is required for chargeback in a multi-tenant product. Per-feature is what tells you whether a capability earns its cost.
Can platform mechanisms provide it, or do you need application metering? Inference profiles give you per-profile spend in cost tooling, which handles team and feature granularity well. Per-request, per-user, or per-conversation attribution needs the application to record it — the platform has no notion of your conversations.
Is this informational or financial? Informational attribution can be approximate. Chargeback must be defensible, which raises the bar on completeness and accuracy considerably.
AWS mechanisms
Application inference profiles. A Bedrock resource wrapping a model, which can be tagged and used in place of a model identifier when invoking. Invocations through the profile carry its tags into cost data. This is the primary mechanism for attributing Bedrock spend, and creating profiles per team, feature, or tenant is what makes Cost Explorer breakdowns possible.
Cost allocation tags, activated in the billing console. Two critical properties: they must be explicitly activated before appearing in cost reports, and they apply from activation forward — they do not backfill. A company that hasn't activated tags cannot attribute historical spend, and that gap is permanent.
AWS Cost Explorer for analysis by service, account, Region, usage type, and tag, with up to thirteen months of history and forecasting.
AWS Cost and Usage Report (CUR) delivered to S3 for detailed line-item analysis — the source of truth when Cost Explorer's granularity isn't enough, queryable with Athena.
The usage block in model responses, carrying input and output token counts, plus cache read and write counts. Persisting this per request with your own dimensions — user, tenant, feature, prompt version, conversation — gives granularity billing data will never reach, and it's free to capture.
Account separation via AWS Organizations, with consolidated billing.
AWS Budgets scoped by tag, account, or service.
The details that decide success
Create inference profiles before launch. Retrofitting attribution means the historical period stays unattributable.
Activate cost allocation tags early. Same reasoning, and it's a single console action people forget.
Persist the usage block. This is the highest-value, lowest-effort instrumentation in the layer. It enables per-tenant chargeback, per-feature ROI analysis, and outlier detection that cost tooling cannot provide.
Reconcile application metering against billing periodically. If your recorded token counts diverge from the bill, something is invoking outside the instrumented path.
Tag everything, not just inference. Vector stores, endpoints, storage, and logs all contribute, and an attribution scheme covering only Bedrock misses a substantial share of the total.
Cost model
Attribution itself is free. Cost Explorer's API has per-request charges for programmatic access. CUR storage and Athena queries are minor.
Failure modes and diagnostics
No dimension exists. Spend unattributable, permanently for the historical period.
Tags activated late. A gap that cannot be filled.
Only inference tagged. Storage, vector stores, and endpoints unaccounted.
Metering drift. Application counts not matching the bill, indicating an uninstrumented path.

Pattern class 2 — Request-level token economics
The problem in full
Every request has a cost determined by its input tokens, output tokens, and the model's rates. Multiplied by volume, this is usually the largest line item in a generative system, and it is almost entirely determined by design decisions made in other layers.
The arithmetic
cost_per_request ≈ (input_tokens × input_rate) + (output_tokens × output_rate)

input_tokens ≈ system_prompt + tool_definitions + history + retrieved_context + question
Two properties matter.
Output tokens are typically several times more expensive per token than input. So a verbose response is disproportionately costly, and controlling output length is a bigger lever than its token count suggests.
In RAG systems, input tokens usually dominate anyway, because retrieved context is large and answers are short. A system retrieving ten 500-token chunks sends 5,000 tokens of context to answer with 200 tokens — the input is 25x the output in volume, and often the larger cost even at a lower rate.
Worked example. A conversational RAG system: 800-token system prompt, 2,400 tokens of tool definitions, 6,000 tokens of history, 5,000 tokens of retrieved context, and a 400-token answer. That is roughly 14,200 input tokens and 400 output tokens per request. At a million requests a month, that's 14.2 billion input tokens — and the single largest contributor is history plus retrieval, both of which are configuration.
The levers, in order of usual impact
Retrieved context size. k × chunk size, decided in B2b and C1. Reducing top-k from 10 to 5, or enabling reranking so 3 chunks suffice where 10 were needed, cuts the largest term. Reranking can pay for itself through this mechanism.
Conversation history policy. Full history grows cumulative cost quadratically with turn count. Compaction converts it to roughly linear. For long conversations this is an order-of-magnitude difference.
Output length. maxTokens, plus prompt design that elicits appropriately concise answers. "Answer in two sentences" is a cost control.
Prompt caching. A large stable prefix — system prompt, tool definitions, a document being asked about repeatedly — cached and billed at a reduced rate. Requires stable-content-first ordering, which is nearly free at design time and disruptive to retrofit.
Tool definition count. Every tool's schema consumes input tokens on every call. Filtering tools by relevance reduces this and improves selection accuracy simultaneously.
System prompt length. Small per request, multiplied by every request.
AWS mechanisms
maxTokens in inferenceConfig bounding output.
Bedrock prompt caching with cache point markers, and the usage block reporting cache read and write token counts — which is how you verify caching is actually working. Zero cache reads means an ordering problem.
Knowledge Bases numberOfResults and reranking configuration controlling retrieved context volume.
Application-level history compaction as covered in C2.
Titan embedding dimension selection affecting vector storage rather than request tokens, but part of the same design conversation.
The details that decide success
Compute the per-request token budget explicitly. Most teams have never done this and are surprised by the breakdown. It takes ten minutes and frequently identifies an obvious reduction.
Watch for drift. Input tokens creep upward as history policies loosen, retrieval settings are tuned for quality, and prompts accumulate instructions. Cost rising faster than traffic is the diagnostic signature, and the token metrics from X6 are where you see it.
Order the prompt for caching from the start. Stable content first, variable content second, question last.
Verify cache hits. Configured caching that never hits is common and silent.
Don't optimize prematurely. At low volume these differences are noise. Compute the monthly figure before investing engineering time.
Cost model
This is the cost model for most generative systems. The relevant exercise: tokens per request × requests per month × rate, decomposed by contributor, with each contributor mapped to the layer that decided it.
Failure modes and diagnostics
Context creep. Cost growing without traffic growing.
Unbounded output. Occasional very long responses.
Cache never hitting. Wrong prompt ordering.
Over-retrieval. Large k compensating for poor ranking, where reranking would be cheaper.
Quadratic conversation cost. No history policy.

Pattern class 3 — Model selection and routing economics
The problem in full
Model prices vary by an order of magnitude or more. Using a frontier model for every request when most requests are simple is the most common and largest source of unnecessary spend in generative systems.
Decision variables
What is the actual quality requirement per request type? A classification, a routing decision, a summary of a short document, or a query rewrite rarely needs a frontier model. Complex reasoning, nuanced synthesis, and open-ended analysis may.
Can requests be classified cheaply? A small model classifying and routing costs a fraction of the generation it directs. Semantic routing by embedding is cheaper still. The routing decision almost always pays for itself, because misdirecting a simple lookup to an expensive path costs far more than the classification did.
Are there internal steps using the wrong model? Query rewriting, history summarization, metadata extraction, and relevance scoring are all internal operations frequently implemented with whatever model the main path uses. Each should use the cheapest model that does the job.
Is a distilled model viable? For a narrow high-volume task, distilling a large model's behavior into a smaller one trades capability breadth for cost — noting the provisioned throughput requirement changes the comparison.
AWS mechanisms
Multiple models via Bedrock, selected per request path — the Converse API's model-agnostic shape is what makes switching a configuration change.
Bedrock Intelligent Prompt Routing routing among models within a family based on predicted complexity. Managed, and it changes which model answers, which disqualifies it where a specific model is required.
Application-level routing with a small classifier model or embedding-based routing.
Step Functions Choice states expressing routing deterministically where the signal is a request attribute.
Bedrock Model Distillation, optionally using production invocation logs as the distillation dataset.
AppConfig holding model identifiers so routing configuration changes without deployment.
The details that decide success
Audit which model each internal step uses. Query rewriting on a frontier model is pure waste, and it's common because nobody chose it deliberately — the model identifier was copied from the main path.
Route by measured need, not assumption. Build an evaluation set, run the cheap model against it, and see where it actually falls short. Frequently it falls short less than expected.
Model identifiers belong in configuration. Hard-coded model IDs make routing changes a deployment.
Re-evaluate when models change. New models shift the price-performance frontier regularly, and a routing configuration set a year ago is probably suboptimal.
Watch the quality floor. Aggressive downgrading saves money and degrades the product. The evaluation set is what keeps this honest.
Cost model
Savings proportional to the fraction of traffic that can use a cheaper model, times the price differential. For a system where 70% of requests are simple, routing those to a model costing a fifth as much cuts inference spend substantially.
Costs: the classification call, and the engineering to build and maintain routing.
Failure modes and diagnostics
One model for everything.
Internal steps on the expensive model.
Routing quality regression from over-aggressive downgrading.
Stale routing configuration after model releases.

Pattern class 4 — Caching
The problem in full
In conventional systems, caching saves latency and database load. In generative systems a cache hit avoids an inference call, which is real money rather than milliseconds. The economics are qualitatively different, and caching deserves more consideration here than elsewhere.
Subtypes
Prompt prefix caching. Server-side reuse of a processed prompt prefix. Reduces cost within a request.
Exact-match response caching. Identical query, cached response. Simple and low hit rate for natural-language input.
Semantic response caching. Embed the query, return a cached response for a sufficiently similar prior query. Catches paraphrases, and hit rates are much higher.
Embedding caching. Embeddings are deterministic per model, so identical text always produces an identical vector. Perfectly cacheable by content hash.
Retrieval result caching. Popular queries retrieve the same chunks; caching the retrieval saves the vector store round trip.
Pipeline artifact caching. Extraction, chunking, and enrichment outputs cached by content hash so pipeline re-runs don't redo expensive work.
Decision variables
What is the repeat rate? Semantic caching only pays if questions repeat. Internal knowledge assistants have high repeat rates — the same questions from different people. Open-ended creative tools have low ones.
Is staleness acceptable? A cached answer derived from documents that have since changed is wrong with no error. Cache invalidation tied to a corpus version, or short TTLs, is required.
Is the response personalized? Then don't cache it. This rules out caching for entitlement-filtered retrieval, where two users asking the same question should legitimately get different answers — a subtle and important constraint.
Does the cache pay for itself? A provisioned cache cluster runs continuously. Below some hit rate it costs more than the inference it saves.
AWS mechanisms
Bedrock prompt caching for prefix reuse, verified through the usage block's cache token counts.
ElastiCache for Redis or Valkey with vector search for semantic caching — embed the query, search cached query vectors, return the associated response above a similarity threshold. This is the sophisticated version and it's implementable with components already in most architectures.
MemoryDB where the cache must be durable.
DynamoDB with TTL for exact-match caching, cheaper at rest than a cache cluster and adequate when sub-millisecond latency isn't required.
S3 with content-hash keys for pipeline artifact caching — extraction outputs, embeddings, generated summaries.
CloudFront for genuinely shared static generated content, which is rare.
The details that decide success
Tune the semantic similarity threshold carefully. Too loose returns answers to different questions; too tight collapses the hit rate. This needs empirical tuning against real query pairs.
Never cache across entitlement boundaries. The cache key must include whatever scopes the answer — tenant, user, permission set — or you have a data leak that looks like a performance optimization.
Invalidate on corpus change. Tie cache keys to a corpus or index version so a re-sync invalidates naturally.
Cache the pipeline, not just the requests. Extraction and embedding caching by content hash is where the largest one-time savings usually are, because pipeline re-runs are inevitable when chunking or embedding changes.
Measure hit rate before and after. A cache with a 5% hit rate on a small cluster may not justify itself.
Cost model
Prompt caching: reduced rate on cached tokens against a possible write premium. Savings scale with prefix size × request rate.
Semantic caching: a full inference avoided per hit — the largest possible per-request saving — against cluster cost and embedding calls for cache lookup.
Pipeline caching: avoided extraction and embedding calls, which for a large corpus re-run can be very large.
Failure modes and diagnostics
Stale cached answers. No invalidation on corpus change.
Cross-tenant cache hits. Entitlement not in the key.
Threshold too loose. Wrong answers returned confidently.
Cache cost exceeding savings at low volume.
Pipeline not cached. Full re-extraction on every chunking experiment.

Pattern class 5 — Capacity mode selection
The problem in full
The same inference can be purchased three ways, differing by an order of magnitude in effective cost: per-token on demand, per-hour committed, or discounted batch. Choosing wrongly is expensive in both directions.
The three modes
On-demand. Pay per token, no floor, no guarantee, absorbs variability. The default and correct for most workloads.
Committed capacity. Provisioned Throughput in Model Units, or SageMaker instance-hours. Guaranteed throughput and consistent latency, paid whether used or not. Mandatory for customized models on Bedrock — which means that cost belongs in the customization decision.
Batch. Asynchronous processing at a substantial discount, with no latency guarantee.
Decision variables
Is anything waiting? If not, batch. This is the single most reliable cost lever in generative systems and it is under-used out of habit. Corpus embedding, bulk classification, enrichment, synthetic data generation, distillation teacher outputs — all belong in batch. The tell is an absence: no stated latency requirement.
What is the utilization? For committed capacity the comparison is straightforward and rarely done:
On-demand: tokens per month × rate
Committed: Model Units × hours per month × hourly rate
Committed wins at high steady utilization and loses badly at low or variable utilization, because you pay for the peak during every idle hour. A workload busy eight hours on weekdays uses roughly 24% of hours and needs substantial peak throughput to justify committing.
How confident are you the workload persists? Term commitments lower the rate and remove flexibility. Models improve fast, and a six-month commitment on a base model superseded in two is a real risk.
Can you split it? A committed baseline with on-demand overflow is often better than either alone — commit to the floor, burst on demand.
AWS mechanisms
Bedrock on-demand, per token.
Bedrock Provisioned Throughput in Model Units, with no-commitment, one-month, or six-month terms. No-commitment is more expensive hourly and right for evaluation.
Bedrock batch inference (CreateModelInvocationJob) — JSONL in S3, results to S3, substantially discounted.
SageMaker real-time endpoints as committed capacity, with Savings Plans discounting steady baseline usage over one or three years.
SageMaker serverless and asynchronous inference for scale-to-zero, converting standing cost to per-request.
SageMaker batch transform and Managed Spot Training for offline and training workloads.
Spot instances for interruption-tolerant self-hosted workloads.
The details that decide success
Identify every latency-insensitive workload and move it to batch. This is usually the single largest saving available and requires almost no engineering.
Compute the utilization crossover before committing. The calculation takes minutes and frequently reverses the decision.
The customization serving tail belongs in the customization decision. A fine-tune's provisioned throughput floor often exceeds the entire on-demand bill it was meant to improve.
Review provisioned capacity on a cadence. Stranded provisioning from concluded work is the most common waste in this layer, and nothing surfaces it automatically.
Savings Plans for steady SageMaker baseline, sized to the floor rather than the peak.
Cost model
The comparison is arithmetic. What makes it hard is that utilization is often unknown before launch — which argues for starting on-demand, measuring, and committing only once the pattern is established.
Failure modes and diagnostics
Real-time used for batch work. Inference spend several times what it needed to be.
Committed capacity at low utilization.
Stranded provisioning. Forgotten resources billing indefinitely.
Term commitment on a superseded model.

Pattern class 6 — Infrastructure right-sizing
The problem in full
Inference is usually the largest cost, and it is not the only one. Vector stores, endpoints, storage, logging, and networking accumulate, and several have costs that are invisible unless you look.
The costs that surprise people
Vector store resting capacity. Provisioned vector stores allocate capacity continuously. OpenSearch Serverless has a minimum compute-unit allocation that puts a floor of hundreds of dollars a month on even a trivial collection. This is the most common cost finding in small RAG deployments, and the alternatives — S3 Vectors for infrequently-queried corpora, pgvector on a modest RDS instance — change the resting cost by an order of magnitude.
Idle endpoints. SageMaker real-time endpoints bill continuously. A dozen endpoints from concluded experiments is a classic finding.
CloudWatch Logs. Ingestion and storage per GB are substantially more expensive than S3. Full prompt and completion logging into CloudWatch Logs with never-expire retention is a large and silent cost.
Unpartitioned Athena queries. Priced per terabyte scanned. Converting to Parquet and partitioning by date typically reduces scan volume by an order of magnitude.
NAT gateway data processing. Charged hourly plus per GB. A workload pulling large volumes from S3 through a NAT gateway is paying for something a gateway VPC endpoint — which is free — eliminates entirely.
S3 versioning without lifecycle rules. Noncurrent versions accumulating indefinitely.
Incomplete multipart uploads. Billed, invisible in the console listing, and cleaned up by a lifecycle rule most people never add.
Orphaned EBS volumes and snapshots.
Transfer Family endpoints. Hourly per protocol whether or not anyone connects.
AWS mechanisms
S3 Intelligent-Tiering for unpredictable access patterns — access-based rather than age-based, which suits a document corpus where you cannot predict which documents get retrieved.
S3 Lifecycle policies for predictable aging — logs, backups, processed originals — including noncurrent version expiration and aborting incomplete multipart uploads.
Gateway VPC endpoints for S3 and DynamoDB, which are free and remove NAT data processing charges. Interface endpoints for other services are charged hourly per AZ plus per GB.
Compute Optimizer for instance right-sizing recommendations.
Cost Explorer rightsizing recommendations for SageMaker and EC2.
Trusted Advisor for idle and underutilized resource checks.
gp3 EBS volumes, which decouple IOPS and throughput from capacity so you no longer over-provision size to get performance.
The details that decide success
Compute vector store resting cost before choosing. Estimate vectors, queries per month, and the provisioned floor, and compare honestly against alternatives.
Set log retention explicitly. Never-expire is the default.
Send invocation logs to S3, not CloudWatch Logs, for anything with long retention.
Partition and convert analytics data.
Audit for idle resources on a cadence. Endpoints, provisioned throughput, vector stores, and Transfer Family endpoints from concluded projects.
Use gateway endpoints for S3 access from VPCs.
Cost model
Highly variable by architecture. The exercise worth running: enumerate every resource that bills continuously regardless of traffic, and justify each one.
Failure modes and diagnostics
Vector store floor on a small corpus.
Idle endpoints.
CloudWatch Logs cost from full logging.
NAT charges avoidable with a free endpoint.
Storage accretion from versioning and multipart debris.

Pattern class 7 — Quota and capacity planning
The problem in full
Capacity is bounded by quotas that are per-account, per-Region, and per-model, and hitting one produces failure rather than a bill. Planning means knowing where the ceilings are before traffic finds them.
Decision variables
What are the current quotas? Bedrock has per-model request and token rate limits. SageMaker has per-instance-type limits. Lambda has account concurrency. Each is a ceiling that can stop a launch.
Which are adjustable, and how long does adjustment take? Many are, and it is not instantaneous — increases go through a request process measured in days. This belongs in project planning, not in incident response.
Is GPU capacity available? Accelerator instance types are genuinely scarce in some Regions and families. Quota approval does not guarantee capacity, and Capacity Reservations are the mechanism where availability matters.
How do workloads share? A batch job can consume the account's model quota and starve interactive traffic. Reserved concurrency and controlled fan-out are the isolation mechanisms.
AWS mechanisms
Service Quotas for viewing and requesting increases, with CloudWatch metrics on quota utilization for supported services — which lets you alarm before hitting a ceiling rather than after.
Bedrock cross-Region inference profiles to absorb bursts beyond one Region's capacity, nearly free and requiring no planning — subject to the data residency constraint.
Bedrock Provisioned Throughput for guaranteed capacity where on-demand quotas are insufficient.
EC2 Capacity Reservations and ML Capacity Reservations for accelerator availability.
Lambda reserved concurrency to guarantee capacity for critical functions and cap others.
SQS buffering to convert a burst into a queue rather than a wall of throttling.
The details that decide success
Check quotas during design, not at launch. A launch blocked by a quota request is an avoidable delay.
Alarm on quota utilization where the metric exists.
Isolate workload classes. Batch and interactive should not compete for the same ceiling without controls.
Remember that scaling out against a rate limit makes throttling worse. More consumers hitting the same quota produce more ThrottlingException, not more throughput. The answer is buffering, and this counterintuitive fact catches experienced engineers.
GPU capacity is a planning constraint, not an operational detail.
Cost model
Quotas themselves are free. Capacity Reservations bill whether used or not, which is the trade for guaranteed availability.
Failure modes and diagnostics
Launch blocked by quota.
Batch starving interactive traffic.
Scaling out into a rate limit.
GPU capacity unavailable despite quota approval.

Pattern class 8 — Anomaly detection and guardrails
The problem in full
Cost can run away faster than billing data reports it. A runaway agent loop, a retry storm, an abusive user, a misconfigured batch job — each can spend a lot in hours, and Cost Explorer will tell you tomorrow.
The detection mechanism must be faster than the billing system.
The detection ladder
   Mechanism Latency Detects     CloudWatch alarm on token metrics Minutes Volume anomalies on a specific model   CloudWatch anomaly detection on tokens or invocations Minutes Deviation from learned normal   AWS Cost Anomaly Detection Hours to a day Spend deviation with root-cause analysis   AWS Budgets forecast alerts Daily Trajectory toward a limit   Cost Explorer Daily Analysis after the fact   The invoice Monthly Too late   The top of that ladder is the one most systems lack. Token metrics are published by Bedrock and alarmable in minutes, and they are the fastest available signal.
Decision variables
Static threshold or anomaly detection? Generative usage varies by day and season, so a fixed threshold either fires constantly or never fires until damage is done. Anomaly detection on token consumption is the better default.
What is the response? Notification, or automated action? A kill switch that disables a feature is drastic and sometimes correct.
Where are the hard limits? Detection tells you after spending starts. Hard caps prevent it.
AWS mechanisms
CloudWatch alarms on InputTokenCount and OutputTokenCount, static or anomaly-detection-based.
AWS Cost Anomaly Detection with monitors scoped by service, account, cost category, or tag, and alert subscriptions with thresholds. Provides root-cause analysis identifying what drove the anomaly.
AWS Budgets with actual and forecast alerts, and budget actions that can apply an IAM policy or stop resources when a threshold is breached — the automated-response mechanism.
Lambda reserved concurrency as a hard cap on how fast a consumer can spend.
API Gateway usage plans with rate and quota limits per API key.
WAF rate-based rules against abusive volume — denial-of-wallet defense, since abuse of a generative endpoint costs inference tokens rather than merely compute.
AppConfig feature flags as a kill switch, disabling a generative feature instantly without a deployment.
Agent iteration and token bounds in application code, preventing runaway loops at the source.
The details that decide success
Alarm on tokens, not just on spend. It is the fastest signal by a wide margin.
Use anomaly detection for variable metrics.
Have hard caps, not just alerts. Reserved concurrency, usage plans, and iteration bounds prevent overspend rather than reporting it.
Build the kill switch before you need it. A feature flag that disables generative functionality is cheap insurance and impossible to add during an incident.
Rate limits are cost controls here. Size them against the inference budget, not just capacity.
Cost model
Alarms and anomaly monitors carry small per-item charges. The value is avoided overspend, which can be large.
Failure modes and diagnostics
Detection only through billing. Days of runaway spend.
Static thresholds on variable usage.
Alerts without caps. Knowing about overspend without stopping it.
No kill switch.
Unbounded agent loops.

Pattern class 9 — Cost governance and unit economics
The problem in full
Beyond controlling spend, the organization needs to know whether the system is worth what it costs, and to make that visible to the people who influence it.
Decision variables
What is the unit? Cost per request is a technical metric. Cost per conversation, per resolved ticket, per document processed, or per active user is a business metric, and it's the one that determines whether a capability is viable. Translating token spend into unit economics is what makes cost conversations productive rather than adversarial.
Who sees it? Engineers who can change chunk size and top-k should see the cost consequence of those settings. A dashboard visible only to finance changes nothing.
Is there chargeback? Real internal billing changes behavior sharply. It also raises the accuracy bar considerably.
What is the budget process? Usage-driven spend is harder to forecast than capacity-driven spend, and budgets should reflect that with ranges rather than points.
AWS mechanisms
Cost Explorer with tag and inference-profile dimensions.
AWS Budgets per team or feature, with forecast alerts.
Cost Categories grouping accounts, tags, and services into business-meaningful buckets.
CUR in S3, queried with Athena, joined against application metering for unit economics — this join is what converts a bill into a cost-per-conversation figure.
Quick Sight dashboards over that analysis for business audiences; CloudWatch dashboards for engineering ones.
AWS Cost Optimization Hub consolidating recommendations.
Application usage data as the numerator source for per-unit calculations.
The details that decide success
Compute unit economics early. Cost per conversation, tracked over time, is the single most useful number for governance, and it makes the effect of technical changes legible.
Show engineers their costs. The people who set top-k and history policy should see what those settings cost.
Forecast with ranges. Usage-driven spend has variance; a point forecast will be wrong.
Review the architecture periodically against the levers. Batch opportunities, routing, caching, and resting capacity all drift as the system evolves.
Model the cost of a change before making it. Increasing top-k for quality has a computable monthly cost, and that should be part of the decision rather than discovered later.
Cost model
Governance tooling is minor. The value is in decisions made with better information.
Failure modes and diagnostics
Cost invisible to engineers.
No unit economics. Spend discussed in absolute terms with no denominator.
Point forecasts.
Optimization as a one-off rather than a recurring review.

The consolidated lever list
Every cost lever in the guide, with the layer that owns it:
   Lever Layer Typical impact     Batch instead of real-time M1 Very large where applicable   Reduce k × chunk size B2b / C1 Large, recurring   Reranking to enable smaller k C1 Large, and can be net negative cost   History compaction C2 Large for long conversations   Model routing by complexity O1 Large   Prompt caching M2 Large with big stable prefixes   Semantic response caching X8 Large with high repeat rates   Cheaper models for internal steps O1 Moderate, easy   Bound maxTokens M2 Moderate   Embedding dimension reduction B2b Moderate on storage   S3 Vectors for cold corpora B2b Large on resting cost   Eliminate idle endpoints and provisioning M1 Large where present   Invocation logs to S3 not CloudWatch X6 Moderate   Parquet and partitioning for analytics X6 Large on Athena   Gateway VPC endpoints Infra Moderate, easy   Spot for training B3 Large on training   Pipeline artifact caching B2a Large on re-runs   Reserved concurrency and rate limits X4 Prevents runaway   The first row is usually the biggest and easiest. Identifying every workload with no latency requirement and moving it to batch is the highest-return exercise in this layer.

Cross-cutting decisions in this layer
Is attribution instrumented, and was it before launch? Inference profiles and activated cost allocation tags, neither retroactive.
Is the usage block persisted with your dimensions? Free, and the only path to real unit economics.
What is the per-request token budget, decomposed? Compute it.
Which workloads have no latency requirement? Move them to batch.
What bills continuously regardless of traffic? Enumerate and justify each.
What is the fastest detection mechanism? Token metrics with anomaly detection, in minutes.
Are there hard caps, not just alerts? Reserved concurrency, usage plans, iteration bounds, kill switch.
What is the cost per meaningful unit, and who sees it?
Interfaces to adjacent layers
From B2b and C1. Chunk size, k, expansion, and embedding dimension set recurring per-request cost and resting storage cost.
From C2. History policy determines whether conversation cost grows linearly or quadratically.
From M1. Capacity mode, packing density, and idle elimination are the largest infrastructure levers.
From M2. Token counts, maxTokens, caching, and prompt ordering are the request-level levers, and the usage block is the measurement.
From B3. Customization commits you to a serving cost floor; that belongs in the customization decision.
From O1. Orchestration choice determines cost predictability — fixed workflows are budgetable, agents are not — and routing is expressed here.
From X6. Token metrics are the fastest cost signal; inference profiles provide the attribution dimension; observability itself has a cost that needs managing.
To X4 (Quotas). Rate limiting is simultaneously a security and a cost control, and denial of wallet is where the two meet.

Unstarred

Dunk (exclude from study)

Cost & Capacity

Review Questions

Todos

Sensemaker