Model Serving

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

Layer Guide — M1: Model Serving
What this layer is responsible for
Where the weights physically run, what capacity is available, and what that capacity costs. Not the call itself — that's M2. The distinction matters because these two layers fail differently and are tuned independently. "The request timed out on a large payload" is a serving problem. "The model won't stop generating" is an invocation problem.
Serving decisions are largely economic. The same model producing the same outputs can differ by an order of magnitude in cost depending on how it's served, and the levers are: whether you pay per token or per hour, how densely you pack accelerators, whether capacity scales to zero, and whether anyone is waiting.
The physics you need, because everything here follows from it
Two facts about transformer inference explain nearly every serving decision.
Inference has two phases with opposite bottlenecks. Prefill processes the entire input prompt at once — highly parallel, compute-bound, saturating the accelerator's math units. Decode generates output tokens one at a time, each requiring a full pass over the model weights — memory-bandwidth-bound, leaving math units mostly idle. Time to first token is dominated by prefill; time between subsequent tokens is dominated by decode.
The consequence: a single request decoding alone wastes most of the accelerator. Throughput comes from batching — running many requests concurrently so each weight-read serves several sequences at once. This is why serving efficiency is fundamentally a batching problem, and why utilization and latency trade against each other rather than moving together.
Accelerator memory holds weights plus KV cache. Weights are fixed: roughly parameters × bytes per parameter. A 7-billion-parameter model at FP16 needs around 14 GB; a 70B model needs around 140 GB, which already exceeds any single accelerator and forces sharding.
The KV cache is the variable part. Every token in every active sequence stores key and value tensors for every layer, and it grows linearly with both sequence length and batch size. Approximately:
KV cache ≈ 2 × layers × kv_heads × head_dim × sequence_length × batch_size × bytes_per_element
The practical form: whatever memory the weights don't occupy is your batching budget, and sequence length spends it. This single relationship is behind most of the serving optimization in this layer.

Pattern class 1 — Managed token-billed serving
The problem in full
You want a model available without operating anything, paying only for what you use, with capacity that absorbs variable traffic without planning.
The variables:
Traffic volume and shape — steady, spiky, or sporadic.
Latency tolerance, including whether streaming is acceptable.
Which model, and whether it's available in your Region.
Data residency constraints.
Throughput requirements relative to account quotas.
Subtypes
Single-Region on-demand. The default. Requests go to the model in your Region, billed per input and output token.
Cross-Region routed. Requests distributed across Regions within a geography to absorb bursts beyond one Region's capacity.
Prefix-cached. Repeated prompt prefixes cached server-side so subsequent requests skip re-processing them.
Decision variables
Is throughput within quota? On-demand capacity is governed by per-model, per-Region quotas on requests and tokens per minute. Below them, on-demand is simply correct. Approaching them, you need a strategy.
Is traffic spiky? On-demand absorbs variability at no standing cost — this is its central advantage over committed capacity and the reason it's right for most workloads.
Does inference data need to stay in one Region? Cross-Region inference profiles move requests across Regional boundaries within a geography. They're nearly free and require no capacity planning, which makes them the natural throttling fix — but they are disqualified outright where residency or sovereignty requirements apply. This is a constraint that overrides the optimization.
Do requests share a long common prefix? A large system prompt, a fixed instruction block, a stable document being asked about repeatedly. Prefix caching turns the prefill cost of that shared portion into a cache hit, reducing both latency and cost on subsequent requests.
AWS mechanisms
Bedrock on-demand invocation. No provisioning. Per-token pricing differing by model and by input versus output tokens — output tokens are typically several times more expensive, which makes output length a real cost lever.
Cross-Region inference profiles. An inference profile identifier used in place of a model identifier; Bedrock routes across Regions in the geography. Same API, same model, no additional charge, no operational overhead — which is why it sits above buffering and below provisioned throughput on the throttling ladder.
Application inference profiles. A taggable wrapper around a model, enabling cost attribution per team, tenant, or feature in Cost Explorer. Without these, Bedrock spend is one undifferentiated line item, and tagging is not retroactive.
Prompt caching. Cache checkpoints marking a reusable prefix. Cached tokens are billed at a reduced rate and skip prefill. Effective when a substantial, stable prefix precedes variable content — ordering matters, since only a prefix can be cached.
Model access grants. Access must be requested per model, per Region, before invocation succeeds. Missing this presents as an authorization failure that looks like an IAM problem.
Service quotas per model and Region, some adjustable by request — which takes time and should be part of capacity planning rather than an emergency response.
The details that decide success
Output tokens dominate cost in generative workloads. Input is usually cheaper per token and often cacheable; output is neither. Controlling output length through maxTokens, prompt design, and output format is frequently the largest single cost lever available, and it lives at the boundary with M2.
Throttling is normal, not exceptional. Design for it from the start: SDK retry with exponential backoff and jitter is the baseline, and the SDKs implement it with configurable modes including adaptive retry that rate-limits client-side based on observed throttling.
Streaming changes perceived latency, not actual latency. Time to first token is what users experience as responsiveness. A response that takes eight seconds but begins rendering in 400 milliseconds feels fast; the same response delivered whole feels broken.
Model availability varies by Region. In a residency-constrained deployment, the model you want may simply not exist where you must run, and that constrains architecture rather than being a detail.
Cost model
Per input token, per output token, at model-specific rates. Cached input tokens at a reduced rate. Cross-Region inference adds no charge. No standing cost whatsoever.
The levers, in rough order of impact: reduce output tokens; cache stable prefixes; choose a smaller model where quality permits; move latency-insensitive work to batch; reduce retrieved context size (which is set in B2b).
Failure modes and diagnostics
Throttling under burst. ThrottlingException. Ladder: backoff → buffering → cross-Region profile → provisioned throughput.
Cost growth outrunning traffic growth. Usually output length or retrieved context creeping upward. Diagnose from token metrics in CloudWatch, broken out by application inference profile.
Unattributable spend. No inference profiles configured, so no dimension exists to break down by. Not retroactively fixable.
Cross-Region routing violating residency. A compliance failure introduced as a performance fix.
Access denied on a model that works elsewhere. Model access not granted in that Region.

Pattern class 2 — Committed capacity serving
The problem in full
Some workloads need throughput guarantees that on-demand quotas don't provide, or must serve a customized model that on-demand doesn't support at all. Committed capacity reserves throughput for a term at a fixed hourly rate.
The variables:
Whether a guarantee is genuinely required, or merely reassuring.
Traffic steadiness, since committed capacity is wasted when idle.
Whether the model is customized, which removes the choice.
Commitment tolerance — how confident you are the workload persists.
Decision variables
Is this forced or chosen? A fine-tuned, distilled, or imported model on Bedrock cannot be invoked on demand. Provisioned throughput is mandatory, and this cost belongs in the customization decision, not discovered afterward.
If chosen, does the arithmetic work? The comparison is straightforward and rarely done:
On-demand cost = tokens per month × per-token rate
Provisioned cost = Model Units × hours per month × hourly rate
Provisioned capacity wins at high, steady utilization. It loses badly at low or variable utilization, because you pay for the peak you provisioned during every idle hour. The break-even is a utilization percentage, and workloads that are busy eight hours a day on weekdays — roughly 24% of hours — need substantial peak throughput to justify committing.
How confident are you? Term commitments of one or six months lower the hourly rate and remove flexibility. No-commitment provisioning is more expensive per hour and appropriate while evaluating.
Is the requirement actually latency consistency rather than throughput? Committed capacity gives predictable latency by removing contention. If that's the real requirement, say so — it's a different justification with a different evaluation.
AWS mechanisms
Bedrock Provisioned Throughput. Purchased in Model Units, each providing a defined throughput allocation for a specific model. Terms: no commitment, one month, or six months. Applies to both base models and custom models, with custom models requiring it.
SageMaker real-time endpoints are the equivalent shape in the SageMaker world — you pay for instances continuously, which is committed capacity by another name, with more control over the hardware.
Savings Plans apply to SageMaker instance usage, giving a discount for committed spend over one or three years. This is the mechanism for reducing steady-state serving cost when you've settled on a configuration.
The details that decide success
Compute the monthly figure before committing, and compare it to actual on-demand spend. This comparison frequently kills a fine-tuning proposal outright, and it should happen during design rather than at the first invoice.
Provisioned capacity does not autoscale. You sized it; traffic above that throttles, traffic below is waste. Combining a provisioned baseline with on-demand overflow is possible at the application level and is often the better shape — commit to the floor, burst on demand.
Idle provisioned capacity is the most common form of stranded spend in generative systems. A model provisioned for an evaluation that concluded, or a project that paused, keeps billing. Tag it, track it, and set a review cadence.
Term commitments assume the model persists. Models improve fast. A six-month commitment on a base model that a better one supersedes in two is a real risk, and it argues for shorter terms unless the workload is genuinely stable.
Cost model
Model-Unit-hours, continuously, at rates decreasing with commitment length. Utilization is the entire story: at 90% utilization committed capacity is usually much cheaper per token; at 10% it is dramatically more expensive.
Failure modes and diagnostics
Stranded capacity. Provisioned and forgotten. Detect with tagging and a regular review of provisioned resources.
Under-provisioned and throttling. Committed capacity exceeded, requests failing, with no elastic overflow.
Committed to a superseded model.
Fine-tune economics discovered late. The training was cheap; the serving floor was not.

Pattern class 3 — Dedicated real-time endpoints
The problem in full
You need a model you control — a custom architecture, specific serving code, an open-weight model Bedrock doesn't host, or a classical ML model — available synchronously with consistent low latency.
You now own the infrastructure decisions: instance type, count, scaling policy, and how densely the accelerator is packed. This is where the physics from the opening section becomes actionable.
The variables:
Model size, which determines minimum accelerator memory and whether sharding is required.
Traffic volume and shape.
Latency requirement, specifically time to first token versus total.
Payload size and processing duration.
Whether GPU is required.
Decision variables
Endpoint type is decided by three numbers, and this is the highest-value table in the layer:
   Type Max payload Max duration Scales to zero GPU     Real-time ~6 MB ~60 s No Yes   Serverless ~4 MB ~60 s Yes No   Asynchronous ~1 GB ~60 min Yes Yes   Batch transform Whole dataset No limit N/A Yes   Work the requirements against it. A workload needing GPU with a 50 MB payload and a fifteen-minute tolerance eliminates serverless (no GPU), real-time (payload and duration both exceeded), and batch transform (it processes datasets, not per-request invocations) — leaving asynchronous. The table does the work; there's no judgment involved once the numbers are known.
How many replicas fit per instance? This is the GPU packing question and it's where most serving money is left on the table. If a model's weights occupy 40 GB and the instance has 8 accelerators of 40 GB each, you can run one replica sharded across four accelerators and a second replica on the other four — or you can naively shard one replica across all eight and halve your throughput.
Tensor parallelism degree should be the smallest that fits the model, not the largest available. Sharding across more devices than necessary adds inter-device communication on every forward pass, which reduces throughput. Model fits in 4 → degree 4, two replicas on an 8-accelerator instance. Degree 8 for that model is slower and serves half the concurrency.
What is the real maximum sequence length? Configured maximum sequence length reserves KV cache capacity. If the configuration allows 32,000 tokens but real traffic peaks at 3,000, you've reserved roughly ten times the KV cache each sequence needs, which caps how many sequences fit in memory and leaves the accelerator underutilized. Right-sizing this to observed traffic directly increases the achievable batch size and therefore throughput per instance.
Note what is not the answer to underutilization: adding instances. That pays for the inefficiency rather than removing it, and it's the intuitive wrong move.
AWS mechanisms
SageMaker real-time endpoints. An endpoint configuration specifying one or more production variants, each with a model, instance type, instance count, and traffic weight. Deployed behind a managed HTTPS endpoint with health checks.
Instance families. P and G families for NVIDIA GPUs; Inf for Inferentia and Trn for Trainium via the Neuron SDK, offering lower cost per token where the model architecture is supported. Compute-only instances for small models and classical ML.
Large Model Inference containers, built on serving stacks such as DJL Serving, vLLM, and TGI, which implement the techniques that matter:
Continuous (rolling) batching — new requests join the running batch as slots free, rather than waiting for a batch to complete. This is the single most important throughput feature for LLM serving.
PagedAttention — KV cache allocated in pages rather than contiguous blocks, dramatically reducing memory fragmentation and raising achievable batch size.
Tensor parallelism across accelerators.
Speculative decoding — a small draft model proposes tokens the large model verifies in batches, reducing per-request latency.
Quantization support, reducing weight memory and freeing KV cache capacity.
Configuration surfaces through serving properties — commonly tensor_parallel_degree, max_rolling_batch_size, and a maximum model or sequence length — plus container environment variables.
Autoscaling via Application Auto Scaling, typically target tracking on SageMakerVariantInvocationsPerInstance or a latency metric, with configurable cooldowns. Step scaling and scheduled scaling for known patterns.
Production variants and shadow variants. Traffic-weighted variants enable canary and blue/green rollout; shadow variants receive a copy of production traffic without returning responses, for silent comparison of a candidate model.
SageMaker inference optimization capabilities for applying quantization, compilation, and speculative decoding to supported models with less manual configuration.
The details that decide success
Container startup time governs scaling responsiveness. A large model container may take several minutes to pull, load weights, and pass health checks. Autoscaling that reacts in minutes cannot absorb a sharp spike. Scheduled scaling ahead of known peaks, or a warm baseline capacity, is the practical answer.
Latency and throughput trade off through batch size. Larger batches raise throughput per accelerator and raise per-request latency. There is no configuration that maximizes both, and knowing which you're optimizing is a prerequisite to tuning anything.
Measure the right latency. For streaming generation, time to first token and inter-token latency are the user-visible quantities. Mean total latency conflates them and hides regressions in either.
Prefill and decode contend. A long prompt's prefill occupies the accelerator and delays token generation for other in-flight requests. Systems with highly variable prompt lengths show latency variability that looks random and isn't.
Health check configuration matters. A container that responds to health checks before it's genuinely ready receives traffic it can't serve, presenting as intermittent failures after scaling events.
Never scale to fix a downstream rate limit. If your endpoint calls something rate-limited, adding replicas produces more throttling, not more throughput.
Cost model
Instance-hours, continuously, per variant. This is the dominant cost and it accrues whether or not requests arrive — idle real-time endpoints are the most common waste in SageMaker environments, and the classic finding is a dozen endpoints from concluded experiments still running.
Levers: right-size the instance; pack replicas correctly; quantize to fit more per accelerator; use Inferentia where supported; apply Savings Plans to steady baseline capacity; and delete endpoints when work concludes.
Failure modes and diagnostics
Underutilized accelerators. Presents as needing many instances for modest traffic. Diagnose with GPU utilization and memory metrics; usual causes are over-configured sequence length, tensor parallelism higher than necessary, or batch size limits set too low.
Scaling too slow for the spike. Errors during ramp while new instances load. Fix with warm baseline or scheduled scaling.
Payload or timeout limit exceeded. Presents as failures on large inputs. The endpoint type was wrong; move to asynchronous.
Idle endpoints accruing cost.
Latency variance from mixed prompt lengths. Prefill contention.
Deployment failures from container contract mismatch. A custom container that doesn't implement the expected inference interface deploys and then fails to serve.

Pattern class 4 — Elastic intermittent serving
The problem in full
Traffic is sporadic or unpredictable, and an always-on endpoint spends most of its life idle. You want capacity that appears on demand and costs nothing when unused.
The variables:
Idle fraction — what proportion of hours see no traffic.
Cold-start tolerance.
Whether GPU is needed.
Model size and memory footprint.
Concurrency requirements.
Decision variables
How idle is idle? If the endpoint serves traffic 5% of hours, always-on capacity is 95% waste and scale-to-zero is compelling. At 60% utilization the comparison narrows and cold starts may not be worth it.
Can the first request wait? Scale-to-zero means the first request after idle pays initialization — loading the container and weights. For a large model that can be tens of seconds. Interactive users notice; background processes don't.
Does it need GPU? This eliminates serverless inference outright for large language models. SageMaker Serverless Inference is CPU-only, which restricts it to smaller models and classical ML. For GPU workloads needing scale-to-zero, asynchronous inference is the option — it scales to zero and supports GPU, at the cost of the deferred-response pattern.
AWS mechanisms
SageMaker Serverless Inference. Configure memory size and maximum concurrency; capacity is provisioned per request and scales to zero. Provisioned concurrency keeps a configured number of workers warm, eliminating cold starts at a standing cost — which partially defeats the purpose but is a legitimate middle ground. No GPU support, which is the binding constraint for LLM serving.
SageMaker Asynchronous Inference. Scales to zero, supports GPU, accepts large payloads, and allows long processing. Requests are queued and results written to S3 with optional SNS notification. Effectively the GPU-capable scale-to-zero option, at the cost of not being synchronous.
Bedrock on-demand is the purest form of this pattern: no capacity to configure, no cold start visible to you, and nothing charged when idle. When the model is available on Bedrock, this pattern is largely solved by using it.
Lambda for very small models or CPU inference, with its own cold-start characteristics and a hard 15-minute ceiling.
The details that decide success
Cold-start cost scales with model size. Container image size, weight loading, and framework initialization all contribute. Smaller images and lazy loading help; a 30 GB model will always take a while.
Cold starts matter less than intuition suggests for generative workloads, because model inference itself takes seconds. A two-second cold start on an eight-second inference is a 25% penalty on the first request only. This is worth knowing because provisioned concurrency is often proposed to fix latency that is actually inference-bound, where it will make no meaningful difference.
Concurrency limits shape burst behavior. Serverless configurations have maximum concurrency; requests beyond it queue or fail.
Prefer the managed API when it exists. Standing up a scale-to-zero endpoint for a model Bedrock already serves is added complexity for no benefit.
Cost model
Per-request and per-duration, with nothing charged at rest. Provisioned concurrency reintroduces a standing cost proportional to warm capacity.
The comparison against always-on is a utilization calculation identical in shape to the provisioned-versus- on-demand one.
Failure modes and diagnostics
Cold-start latency on interactive paths. Presents as sporadic slow first requests.
GPU requirement discovered after choosing serverless. The model simply cannot deploy.
Concurrency ceiling under burst.
Provisioned concurrency negating the savings. Configured warm capacity approaching what always-on would have cost.

Pattern class 5 — Deferred-response serving
The problem in full
Processing takes longer than any synchronous transport tolerates, or payloads exceed what one can carry, but the work is still per-request rather than a batch job. The caller submits, gets an acknowledgment, and collects the result later.
The variables:
Processing duration per request.
Payload size.
How the caller learns of completion — polling, notification, or push.
Volume and burstiness.
Result retention requirements.
Decision variables
Is anyone waiting on the connection? If not, this pattern removes every timeout constraint at once.
Duration and payload against the endpoint table. Asynchronous inference accommodates roughly 1 GB payloads and processing approaching an hour, which covers essentially all per-request generative work including long document processing and extended generation.
How does the caller find out? Polling a status endpoint is simplest. SNS notification is push-based and works for backend consumers. A WebSocket push to a stored connection identifier is what you use when a browser is waiting — which links this pattern to the transport layer.
AWS mechanisms
SageMaker Asynchronous Inference. Input from S3, results written to S3, optional SNS notification on success and failure, an internal queue with configurable maximum concurrent invocations per instance, and autoscaling including to zero when the queue drains. The natural fit for large-payload, long-duration, GPU-requiring inference.
Bedrock with an application-level async pattern. Bedrock's synchronous API is fast enough for most generation, but when orchestration around it is long — retrieval, several model calls, post-processing — the async shape moves to the application: API Gateway accepts the request, writes a job record to DynamoDB, enqueues to SQS, returns a job identifier; a worker processes and updates status; the client polls or is pushed to.
SQS as the queue, with visibility timeout exceeding worst-case processing time. This is the detail that bites: if extraction and generation take ninety seconds and the visibility timeout is thirty, the message reappears twice and you pay for three inferences of the same request.
Step Functions when the deferred work is multi-stage with retries and error handling per stage, and especially when a human approval sits in the middle — the callback pattern holds an execution open for up to a year.
EventBridge and SNS for completion notification fan-out.
The details that decide success
Idempotency is mandatory. At-least-once delivery plus expensive inference means duplicate work is a cost problem as well as a correctness one. Key on a deterministic request identifier and check before invoking.
Return the job identifier immediately. The acknowledgment should not wait on any downstream work.
Design the failure path. A dead-letter queue for requests that repeatedly fail, and a status value the client can distinguish from "still working." Silent permanent pending is a bad failure mode.
Set result retention deliberately. Results in S3 accumulate; lifecycle rules should age them out, and sensitive results should carry the same encryption and retention posture as any other content.
Alarm on queue age, not depth. ApproximateAgeOfOldestMessage tells you whether the system is keeping up; depth alone doesn't distinguish a burst from a stall.
Cost model
Instance-hours only while processing, scaling to zero when idle — which makes this substantially cheaper than always-on for bursty workloads. Plus queue, storage, and notification costs, all minor.
The waste to avoid is duplicate processing from visibility-timeout misconfiguration.
Failure modes and diagnostics
Duplicate inference. Visibility timeout shorter than processing. Presents as inflated cost and duplicate outputs.
Queue growth. Consumers not keeping up. Alarm on age.
Silent pending. Failures with no terminal status.
Unbounded result storage.

Pattern class 6 — Offline batch scoring
The problem in full
A large body of inputs must be processed, and nothing is waiting. This is the cheapest way to run inference by a wide margin, and it is under-used because teams reach for the real-time path by habit.
The variables:
Volume.
Deadline, which is usually loose.
Per-item processing complexity — one model call, or a pipeline.
Failure isolation requirements.
Decision variables
The tell is an absence. A requirement describing a large corpus with no latency requirement is a batch requirement, and the absence of a stated deadline is the signal. This is easy to miss because absences are less noticeable than presences.
One call per item, or a pipeline? Pure inference over many inputs is Bedrock batch inference — simplest and cheapest. Multi-stage processing per item, with per-item error isolation and non-Bedrock stages, is Step Functions Distributed Map fanning out over S3 objects.
Is throughput or cost the driver? Batch inference trades latency for a substantial discount. If you need the volume processed quickly rather than cheaply, parallel real-time invocation may be the answer despite costing more — but verify against quotas first.
AWS mechanisms
Bedrock batch inference (CreateModelInvocationJob). JSONL input in S3, asynchronous processing, results to S3, at a substantial discount to on-demand rates. No infrastructure. The natural choice for corpus embedding, bulk classification, bulk summarization, synthetic data generation, and teacher output generation for distillation.
SageMaker batch transform. Processes a dataset in S3 through a model on ephemeral instances that terminate when finished. No endpoint, no idle cost. The equivalent for models you host yourself.
Step Functions Distributed Map. Fans out across many parallel child executions over objects in S3, with per-item error isolation, controlled concurrency, and result aggregation. Choose it when each item needs multiple stages.
Managed Spot for batch transform, since interruption is tolerable when nothing waits.
The details that decide success
Batch is the most reliable cost lever in generative systems. Any corpus-scale processing — embedding, classification, enrichment, distillation teacher outputs — belongs here, and running it through real-time inference is straightforwardly wasteful.
Concurrency control protects shared quotas. A large batch job can consume the account's model quota and starve interactive workloads. Distributed Map's maximum concurrency setting, and reserved concurrency on consumers, are the controls.
Partial failure is normal at scale. Design for it: per-item outcome reporting, a failure manifest, and a re-run path over just the failures. Both Bedrock batch and Distributed Map produce per-item results; use them.
Idempotency lets you re-run safely. Keyed on input identifier, so a re-run skips completed items.
Batch and real-time can share a model and a prompt. Keeping them consistent means a prompt change applies to both, which is an argument for prompts as managed resources rather than inline strings.
Cost model
Substantially discounted per-token pricing for Bedrock batch. Instance-hours only for the job duration with batch transform, further reduced by Spot. No standing cost in either case.
This is where the largest cost reductions in a generative system usually come from, and the analysis is simply identifying which workloads have no real latency requirement.
Failure modes and diagnostics
Real-time used for batch work. Presents as inference spend far above expectation for the volume.
Quota starvation. A batch job consuming shared capacity, degrading interactive workloads.
Partial failure unnoticed. Some items never processed, discovered downstream as gaps.
No re-run path. A failure means reprocessing everything.

Pattern class 7 — Density patterns
The problem in full
Many models, each with modest traffic. Dedicating an endpoint to each is prohibitively expensive — a hundred models at one endpoint each is a hundred instances mostly idle.
The variables:
Model count and traffic distribution across them.
Model size, and whether several fit in one accelerator's memory.
Whether models share a base, as fine-tuned variants do.
Latency tolerance for loading a model not currently resident.
Subtypes
Multi-model endpoints. Many models share instances, loaded from S3 on demand and evicted under memory pressure.
Inference components. Multiple models on one endpoint with explicitly allocated resources per component and independent scaling.
Multi-adapter serving. One base model deployment serving many LoRA adapters, each small, swapped per request.
Multi-tenant single model. One model serving all tenants with tenant context supplied per request — the cheapest option when it's viable, and often overlooked in favor of unnecessary per-tenant models.
Decision variables
Do the models share a base? If they're LoRA adapters over one base model, multi-adapter serving is dramatically more efficient than any other option — the base weights load once and each adapter is small. This is a strong argument for parameter-efficient fine-tuning back in B3, and it's a serving consequence of a training decision.
What's the traffic distribution? A long tail of rarely-invoked models suits on-demand loading, since most stay evicted. Uniformly busy models don't benefit and should be separately resourced.
Is a load-latency penalty acceptable? Multi-model endpoints load a model from S3 on first request if it isn't resident. Cold model invocation is slow. Acceptable for background work, often not for interactive.
Do you actually need separate models? Per-tenant fine-tuned models are sometimes proposed where prompting with tenant context or retrieval filtering would serve. The simplest density pattern is not needing density.
AWS mechanisms
SageMaker multi-model endpoints. Models in S3, loaded into instance memory on demand, evicted under pressure. One endpoint, one instance fleet, many models. Best for many models with sparse traffic.
SageMaker inference components. Each component has a model, allocated CPU/GPU/memory, and its own scaling configuration, packed onto shared instances. More control than multi-model endpoints and better suited to models with meaningful traffic.
Multi-adapter serving in the large-model inference containers, hosting a base model with multiple LoRA adapters selected per request.
Bedrock sidesteps the problem for base models: many models, one API, no capacity per model. Density concerns arise only for customized models, which each require their own provisioned throughput.
The details that decide success
Memory is the packing constraint. Total resident model memory plus KV cache must fit. Quantization directly increases how many models fit.
Cold-load latency is the trade. Measure it, and decide whether the traffic pattern tolerates it.
Noisy neighbors are real. One model's traffic spike affects co-resident models. Inference components with explicit resource allocation mitigate this; multi-model endpoints do not.
Per-tenant cost visibility gets harder when models share infrastructure. Attribution requires application-level accounting rather than infrastructure-level.
Cost model
Instance-hours for the shared fleet, amortized across models. The saving relative to dedicated endpoints is roughly the ratio of models to instances, and it can be very large.
Failure modes and diagnostics
Thrashing. Working set exceeds memory, models constantly loading and evicting. Presents as high latency and low throughput. Diagnose with model load metrics.
Noisy neighbor latency.
Cold-load latency on interactive paths.

Pattern class 8 — Self-managed serving
The problem in full
Neither Bedrock nor SageMaker fits: you need a specific serving framework, unusual hardware, a deployment target outside AWS's managed surface, or integration with an existing container platform.
The variables:
Why managed doesn't fit — and this should be a specific, articulable reason.
Team capacity for operating inference infrastructure.
Deployment target — cloud, edge, on-premises.
Scale, which determines whether the operational burden amortizes.
Decision variables
Is there a genuine reason? Self-managed serving means owning autoscaling, health checking, rolling deployment, observability, and capacity planning for GPU infrastructure. The reasons that justify it: a serving framework with capabilities the managed containers lack, hardware not available through SageMaker, an existing Kubernetes platform with GPU scheduling already solved, or a deployment target outside AWS Regions.
Kubernetes or simpler? If the organization already runs EKS with GPU node groups and a platform team, adding inference workloads is incremental. If not, adopting Kubernetes to serve a model is a large commitment for a narrow benefit.
Does it have to run somewhere specific? Data residency requiring on-premises, or latency requiring edge proximity, are the constraints that force this and cannot be satisfied any other way. Note that Bedrock is a Regional cloud service — a requirement that inference happen on-premises means self-hosting an open-weight model there, with all the operational consequences.
AWS mechanisms
ECS or EKS on GPU instances, running a serving framework — vLLM, TGI, DJL Serving, Triton — in containers from ECR. Karpenter on EKS for GPU node provisioning. Fargate is not an option for GPU workloads.
EC2 directly with your own orchestration, when even container orchestration is more than the workload needs — a single large instance running a serving process behind a load balancer is a legitimate and frequently sufficient architecture.
Application Load Balancer in front, with a configurable idle timeout well beyond API Gateway's 29-second integration limit, which is one reason self-managed serving sometimes pairs with ALB rather than API Gateway.
AWS Outposts for on-premises deployment with AWS APIs, and Wavelength for 5G edge proximity — both narrow, both answers to physical-location constraints rather than technical ones.
SageMaker Neo and Neuron for compiling models to constrained or purpose-built targets, including edge devices.
Inferentia and Trainium accessible through EC2 directly as well as through SageMaker.
The details that decide success
You now own everything the managed services were doing. Health checks that account for model loading time, graceful draining during deployment, GPU-aware autoscaling metrics, and observability into batch sizes and cache utilization. Budget for it.
GPU capacity is genuinely scarce. Instance availability varies by Region and family, and quota increases take time. Capacity Reservations are the mechanism where availability matters, and this is a planning constraint rather than an operational detail.
Serving framework choice is consequential. The frameworks differ meaningfully in batching implementation, quantization support, and adapter handling, and switching later is real work.
The physics from the opening section is entirely yours to manage. Tensor parallelism degree, maximum sequence length, batch size limits, quantization, and KV cache configuration are all your settings, and defaults are frequently wrong for your traffic.
Cost model
Instance-hours, with Spot available for interruption-tolerant workloads and Savings Plans or Reserved Instances for steady baseline. Plus the cluster control plane if using EKS. Plus engineering time, which is usually the dominant real cost and never appears in a bill.
Failure modes and diagnostics
Operational burden underestimated. The most common failure, and it presents as slow delivery rather than as an incident.
GPU capacity unavailable. Scaling silently stops at a quota or capacity ceiling.
Misconfigured serving parameters. Underutilized accelerators, per the packing discussion.
Deployment downtime. Rolling deployment of large model containers without sufficient spare capacity.

Cross-cutting decisions in this layer
Who is waiting, and for how long? This one question routes to the pattern class: nobody at all → batch; nobody on the connection → deferred; somebody watching → real-time with streaming; somebody occasionally → elastic.
Per-token or per-hour? The fundamental cost-shape decision. Per-token has no floor and no ceiling guarantee; per-hour has both. Utilization decides which is cheaper, and the calculation is simple enough that there's no excuse for not doing it.
What did the customization layer already decide? A customized Bedrock model mandates provisioned throughput. LoRA adapters enable multi-adapter density. Quantization changes packing. The serving decision is frequently made upstream.
How dense is the packing, and has anyone checked? GPU utilization and memory metrics answer this, and they're rarely examined until someone questions the bill.
What is the throttling posture? Backoff, buffering, cross-Region routing, or committed capacity — in that order of escalation, with residency constraints potentially removing the third.
What is the idle-resource review process? Endpoints and provisioned throughput from concluded work are the most common waste in this layer, and they persist because nothing surfaces them.
Interfaces to adjacent layers
From B3 (Customization). Determines whether managed on-demand serving is even available, and what the capacity commitment will be. This coupling is tighter than any other in the guide.
To M2 (Invocation). Serving determines which invocation patterns are possible — streaming requires a serving path that supports it; batch requires the batch API; provisioned capacity is addressed through a different identifier. Conversely, invocation parameters like maxTokens and prompt length directly drive the serving layer's memory and throughput behavior. The two layers are tuned together.
To O1 (Orchestration). Serving latency and failure modes shape orchestration design: slow inference pushes toward asynchronous workflows; throttling pushes toward buffering; long-running agent loops require a hosting model that tolerates their duration.
To X8 (Cost). This layer contains the largest cost decisions in most generative systems, and the levers are concentrated here: batch versus real-time, committed versus on-demand, packing density, and idle elimination.
To X6 (Observability). GPU utilization, memory usage, batch size, queue depth, token throughput, and time to first token are the metrics that make this layer tunable. Without them, capacity decisions are guesses.

Unstarred

Dunk (exclude from study)

Model Serving

Review Questions

Todos

Sensemaker