Model Invocation

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

Layer Guide — M2: Model Invocation
What this layer is responsible for
Constructing the request, making the call, and interpreting the response. Everything above this layer is a composition of operations at this layer — an agent is a tool-calling invocation in a loop, a workflow is a sequence of invocations, a RAG system is an invocation with retrieved context assembled into it. Get the mental model here right and the layers above stop being separate topics.
The layer has four distinct concerns that are easy to conflate:
What goes in. Prompt construction, context assembly, and the budget that constrains both. How generation is controlled. Parameters governing length, randomness, and termination. What comes back, and why it stopped. Response structure, and stop reasons that carry meaning. What shape the interaction takes. Single-turn, conversational, structured-output, tool-calling, or streaming.
The two things worth internalizing before the pattern classes
The model is stateless. Every call carries the entire context — system instructions, all prior turns, retrieved passages, tool results. Nothing persists between calls. Conversation is an illusion maintained by resending history, which is why context grows with every turn, why long conversations get expensive, and why memory is a real engineering problem rather than a configuration setting.
The context window is a budget, and everything competes for it. System prompt, tool definitions, conversation history, retrieved context, the user's question, and the space reserved for the response all draw from one pool. Adding retrieval means less room for history. Adding tools means less room for both. Nothing is free, and when the budget is exceeded, behavior ranges from silent truncation to an error depending on where you exceeded it.

Pattern class 1 — Context assembly and budgeting
The problem in full
Deciding what to include in a request, in what order, within a fixed token budget — and doing it adaptively, because the inputs vary in size from request to request.
This is the most consequential and least-discussed concern in the layer. It determines cost per request, answer quality, and whether the system degrades gracefully or fails when inputs are large.
The variables:
Window size of the target model.
Required output length, which must be reserved.
Retrieved context volume, set by chunk size and top-k in B2b.
Conversation history length, which grows without bound.
Tool definitions, which consume tokens on every call when tools are configured.
Variability — a system whose inputs are usually small but occasionally huge needs adaptive handling.
The arithmetic
The budget equation:
window ≥ system_prompt + tool_definitions + conversation_history
       + retrieved_context + user_message + reserved_output
reserved_output is the part people forget. The window is shared between input and output; if you fill it with input, there is no room to generate, and you'll get truncation at best. maxTokens must fit in what remains, and the assembly logic must guarantee that.
A worked example for a conversational RAG system on a 200,000-token window:
   Component Tokens     System prompt 800   Tool definitions (6 tools) 2,400   Conversation history (10 turns) 6,000   Retrieved context (10 chunks × 500) 5,000   User message 150   Reserved output 2,000   Total 16,350   Comfortable. Now the same system after fifty turns, with hierarchical chunking returning 2,000-token parents:
   Component Tokens     System prompt 800   Tool definitions 2,400   Conversation history (50 turns) 30,000   Retrieved context (10 × 2,000) 20,000   User message 150   Reserved output 2,000   Total 55,350   Still fits, and costs roughly three and a half times as much per request. This is how generative systems get expensive without anyone changing anything — the growth is emergent, not deliberate.
The cost per request is dominated by input tokens in RAG systems, and the two largest terms are retrieved context and history. Both are controllable, and neither is controlled by default.
Subtypes of assembly strategy
Fixed assembly. Same structure every time, sized so the worst case fits. Simple, wasteful when inputs are usually small.
Budget-aware assembly. Compute available budget, allocate across components by priority, truncate the lowest-priority component that exceeds its allocation. The robust approach.
Adaptive retrieval. Vary top-k by remaining budget — fewer chunks when history is long.
History compaction. Summarize or drop old turns rather than truncating mid-message.
Multi-pass. When content genuinely exceeds the window, process in pieces and synthesize — map-reduce over chunks, or iterative refinement.
Decision variables
What gets cut first when the budget is tight? This is a priority ordering you must define explicitly: the system prompt and current user message are non-negotiable; retrieved context and history compete; tool definitions are fixed unless you filter tools dynamically. Most systems truncate history first, which is usually right and occasionally wrong when the conversation carries essential context the retrieval doesn't.
Where in the prompt does each component go? Order matters for two reasons. Models attend more reliably to material at the beginning and end of a long context than the middle, so burying critical instructions mid-prompt degrades adherence. And prefix caching only works on a prefix, so anything you want cached must come first and must be stable.
The implied layout: stable content first (system prompt, tool definitions, static context — cacheable), then variable content (retrieved passages, history), then the current question last.
Do you count tokens, or estimate? Estimating by character count is fast and wrong by enough to matter at the margin. Counting properly requires a tokenizer matching the model. Systems that estimate need generous safety margins, which wastes budget.
AWS mechanisms
The Converse API structures the request explicitly, which makes assembly tractable:
system — a list of system content blocks, separate from the conversation.
messages — an ordered list of {role, content} where role alternates user and assistant, and content is a list of blocks that may be text, images, documents, tool uses, or tool results.
inferenceConfig — maxTokens, temperature, topP, stopSequences.
additionalModelRequestFields — provider-specific parameters not in the common surface, such as topK for models that expose it.
toolConfig — tool specifications and choice policy.
guardrailConfig — guardrail identifier, version, and trace setting.
This unified structure is the practical reason to prefer Converse over InvokeModel: the assembly code is model-agnostic, so changing models is a configuration change rather than a rewrite of prompt construction.
The usage block in every response returns inputTokens, outputTokens, and totalTokens, plus cache read and write counts when prompt caching is used. Persisting this per request is how you build real cost accounting at a granularity billing data never reaches — per feature, per tenant, per prompt version.
Bedrock Prompt Management lets the prompt template live outside code as a versioned resource, invoked by passing its ARN in place of a model identifier with promptVariables for substitution. The assembly logic still lives in your code; what moves is the template text.
Knowledge Bases RetrieveAndGenerate does assembly for you — retrieval, prompt construction, and invocation in one call. Less control, less code. Retrieve returns chunks and leaves assembly to you, which is the right choice when the budget logic is non-trivial or the model isn't in Bedrock.
The details that decide success
Reserve output space explicitly and verify it. The most common assembly bug is filling the window with input and discovering the response is truncated at generation time.
Truncate at semantic boundaries. Cutting history mid-message, or a retrieved chunk mid-sentence, produces confusing context. Drop whole turns and whole chunks.
Put the question last. Recency matters for instruction adherence, and a question buried above ten thousand tokens of context gets followed less reliably.
Label the components. Delimiting retrieved context explicitly — with markers, or as document content blocks — helps the model distinguish source material from instructions and from conversation, and it makes grounding checks more reliable.
Measure token distribution in production. The p50 and p99 of input tokens tell you whether your budget assumptions hold. A p99 near the window limit means occasional failures you haven't seen yet.
Assembly is where cost is decided. Every optimization elsewhere is secondary to not sending tokens you didn't need. Reducing top-k from 10 to 5, or compacting history at 20 turns instead of 50, are single-line changes with immediate proportional effect.
Cost model
Input tokens at the model's input rate, on every request. In RAG systems this typically dominates output cost, because retrieved context is large and answers are short.
Levers, in order of usual impact: retrieved context size (k × chunk size); conversation history policy; prefix caching for stable portions; tool definition count; and system prompt length, which is small per request but multiplied by every request.
Failure modes and diagnostics
Silent truncation. Input exceeds the window and is cut, or output is truncated because insufficient space was reserved. Presents as answers that stop mid-sentence or ignore the end of the context. Check stopReason for max_tokens.
Context creep. Input tokens grow over weeks as history policies and retrieval settings drift. Presents as cost rising without traffic rising. Diagnose from token metrics over time.
Lost-in-the-middle. Critical instructions buried in a long context are followed inconsistently. Fix by moving them to the start or end.
Cache never hitting. Variable content placed before stable content, so no reusable prefix exists.
Estimation error. Character-based token estimates diverging enough to overflow the window on unusual inputs.

Pattern class 2 — Single-turn completion
The problem in full
One request, one response, no history. Classification, summarization, extraction, rewriting, translation. The simplest pattern and the foundation of the others.
The variables:
Determinism requirement — must the same input produce the same output?
Output length and how it's bounded.
Termination condition — how you know the response is complete.
Output format requirements.
Decision variables — the generation parameters
These control how tokens are sampled, and understanding what each does prevents the common error of reaching for the wrong one.
At each step the model produces a probability distribution over the vocabulary. The parameters shape how a token is chosen from it.
temperature scales the distribution's sharpness before sampling. Low values concentrate probability on the most likely tokens, producing focused, repetitive, predictable text. High values flatten the distribution, producing varied and more erratic text. Temperature 0 approximates greedy selection — always the most likely token — though in practice exact reproducibility isn't guaranteed even at 0, because floating-point non-determinism and batching effects introduce variation. Design for approximate rather than exact determinism.
topP (nucleus sampling) restricts sampling to the smallest set of tokens whose cumulative probability exceeds P. Adaptive: where the model is confident, few tokens qualify; where it's uncertain, many do. Generally a better diversity control than temperature alone.
topK restricts sampling to the K most likely tokens regardless of their probabilities. Blunter than topP, and exposed through additionalModelRequestFields on models that support it rather than in the common inference config.
maxTokens is a hard ceiling on output. It does not make the model conclude — it cuts generation off. A response truncated by maxTokens ends mid-thought, and you detect this from the stop reason, not from the text.
stopSequences halt generation when a specified string is produced. This is the deterministic, structural control over where output ends, as distinct from the probabilistic parameters that affect what gets generated. Use it when the output has a known terminator — a closing delimiter, a section marker, a role label in a transcript format you're constraining.
The distinction that matters: temperature, topP, and topK influence what is generated; stopSequences and maxTokens determine when generation ends. And putting "stop after this phrase" in the prompt is an instruction the model may or may not follow — a request, not a control. This mirrors the general principle that enforcement lives outside the model.
AWS mechanisms
Converse with inferenceConfig carrying maxTokens, temperature, topP, and stopSequences; provider-specific parameters through additionalModelRequestFields.
InvokeModel with the provider's native body format — required for embeddings and image generation, which don't fit the conversational message shape.
The response structure from Converse:
output.message — the assistant's message with its content blocks.
stopReason — why generation ended. The values carry real meaning and must be handled:
end_turn — the model finished naturally. The normal case.
max_tokens — truncated by your limit. The response is incomplete, and treating it as complete is a bug.
stop_sequence — a stop sequence was produced.
tool_use — the model wants to call a tool. Central to pattern class 5.
guardrail_intervened / content_filtered — policy blocked it.
usage — token counts.
metrics — latency.
Always branch on stopReason. Systems that read output.message and ignore why generation stopped will silently serve truncated answers and misinterpret tool-use responses as final answers.
The details that decide success
Change one parameter at a time. Temperature and topP interact; adjusting both simultaneously makes their effects uninterpretable. Most tasks are well served by setting one and leaving the other at default.
Low temperature for extraction and classification, higher for creative generation. Obvious but frequently unset, leaving defaults tuned for conversation applied to tasks needing determinism.
maxTokens is a cost control as well as a safety valve. Output tokens are typically the most expensive per unit, and an unbounded maxTokens on a task that should produce a short answer occasionally produces a very long one.
Approximate determinism is achievable; exact is not. For workflows requiring reproducibility — audit, testing, regulatory evidence — the answer is capturing inputs and outputs, not relying on the model to reproduce them.
Cost model
Input tokens plus output tokens, with output typically several times more expensive. The direct levers are maxTokens and prompt design that elicits appropriately concise responses.
Failure modes and diagnostics
Truncation treated as completion. stopReason ignored. Presents as answers cut off mid-sentence.
Unwanted variability. Temperature too high for a task needing consistency.
Repetitive or degenerate output. Temperature too low, or an unfortunate topP.
Runaway generation. No maxTokens, an unusual input, and a very long expensive response.
Stop sequence appearing in legitimate content. Generation halts early because the terminator occurs naturally in the output.

Pattern class 3 — Multi-turn conversation
The problem in full
Maintaining a coherent exchange across turns, given a stateless model. The application resends the history each time, which means history management — what to keep, what to compact, what to drop — is an application responsibility with direct cost and quality consequences.
The variables:
Expected conversation length.
Whether early context stays relevant — some conversations reference the opening throughout, others move on.
Cost tolerance, since history is resent on every turn.
Whether history must be retained beyond the session, for audit or personalization.
Subtypes
Full history. Resend everything. Correct for short conversations, unsustainable for long ones.
Sliding window. Keep the last N turns. Simple, and loses early context abruptly.
Summarized history. Compact older turns into a running summary, keep recent turns verbatim. Preserves gist at much lower token cost.
Selective retrieval over history. Treat past turns as a corpus and retrieve the relevant ones. Powerful for long-running relationships, and effectively RAG over conversation.
Structured state extraction. Rather than keeping conversational text, extract the facts established — entities, preferences, decisions — and carry those forward compactly.
Decision variables
How does cost grow? With full history, each turn resends everything before it, so cumulative token cost over a conversation grows quadratically with turn count. This is the fact that makes history policy a first-order concern rather than a detail — a fifty-turn conversation with full history costs vastly more than fifty independent questions.
When does early context stop mattering? Task-dependent. A troubleshooting conversation usually needs the original problem statement throughout. A wide-ranging exploration usually doesn't. Sliding windows suit the second and break the first.
Where does compaction happen? Summarizing history costs a model call. Doing it every turn is expensive; doing it on a threshold — every N turns, or when tokens exceed a bound — amortizes it.
AWS mechanisms
The messages array in Converse, with alternating user and assistant roles. Assembly is yours; the API just carries what you send.
sessionId on RetrieveAndGenerate for managed multi-turn in Knowledge Bases — Bedrock maintains the session context, which removes the work and the control.
DynamoDB as the history store: partition by session or user, sort by timestamp, TTL for expiry. The 400 KB item limit means long conversations store per-message rather than as one document, or spill to S3.
AgentCore Memory for managed short-term and long-term memory with built-in strategies — semantic facts, user preferences, and running summaries — which is the managed answer to this pattern class for agents.
ElastiCache when history must be read with very low latency on every turn.
Bedrock invocation for the summarization step in compaction, ideally with a small cheap model since summarizing a transcript doesn't need a frontier model.
The details that decide success
Store more than you send. Persist the full conversation for audit, analytics, and evaluation; send a compacted view to the model. Conflating the two loses data you'll want later.
Compaction is lossy and should be tested. A summarization prompt that drops the user's actual goal produces a conversation that quietly goes off the rails at turn thirty. Test by checking whether the compacted context still supports answering questions about earlier turns.
Include enough to resolve references. "Do that for the other one too" requires knowing what "that" and "the other one" refer to. Aggressive compaction breaks anaphora, and the failure is subtle.
Watch the interaction with retrieval. Both history and retrieved context consume the budget. Long conversations should retrieve less, or the assembly will overflow. This coupling is why budget-aware assembly matters.
TTL is a privacy control. Retention limits on conversation data are expressed natively through DynamoDB TTL rather than through a cleanup job.
Cost model
Cumulative token cost grows quadratically with turn count under full history. Compaction converts this to roughly linear at the cost of periodic summarization calls.
The calculation worth doing: average conversation length × average tokens per turn, under each policy. The difference between full history and summarized history at fifty turns is usually an order of magnitude.
Failure modes and diagnostics
Quadratic cost growth. Presents as cost per conversation far exceeding cost per question.
Context overflow on long conversations. Presents as errors or degraded answers late in sessions.
Lost early context. A sliding window drops the original goal; the assistant drifts.
Broken references after compaction. Pronouns and demonstratives no longer resolve.
History leaking across sessions or users. A session-key bug with serious privacy consequences.

Pattern class 4 — Structured output
The problem in full
Downstream code needs a specific data structure, not prose. Extraction into fields, classification into an enum, routing decisions, anything feeding a database or an API.
Asking for JSON in the prompt mostly works and fails unpredictably: markdown fences appear, fields are renamed, optional fields vanish, an explanatory sentence precedes the object. Parsing failures then occur at a low rate, which is worse than a high rate because it evades testing.
The variables:
Schema complexity — flat fields, nested objects, arrays, unions.
Failure tolerance — is a parse failure recoverable or user-visible?
Volume, which determines whether a low failure rate is acceptable in aggregate.
Whether fields may be absent and how that's expressed.
Subtypes
Prompted JSON. Ask for JSON, parse, retry on failure. Simple, imperfect.
Tool-based schema enforcement. Define a tool whose input schema is the structure you want, force the model to call it, and read the arguments. The model's tool-calling training makes it adhere to schemas far more reliably than free-form instruction.
Prefill or delimiter constraint. Begin the assistant's response with an opening brace, or use a stop sequence at the closing one, to bound the output structurally.
Validate-and-repair. Parse, validate against a schema, and on failure send the errors back for correction.
Two-stage. Generate prose, then a second call extracts structure from it. More calls, sometimes better quality when the reasoning and the formatting conflict.
Decision variables
Use tool-calling for schema adherence. This is the single most useful technique in this pattern class and it's under-used because the feature is named for a different purpose. Defining a tool with your target schema and setting toolChoice to require it turns structured output into the well-trained tool-calling path. Adherence is substantially better than prompted JSON.
How much reasoning does the task need before the structure? Forcing immediate structured output can degrade quality on tasks needing deliberation, because there's no room to think. Either allow a reasoning field within the schema, or use the two-stage approach.
What happens on failure? A retry with the validation errors included usually succeeds. Design the retry rather than letting the exception surface.
AWS mechanisms
toolConfig in Converse:
tools — a list of toolSpec entries, each with name, description, and inputSchema expressed as JSON Schema.
toolChoice — auto (model decides), any (must call some tool), or a specific tool by name (must call that one). Specifying a tool by name is the schema-enforcement mechanism.
The response arrives as a toolUse content block with toolUseId, name, and input — the input being your structured data, already parsed as an object.
stopSequences to bound output where a delimiter approach is used.
Guardrails with a schema-validating downstream step, since guardrails check content policy rather than structure.
Application-side validation with a JSON Schema library, feeding errors into a repair call.
The details that decide success
Descriptions in the schema matter as much as types. The model reads field descriptions; a well-described schema produces better extraction than a bare one. Treat the schema as prompt text.
Make optionality explicit. Ambiguity about whether a field may be absent produces inconsistent behavior — sometimes null, sometimes omitted, sometimes an invented value. Say so in the description.
Constrain enums in the schema, not the prompt. Enumerated values in the schema are respected more reliably than the same list in instructions.
Always validate. Schema adherence via tool-calling is much better, not perfect. Validate and handle failure explicitly.
Watch for truncation. A large structured output hitting maxTokens produces invalid JSON. Check stopReason before parsing — this is a concrete case where ignoring stop reason produces a confusing downstream error.
Cost model
Tool definitions consume input tokens on every request, which is a real cost when many tools are defined. Retries on validation failure cost a full additional invocation. Two-stage approaches double the calls.
Failure modes and diagnostics
Intermittent parse failures. Prompted JSON with a low failure rate that evades testing and surfaces in production.
Truncated structure. maxTokens reached mid-object.
Schema drift. Fields renamed or types varying between calls, usually from an under-specified schema.
Degraded reasoning quality. Forcing immediate structure on tasks needing deliberation.
Hallucinated enum values. Values outside the permitted set, when the constraint lived in the prompt rather than the schema.

Pattern class 5 — Tool invocation
The problem in full
The model needs to do something it cannot do by generating text: look up current data, perform a calculation, call an API, query a database. Tool calling is the protocol by which a model requests an action and receives the result.
This pattern class is the foundation of agents, and understanding it as a protocol at the invocation layer — rather than as an agent feature — is what makes the orchestration layer comprehensible. An agent is this loop, run until the model stops asking for tools.
The variables:
Number of tools, which affects both token cost and selection accuracy.
Tool execution latency and reliability.
Whether tools have side effects, which changes the safety posture entirely.
Loop bounds — how many iterations are permitted.
Parallelism — can multiple tools be called at once?
The protocol, precisely
This is worth stating step by step because everything agentic builds on it:
You send a request including toolConfig with tool specifications.
The model responds with stopReason: "tool_use" and one or more toolUse content blocks, each containing a toolUseId, the tool name, and an input object matching the tool's schema.
You execute the tool. The model does not call anything — it emits a request, and your code decides whether and how to fulfill it. This is the central security fact of the pattern.
You append the assistant's message to the conversation, then append a new user message containing toolResult blocks, each referencing the toolUseId and carrying the result content and a status.
You call the model again with the extended conversation.
The model either responds with a final answer (stopReason: "end_turn") or requests more tools, returning to step 2.
The loop terminates when the model stops requesting tools, or when your bound is reached.
Decision variables
How many tools? Every tool definition consumes input tokens on every request, and selection accuracy degrades as the count grows — a model with fifty tools chooses worse than one with five. Beyond a modest number, the answer is filtering tools by relevance before the call rather than presenting all of them.
Are the tools read-only or do they mutate state? Read-only tools are relatively safe to expose. Tools with side effects — sending email, modifying records, spending money — require deliberate design: scoped permissions, confirmation for consequential actions, and idempotency. The model may be influenced by content it reads, so tool exposure defines the blast radius of a prompt injection.
What bounds the loop? An unbounded loop can iterate indefinitely, each iteration costing a model call with a growing context. A maximum iteration count is mandatory, not optional, and hitting it needs a defined behavior.
Parallel or sequential? Models can request multiple tools in one response when the calls are independent. Executing them concurrently reduces latency materially.
AWS mechanisms
toolConfig in Converse as described in pattern class 4, with toolChoice: auto for genuine tool use.
toolResult content blocks carrying toolUseId, content (text or JSON), and status (success or error). Returning errors as tool results rather than throwing lets the model adapt — it can retry with different arguments or explain the failure to the user, which is usually better than an exception propagating to the caller.
Lambda as the typical tool implementation, invoked by your loop code.
AgentCore Gateway converts existing APIs, Lambda functions, and MCP servers into tools behind one endpoint, and provides semantic tool search to address the tool-count problem — surfacing only relevant tools rather than presenting the whole catalog.
Bedrock Agents implements the loop for you with action groups, which is the managed alternative to writing it yourself.
Guardrails applied to the invocation, noting that tool results enter the context and are therefore content the model reads — and content an attacker may control if the tool reads external data.
The details that decide success
You are the enforcement point, not the model. The model requests; your code decides. Validate tool arguments, check authorization for the user rather than the application, and refuse requests that shouldn't be honored. Treating the model's request as authorization is the fundamental agentic security error.
Scope permissions to the user. If the tool queries a database, it should query with the calling user's entitlements. Otherwise the model can be induced to retrieve data the user can't access — which connects this pattern directly to the identity layer.
Tool results are untrusted input. A tool that fetches a web page or reads a document returns content that may contain instructions. This is the primary prompt injection vector in agentic systems, and the mitigations are guardrails on the assembled context and, more importantly, permissions narrow enough that a successful injection can't do much.
Write good tool descriptions. Selection accuracy depends heavily on them. Describe when to use the tool, not just what it does, and be explicit about what it doesn't do.
Handle the max_tokens case mid-loop. A truncated tool-use response produces malformed arguments.
Log the full trajectory. Debugging a tool-calling system means seeing which tools were selected, with what arguments, returning what results, in what order. Without this, failures are inscrutable — and it's why agent observability is a distinct concern from conventional tracing.
Cost model
Each iteration is a full invocation with the accumulated conversation, so context grows with every tool call. A five-iteration loop costs substantially more than five times a single call, because each iteration carries everything before it.
Tool definitions add fixed input tokens per call. Filtering tools reduces this.
The cost of an agentic request is inherently variable, which makes budgeting harder than for fixed pipelines — and is one of the honest arguments for using a workflow when the sequence is known.
Failure modes and diagnostics
Runaway loops. No iteration bound, or the model repeatedly calling the same tool. Presents as high latency and cost. Bound it, and detect repetition.
Wrong tool selected. Poor descriptions or too many tools. Diagnose from trajectory logs.
Malformed arguments. Schema under-specified, or truncation.
Prompt injection via tool results. Content read by a tool influencing subsequent behavior.
Over-privileged execution role. The blast radius of any of the above.
Silent tool failures. An error swallowed and returned as success, so the model reasons on a false premise.

Pattern class 6 — Streaming emission
The problem in full
Generation takes seconds to tens of seconds. Delivering the whole response at once means the user stares at nothing throughout. Streaming emits tokens as they're produced.
Streaming does not make generation faster. It reduces time to first token, which is what users experience as responsiveness.
The variables:
Whether a human is watching.
Whether downstream processing needs the complete response before acting.
Transport capability at the edge.
How errors mid-stream are handled.
Decision variables
Is a human watching in real time? If yes, stream. If the consumer is code that needs the whole response — parsing structured output, for instance — streaming adds complexity for no benefit.
Can partial output be shown safely? Streaming means the user sees text before any output-side guardrail has evaluated the complete response. For applications where a policy violation appearing briefly is unacceptable, this is a genuine tension between responsiveness and control.
What happens if the stream fails mid-response? The user has a partial answer and the connection dropped. The application needs a defined behavior — retry, mark incomplete, or discard.
AWS mechanisms
ConverseStream returns an event stream. The event types:
messageStart — role information.
contentBlockStart — a content block begins, including tool-use blocks.
contentBlockDelta — incremental content. This is the event carrying text to render.
contentBlockStop, messageStop — with the stop reason.
metadata — usage and latency, arriving at the end.
Note that usage arrives in the final metadata event, so token accounting in streaming requires consuming the stream to completion rather than abandoning it when the user navigates away — a subtle source of missing cost data.
InvokeModelWithResponseStream as the provider-native equivalent.
Transport options, which belong to the edge layer but are chosen together with this: API Gateway WebSocket for interactive sessions, Lambda function URLs with response streaming for the simplest single-response case, AppSync subscriptions for GraphQL clients.
The details that decide success
Time to first token is the metric. Measure it separately from total latency. It's what determines perceived responsiveness and it's affected by different things — prompt length drives prefill, which drives TTFT.
Buffer for rendering. Emitting every token individually to a browser produces excessive network chatter. Small buffering windows improve efficiency imperceptibly to the user.
Tool use arrives in stream form too. A streaming response may contain tool-use blocks assembled across deltas, which the loop must reconstruct before executing.
Guardrail interaction needs a decision. Output-side policy evaluation on a complete response conflicts with streaming it. Some designs stream optimistically and retract on violation; others buffer. Choose deliberately rather than discovering the behavior.
Consume the stream fully for accounting, per the metadata note above.
Cost model
Identical token pricing. The operational difference is connection duration — WebSocket connections bill by connection-minute plus messages, which is a small but real cost for long-lived sessions.
Failure modes and diagnostics
Abandoned streams losing usage data. Cost accounting under-reports.
Mid-stream failures with no defined handling. Users left with partial answers.
Policy violation visible before retraction.
Streaming used where the consumer is code, adding assembly complexity for nothing.

Pattern class 7 — Prefix reuse
The problem in full
Many requests share a large identical opening — a long system prompt, a fixed set of tool definitions, a document being asked about repeatedly, few-shot examples. Reprocessing that prefix on every request costs prefill compute and input tokens.
Prompt caching stores the processed prefix so subsequent requests skip it, reducing both latency and cost.
The variables:
Size of the shared prefix. Below some threshold it isn't worth caching.
Request frequency, since caches expire.
How stable the prefix is.
Whether the prompt can be ordered with stable content first.
Decision variables
Is there a substantial stable prefix? The saving is proportional to its size. A 200-token system prompt isn't worth it; a 20,000-token document being asked about repeatedly clearly is.
Can you order the prompt correctly? Only a prefix can be cached. If variable content — retrieved passages, conversation history — sits before the stable content, nothing is cacheable. This frequently requires restructuring prompt assembly, and it's the most common reason caching doesn't work.
Is request frequency high enough? Caches have a lifetime. Sporadic requests may miss consistently, producing cache-write costs with no reads.
AWS mechanisms
Bedrock prompt caching using cache point markers in the request, indicating where a cacheable prefix ends. Cached input tokens are billed at a reduced rate; a cache write may carry a premium. The usage block reports cache read and write token counts, which is how you verify it's working.
Support varies by model, so this is a model-selection consideration.
Application-level caching for a different problem: caching complete responses to repeated queries. Exact-match caching in ElastiCache or DynamoDB works for identical questions; semantic caching — embedding the query and returning a cached response for a sufficiently similar prior query — catches paraphrases and is implementable with a vector store fronting the model.
Note these are different mechanisms at different layers. Prompt caching reduces the cost of processing a request; response caching avoids the request entirely.
The details that decide success
Structure the prompt for caching from the start. Stable content first: system prompt, tool definitions, static documents, few-shot examples. Then variable content. Then the question. Retrofitting this ordering is more disruptive than designing for it.
Verify with the usage block. Cache read token counts confirm hits. A system configured for caching with zero cache reads has an ordering problem.
Response caching needs an invalidation story. If underlying documents change, cached answers go stale with no error. Tie cache keys to a corpus version, or use short TTLs.
Semantic caching has a similarity threshold to tune. Too loose and you return answers to different questions; too tight and hit rates collapse.
Don't cache personalized or non-deterministic responses. If the answer depends on who's asking or is expected to vary, caching is wrong.
Cost model
Prompt caching: reduced rate on cached input tokens, against a possible write premium. Savings scale with prefix size × request rate.
Response caching: a full inference avoided per hit — the largest possible saving per request — against cache infrastructure cost and staleness risk.
Failure modes and diagnostics
Cache never hitting. Wrong ordering, or frequency below expiry. Diagnose from usage token counts.
Stale cached responses. No invalidation on corpus change.
Semantic cache returning wrong answers. Threshold too loose.
Cache infrastructure exceeding the savings at low volume.

Pattern class 8 — Non-text invocation
The problem in full
Embeddings, image generation, and multimodal inputs use the same service and different call shapes. The conversational API doesn't fit all of them.
Subtypes and mechanisms
Embedding invocation. InvokeModel with a provider-specific body — Titan takes inputText with optional dimensions and normalize; Cohere takes texts with an input_type distinguishing search_document from search_query. That input type must match between ingestion and query, and a mismatch degrades retrieval silently. There's no conversational shape here, so Converse doesn't apply.
Image generation. InvokeModel with a provider body specifying prompt, negative prompt, dimensions, and generation parameters. Returns base64-encoded images. Titan Image Generator embeds an invisible watermark with a corresponding detection capability, which is the mechanism for verifying machine origin.
Multimodal input. Converse supports image and document content blocks within messages, so a request can carry text and images together. Images consume input tokens as a function of resolution — a high-resolution image can cost more than a page of text, which makes image sizing a cost decision.
Video and audio generally route through the purpose-built services — Transcribe, Rekognition — or through frame extraction feeding multimodal invocation, rather than being handled natively at this layer.
The details that decide success
Directionality. Generation models generate; they do not analyze. Using an image generation model for classification or description is a capability error, not a quality trade-off. Image understanding is Rekognition for known categories or a multimodal model for open-ended interpretation.
Resize images before sending. Token cost scales with resolution, and most tasks don't need full resolution.
Embedding calls are throttled like any other invocation, and a corpus embedding pass will hit limits. Batch inference sidesteps this.
Embedding output is deterministic, which makes it perfectly cacheable by content hash.
Failure modes and diagnostics
Input type mismatch on asymmetric embedding models. Uniformly mediocre retrieval with no visible error.
Image token cost surprise. High-resolution images inflating input costs.
Generation model used for analysis.

Cross-cutting decisions in this layer
Converse or InvokeModel? Converse for anything conversational, for the model portability and the uniform assembly code. InvokeModel where the modality doesn't fit — embeddings, image generation.
What is the context budget policy, and what gets cut first? Define it explicitly with a priority ordering, implement it as budget-aware assembly, and reserve output space.
Is stopReason handled everywhere? Truncation and tool-use are both signalled there and both are bugs when ignored.
Is usage persisted? It's the foundation of real cost attribution and it's free to capture.
Where do prompts live? Inline strings are fine for one application; versioned resources matter when prompts are iterated by non-developers or must be auditable.
Is the prompt ordered for caching? Stable first, variable second, question last. Nearly free if done from the start.
Interfaces to adjacent layers
From M1 (Serving). Serving determines which invocation modes exist: streaming requires a serving path that supports it, batch requires the batch API, provisioned capacity is addressed differently. Conversely, invocation parameters drive serving behavior — prompt length drives prefill and therefore KV cache pressure; maxTokens bounds decode duration.
From C1 (Retrieval). Retrieved context enters the budget here, and retrieval's k and chunk size — decided back in B2b — are the largest input-cost term in most RAG systems.
From C2 (State). Conversation history enters the budget here, and the history policy is a cost decision made at this boundary.
To S1 (Safety). Guardrails attach to the invocation and evaluate the assembled request and the response. Tool results entering the context are content the policy layer should see.
To O1 (Orchestration). The tool-calling protocol in pattern class 5 is the agent primitive. An agent is this loop with a termination condition; a workflow is a sequence of invocations with the sequence fixed in advance. Orchestration is composition over this layer, which is why this layer comes first.
To X8 (Cost). Nearly every cost lever in a generative system is expressed here: token counts, cache hits, output length, and the assembly decisions that determine input size.

Unstarred

Dunk (exclude from study)

Model Invocation

Review Questions

Todos

Sensemaker