Building Production-Grade Agentic Systems on AWS

Thoughts on building production agentic systems on AWS.

The $40,000 Weekend

A team I know shipped a "lightweight research assistant" on Lambda. One tool, one prompt, no recursion_limit. On a Friday afternoon, a malformed input sent the agent into a loop. It kept calling its summarization tool against its own output. By Monday morning, the bill had crossed $40,000. The kill switch was an AWS Support ticket.

The agent worked as designed. The architecture did not.

That story has a hundred versions. Too many tools in the prompt. No prompt cache. A judge grading its own model family. "Just one more sub-agent." Each one looks small in isolation. Together, they produce the same failure: a clean demo that breaks expensively once real users arrive.

This is the path I would take in 2026 to avoid that outcome. It compresses a 213-KB research doc into the choices that matter, using a 9-layer reference architecture you can adapt to most serious agent products.

TL;DR: The Defaults I Would Start With

For a senior engineer building an agent product that needs to survive more than a quarter, these are the defaults I would write down first. Every default needs a clear reason to change.

Compute: Use AgentCore Runtime for user-facing agents that need sessions longer than 15 minutes, per-tenant isolation, or active-only billing. Use Lambda for stateless, sub-15-minute, low-QPS tool callers.
Framework: Use Strands Agents wrapped with BedrockAgentCoreApp. Stay single-agent until eval data proves a split is worth it. Use LangGraph when the flow is a real state machine with durable checkpoints and time travel.
Cognitive pattern: Start with ReAct, hard step caps, and summarization. Add Plan-and-Execute for batch or research jobs. Add Reflection only when you have a measurable rubric. Use CodeAct for data analysis behind a real sandbox. Avoid Tree-of-Thoughts in production.
Models: Use Sonnet 4.6 as the workhorse. Route cheaper steps to Haiku 4.5, Nova Pro, or Nova Lite. Keep Opus 4.7 for hard reasoning checkpoints. Aim to route about 80% of agent steps through cheaper paths, then escalate on confidence.
Memory: Use AgentCore Memory for short-term state, summaries, and user preferences. Add episodic memory when volume justifies it. Drop to DynamoDB + OpenSearch only when you need custom extraction or sub-50 ms p99.
Tools: Use AgentCore Gateway as the canonical MCP surface once you have more than one agent or framework. Past 30 tools in one prompt, semantic tool retrieval is required.
Data plane: Use Bedrock Knowledge Bases on OpenSearch Serverless with Titan Embed v2 and Amazon Rerank 1.0. Turn on hybrid search. Apply contextual retrieval when the budget allows because the retrieval lift is large enough to matter.
Observability + eval: Use AgentCore Observability and self-hosted Langfuse on the same OTel stream. Put Ragas + DeepEval in CI. Use Bedrock Model Evaluation for managed regression jobs. Do not judge with the same model family you ship.
Security: Treat Bedrock Guardrails as defense-in-depth. The real control is structural: do not let private data, untrusted content, and external communication meet in one agent path. Use AgentCore Identity for credentials and cell-based isolation for regulated tenants.
Cost ceilings: Set them before launch. Use a $0.50 per-session hard cap, $5 per user per day, and a kill switch wired to Cost Anomaly Detection. Missing recursion_limit is still the most expensive bug in this class of system.

When Agents Pay Off

The useful question is not "should this be an agent?" The useful question is:

Is the cheapest deterministic alternative actually worse?

Agents earn their cost when the task has uncertainty you cannot enumerate, when the model needs to compose three or more different tools, when the state space is too large for a hand-written workflow, and when tool feedback helps the model recover from errors.

They do not earn their cost for deterministic ETL, pure RAG Q&A with no actions, one-shot classification, sub-second responses at extreme TPS, or compliance regimes that punish nondeterminism. Use Step Functions, Bedrock Knowledge Bases, or a fine-tuned small model for those jobs.

A customer-facing research or support assistant sits on the agent side when the answer needs multiple private systems, a few external sources, user-specific context, and judgment about which tool to call next. You can build a scheduled summary or ticket triage job as a Step Functions DAG. That DAG stops being useful when the user asks a follow-up that changes the plan halfway through.

Half the products marketed as agents in 2026 should be Step Functions with a system prompt. The other half need the layers below.

The 9 Layers

Layer 1: Compute / Runtime

Agents need a place to run where the loop can last minutes or hours, hold per-session state, isolate tenants at the kernel boundary, and bill for work instead of waiting time. Lambda was not built for that shape.

My default is AgentCore Runtime. It gives you per-session Firecracker microVM isolation, 8-hour lifetimes, and active-consumption billing in one managed surface. Pricing is $0.0895 per vCPU-hour + $0.00945 per GB-hour, with a 1-second minimum and 128 MB memory floor. CPU is billed only while the process is executing, so the 60-80% of wall-clock time spent waiting on models or APIs is not charged. Plan around the hard limits: 15-minute sync, 60-minute streaming, 8-hour async, 25 TPS per agent, 1,000 concurrent sessions in IAD/PDX, and 15 GA regions as of May 2026.

The tradeoff is control. Bedrock Agents runs the loop for you on a fixed model catalog. AgentCore makes you write the loop, then gives you compute, isolation, identity, memory, and gateway. You give up the console wizard, but you get model-agnostic deployment, longer sessions, and per-session microVMs.

Use Lambda for a 200-millisecond stateless tool call. Do not use it for a research session that can run beyond 15 minutes. That limit will not hurt in a demo. It will hurt the first time a user asks a broad question in production.

For a real customer-facing assistant, AgentCore Runtime fits when sessions can run 30+ minutes across retrieval, tool calls, and multi-turn dialogue. Each tenant or user gets stronger isolation, and most of the wall-clock time is spent waiting on APIs and models.

Layer 2: Orchestration Framework

The framework is the code surface where you define the agent, register tools, manage state, and emit traces. Pick the wrong one and you spend the next year paying abstraction tax.

I would start with Strands Agents. It has three primitives: model, system prompt, and tools. It is Apache-2.0, model-agnostic, OTel-native, and used in production by Kiro, Amazon Q, AWS Glue, and VPC Reachability Analyzer. The documented "40 lines of LangGraph to 3 lines of Strands" example is about the 80% case, not every case. Wrap Strands with BedrockAgentCoreApp and deploy it unchanged so Memory, Identity, Gateway, and Observability attach without glue code.

Strands assumes the model can drive the loop well. If you save money with a weaker model, you may spend it back on tool-call errors and recovery turns. LangGraph is the better choice when the flow is genuinely a state machine with branches, durable checkpoints, time travel, or human edit-and-continue. Do not pay for LangGraph's verbosity when the product is a normal ReAct loop.

Strands works when the conversational loop is model-driven: intent, tool, observation, reply. If a scheduled or replayable workflow appears later, that part can move to LangGraph. The interactive loop can stay on Strands.

Layer 3: Cognitive Pattern

Pattern choice affects token cost, latency, and quality more than the framework does.

Start with ReAct: reason, act, observe, repeat. Add hard step caps and summarization. Stay single-agent until evidence says otherwise. Anthropic's June 2025 multi-agent research system reported a 90.2% lift over a single agent at roughly 15x the token cost. That is the bar you need to clear before splitting. Most teams do not have an eval set strong enough to prove that lift.

Use Plan-and-Execute for batch or research tasks with stable sub-goals. Use Reflection only when you have a measurable rubric and a critic that does not prefer its own family. Use CodeAct for data-shaped tasks because pandas beats a pile of SQL tools, but put it behind AgentCore Code Interpreter, not exec(). Avoid Tree-of-Thoughts in production unless you have a domain like theorem proving where the 10-100x token bill is justified.

The single-agent path keeps context coherent and makes debugging easier. Multi-agent buys you per-domain tone, tools, and models, but it adds handoff loss, harder traces, and a much larger token budget. Shape the first version like a supervisor with agents-as-tools so you can split later without a rewrite.

The right day-one shape is usually a single ReAct agent with a supervisor-style boundary. Later, domain specialists can become separate tools if evals prove the split helps.

Layer 4: Model Layer

The model layer decides which foundation model executes each step. It is usually the biggest cost lever and the place where quality problems show up first.

My default is Bedrock managed inference with a cheap-first router. Use Haiku 4.5, Nova Pro, or Nova Lite for the common steps. Use Sonnet 4.6 for planning and tool-error recovery. Reserve Opus 4.7 for hard reasoning checkpoints. Turn on cross-region inference by default because it is free, smooths regional throttling, and is required for the newest models. Bedrock Intelligent Prompt Routing for Claude-to-Haiku and Nova-Pro-to-Lite can save up to 30% in practice, with 43.9% reported in the paper, at sub-150 ms overhead.

Prompt caching is the cost lever teams skip too often. Bedrock supports a default 5-minute TTL, with optional 1-hour TTL for Opus 4.5, Sonnet 4.5, and Haiku 4.5. The pricing pattern is simple: cache writes cost 25% more than normal input, and cache reads cost roughly 10% of input. One published production case reported 86% lower data-processing cost and 66% faster processing on stable document prefixes. Use Bedrock Batch for bulk evals, offline RAG indexing, and dataset prep when a 24-hour SLA is fine.

Cheap-first routing means your eval harness now has multiple paths to cover. A single model everywhere is simpler and more predictable, but usually costs 3-5x more. Route only when you can prove quality parity for each tier.

For a typical production assistant, I would use Nova Lite for intent classification, Sonnet 4.6 as orchestrator, Haiku 4.5 for cheap specialist steps, and Opus 4.7 only for offline or high-stakes reasoning checkpoints. Cache the system prompt, tool definitions, and ReAct history at the breakpoint. The target per-turn cost is about $0.0046 on a 6.2K-token cached prefix.

Layer 5: Memory Layer

Agents need working state, cross-session recall, and replay-grade audit. Without explicit memory, every session starts from zero and every long task becomes context-window roulette.

Start with AgentCore Memory for short-term memory, summaries, and user preferences. Add episodic memory once volume justifies it. The built-in strategies are UserPreferenceMemoryStrategy, SemanticMemoryStrategy, SummaryMemoryStrategy, and EpisodicStrategy. Per-actor namespacing with /{actorId}/... gives you tenant isolation by design. Pricing is $0.25 per 1,000 short-term events, $0.75 per 1,000 long-term records per month for built-in storage or $0.25 for override/self-managed, and $0.50 per 1,000 retrievals. AWS's worked example, 100k events + 10k long-term records + 20k retrievals, lands around $42.50/month. KMS at rest and 7-365 day expiry windows are available.

Long context does not replace RAG. A 1M-token window costs roughly 1,250x more and runs 30-60x slower than retrieval. Accuracy also drops by more than 20 percentage points when the key fact moves from the front to the middle. Put old turns into memory and retrieval, not the prompt window.

The dangerous shortcut is one shared vector index across tenants with metadata filters as the only isolation. Agents forget filters, attackers craft inputs, and cross-tenant leaks become possible. Enforce isolation through namespaces or indexes, not through a reminder in the prompt.

For most SaaS agents, I would use AgentCore Memory, a customer-managed KMS key, per-actor namespaces like /{user_id}/{strategy}/..., 90-day retention, and a working /forget endpoint that cascades DeleteMemoryRecord by actor_id. Keep the built-in strategies on, then add domain-specific memory only when you can name exactly what should be extracted.

Layer 6: Tools / Integration Layer

Tools are the agent's API surface. This layer determines tool count, auth, traces, reuse across agents, and lock-in.

Use AgentCore Gateway as the canonical MCP surface once you have more than one agent or more than one framework. You get one endpoint, IAM + JWT/OAuth at the edge, on-behalf-of tokens downstream, semantic tool search, and CloudWatch traces. Pricing is $0.005 per 1,000 invocations, $0.025 per 1,000 search API calls, and $0.02 per 100 tools per month for indexing. AWS's HR-assistant example, 50M interactions per month across 200 tools, lands around $2,250/month. An in-process registry is fine for one team and fewer than 20 tools. Move to Gateway when a second consumer or non-Python caller appears.

Past 30 tools, dumping every definition into the prompt falls apart. RAG-MCP (Gan & Sun, 2025) measured all-tools-in-prompt at 13.62% correct selection. Retrieval-augmented tool selection hit 43.13%. Same tool set, different visible subset. Each tool definition costs 200-500 tokens, so 50 tools burn 10-25K tokens before the user even asks a question.

Do not put raw tokens in tool inputs or outputs that the model can see. Anything in context can appear in output. AgentCore Identity's token vault exists so credentials stay out of the LLM context entirely.

In a general SaaS product, Gateway becomes the MCP front door for CRM, billing, ticketing, docs, data warehouse, and internal APIs. Per-domain specialists keep each prompt under six tools. Semantic tool search can stay disabled until the surface reaches roughly 25 tools, but the architecture should be ready for it.

Layer 7: Data Plane (RAG)

Models do not know your private data. Long context is expensive and lossy. Fine-tuning to inject facts produces hallucinated variants. The data plane gives the agent grounded, citable, freshly updated knowledge.

Start with Bedrock Knowledge Bases on OpenSearch Serverless, Titan Embed Text v2 at 1024 dimensions, and Amazon Rerank 1.0. Turn on hybrid search with dense retrieval, BM25, and RRF fusion. Default chunking should be sentence-aware, roughly 300 tokens, with 15% overlap. Rerank whenever you retrieve more than 10 docs. Skipping rerank leaves too much recall behind.

Apply Anthropic's contextual retrieval when the index budget allows. Prepend an LLM-generated summary to each chunk before embedding. The reported numbers are large: 35% fewer failures with contextual embeddings alone, 49% with contextual BM25 added, and 67% with reranking on top. At roughly $1.02 per million document tokens with prompt caching, it is usually a one-time index cost that pays back quickly.

Dense-only search misses exact matches like error codes, SKU IDs, and names. Pure BM25 misses paraphrase. Hybrid search costs one extra index and one extra query branch, but it recovers accuracy that dense-only systems silently lose.

For a multi-tenant SaaS agent, I would use S3 Vectors or OpenSearch namespaces per tenant for private knowledge, plus Aurora PostgreSQL for transactional state like accounts, permissions, subscriptions, and scheduled jobs. Apply contextual retrieval at ingest. Run Cohere Rerank 3.5 on every query: top 20 in, top 5 out.

Layer 8: Observability + Eval

Classic APM assumes deterministic services. Agents break that assumption. P99 latency is not enough when output quality changes from turn to turn. A tool 502 at step 3 can corrupt the planner at step 4, and the user sees a hallucination instead of an HTTP error. Without evals, every prompt change is guesswork.

Use AgentCore Observability, CloudWatch, X-Ray, and ADOT as the substrate. Add self-hosted Langfuse in the same VPC for prompt registry, datasets, and eval workflows. Both should consume the same OTel stream. Add aws-opentelemetry-distro, run with opentelemetry-instrument python my_agent.py, and propagate session IDs through X-Amzn-Bedrock-AgentCore-Runtime-Session-Id or OTel baggage.

Put Ragas + DeepEval in CI for faithfulness and tool-call F1. Use Bedrock Model Evaluation for managed regression jobs and judge fan-out. Run Inspect AI on a separate cadence for safety and red-team work. Dimension histograms on tenant_id, feature_id, agent_id, model_id, and prompt_version. Sample online evals at 10-30%, not 100%.

Do not judge with the same model family you ship. GPT-4-class judges can reach roughly 80% agreement with humans, but self-preference bias is real. If you ship Claude, judge with Nova or Gemini. That is how you keep the judge from grading its own homework.

Plain logs are not enough. Agent debugging needs a per-session span tree with prompts, tool arguments, observations, and decisions inline. The incident question is almost always, "why did it take that path on turn 4?"

The OTel rail should run under every layer. CloudWatch GenAI dashboard handles infra and alarms. Langfuse handles prompt registry, dataset replay, and human-anchor labels. Cross-family judges compare Nova against Claude and Claude against Nova. Every prompt change gets a 5% online shadow.

Layer 9: Security + Identity + Guardrails

Traditional appsec still applies. Agents add prompt injection, indirect injection, excessive agency, sensitive data disclosure, and cross-tenant context bleed. Guardrails and content filters help, but they do not solve prompt injection.

Use three layers. First, Bedrock Guardrails on inputs and outputs for PII redaction, contextual grounding, prompt-attack filtering, and denied topics. Second, AgentCore Identity for token vaulting, inbound JWT authorization at Runtime ingress, and OBO 3LO OAuth for per-user downstream calls. Third, structural separation. This is the layer that matters most.

Simon Willison's "lethal trifecta" is the right mental model: any agent path that combines private data, untrusted content, and external communication is trivially exfiltratable. His framing is blunt: once an LLM agent has ingested untrusted input, it must be constrained so that input cannot trigger consequential actions. In appsec, 99% is a failing grade. A guardrail that catches 95% of attacks is still exploitable. Break the trifecta with dual-LLM patterns, CaMeL-style capability-tagged data flow, or capability separation with an egress allowlist.

Treat Bedrock Guardrails as defense-in-depth, not the security control. The structural split is what stops indirect injection. Document residual risk explicitly because reviewers should expect both layers.

For a support or research assistant, the summarizer that reads untrusted documents should have no external-action tools and should return only schema-validated summaries. The tool-calling orchestrator should not see raw untrusted content if it also has outbound authority. Bedrock Guardrails handle policy checks, and an independent small-model classifier can inspect Gateway tool inputs outside the agent loop. VPC egress should be allowlisted to the systems the product actually needs.

Cross-Cuts

Cost Engineering

Model routing + prompt caching from day one. Cheap-first routing can return 30-60%. Prompt caching can return up to 90% on cached input. Together, they explain most documented cost wins. The 86% cost reduction and 66% speedup reported in production are achievable on stable system prompts and tool schemas if the cache breakpoint sits after system + tool definitions and before conversation history.
Cost ceilings + kill switch. Use $0.50 per session, $5 per user per day, and per-tenant daily limits from the contract. Wire Cost Anomaly Detection to a Lambda that calls UpdateAgentRuntimeEndpoint to disable the endpoint past threshold. A rough baseline of 10k DAU x 4 turns/day x $0.008 is about $32/day and $960/month. A $50K/month ceiling gives about 52x headroom.
Bedrock Batch for offline work. Use the 50% discount for inference with a 24-hour or shorter SLA: bulk evals, offline RAG indexing, and weekly recap generation.

Reliability

Hard recursion_limit and max_iterations on every loop. Default to 8 unless you have data that says otherwise.
Fallback model chain validated against the same eval set as primary. Sonnet 4.6 to Nova Pro to a cached degraded message is a reasonable starting chain. Use CRIS by default to absorb regional throttling. Add per-tool circuit breakers that open after 5 failures in 30 seconds.
Adaptive retry x3 on every model and tool call. Give each tool a deadline, with 30 seconds as the default. Put a per-tenant token bucket at the front door: 60 requests/minute per user and 1,000 requests/minute per tenant. AgentCore Memory is regional, so active-active requires DynamoDB Global Tables or an explicit memory-loss-on-failover decision.

Security

A single agent path should never combine private data, untrusted content, and external communication. Break one leg. The published patterns with stronger guarantees are CaMeL-style capability-tagged data flow and capability separation with an egress allowlist. Bedrock Guardrails plus an independent classifier are useful, but insufficient. For regulated tenants, use Object-Lock audit logs through Firehose to S3 in COMPLIANCE mode, per-tenant KMS CMKs for crypto-shredding, and a DSAR-friendly delete-by-user-id path across every memory store and knowledge base.

Migration Playbook

Stage 0: Hack / toy, week 0-1. Start with a notebook, Bedrock Converse, and two or three Python tool functions. Defer everything else. Exit when the demo answers five hand-picked questions end to end. The common mistake is reaching for LangChain before you know the shape of the problem.

Stage 1: Prototype, week 1-4. Add Strands locally, 5-10 schema-typed tools, an in-memory buffer, manual eval on 10-20 prompts, and basic logging. Defer AgentCore, Knowledge Bases, Guardrails, and multi-tenant concerns. Exit when you pass 10/10 happy-path cases and know your cost per session. The common mistake is more than 30 tools with no eval set and no max_iterations.

Stage 2: MVP, month 1-3. Add AgentCore Runtime, Memory, and Gateway. Add Bedrock Guardrails for PII and violence. Add Observability + the GenAI dashboard, at least 50 eval cases in CI, least-privilege IAM pinned to model and inference-profile ARNs, and CRIS. Defer multi-region, Provisioned Throughput, and fine-tuning. Exit when 50+ eval cases pass at 90%+, four incident runbooks exist, and the CI gate is live. The common mistake is skipping the eval gate, using bedrock:InvokeModel *, or missing tenant_id in OTel.

Stage 3: Production-grade, month 3-6. Add multi-tenant isolation from JWT to IAM session tag to Cedar to Memory namespace. Add identity propagation, CI eval gates that block regressions, shadow + canary rollout from 5% to 25% to 100%, fallback model chain, cost ceilings, kill switch, chaos tests, VPC, WAF, and CloudTrail data events. Defer active-active multi-region and fine-tuning. Exit when four runbooks have been exercised in a game day, per-tenant cost attribution is within 5%, and mean rollback is under 5 minutes. The common mistake is tenant ID only in Cognito, guardrails stuck in shadow mode, or no chargeback.

Stage 4: Scale, month 6+. Add multi-region, warm standby first. Add Provisioned Throughput on the steady-state baseline, distilled Haiku for hot paths, advanced RAG with hybrid search + rerank + query rewriting + parent documents, tiered KB storage, and an A/B framework tied to eval and business metrics. Exit when Provisioned Throughput break-even is verified monthly, failover is tested quarterly, and the fine-tuned model holds the baseline. The common mistake is buying PT before traffic data, fine-tuning before prompt caching and routing, or going active-active without sticky tenant routing.

Skipping stages is the normal failure mode. Move forward when the exit criteria are met, not when the roadmap says it is time.

The 10 Mistakes I Would Avoid

No recursion_limit / max_iterations. This is the expensive missing line.
The lethal trifecta in one path. Private data + untrusted content + external communication is exfiltration waiting to happen.
Treating Bedrock Guardrails as the security control. It catches obvious jailbreaks. It does not solve indirect injection.
Premature multi-agent. A 90.2% lift at 15x tokens must be earned with evals.
Metadata-filter-only tenant isolation. Use per-tenant indexes or namespaces and enforce isolation outside the prompt.
No prompt cache. Production users have reported 86% cost reduction and 66% faster processing on stable prefixes. This is not theoretical.
Judging with the same model family you ship. Ship Claude, judge with Nova. Or vice versa.
Adding tools past 30 without retrieval. RAG-MCP measured 13.62% correct selection with all tools in prompt versus 43.13% with retrieval.
Treating 1M context as a RAG replacement. It is slower, more expensive, and less accurate for buried facts.
Static credentials in env vars or raw tokens in prompt-visible tool output. Use AgentCore Identity and keep secrets out of model context.

Where to Start Tomorrow

Fill out the 9-layer table for your product. For each layer, write your actual choice and the trigger that would make you change it. If you cannot state the trigger, you do not own the choice yet.
Wire recursion_limit, a per-session dollar cap, and a Cost Anomaly Detection kill switch before the first real prompt. This is a one-day task. It prevents the $40K weekend.
Stand up the OTel rail and a 50-case eval set in CI. Block on faithfulness and tool-call F1 regression. Pin prompt_version and tenant_id to every span.
Put the prompt-cache breakpoint after system + tool definitions and before history. Assert on cache_read_input_tokens in observability. If the metric is zero, the cache is broken and you are paying full price.
Draw your lethal-trifecta containment line. Find every path where private data, untrusted content, and external communication meet. Remove one leg, gate the action with HITL, or document the residual risk in writing.

The defaults are only defaults. Start with them, measure what your workload actually does, then deviate when the evidence says to. That is the job.