Your LLM picks the right answer on a benchmark. Your agent still makes the wrong decision in production. The gap between those two outcomes is almost never the model. It is what the model sees when it has to decide.
Google DeepMind’s Philipp Schmid put it bluntly: “Most agent failures are not model failures anymore, they are context failures.” That observation, from someone who works on frontier models daily, should reframe how you think about building agents. The model is the engine. Context engineering is the road, the fuel, and the GPS.
Context engineering is the discipline of designing, managing, and optimizing the entire information environment an AI agent operates in. Not just the prompt. The system instructions, conversation history, retrieved documents, tool definitions, memory, structured state, and output schemas. All of it, orchestrated so the right tokens arrive at the right time in the right order.
Why Prompt Engineering Hit a Ceiling
Prompt engineering was the right discipline for 2023. You had a single model call, a static system prompt, and one shot to get the answer right. Tweaking word choices, adding few-shot examples, rewriting instructions in second person vs. third person: those tricks worked when the context window was 4,096 tokens and the model saw everything at once.
Agents broke that model. A production agent running a 14-step task does not operate in a single call. It accumulates state across turns, calls tools that return unpredictable payloads, retrieves documents of varying relevance, and carries conversation history that grows with every interaction. By step 10, the context window is a landfill of stale tool outputs, irrelevant history, and redundant instructions.
Harrison Chase, CEO of LangChain, described the core problem on the Sequoia Capital podcast: “You don’t actually know what the context at step 14 will be, because there’s 13 steps before that that could pull arbitrary things in.” The prompt you wrote is one layer. The other six layers of context, the ones you did not explicitly design, determine whether the agent succeeds or wanders off a cliff.
Prompt engineering is about what you say to the model. Context engineering is about what the model knows when you say it.
The Seven Layers of Agent Context
InfoWorld identified seven distinct context layers that production agents must manage. Understanding them explains why “just write a better prompt” stopped being useful advice.
Layer 1-2: System and User Prompts
These are the layers prompt engineering already handles. The system prompt defines the agent’s role, boundaries, and behavioral guidelines. The user prompt carries the immediate task. Together, they are maybe 5% of a production agent’s context budget. The other 95% comes from the layers below.
Layer 3-4: State and Long-Term Memory
Short-term state is the conversation’s scratch pad. LangGraph implements this through checkpointing: persisting agent state across every step so the agent can write intermediate results, track progress, and pick up where it left off after interruptions.
Long-term memory is what the agent remembers across sessions. Zep’s Graphiti temporal knowledge graph delivers sub-200ms retrieval with an 18.5% accuracy improvement over baseline approaches, because it tracks how facts change over time rather than treating memory as a static dump.
Layer 5-7: Retrieved Knowledge, Tools, and Output Schemas
RAG, tool definitions, and structured output specifications round out the context. Each competes for the same finite token budget. A RAG retrieval that pulls 10 documents at 1,500 tokens each burns 15,000 tokens before the model even sees the question. Multiply that by tool definitions and conversation history, and you understand why context management is an engineering discipline, not a prompt-writing exercise.
Four Failure Modes That Kill Agents
Context does not just overflow. It rots. InfoWorld categorizes four distinct failure patterns that explain most production agent breakdowns:
Context poisoning happens when the model hallucinates a fact in step 3, treats it as ground truth in step 7, and builds a cascade of wrong decisions on top of it. Without explicit verification checkpoints, hallucinations compound.
Context distraction occurs when an agent with too much history fixates on an irrelevant earlier exchange instead of the current task. The “lost-in-the-middle” effect is well-documented: models prioritize information at the beginning and end of their context, so critical facts buried in the middle get ignored.
Context confusion is what happens when retrieved documents conflict with the current task or with each other. An agent asked to summarize a legal document might retrieve three versions and produce an answer that blends contradictory clauses.
Context clash is the subtlest failure. New information contradicts earlier context, but the model cannot distinguish which is authoritative. Without explicit precedence rules, it averages the contradiction instead of choosing.
How Production Teams Actually Build Context Engineering
The teams building the most capable agents in 2026, Manus, Factory’s Droids, and Anthropic’s Claude Code, converge on the same core strategies. Anthropic formalized them as five operations: select, compress, order, isolate, and format.
Select: Less Context Is Usually Better
The instinct to give the agent “everything it might need” is reliably wrong. Five Sigma Insurance found that curating a targeted schema of policy data, claims history, and relevant regulations reached over 95% accuracy, while feeding the full document corpus achieved far less. The signal-to-noise ratio matters more than total information.
Just-in-time loading is the production pattern. Instead of pre-loading documents, maintain lightweight identifiers (file paths, URLs, database queries) and fetch data only when the agent’s current step requires it. Claude Code uses bash commands like head and tail to sample large datasets without loading full files into context.
Compress: The Art of Strategic Forgetting
LangChain’s Deep Agents harness implements a three-tier compression strategy. When a tool response exceeds 20,000 tokens, it gets offloaded to the filesystem with a file path reference and a 10-line preview. When the context hits 85% of the model’s available window, file operation inputs get truncated. When neither is enough, the system generates a structured summary: session intent, artifacts created, next steps. The full history remains on disk for retrieval if needed.
Semantic compression can reduce token usage by 50-80% while preserving the information the model actually needs. Redis reports that semantic caching through their LangCache product delivers 50-80% cost savings by avoiding redundant computations.
Order: Where Information Sits Matters
Manus’s todo.md pattern is instructive. The agent creates and constantly rewrites a task list file, pushing the global plan into the model’s most recent attention window. This exploits the recency bias of transformer attention: information at the end of the context gets disproportionate weight. By rewriting the plan at every step, Manus ensures the agent always has the strategic picture in its highest-attention zone.
Isolate: Sub-Agents as Context Firewalls
When a task requires exploring a large codebase, parsing a long document, and writing a summary, a single agent with all three jobs will have a polluted context by the time it reaches the summary. The isolation pattern delegates each subtask to a separate agent with a clean context window. The parent agent receives only the output, not the full working memory of each child.
Anthropic’s data shows that sessions stopping at 75% context utilization produce higher-quality, more maintainable output than sessions that push to the limit.
Memory Architecture: The Four-Tier Model
Production agents need four distinct memory tiers, each with different latency, persistence, and retrieval characteristics:
| Tier | Scope | Latency | Persistence | Example |
|---|---|---|---|---|
| Working memory | Current context window | Zero | Volatile | Active conversation |
| Short-term memory | Session-persistent | Low | Session | LangGraph checkpoints |
| Long-term memory | Cross-session | Medium | Semi-permanent | User preferences, project context |
| Permanent memory | Archival | Higher | Permanent | Compliance logs, training data |
The infrastructure layer is evolving fast. Redis combines vector search, semantic caching, and session management with sub-millisecond latency. MongoDB integrates with LangGraph for cross-thread persistence. Zep builds temporal knowledge graphs that track entity relationships over time, delivering 18.5% accuracy improvements with 90% latency reduction versus baseline RAG.
The Cognitive Workspace study (2025) found a 58.6% memory reuse rate for agents using structured state-based memory, compared to 0% for classical RAG. That number captures the core insight: agents that remember structured facts outperform agents that re-retrieve everything from scratch.
The Infrastructure Is Catching Up
Context engineering is not just a software pattern. The hardware is adapting too.
NVIDIA’s Rubin CPX GPU, announced at CES 2026, is purpose-built for massive-context inference. The Vera Rubin system introduces “context storage” as a first-class infrastructure component, with BlueField-4 DPUs enabling KV cache to spill across the network into shared NVMe pools instead of being confined to local GPU memory.
On the protocol side, MCP (Model Context Protocol) was donated to the Linux Foundation’s Agentic AI Foundation in December 2025. Harrison Chase noted that “it’s hard to separate the term context engineering from MCP,” since MCP standardizes how agents access external tools and data sources, which is the retrieval layer of context engineering.
Context windows themselves keep growing. Gemini 3 Pro supports 1 million tokens. Llama 4 Scout handles 10 million. But bigger windows do not solve context engineering problems. They make them worse, because the “lost-in-the-middle” effect scales with window size, and token costs grow linearly.
Getting Started: A Pragmatic Checklist
If you are building agents today, here is what the research and production experience converge on:
Audit your context budget. Before adding anything, calculate how many tokens your system prompt, tool definitions, and typical conversation history consume. Most teams discover they are burning 40%+ of their budget before the agent does any real work.
Implement compression early. Do not wait until your agent hits context limits. Set offloading thresholds at 20K tokens for tool outputs and 85% for total context utilization. LangChain’s Deep Agents harness is a solid reference implementation.
Tier your memory. Not everything needs to be in the context window. Use checkpointing for session state, a vector store for cross-session facts, and structured state for high-priority rules.
Measure context quality, not just quantity. Track how often your agent references stale information, how frequently it re-retrieves already-available data, and where in the context window critical facts land.
Start with isolation for complex tasks. If a workflow has more than 5 sequential tool calls, consider splitting it across sub-agents with clean context boundaries.
Anthropic’s engineering guidance distills it well: “Do the simplest thing that works.” Context engineering is about finding the smallest possible set of high-signal tokens that maximize the likelihood of a good outcome.
Frequently Asked Questions
What is context engineering for AI agents?
Context engineering is the discipline of designing, managing, and optimizing the entire information environment that an AI agent uses to make decisions. It encompasses system prompts, conversation history, retrieved documents, tool definitions, memory, and structured state. Unlike prompt engineering, which focuses on what you say to the model, context engineering focuses on what the model knows when it has to act.
How is context engineering different from prompt engineering?
Prompt engineering optimizes a single input-output pair: the words you send to the model. Context engineering manages the full information stack across multi-step agent workflows, including state management, memory tiers, context compression, tool integration, and retrieval orchestration. Prompt engineering is a subset of context engineering, not a replacement for it.
Why do AI agents fail despite using large context windows?
Large context windows create four failure modes: context poisoning (hallucinations treated as facts), context distraction (fixating on irrelevant history), context confusion (conflicting retrieved documents), and context clash (new information contradicting earlier context). The lost-in-the-middle effect also means models pay less attention to information in the center of the context, regardless of window size.
What tools and frameworks support context engineering?
LangGraph provides checkpointing and memory for agent state management. Redis offers sub-millisecond vector search and semantic caching. Zep builds temporal knowledge graphs with 18.5% accuracy improvements over baseline. MCP standardizes tool and data access. NVIDIA’s Rubin CPX GPUs are purpose-built for massive-context inference workloads.
What is the 75% context utilization rule?
Anthropic’s research shows that agent sessions stopping at 75% context window utilization produce higher-quality, more maintainable output than sessions that push to the limit. This is because the remaining 25% provides headroom for the model to process and reason about the existing context without quality degradation.