Memory and State Across Turns

Studies on needle-in-a-haystack benchmarks surfaced a pattern Anthropic calls context rot: as the number of tokens in a model's context window grows, its ability to accurately recall information from that context decreases. Every model shows this to some degree — some more gently than others — which means context can't be treated as a free resource you simply keep adding to. It behaves more like a limited attention budget: every new token you put in front of the model draws down that budget by some amount.

The underlying reason is architectural. A transformer lets every token attend to every other token, which creates a relationship for every pair of tokens in the context — a number that grows much faster than the context itself. As a conversation or an agent's history grows, that budget gets stretched thinner across more pairs, and precision on any one piece of information degrades. This is exactly why a long-running agent needs a deliberate strategy for what stays in context and what doesn't, rather than just letting history accumulate.

Anthropic describes three techniques for keeping a long-running agent coherent once a task outgrows a single context window. Compaction takes a conversation nearing its context limit, has the model summarize it, and reinitiates a new context window built from that summary. In Claude Code, this means passing the message history to the model to compress: architectural decisions, unresolved bugs, and implementation details are preserved, while redundant tool outputs and exploratory dead ends are dropped, and the agent continues with the compressed summary plus its five most recently accessed files.

Getting this right takes iteration. Anthropic's own guidance is to first tune a compaction prompt for recall — make sure it captures everything that later turns out to matter — and only then trim for precision by cutting what turned out to be superfluous. Tool result clearing (dropping old tool outputs from history once a fresher result has superseded them) is described as one of the lightest-touch, safest forms of this.

def maybe_compact(messages, model, threshold_tokens=150_000):
    if count_tokens(messages) < threshold_tokens:
        return messages

    summary = model.summarize(
        messages,
        instructions="Preserve decisions, unresolved issues, and key details.",
    )
    return [{"role": "user", "content": summary}] + messages[-5:]

Two other techniques handle the same problem differently. Structured note-taking — Anthropic also calls this agentic memory — has the agent periodically write notes to storage outside the context window, then read them back in later. Anthropic's own beta-released memory tool implements this as a file-based system an agent can consult across sessions; a simpler version of the same idea is an agent maintaining its own running NOTES.md or to-do list. In one striking example, an agent playing a video game over thousands of steps kept precise tallies — "for the last 1,234 steps I've been training my Pokémon in Route 1" — entirely through self-directed notes, with no prompting about how to structure its own memory.

Sub-agent architectures solve it by not putting everything in one context window at all. A lead agent delegates a focused piece of work to a sub-agent, which might explore extensively — tens of thousands of tokens of searching and reading — but returns only a condensed summary, often in the 1,000–2,000 token range, back to the lead agent. The detailed exploration stays isolated in the sub-agent; only the distillation crosses back.