Compaction, offloading, and retrieval

#The problem all three solve

An agent’s context only grows (see the agent-loop refresher), but the context window is a fixed budget. Every context-management method is built from three basic moves:

• Compaction shrinks history in place.
• Offloading moves content out of the context into external storage.
• Retrieval brings stored content back when it’s needed.

A useful mental model is a person at a desk. Compaction is condensing your notes into a summary. Offloading is filing papers in a cabinet. Retrieval is going back to the cabinet to look something up. Offloading and retrieval come as a pair: storing something you can never find again is just deleting it slowly.

#Compaction: rewrite history into something shorter

How it works. Pick a span of the context, usually everything except the system prompt and the task. Ask the model to summarize that span, then replace the span with the summary:

Designs differ along three axes:

• When. At a fixed threshold, as in Codex-style harnesses that compact at about 75% of the budget. At every turn, as in MEM1. Or whenever the model decides, as in Self-Compact and AutoCompact.
• What. The whole history, or only part of it. Context-as-a-Tool compacts a predefined portion; Sculptor works at the level of fragments; Context Folding branches into a sub-task and folds that branch into a summary when it returns.
• How. Usually a fixed, prescribed summarization prompt.

A common practical variant keeps the most recent turns verbatim and summarizes only the older ones.

How it fails.

1. It’s lossy and irreversible. Exact strings such as IDs, hashes, numbers, and file paths get paraphrased or dropped, and there’s no way back.
2. It can hallucinate. A summary can state things the history never said.
3. It bets on the future. At compaction time, the model has to guess what the rest of the task will need.
4. Loss compounds. Over a long run you end up summarizing summaries, and each round loses a little more.

#Offloading: move content out, leave a pointer

How it works. Write a block to external storage (a file, a database, a memory store), and replace it in the context with a short pointer describing what was saved and where:

def offload(block_id, text):
    path = f"/tmp/store/{block_id}.txt"
    write_file(path, text)
    return f"[{block_id}: {count_lines(text)} lines saved to {path}; grep to recall]"

Unlike compaction, offloading is lossless: the content still exists, verbatim, outside the context. What the model knows in the moment shrinks to whatever the pointer says, though, so writing a good pointer is a skill in itself. ACM adds model-triggered offloading as a tool; MemGPT builds a whole memory hierarchy around the idea; and any agent that writes notes or TODOs to a file is offloading.

The subtle catch: copying is not evicting. In a normal harness, a tool’s output has already been appended to the context before the model can act on it. Writing that output to a file creates a second copy on disk, but the original stays in the context. Real offloading needs a way to remove the original, and most tool interfaces don’t provide one.

How it fails. The pointer gets lost (for example, summarized away by a later compaction), the original is never actually evicted, or the pointer is too vague for the model to know what’s behind it.

#Retrieval: bring the right piece back

How it works. When the model needs stored content, it queries the store and the result is appended to its context. Queries range from exact lookup (grep for a key) to structured queries (SQL, key-value) to semantic search over embeddings.

Recursive Language Models (RLMs) take a retrieval-first approach: they keep a huge input outside the context, as a variable in a Python REPL, and read slices of it on demand.

How it fails.

1. Forgetting the content exists. If no pointer survives, the model never thinks to look.
2. The wrong query, which pulls back the wrong thing.
3. Recalling too much: a broad search that dumps thousands of lines back into the context.

Retrieved content also grows the context again, so retrieval needs its own eviction story, which brings you back to the “copying is not evicting” problem.

#Side by side

	Compaction	Offloading	Retrieval
What moves	History becomes a summary, in place	Content goes to an external store	Stored content comes back in
Lossless?	No	Yes (outside the context)	Only as good as the query
Who decides	The harness, or the model at a trigger	The model via a tool, or the harness	The model
Typical failure	Drops or invents details	Pointer lost, or original never evicted	Wrong or too much recall
Prefix-cache cost	Small re-prefill (the summary is short)	Re-prefill of everything after the evicted block	Cheap (it’s an append)

The last row connects to the KV-caching refresher. An edit costs roughly the number of tokens after the edit point. Retrieval appends, so it’s cheap. Compaction rewrites almost everything but leaves very little behind, so it’s also fairly cheap. Evicting a block from the middle of a long context is the expensive one.

#The common thread: a fixed menu

In every system above, each move is a predefined operation with fixed semantics that the harness exposes as a tool. The model may choose when to call it, and sometimes which span it applies to, but it can’t change what the operation does. A tool such as “summarize everything since my last call” can’t be told to keep line 14 verbatim and delete lines 15 through 200.

That limitation is the gap the CLM paper targets. If the context is an editable file, every move above becomes just one possible edit, and the model can combine them or invent new ones.

#Where this shows up in the CLM paper

The paper’s diagnostic benchmark, ContextBench, is built to stress exactly these moves:

• Needle Retention tests verbatim keeping, which is compaction’s weak spot.
• Sudoku Sketchpad needs small in-place updates, which none of the three moves provides.
• KV Store and Log Triage need offloading, retrieval, and eviction.

The paper’s Appendix D, Table 4, shows each method’s instructions for the KV Store task side by side. It’s the clearest way to see how differently these mechanisms work in practice.