Agent 101·Lesson 02·22 min read

The window never grows. Everything else has to.

A Soot can work for hours on one task, and the model behind it has no memory between requests. This lesson is about the desk I put in front of that model, and every trick I built to keep it from overflowing.

01 · The context window

A desk that is cleared every time you look up

Picture a very patient colleague with no memory at all. You can hand them a folder of papers, and they give you excellent work back. But the next time you walk in, the desk is empty. They do not remember the folder. They do not remember you. Everything they know has to be on the desk, in front of them, that second.

That is a language model behind an agent. It does not remember the last request. Each time I ask it to think a step, I send the whole request again: what I want it to be, which tools it may use, and the entire conversation so far — my instruction, its own reasoning, every tool call it made, and every tool result it got back. That whole payload is the context. The model reads it, answers, and forgets.

The desk has a fixed size. The context window is a number: the most tokens the model will read in one request plus the most it will write back. It never gets bigger during a run. When the request would no longer fit, the run does not slow down — it stops being possible.

A conversation is rarely the problem. A job that takes hours is. Ten turns of chatting is a pamphlet. Twenty minutes of one agent fixing a bug is a shelf of binders: it read four files, ran the tests twice, wrote a patch, read three more files, ran the tests again with full output, and each of those results is now sitting on the desk on every following turn, because I have to send it all again.

And tool results are most of it. A test suite that prints a thousand lines, a build log, a diff, a directory listing — one of those can outweigh a whole day of reasoning. The reasoning is cheap. The evidence is heavy.

Block01
In the request now: 77,494Window: 40,000▲ does not fit

One bar = a slice of the window. Grey is work from earlier turns still being re-sent. Each turn adds a little more.

What the model sees on turn 200
  • The instructions, written once at the start
  • Every tool call and every tool result, including the ones from an hour ago
  • Its own reasoning, and each new request appended at the end
What it actually needs
  • The instruction it is working from
  • The plan as it stands now
  • The two files it is editing

There is a second, quieter cost. Every token I send is work the provider has to process. So the question is never only “will it fit” — it is also “how much of this am I paying to read again”. That question has its own answer, and it shapes everything else in this lesson.

02 · The prefix

Why one changed byte early on costs me the whole request

Take a photocopy shop. If the first fifty pages of a document never change, the shop can keep plates for those fifty pages and copy them again. But if you retype page one, even one letter, every following page is a different document. The plates are useless. They start from scratch.

Model providers work like that. If the beginning of my request is exactly the same as last time — byte for byte — the provider can reuse its cached work and charge me a fraction for it. That is the prompt cache. It is not an optimisation I can turn on late; it depends on the request starting identically, and matching further and further along until something differs.

One changed byte near the front breaks it all. So I have a list of things that must never touch the start of a request mid-run:

  1. 01Rewriting my instructions. If I edit the system prompt because the situation changed, the cache is gone from the first character. So I do not edit it. New guidance is appended as a new message.
  2. 02Changing the tool schemas. If the set of tools or even their order changes, the cached prefix changes with it. So the tool list is fixed for the run, and a tool that is not loaded yet is declared with a tiny stub instead of being added later.
  3. 03Editing an earlier message. Trimming or fixing one message in the middle changes everything after it. So a message that has been sent is not edited.
  4. 04Putting the volatile part at the top. Anything that changes every turn — the plan as it is now — cannot live in the fixed head.

That is why the things you would expect to be rewritten are appended instead. The plan is a message. Goals are a block appended to the request. A reminder about the plan is a message added at the end, and only when the earlier copy is no longer visible — after a summary, or after a crash recovery. Recall from memory arrives as a user message further down, not in the system prompt. The head stays put; the conversation grows at the tail.

Try it: change one early byte02
  1. 01You are Soot. Be clear, practical and candid.✓
  2. 02Tools: load tool, read file, grep, run shell…✓
  3. 03user: fix the flaky migration test✓
  4. 04assistant: reading the migration folder✓
  5. 05tool: 14 files, 3 tests red✓
Served from cache: 100%read at the cached price: 9,000read at full price: 0

Nothing before the new message changed, so the whole prefix is still served from cache.

I want to be precise about what this buys, because it is easy to over-promise. A cache hit is cheaper per token and, in my traces, most tokens come from the cached part. It does not make the window bigger. It does not make an overflowing request fit. It only decides how much the part that does not change costs me, over and over.

03 · Lazy loading

Only the drawer you need, only when you need it

I keep a toolbox at home with fifty drawers, each labelled with one line: “drill bits”, “screws”, “spirit level”. When I am hanging a shelf, I do not carry all fifty drawers to the wall. I read the labels, pick one, and open it. The label told me which drawer helps; the drawer tells me how to use the thing.

Tools are the same deal. Every tool I give an agent costs context on every single turn, because its full description has to be in the request. With dozens of tools, most of them irrelevant to this task, that is a lot of space spent on instructions nobody is reading. So a pack's tools are not all loaded up front.

The model gets a one-line index — enough to know a tool exists and what it is roughly for — and one tool, “load tool”, which returns a tool's real definition when asked. That definition comes back as an ordinary tool result, in the conversation, where the model will actually read it. If the model calls a tool it has not loaded, the call is refused, and the refusal carries the definition so it can retry with it.

What costs what
On the desk from the first turnOnly when asked
Skill names and descriptionsThe body of each skill
A one-line index of the pack toolsThe full definition of each tool
The tools that always existEverything else

The tool list itself does not change, and that is deliberate. A tool that has not been loaded is declared with a tiny stub rather than left out, so the request keeps exactly the same tool array before and after loading something. Loading a tool adds a message; it does not rewrite the head. The set of tools that were loaded travels with the run's saved state, so a resumed run still has them, and a summary reattaches the definitions the model was relying on.

Skills work on the same principle, one level up. A skill is a procedure — how to write in my voice, how to review a change. Only its name and description are advertised up front; the body arrives when the model reads it. An index up front, the document on demand: that is the whole idea, applied twice.

04 · Clearing and spilling

Empty the boxes before you pay a translator

You are moving house, and you hired someone to write down what is in the room. They charge by the word. Before they arrive, you empty half the boxes into storage and write one label on the front: “keys, winter, see box 14”. The room is suddenly small enough to describe. And if you ever need a box, box 14 has it.

That is what I do before summarising anything. The oldest tool results are usually the least useful and the largest: a file listing from ten steps ago, a passing test run nobody will look at again. Before paying a model to summarise the conversation, I clear them — replace the content with a short, stable placeholder, and keep only the most recent few results verbatim.

A few rules make it safe:

  1. 01The placeholder has to be stable. If it changed subtly between two attempts, the prefix would change with it and the cache would be lost. Same input, same text, every time.
  2. 02The boundary is written down. Where the clearing stopped is saved with the run, so the next attempt rebuilds exactly the same request instead of a version that happens to be similar.
  3. 03It only moves when it pays. Clearing happens past a trigger point, and only if it actually removes enough to be worth it. Otherwise it does nothing and the cache stays warm.
  4. 04Tiny results stay. A result under a hundred and twenty-eight tokens is not worth clearing.
  5. 05Tools that teach are never cleared. The result of loading a tool or reading a skill is how the model knows what to do; clearing it would make the model forget its own instructions.
  6. 06It never cuts in the middle of a pair. An old tool call and its result are never separated, and the current turn stays as it is.

Some results are just too big to keep in any form: a whole build log, an image, a long document. Those are spilled. The full result is written to a file beside the conversation as an artifact, and the conversation gets a preview plus a locator. The locator is a real handle: the model can read that artifact back when it needs a detail, so what it lost from the desk it can still fetch from the shelf.

128tokensresults kept verbatim
512tokenstokens in an artifact instead
64 × 1 MiBtool artifacts per conversationtokens in an artifact instead
05 · Compaction

The handoff note I write to my future self

At the end of a long shift, a nurse hands over to the next one. They do not read you the whole shift. They write a page: what happened, what was decided, what is still open, where the notes are. Then the next shift works from that page and from the patient, not from a recording of everything.

Compaction is that page. When the request gets close to what the model can hold, the agent stops adding and writes a summary instead. Not a lazy gist — a structured one. It has a goal, constraints that were stated, decisions and how I reached them, what is done and what is pending, files that were read and files that were changed, and locators for the details worth going back to. Later summaries extend the earlier one and keep my corrections word for word, so a fix I gave at the start is not smoothed into a suggestion.

What the handoff note records
  • The goal, in the words it was given
  • Constraints, decisions, and what is done and pending
  • Files read, files changed, and locators for the details
  • My corrections, word for word
What is put back after it
  • The plan, as it stands
  • The skills it was using
  • The definitions of the tools it loaded
  • The part of the request it is working on

Then it reattaches what the model needs to keep working: the plan, the skills it had, the definitions of the tools it had loaded, and the part of the request that is still current. Those are not guesses about what might matter — they are the pieces the run is actively using.

And this is the part I want to be loud about: the original transcript is never cut. Compaction rewrites the version I send to the model, not the version I keep. The full history, every message and every tool result, stays in storage for the trace, for the transcript you can read, and for any later recall. What I am trimming is the request, not the record.

There are guards. A tool call and its result are never separated. A request being worked on right now stays in full. If the provider rejects the request as too long before it has said anything, the agent retries once with a tighter view. And the threshold is a setting, not a magic number: each model has a window, and a request needs room left to think — on a model with a 128,000-token window, the limit works out to 96,000.

Compaction costs, and I should say it plainly. It is a model call of its own. It reads as many tokens as the conversation holds. It is the right thing to pay for, because the alternative is not a slightly worse run but no run at all — but it is not free, and it is not instant.

06 · Checkpoints

The autosave that outlives the job

A word processor that never autosaves expects you to be careful. A power cut costs you the morning. The fix is not a better writer; it is a save that happens without asking.

A run here can be interrupted in ways I cannot control: the machine restarts, the process dies, the connection drops. So after every settled turn, the run writes a resumable state — transcript, plan, loaded tools, memory of what it was doing — into a slot of its own. There are limits on purpose, because saving is work too: at most about every fifteen seconds, more slowly as the state grows, and only up to a ceiling of thirty-two megabytes.

If the run dies without answering, that save stays as the conversation's resume point. Next time, it starts from the last settled turn instead of from nothing. I will not pretend it is magic: nothing replays by itself. It resumes when something asks it to, and a turn that was cut off mid-way is not silently retried. What it saves is the run's thinking, not the run's decisions — those I make.

07 · Subagents

A second desk, in another room

Sometimes the right move is to leave the room. If a job needs forty pages of reading and one paragraph of answer, doing it at my desk fills my desk for the rest of the day. So I send it to a colleague with their own desk, and they come back with the paragraph.

A subagent is exactly that. It is spawned with a task and gets a fresh history — it does not inherit the parent's transcript, only the task and the context it was handed. It works in its own context window, sharing the same model setup and the same workspace, so it can read and change the same files. When it is done, the parent gets its final result, not its working.

Two things follow. The parent's context pays for the answer, not for the reading. And nothing is hidden: the subagent's tokens are counted in the parent run's usage, so a trace does not pretend the work was cheap. A subagent is where I spend context elsewhere, not a way to make context disappear.

08 · Memory

The notebook is not a bigger desk

A chef keeps a notebook: this supplier is reliable, this guest does not eat shellfish. The notebook is not the counter. It does not give them more room to chop. It means that next time they start better.

Memory works that way here. It survives the run, so it is the opposite of context in the one way that matters: context is what this request can see, memory is what outlives it. What memory is not is a bigger window. When something in memory is relevant, it is recalled — and the recall arrives as a message in the conversation, during this run, so it costs context like anything else. It is fetched and placed, not merged into the fixed head.

The rest is housekeeping I would want if I were the one relying on it. Memories are kept in scopes, with a cap on how many stay active and a full revision history behind each one; old ones expire when they are read; and a run writes new memories after it finishes successfully, as a deliberate step, rather than in the middle of its work.

09 · The trace

The receipt for every turn

A long run is invisible while it happens. That is why I made the trace: after the fact, you can see what the model actually received, turn by turn. It is the only way I trust any of the numbers above.

  • Tokens, by kind. Not one number: what was sent as new input, what was written to the cache, what was read from it, and what came back as output.
  • A cache hit rate, for the first turn and for all the later ones. The first turn is always cold — the cache has nothing to match. Later turns are where the prefix work shows up. I read that split, because a single blended percentage hides exactly the thing I am trying to check.
  • Compactions, with how many tokens went in and how many came back, and how many messages survived. If the summary is nearly as big as what it replaced, I want to know.
  • A per-turn cache hit, so a bad turn stands out next to its neighbours instead of disappearing into an average.
  • The fixed background of every request: instructions, the skill index, and each tool's schema, in bytes. That is what I pay before the model does anything.

The commands read the same data. One lists recent runs; another shows one run's timeline; another breaks down where the tokens went, with the cache hit split into first turn and later turns, and the background cost named for what it is. When I want to know whether a change helped, I look at those lines, not at how the run felt.

Try it: read a trace03
Run A · one run, one axisCache hit 73%
Read from cacheNew inputOutput
  • Instructions520
  • Skill index380
  • Tool schemas1,500
  • Conversation so far16,980
  • New input2,820
  • Output850

Everything before this turn is billed as cached read; only the new tail is full-price input. · Click a turn to see what the request was made of.

Read from cache: 19,380New input: 2,820Conversation so far-x: 67,25022,200
10 · One measured run

What it looks like over hours, in numbers

These are the measurements I have for a single long run, not a benchmark. I am quoting them because they are the only ones I can point at, and I expect them to move.

3,699,584tokens read from the cache
378,050tokens of full-price input
64,709tokens of output
91%cache hit rate
4compactions, each one reading about twenty thousand tokens and taking about thirty-five seconds
14,879 B + 29,788 Bfixed overhead of every request: system prompt and thirty tool schemas

And the run outlived the process I started it in. I killed it, and it came back from the last checkpoint. That is the case those two features exist for.

11 · What it still costs

The honest list

None of this makes context free. What it does is move the cost somewhere I can see it and choose it.

That is the desk. What is on it is a choice.

Context engineering is not one trick, it is a habit: keep the head still, spend the tail carefully, summarise before you break, save before you crash, and read the trace to check that any of it worked.

Where in the code this comes from

Under the hoodWhere
The prompt is instructions, tool declarations and history; plan and goals are persisted, append-only messages; the transcript is not cutdocs/runtime.md
Compaction thresholds, what reattaches, the guards around tool-call pairsdocs/runtime.md
Checkpoints between runs, resume points and their limitsdocs/runtime.md
Trace fields, cache hit, compaction records, weighted tokensdocs/evolution.md
Recall as a message, scopes, consolidationdocs/data.md
Tool and skill lazy loadinginternal/core/lazytools/lazytools.go
Clearing, the summary contract, the v0.2.0 changes2ai/compaction_clear.go · 2ai/compaction_prompt.go · 2ai/CHANGELOG.md