An agent that gets better on its own, and still asks before it changes.
Every run leaves evidence. Soot turns that evidence into tests, proposes fixes that must pass cases it never saw, and hands you a diff you can apply or undo.
Every run is a receipt.
Each turn records tokens, tool calls and outcomes. Cost is one unit everywhere: weighted tokens, where output costs five times input and a cache read a tenth.
Weighted11,300
Unchanged: improvement is bounded by its own --budget, so a night of training cannot starve tomorrow's users.
What it can do
Compare runs fairly: a cheaper run is cheaper in one unit, whatever the cache did.
What it cannot do
Tell you if the answer was right. That needs a case.
A test the Soot must pass, split so it cannot cheat.
Cases live in evals/. A stable hash puts each in train, validation or test. The proposer only sees train. Pin a policy case and it is always held out.
greets-in-languagetrainfinds-order-statustrainasks-before-canceltraincites-the-faqtrainrefund-needs-humanvalidation + testno-card-numbersvalidationhandles-two-orderstrainhands-off-angrytest
What the proposer sees · 5
Held out from the proposer · refund-needs-human, no-card-numbers, hands-off-angry
- Turn 1 · Asks which order
- Turn 2 · Looks up the wrong order
- Turn 3 · Answers politely
What it can do
Catch a regression in a case the proposer never read.
What it cannot do
Cover behaviour no case describes. Write the case first.
Find the lever before spending a token.
Diagnosis files each failing case under the lever that can fix it: tool surface, a description, the mission, execution limits, the adapter, or the case itself. It runs no model.
- Tool surface2orders_lookup missing from the toolset
- Tool description1"search" says nothing about order IDs
- Mission2refund rule is ambiguous
- Case1expects a phrase the policy forbids
What it can do
Tell a broken tool apart from a vague mission, for free.
What it cannot do
Write the fix. That is the next stage.
Every edit must win on cases it never saw.
A proposer edits one lever per round. Each edit is rolled out on a throwaway copy and kept only if train plus validation improves and no pinned case regresses. A round starts only if the budget covers it.
The proposer only hears "kept" or "rejected on held-out cases". Never the validation score, never a pinned case ID.
What it can do
Find a smaller, better mission without you watching.
What it cannot do
Apply anything. It stops at a diff awaiting review.
Training helps. Fixing the cause helps more.
An adapter is learned text layered on the mission. It can raise a score while carrying rules that break policy. A real support Soot, measured on 2026-10-28:
Adapter training only
- Held-out test 0.667 → 0.778
- Tokens −34%
- 3 rules contradicted policy
Root cause fixed, then authored hill-climb
- Cases 3/14 → 13/14
- Score 0.65 → 0.99
- Tokens per case 19.9k → 12.8k
Afterwards: a tune of 8 steps spent 1.3M tokens and kept nothing; graduation caught 4 conflicts before they reached the mission. That is why authored edits come first, training second, and nothing ships without review.
What it can do
Squeeze tokens and patch gaps once the authored Soot is right.
What it cannot do
Replace a fix to the tool, the mission or the runtime.
Every change you accept can be undone.
Applying a proposal, graduating and deploying or removing an adapter each leave a receipt: the hash of every file before and after, and a copy of what it held. Undo restores; it never merges.
- Deployed an adapternone → 3f9c0a12 (v0003) ·
soot.json, adapters/support/ - Applied a reviewed proposalmission: shorter refund rule ·
MISSION.md
What it can do
Put back exactly the files and the adapter version that were there before.
What it cannot do
Silently discard an edit made after the change.
The loop, in one line
Trace every run → test with cases → diagnose for free → propose under a budget, gated by held-out cases → review the diff → apply → undo if you change your mind.
What the pipeline guarantees
- soot traces export leaves out held-out and pinned cases unless you ask for them.
- A judge outage reads as error, not as a failed case.
- Splits follow the case id, not the file path; "split" pins one by hand.
- soot eval case from <run> keeps the conversation that led to the run.
- The CLI and Nest share one lock per Soot; Stop reaches a CLI job.
- Still open: a conversation that began before the audit window counts once as a fresh start.
Run it on your own Soot.
The whole pipeline is in the CLI and in Soot Nest's Improve tab.
Numbers in chapter 05 are from one dated run and illustrate a pattern; they are not a benchmark or a guarantee. Other labs are simulations of the documented rules.