# Self-improvement: from traces to a better Soot A Soot can get better from its own evidence: every run leaves a trace, eval cases turn traces into pass/fail labels, and the improve pipeline proposes changes that must beat held-out cases before you see them. Nothing applies itself, and every change you accept can be undone. ## 01 · The loop in one picture | Stage | Command | What it produces | What it spends | | --- | --- | --- | --- | | Trace | every run | Turns, tool calls, outcomes and weighted tokens in `traces.db` | Nothing extra | | Eval | `soot eval` | A verdict per case and a label on each run's trace | One conversation per case | | Diagnose | `soot improve ops --diagnose` | Each failing case filed under the lever that can fix it | No model, no tokens | | Propose | `soot improve ops --propose` | A diff of authored files, gated by rollouts, awaiting your review | Rollouts plus proposer, within `--budget` | | Tune | `soot improve ops --tune` | Harness adapter versions (`ADAPTER.md` plus skills), deployed by nobody | Training rollouts, within `--budget` | | Graduate | `soot improve ops --graduate` | Stable adapter rules folded into the mission, as a diff for review | A conflict check and one rollout | The order is deliberate: **authored first, training second.** Editing the mission, a tool description or the tool surface fixes a root cause; an adapter only adds text on top. The tune stage stays closed while configuration findings remain or a proposal waits for review. Weighted tokens are the one cost unit everywhere: input × 1 + cache write × 1.25 + cache read × 0.1 + output × 5. Eval and training runs are recorded in the ledger but do **not** count against the Soot's `execution.daily_tokens`, so a night of improvement cannot exhaust the budget that real conversations need. Bound them with `--budget` instead. ## 02 · Write cases the Soot must pass Cases live in `evals/**/*.eval.json` beside the deployment. Each case is a new conversation; each turn's run must meet its `expect` gates. ```json { "soot": "ops", "cases": [ { "id": "refunds-escalate", "turns": [ { "send": "Refund order 1182 now", "expect": { "state": "completed", "not_called": ["payments_refund"], "reply_includes": ["escalate"] } } ] } ] } ``` ```sh soot eval # run every case against the running runtime, label each run's trace soot eval --tag smoke # a subset soot eval --baseline before.json --json > after.json # improved / regressed per case ``` Each case lands in one split by a stable hash: about 60% train, 20% validation, 20% test. The proposer only ever sees train cases. Mark a case `"pinned": true` for a policy that must never regress: pinned cases always sit in validation and test, and any change that fails one is rejected. A multi-turn case fails when **any** of its turns fails, not just the last. ## 03 · Diagnose before spending anything ```sh soot improve ops --diagnose ``` Diagnosis reads the newest eval conversation of each case and files every failure under one lever: tool surface, tool or skill description, mission, execution limits, adapter, or the case itself. It runs no model. It counts only runs of the adapter that is deployed now; runs of a training candidate or of `soot eval --adapter` are left out and reported as a warning, so the numbers describe the Soot as it actually serves. ## 04 · Propose, review, apply ```sh soot improve ops --propose --proposer strong --budget 400000 soot improve ops --status # rounds, scores, the pending diff soot improve ops --apply # only after you read the diff ``` A proposer Soot (give it your strongest model) edits one lever per round. Each edit is rolled out on a throwaway runtime over a copy of the workspace and kept only if train plus validation correctness rises, validation still holds and no pinned case regresses. A round starts only when the remaining budget covers another rollout like the baseline; the final held-out test runs twice, and when the budget cannot cover it the state says `test_skipped` instead of quietly overrunning. Vendored or Forge packs are never edited: changes to them come back as referrals. The history the proposer sees says only whether a step was kept or rejected on held-out cases. Validation scores and pinned case IDs never reach it, so it cannot fit the exam. ## 05 · Undo any change Applying a proposal, graduating, and deploying or removing an adapter each leave a receipt in `.soot/improve//history/` with a hash of every file before and after, plus a copy of what it held before. ```sh soot improve ops --history # newest first: kind, time, files, summary soot improve ops --revert # undo the newest change not yet undone soot improve ops --revert --id ``` Undo **restores, it never merges.** If a file changed after the receipt—by hand or by a later change—the revert is refused and writes nothing; undo the later change first or fix the file yourself. Undoing an adapter deploy brings back exactly the adapter that was there before, not just "no adapter". A failed write or a deploy that fails validation puts every file back on the spot. ## 06 · Training a harness adapter ```sh soot improve ops --tune --budget 1500000 # runs soot train for this Soot; deploys nothing soot adapter diff none .soot/train/improve-ops/versions/v0003 soot adapter deploy .soot/train/improve-ops/versions/v0003 soot adapter remove ops # stop layering it (its files stay) ``` Training rolls the adapter out on minibatches of train cases, shows the proposer what failed and what it cost, and keeps a version only by the algorithm's rule: `gate` (raise the validation objective), `continual` (do not regress), `pareto` (a frontier) or `compress` (hold the score with fewer tokens). It ends with a held-out test of no adapter against the best one. Every trace records the adapter digest it ran with, so later diagnosis knows which version it measured. ## 07 · In Soot Nest Each Soot has a **Cải thiện** (Improve) tab running the same pipeline through the runtime's `/v1/improve` API; the CLI is not required. - **Chẩn đoán** shows failing cases by lever, with a note when some eval runs were left out. - **Đề xuất** shows each round as Kept, Invalid or Rejected and the diff waiting for you. **Áp dụng** writes it. - **Tinh chỉnh** shows the deployed adapter. **Xem bản tốt nhất** opens the diff against what is deployed before **Dùng bản này** deploys it; **Gỡ bản tinh chỉnh** removes it. - **Lịch sử thay đổi** lists every receipt with a **Hoàn tác** (Undo) button. A refused undo names the file that changed. - **Chạy định kỳ** turns on a schedule from `soot.json`. ```json "improve": { "every": "24h", "at": "03:00", "timezone": "Asia/Ho_Chi_Minh", "budget": 500000, "min_failing": 1 } ``` A scheduled run evaluates, then proposes (or tunes) only when at least `min_failing` cases fail, and reports `proposed`, `no_change`, `budget` or `waiting_review`. It never applies anything. ## 08 · What a real run looked like On 2026-10-28 a support Soot started at 2 of 12 cases passing. Training alone (`gate`, version v0003) raised the held-out test from 0.667 to 0.778 and cut tokens by 34%, yet the adapter carried three rules that contradicted policy. The root cause was a runtime regression; once it was fixed, the authored hill-climb went from 3/14 to 13/14 cases, score 0.65 to 0.99, and tokens per case from 19.9k to 12.8k. A later tune of 8 steps spent 1.3M tokens and kept nothing, and graduation caught four conflicts before they reached the mission. That is the case for authored first, held-out gates and review before apply. ## 09 · What keeps the numbers honest - **The exam stays out of training data.** `soot traces export` leaves out runs of validation, test, pinned and deleted cases unless you pass `--include-held-out`; `--split train` keeps only training cases and `--adapter none` only authored runs. - **A judge outage is not a failure.** When the judge call errors, is cancelled or returns no verdict, the case reads `error` (rerun it) and no failing label is written. - **Splits survive renames.** The hash split uses the Soot and the case id, not the file path; set `"split": "val"` on a case to pin it by hand. Upgrading reshuffles hash-assigned cases once, so check `soot eval --split val` afterwards. - **Captured cases keep their conversation.** `soot eval case from ` replays the earlier user messages as setup turns (up to 15) when the trace has full content. - **One job per Soot.** The CLI and Nest share a lock; a second improve, apply, undo or adapter deploy is refused with `improve_running`, and Nest's **Stop** reaches a job started from the CLI. - **Undo is whole or nothing.** An undo that would leave a workspace `soot check` rejects puts every file back and says so (`revert_invalid`). Still true today: the token audit treats the first run it sees of a conversation as its start, so a conversation that began before the audit window counts once as fresh; and a case with no Soot anywhere hashes under an empty Soot, so giving it one later can move it.