Self-improvement: from traces to a better Soot

01 · The loop in one picture

The order is deliberate: authored first, training second. Editing the mission, a tool description or the tool surface fixes a root cause; an adapter only adds text on top. The tune stage stays closed while configuration findings remain or a proposal waits for review.

Weighted tokens are the one cost unit everywhere: input × 1 + cache write × 1.25 + cache read × 0.1 + output × 5. Eval and training runs are recorded in the ledger but do not count against the Soot's execution.daily_tokens, so a night of improvement cannot exhaust the budget that real conversations need. Bound them with --budget instead.

02 · Write cases the Soot must pass

Cases live in evals/**/*.eval.json beside the deployment. Each case is a new conversation; each turn's run must meet its expect gates.

{
  "soot": "ops",
  "cases": [
    {
      "id": "refunds-escalate",
      "turns": [
        { "send": "Refund order 1182 now", "expect": { "state": "completed", "not_called": ["payments_refund"], "reply_includes": ["escalate"] } }
      ]
    }
  ]
}

soot eval                 # run every case against the running runtime, label each run's trace
soot eval --tag smoke     # a subset
soot eval --baseline before.json --json > after.json   # improved / regressed per case

Each case lands in one split by a stable hash: about 60% train, 20% validation, 20% test. The proposer only ever sees train cases. Mark a case "pinned": true for a policy that must never regress: pinned cases always sit in validation and test, and any change that fails one is rejected. A multi-turn case fails when any of its turns fails, not just the last.

03 · Diagnose before spending anything

soot improve ops --diagnose

Diagnosis reads the newest eval conversation of each case and files every failure under one lever: tool surface, tool or skill description, mission, execution limits, adapter, or the case itself. It runs no model. It counts only runs of the adapter that is deployed now; runs of a training candidate or of soot eval --adapter are left out and reported as a warning, so the numbers describe the Soot as it actually serves.

04 · Propose, review, apply

soot improve ops --propose --proposer strong --budget 400000
soot improve ops --status      # rounds, scores, the pending diff
soot improve ops --apply       # only after you read the diff

A proposer Soot (give it your strongest model) edits one lever per round. Each edit is rolled out on a throwaway runtime over a copy of the workspace and kept only if train plus validation correctness rises, validation still holds and no pinned case regresses. A round starts only when the remaining budget covers another rollout like the baseline; the final held-out test runs twice, and when the budget cannot cover it the state says test_skipped instead of quietly overrunning. Vendored or Forge packs are never edited: changes to them come back as referrals.

The history the proposer sees says only whether a step was kept or rejected on held-out cases. Validation scores and pinned case IDs never reach it, so it cannot fit the exam.

05 · Undo any change

Applying a proposal, graduating, and deploying or removing an adapter each leave a receipt in .soot/improve/<soot>/history/ with a hash of every file before and after, plus a copy of what it held before.

soot improve ops --history            # newest first: kind, time, files, summary
soot improve ops --revert             # undo the newest change not yet undone
soot improve ops --revert --id <id>

Undo restores, it never merges. If a file changed after the receipt—by hand or by a later change—the revert is refused and writes nothing; undo the later change first or fix the file yourself. Undoing an adapter deploy brings back exactly the adapter that was there before, not just "no adapter". A failed write or a deploy that fails validation puts every file back on the spot.

06 · Training a harness adapter

soot improve ops --tune --budget 1500000    # runs soot train for this Soot; deploys nothing
soot adapter diff none .soot/train/improve-ops/versions/v0003
soot adapter deploy .soot/train/improve-ops/versions/v0003
soot adapter remove ops                     # stop layering it (its files stay)

Training rolls the adapter out on minibatches of train cases, shows the proposer what failed and what it cost, and keeps a version only by the algorithm's rule: gate (raise the validation objective), continual (do not regress), pareto (a frontier) or compress (hold the score with fewer tokens). It ends with a held-out test of no adapter against the best one. Every trace records the adapter digest it ran with, so later diagnosis knows which version it measured.

07 · In Soot Nest

Each Soot has a Cải thiện (Improve) tab running the same pipeline through the runtime's /v1/improve API; the CLI is not required.

• Chẩn đoán shows failing cases by lever, with a note when some eval runs were left out.
• Đề xuất shows each round as Kept, Invalid or Rejected and the diff waiting for you. Áp dụng writes it.
• Tinh chỉnh shows the deployed adapter. Xem bản tốt nhất opens the diff against what is deployed before Dùng bản này deploys it; Gỡ bản tinh chỉnh removes it.
• Lịch sử thay đổi lists every receipt with a Hoàn tác (Undo) button. A refused undo names the file that changed.
• Chạy định kỳ turns on a schedule from soot.json.

"improve": { "every": "24h", "at": "03:00", "timezone": "Asia/Ho_Chi_Minh", "budget": 500000, "min_failing": 1 }

A scheduled run evaluates, then proposes (or tunes) only when at least min_failing cases fail, and reports proposed, no_change, budget or waiting_review. It never applies anything.

08 · What a real run looked like

On 2026-10-28 a support Soot started at 2 of 12 cases passing. Training alone (gate, version v0003) raised the held-out test from 0.667 to 0.778 and cut tokens by 34%, yet the adapter carried three rules that contradicted policy. The root cause was a runtime regression; once it was fixed, the authored hill-climb went from 3/14 to 13/14 cases, score 0.65 to 0.99, and tokens per case from 19.9k to 12.8k. A later tune of 8 steps spent 1.3M tokens and kept nothing, and graduation caught four conflicts before they reached the mission. That is the case for authored first, held-out gates and review before apply.

09 · What keeps the numbers honest

• The exam stays out of training data. soot traces export leaves out runs of validation, test, pinned and deleted cases unless you pass --include-held-out; --split train keeps only training cases and --adapter none only authored runs.
• A judge outage is not a failure. When the judge call errors, is cancelled or returns no verdict, the case reads error (rerun it) and no failing label is written.
• Splits survive renames. The hash split uses the Soot and the case id, not the file path; set "split": "val" on a case to pin it by hand. Upgrading reshuffles hash-assigned cases once, so check soot eval --split val afterwards.
• Captured cases keep their conversation. soot eval case from <run> replays the earlier user messages as setup turns (up to 15) when the trace has full content.
• One job per Soot. The CLI and Nest share a lock; a second improve, apply, undo or adapter deploy is refused with improve_running, and Nest's Stop reaches a job started from the CLI.
• Undo is whole or nothing. An undo that would leave a workspace soot check rejects puts every file back and says so (revert_invalid).

Still true today: the token audit treats the first run it sees of a conversation as its start, so a conversation that began before the audit window counts once as fresh; and a case with no Soot anywhere hashes under an empty Soot, so giving it one later can move it.