Soot DocsHow Soot is designedMarkdown ↗

02 / Self-improvement

Self-improvement: evals, traces and adapters

How Soot improves from evidence: evals, traces and tokens, soot improve, adapters, review and undo.

A Soot running for real will be wrong in ways nobody thought of when the mission was written. Soot improves itself on evidence, not on feel: every failure becomes an eval case, every run leaves a trace with tokens, a change is kept only when measuring again shows it is better, and every applied change can be reverted. This is the pillar "you stay in charge, Soot improves on evidence" in Soot's positioning (AGENTS.md, BRANDING.md §21).

Everything here already runs in code (except the "Not yet available" part at the end). The general principle: nothing applies itself. Soot only produces proposals with measurements, and a person decides. There is one exception, and a person has to switch it on: a tuned adapter is used automatically when improve.auto_deploy is in soot.json and the held-out check shows it is better. Soot then watches that version and reverts it by itself if the work gets worse (Auto-using a tuned version). Authored changes (stage 1, stage 3) never apply themselves.

Overall model

Sample text · not a command
        real run / eval run
              │ writes
              ▼
   ┌─────────────────────┐   "Không đúng ý"      ┌──────────────────┐
   │ trace + tokens      │ ───────────────────►  │ eval case        │
   │ (traces.db)         │   soot eval case from │ soots/<id>/evals │
   └─────────────────────┘                       └────────┬─────────┘
              ▲                                           │ runs through conversation API
              │ score labels, tokens                      ▼
              │                                  ┌──────────────────┐
              └───────────────────────────────── │ soot eval        │  score, tokens, adapter digest
                                                 └────────┬─────────┘
                                                          │ failing case
                                                          ▼
   0 Diagnose ─► 1 Fix authored ─► 2 Tune adapter ─► 3 Merge into authored
   rules, free   proposer + gate    soot train (gate)    durable rules → mission
                 diff awaiting      deploy when a        diff awaiting review
                 review             person chooses
                          │                    │                     │
                          └── Áp dụng / Dùng bản này (a person) ─────┘
                                           │
                                  receipt ─► --revert

The two harness layers of a Soot, and who edits which:

LayerLives inWho writesEdited by
Authored: soul, mission, skill, packs, execution, tool descriptionssoots/<id>/, packs/A person, or the authored proposer (stages 1, 3)Diff, reviewed like code
Adapter: learned guidance and skillssoots/<id>/adapter/soot train (stage 2)Versions, deploy / remove
Eval case: the examsoots/<id>/evals/, evals/A person, or soot eval case from <run>Never by the proposer

An adapter only adds to authored and never edits it. Authored fixes the root cause; the adapter only adds text. So the order is always authored first, adapter second.

Concepts

Eval case. A fixed input with hard gates (state, called, not_called, order, max_tool_calls, reply_includes…) and optionally a judge. It runs through the conversation API of the running runtime, the same path soot run takes. There is no separate model loop and no mocked tools. Full list of gates: soot schema eval.

Trace. The record of one run: a header (Soot, origin, model, adapter digest, config revision), each model turn with tokens, each tool call (ok, error, blocked, repeated), the context size of each turn, and the labels attached afterward. The trace is the evidence that evals, diagnosis and training all read.

Weighted tokens. The shared unit for comparing cost: input ×1 + cache write 5 minutes ×1.25 + cache write 1 hour ×2 + cache read ×0.1 + output ×5. Budgets, soot usage, eval reports and training all use this unit. For comparing candidates only, training uses comparable tokens (every prompt token ×1, output ×5) so a candidate's cold cache is not penalized unfairly.

Lever. Something that can be edited to fix a failure: wiring (groups, packs), the tool surface, a tool description, a skill description, the mission, execution, or the adapter. Diagnosis assigns each failing case to exactly one lever.

Adapter. The directory soots/<id>/adapter/ containing adapter.json (the operator's manifest: which parts may be learned, byte limits), ADAPTER.md (appended to the end of the mission under ## Learned guidance), skills/*/SKILL.md, and CARD.md (provenance). Every trace records the digest of the adapter that was running.

Proposer. Whatever proposes edits. Either a Soot on the same runtime (preferably with the strongest model), an operator's program ({"program": [...]}: material in on stdin, proposal out on stdout), or a coding agent following the soot-hillclimb skill.

Gate. The rule for accepting or rejecting a change after re-rolling out the cases. Correct first, cheap in tokens second: tokens never buy back correctness.

Held-out. The val, test and pinned cases. The proposer never sees them; they decide acceptance or rejection.

Evals: locking in behavior

Where cases live, how IDs work

Each Soot keeps its own cases in soots/<id>/evals/**/*.eval.json (soot spawn creates smoke.eval.json). Cases that span several Soots live in evals/ next to deployment.json. ID = Soot ID + the file path without .eval.json + the case id (notes/smoke/answers); index.eval.json takes the directory name. A duplicate ID is the error duplicate_eval. iterations: 3 multiplies one case into three runs to measure the stability of a non-deterministic model.

Terminal · review before running
soot eval                       # every case
soot eval --tag smoke           # by tag
soot eval --soot ops            # every case of one Soot
soot eval notes/delete/refuses  # one case or a whole branch
soot eval --json > base.json && soot eval --baseline base.json

Exit 0 when every case passes; 1 when there is a failed (a gate not met) or an error (no run to grade: refused, timed out, connection lost, broken judge). Flags: soot eval --help.

Evals do no harm

Each case runs in a conversation with origin `eval`:

  • Isolated memory. The Soot neither reads nor writes user/workspace memory; evals teach the Soot nothing about you, and results do not depend on earlier runs. Knowledge gaps and feedback from evals do not reach the team.
  • Dry run by default ("effects": "dry_run"). A pack's write/repair tools run only the tool.preflight hook; the Soot receives dry run: … with a preview or a refusal. Host tools with effects (write, bash, http_request…) also return dry run:. Read tools run for real. So called / order can still lock in the decision to call a write tool, and rerunning any number of times is safe. "effects": "live" allows real execution for end-to-end checks on test data.
  • Not in Cần bạn ("Needs you"). An eval conversation does not appear in Cần bạn (see it with soot attention list --include-evals); clean up with soot conversations close --origin eval.
  • Public events only (tool.started, tool.failed): tool name and order, never arguments or output.
  • Needs an operator token and costs real model calls. Run it on a dev workspace.

Judge

For quality that cannot be written as a hard gate. A judge is an ordinary Soot (for example grader, no tools) running in its own conversation; the answer being graded is fenced with a random fence and sent as data, and the verdict is read only from the first line (PASS or FAIL: <reason>). A broken judge is not a failed case: the case becomes error and is not labeled as failing (so training and diagnosis data stay clean). Prefer hard gates and use judges sparingly.

Score, tokens, labels

A report has a score (the average, over cases, of the fraction of checks passed, finer than pass/fail) and weighted_tokens for the whole suite and for each case. With --label (the default) each run gets a label source: eval (name = case ID, score, pass, failed checks) on its trace, so soot traces export --labeled can pull a dataset and soot usage --origin eval shows the suite's cost. --baseline compares two reports: improved, regressed, new, missing. A hillclimb step passes when the target case is improved and nothing regressed.

--adapter <dir|none> runs the cases with a candidate adapter without deploying it (the overlay is fixed when the conversation is created, and the runtime rechecks the digest). That is how an adapter is measured on the held-out set and it is the rollout mechanism of soot train.

Case store: cheaper than running training

Evals are the main asset; the algorithm is only as good as the case set. So creating cases has to be cheap.

  • From a real run. In Nest, the Không đúng ý ("That's not right") button on an answer creates a case from that run. On the CLI: soot eval case from <run> --not-called spawn_subagent --reply-excludes "sẽ sửa" --pinned. The command reads the messages from the trace (needs traces.content: full, or pass --send), appends the case to soots/<soot>/evals/captured.eval.json with the tag from-run. A run in the middle of a conversation becomes a multi-turn case (the early turns only check state). Review the file like code.
  • Automatic split. Priority order: pinned > "split" in the case > tag train/val/test > hash of <soot>/<case id> (60% train, 20% val, 20% test, stable between runs; renaming or moving a file does not change the set). With fewer than 8 val cases, soot train and soot improve --diagnose warn.
  • `pinned: true` marks a policy rule ("never promise refunds", "money that was charged must go to a person"). A pinned case always belongs to val and test, never enters the proposer's material, and a candidate that fails a pinned case its parent passed is rejected regardless of score.
  • Do not leak the exam. soot traces export by default leaves out runs of val, test and pinned cases, and of deleted cases (fail closed: deleting a case does not turn it into training data).

Replay: measuring on recorded conversations

soot eval replay --soot le-tan --cases cases.jsonl --report replay.json runs a Soot on real recorded conversations (for example a customer-support export), each case being a dry-run eval conversation, and read tools with a matching fixture return the fixture. The report computes escalation_recall, over_escalation, forbidden_calls, unsourced_numbers, run_error_rate, p95_first_reply_ms, cost_usd; each run is also labeled onto its trace. Exit 0 means the replay finished running; the gate for accepting or rejecting is the report reader's job. Full contract and fixtures: contracts/eval-replay.md.

Traces and tokens: the evidence

Every run is recorded as a trace from the moment it starts. Recording a trace adds no model call, changes no behavior, and a write failure does not break the run. The data is in traces.db next to the runtime's state, read through the operator API.

Terminal · review before running
soot traces list --since 24h
soot traces show <run>
soot traces label <run> --score 0.8 --passed --feedback "correct but long"
soot traces export --format chat --passed --out sft.jsonl
soot usage --since 7d

What content to keep

traces.content in the deployment:

ValueWhat is kept
shape (default)Tokens, timing, tool names, status, sizes, argument hashes. No text.
fullAdds input, output, arguments and tool results, with secrets redacted and truncated.
offOnly the daily token ledger.

Two rules override the config: runs from public customers are always shape; runs with origin: eval are always full (unless off) because they are the operator's fixtures and training needs the content. Retention deletes old traces first, then shrinks large traces without dropping the numbers.

What a trace contains

Header (Soot, origin, model, adapter digest), tokens (input, output, cache_read, cache_write, reasoning, weighted), context per turn, each turn, each tool call (ok|error|blocked|invalid, repeat), compaction and labels. A long run split across several slices is still one trace. If the process dies mid-run, on the next start the runtime closes it as failed / runtime_interrupted, keeping the numbers already recorded; each token enters the ledger exactly once. Details: soot traces show <run>, internal/feature/traces/types.go.

Budget

execution.max_run_tokens and execution.daily_tokens (weighted tokens, set in the deployment or per Soot). A run about to hit the limit gets one final turn with no tools and then stops with the stop reason budget_exhausted; a new run of a Soot that has used up its daily budget is rejected with token_budget_exhausted. Runs with origin: eval go into the ledger (eval_weighted_tokens) but do not count against daily_tokens, so overnight improvement does not use up the real chat budget; they are limited by the train/improve budget.

soot usage: token audit

Reads traces over a window and prints a token-audit template: input / cache / output split, cache hit (first turn and later turns), per-turn background (instructions, skills, tool schemas, the part of the system prompt the runtime adds), carry (large tool results carried into later turns), friction (tool errors, blocked calls, repeated calls, stops from budget) and a list of proposals ranked by saving ÷ risk, each line naming the exact setting to change. --json for automation. The suggestions of soot usage feed stage 0 diagnosis as stage 1 levers. It is an input for measuring tokens; it changes nothing by itself.

Outcomes

Outcomes (units of finished work declared by a pack, counted with their cost) are not yet an input to the improvement loop in code: improve and train do not read them. Nest shows the self-improvement report next to outcomes (the Kết quả ("Results") section), but wiring "outcome drops, so open a case" is still a direction (workspace.md).

Improve: the four-stage pipeline

soot improve <soot> and the Cải thiện ("Improve") button on a Soot's tab in Nest run the same pipeline. Stage 0 is free, stages 1 and 3 produce diffs awaiting review, stage 2 produces an adapter version.

Stage 0: Diagnose (rules, no model)

soot improve ops --diagnose assigns each failing case to one lever, from signals in the trace and soot info. It counts only eval runs of the currently deployed adapter; a judge error does not count as failing; a multi-turn case fails when any turn fails.

Signal in the traceLeverStage
A tool returns "not found", group members cannot be called by nameWiring: groups, default_groups, packs1
Calling tools outside the Soot's job (spawn_subagent, learn, memory_edit, ask), a large tool surfaceTool surface: packs (pack versus pack:read), execution1
A tool fails on arguments, repeated callsTool description/arguments (third-party pack: propose to the pack owner)1
Reads the wrong skill or reads many skillsSkill description1
Fails a pinned case, answers against a ruleMission/skill1
Stops on turn limit/budget, gets asked againexecution, runtime contract1
Right tool, right rule; wrong detail or token-heavyAdapter2
The case's expectation contradicts itself or is wrongFix the casePerson

Why this stage exists: the baseline for the Lohi customer-support eval set passed only 2/12 cases. Training an adapter "raised the score", but mostly by noise, because the root cause was wiring (one group made the pack lose all its tools). Removing that group with an authored fix took it to 13/14. An adapter cannot fix a wiring error, and a small val set accepts noise too. That is why diagnosis reads the tool result ("not found") and not just the tool name.

Stage 1: Fix authored (strong proposer, needs review)

soot improve ops --propose --proposer <strong-soot>:

  1. The proposer reads the whole authored part, the diagnosis result and the traces of the failing cases, then proposes one lever per round (at most 3 rounds) as a patch. The usual format is LEVER: / RATIONALE: / === EDIT <file> with --- FIND / --- REPLACE, or a JSON object; REFER <pack>: … for third-party packs. The proposer may not invent facts (opening hours, prices, names) that the files and cases do not state.
  2. Each patch is built into a trial workspace and rolled out on every train + val case (with iterations) on a soot serve --trial runtime: a temporary runtime that only answers the conversation API and does not run the copy's schedules or periodic improve.
  3. Gate: a patch is rejected if it fails a pinned case or does not raise the score. The decision is measured twice (the fix the gate is about to accept is rolled out again, since a single rollout of a non-deterministic model may just be luck); --no-confirm skips this step, cheaper but noisier.
  4. A fix blocked by a pinned case is held for one round: sometimes it exposes a bug that it was hiding, and the next round pulls the mission lever on top of that fix so the two changes are gated as one.
  5. Finally the test set (held back until now) is scored, and .soot/improve/<soot>/proposal/{proposal.json,changes.diff} is written. That is the diff awaiting review.

Edit scope: soots/<id>/ (mission, soul, packs, execution, the Soot's own skills) and local packs in packs/. It never edits evals/ (they grade, they are not a lever), adapter/ (stage 2's) or bin/. Vendored or Forge packs are never edited in place.

soot improve ops --status shows the result; --apply is a human action: it writes the approved proposal (or the Áp dụng ("Apply") button in Nest). The --budget / improve.budget budget is in weighted tokens.

Stage 2: Tune the adapter (soot train)

Opens only when diagnosis has no stage 1 cases left and there is no authored diff awaiting review (--tune, locked behind stage 1; an operator can force it, and the result carries a warning label). soot train also runs on its own:

Terminal · review before running
soot adapter init ops                          # empty soots/ops/adapter/
soot train init nightly --soot ops             # train/nightly.train.json
soot train nightly                             # evolve the adapter, checkpoint every step
soot adapter diff none .soot/train/nightly/versions/v0005
soot eval --adapter .soot/train/nightly/versions/v0005 --tag test
soot adapter deploy .soot/train/nightly/versions/v0005
soot adapter remove ops

Each step: sample a minibatch from train, build the learning material (the current adapter and its limits, the authored mission excerpt, the list of authored skills, the failing train cases with the shape of their tool calls, at most 5 passing cases, token data per case and per tool, the last 5 steps), the proposer returns rationale + guidance + skills, the runtime checks (inside the manifest, an edit bounded by max_edit_bytes, repaired up to 2 times if wrong), rolls out the candidate on the same minibatch, and the algorithm decides.

algorithmKeep the candidate whenFits
continualthe minibatch objective does not drop; no val neededCheapest, continuous learning
gate (default)the minibatch does not drop and the val objective rises by more than min_gainSafe default
paretothe minibatch objective rises; the parent is drawn from the frontier by the number of val cases it is best onMany kinds of cases
compressthe score does not drop and tokens per case dropAfter the score is reached: cutting tokens

Two objectives, correct before cheap. If the score drops by more than score_tolerance the candidate is rejected, no matter how many tokens it saves. Otherwise it must raise objective = score − token_weight × (token/case ÷ initial token/case) (token_weight defaults to 0.1): the same score with fewer tokens counts as progress.

Guarding against overfitting and against learning wrong rules:

  • The proposer sees only train; val decides; test is scored only once at the end (for none, the initial adapter and the best).
  • pinned cases do not enter the material; a candidate that fails one is rejected. When there is no val (continual, compress), the runtime still scores the pinned cases separately for the parent and the candidate.
  • The step history the proposer sees contains only the kind of decision, no val scores and no pinned case IDs.
  • The material states the runtime contract (the final answer must contain STATUS: DONE; an adapter that teaches a different last line gets asked again and doubles the tokens), the list of authored skills, and the rule "an adapter never goes against authored". That rule exists because one adapter once learned to forbid read_skill and promised to "will fix" a faulty chapter, against the mission. Lock policy with pinned cases, and always review soot adapter diff before deploying.

State lives in .soot/train/<name>/ (state.json, index.jsonl, report.md, versions/vNNNN/, material/), not in the bundle. It stops at budget.steps, budget.tokens (rollout plus proposer) or budget.rollouts; Ctrl-C keeps the state, and --resume redoes the interrupted step. soot train status <name> prints the report.

Stage 3: Merge into authored

soot improve ops --graduate: rules of the adapter that hold up across many versions are rewritten into the mission/skill by the authored proposer (a diff awaiting review), and then the adapter starts over from empty. So the adapter stays short and proven rules become real config. Two lines count as one rule when they share at least half their words and negate the same things (không, chưa, cấm, never…), so "never promise refunds" is not treated as the same rule as "always promise refunds". The contradiction check step compares rules against the mission/skill; if there is a contradiction it stops at conflicts and creates no diff.

Which model for which role

RoleModel
DiagnosisNo model (rules)
Authored proposer (stages 1, 3) and the contradiction checkThe strongest the workspace has
Adapter proposer (stage 2)Medium or better
Rollout (running cases)Always the Soot's model: measuring with another model measures the wrong thing

The proposer is only about a tenth of the cost (most of it is rollouts), so to make it cheaper reduce rollouts and do not downgrade the proposer. Choose the proposer with --proposer, or proposer.soot in the train file.

Who triggers what

WhoHowLimits
Person (CLI)soot improve …, soot train …, soot adapter deploy/remove, soot eval case fromThe only party that applies and reverts
Person (Nest)The Soot's Cải thiện tab: Chẩn đoán (Diagnose) → Sửa cấu hình (Fix config) → Tinh chỉnh (Tune) → Gộp (Merge), the Áp dụng / Bỏ (Discard) / Hoàn tác (Revert) buttons, Không đúng ý on an answerThrough POST /v1/improve (operator)
Lead (the workspace's coordinator)soot manage has two commands: improve.status (read, with the signals, the number of drafts and the auto-used version) and improve.run (an automatic run, soot improve <soot> --auto, trigger lead, default budget 500000, maximum 2000000)Does not apply, discard, schedule or accept drafts. Refuses when there are no cases (no_eval_cases), one is running (improve_running) or a proposal is already waiting (improve_pending)
Schedule"improve" in soot.json; the runtime runs soot improve <soot> --scheduledOnly creates proposals; an adapter is used automatically only with auto_deploy
Watch (the runtime reads traces itself)"improve": {"watch": {…}}; when a signal passes its threshold, the runtime runs soot improve <soot> --auto --trigger watchSame limits as the schedule; a cooldown between two runs; does not run while the schedule is paused, a job is running, a proposal is waiting or an auto-used version is being watched

The improve schedule (optional, one per Soot, set only by an operator):

JSON sample · not a command
"improve": {"every": "24h", "at": "03:00", "timezone": "Asia/Ho_Chi_Minh",
            "budget": 500000, "min_failing": 1}

Each run: run all evals; below min_failing it is healthy; if there is an unreviewed proposal it is waiting_review (no overwrite); if config errors remain it runs stage 1; otherwise stage 2. every is at least 6h; budget is required and covers both evals and fixes; a missed run is not made up; it does not overlap a running job. Dead ends are not rerun: each run records a fingerprint (the content of files in the edit scope, the eval cases, the proposer); if the last run ended no_change and the fingerprint has not changed, this run costs nothing.

A Soot has only one improve job at a time: propose, tune, graduate, scheduled, auto, apply, revert and adapter deploy/remove all hold the lock .soot/improve/<soot>/job.lock; a second command is rejected with improve_running (HTTP 409). Within one process the lock is re-entrant: an automatic run that holds the lock can still call soot adapter deploy.

soot improve <soot> --auto is exactly a scheduled run, but it starts now, called by a person (--trigger manual), Lead (lead) or watch (watch, with --signal). A Soot with no schedule uses --budget (default 500000). When a person or Lead asks, Soot skips the "nothing has changed" shortcut: asking means they want a fresh measurement.

Watching signals

A schedule runs by the clock; watch runs by evidence. On each check, the runtime reads the traces of real runs inside the window (skipping eval runs, dry runs and runs still in progress) with no model call and no tokens:

SignalWhat it measuresDefault threshold
failed_rateShare of broken runs: failed, out of token budget, interrupted (a run cancelled by a person does not count)above 20%
tool_error_rateShare of tool calls that errored or were blockedabove 20%
memory_writes_per_runCalls to learn / memory_edit per run. A Soot that keeps taking notes for itself is patching its way of working one run at a time, which is worth fixing at the root in the configabove 1
pass_rateShare of cases that pass in the latest eval on the adapter in usebelow 80%
JSON sample · not a command
"improve": {"every": "24h", "budget": 500000,
            "watch": {"window": "24h", "check": "1h", "cooldown": "24h", "min_runs": 20,
                      "failed_rate": 0.2, "tool_error_rate": 0.2,
                      "memory_writes_per_run": 1, "pass_rate": 0.8, "draft_cases": 3}}

Every field is optional; a threshold of 0 turns that signal off. The first three rates are considered only when there are at least min_runs runs (a few stray runs are just noise). pass_rate is measured on the eval set, so it needs no run count. When a signal passes its threshold the runtime starts an automatic run, and signal records those signals as name=value/threshold joined by commas (for example failed_rate=0.35/0.2) so Nest can word it itself. That run still measures the eval set first: if the eval set does not catch what watch saw, it ends healthy and says so, with a reminder to accept the drafts below. On the next watch run, if nothing has changed (same fingerprint: authored files, cases, proposer), it does not measure again but records unchanged with basis: "healthy", at no token cost. Only watch skips this way; the schedule still measures on its own beat, and a person or Lead asking always measures again. Accepting a draft changes the case set, so the next watch run measures again. check is at least 15 minutes, cooldown at least 6 hours, and it counts the schedule's runs too.

soot improve <soot> --signals (or GET /v1/improve/{soot}/signals) prints exactly the numbers watch reads. The latest reading, with what watch did or why it did not, lives in .soot/improve/<soot>/watch.json and in signals of GET /v1/improve. The reason for not acting comes in two forms: held is an English sentence for the log, hold is a code for the screen (guarding, paused, cooldown, busy, review, no_cases, failed), with hold_until when the end time is known.

In Nest, the Improve tab has a Tự phát hiện vấn đề ("Detect problems automatically") block. The switch saves improve.watch through the same soot.improve as the schedule: turning it on when there is no schedule also turns on self-improvement (every day at 03:00, default budget); turning it off removes watch and auto_deploy but keeps the schedule. The block shows the four metrics against their thresholds (Vượt ngưỡng "Over", Ổn "OK", Chưa đủ dữ liệu "Not enough data", Đã tắt "Off"), the number of runs read, the last reading, the reason it did not run, the number of waiting drafts with a button that opens Cần bạn, and the Tự dùng bản tinh chỉnh ("Auto-use the tuned version") box (auto_deploy).

Draft tests

Watch also drafts cases from the broken runs, newest first, at most draft_cases per reading and no more than 20 waiting. A draft is built the same way as the Không đúng ý button: the run's message becomes send, the condition is that the run finishes, and if a tool errored then no tool may error. Drafts live in .soot/improve/<soot>/drafts/, show in Cần bạn (Nhận vào bộ kiểm tra "Add to the eval set" / Bỏ "Discard") and in soot improve <soot> --drafts [--accept <id> | --discard <id>].

The card in Cần bạn speaks in plain words: the message, the answer Soot gave (stored with the draft, at most 2000 characters), when the case passes, and a warning when the case only checks "finished, no error" without looking at the content of the answer. The raw data of the case sits in a collapsed section. These fields travel in the item's payload (send, earlier, reply, checks, loose, problem, run).

Until a person accepts it, a draft is not measured, not used for training and does not block any change. The eval set is the exam, and Soot does not write its own exam. Once accepted, the case goes into captured.eval.json like any other and is split into train/val/test by hash as usual. Watch does not draft a run it already drafted, and does not draft two cases for the same message. A run with no content (traces.content is not full, or an anonymous visitor) cannot be drafted.

Auto-using a tuned version

JSON sample · not a command
"improve": {"every": "24h", "budget": 500000,
            "auto_deploy": {"min_gain": 0.05, "min_test_cases": 5, "hold": "72h"}}

With auto_deploy, when an automatic run (from the schedule, watch or Lead) finishes training in stage 2, it deploys the best version by itself if it passes all of these gates:

  • Training has finished and the version has a score on the held-out check (test). The best version is not v0000 and is not already in use.
  • The test part has at least min_test_cases cases, and the score rises by at least min_gain over the adapter in use.
  • Every pinned case passes on the test part.
  • This version has never been reverted for making Soot worse, and no other auto-used version is still being watched.

The deploy goes through exactly soot adapter deploy, so it has a receipt and can be reverted. After that, within hold, watch compares Soot with how it was before the change: the eval set passes less than in the eval at deploy time, or the share of broken runs since deploy exceeds both the failed_rate threshold and the share before deploy. On either, watch reverts through the receipt (passing soot check like every revert) and records the digest in the list of versions that are not used automatically again. If hold ends and it is not worse, the version stays. During hold, watch starts no other improvement run, so only one change is measured at a time.

Cần bạn reports a version that was just auto-used (Giữ "Keep" / Hoàn tác "Revert"), a version that watch reverted, or a revert that could not be done because a file was edited afterwards (then a person reverts it in the Improve tab). If a person reverts it, that version is not auto-used again either. The card has the numbers to compare: the separate exam before and after, the eval pass rate and the broken-run rate at deploy time against watch's latest reading (suite_now, failed_now; "not measured yet" when there is no eval on the new version or not enough runs), and the watch deadline. When watch reverts, worse records which signal got worse, from what to what. The card calls the Soot by the name its owner gave it (not the id). A successful automatic revert is a notice (info): nothing broke and it has nothing to do with the budget; only revert_failed is an error. State lives in .soot/improve/<soot>/auto_deploy.json: watching, kept, reverted, undone, revert_failed. Watch and the person's answer (Giữ, Đã hiểu "Got it", Hoàn tác) write this file under the same lock and re-read it before writing: if a person kept it, watch does not overwrite it and does not revert it. Authored changes never go this way.

Improve the whole workspace from flows

The four-stage pipeline above improves one Soot from its evals. What breaks in a group of Soots is usually between teams or in the team config (a relation missing a field, a per-turn limit too small, a process with no way to finish, a team that does not know something everyone assumes it knows). A flow (plans/team-flows.md) is the test of the whole workspace, so there is a separate loop that improves the workspace from flows: soot improve --flows. The package is internal/feature/improve/wsimprove, and it reuses improve.Gate (the same gate as the Soot loop: val does not drop, train+val rises by more than MinGain, no pinned flow that was passing now fails), improve.Snapshot/Revert (receipts and drift-safe revert of files) and the job lock .soot/improve/workspace/.

The loop: measure train+val k times on a copy of the workspace (each try is a trial runtime), diagnose (soot flow diagnose), apply every no-model fix as one round (measure again; if it only raised a limit, plan again from the candidate's own measurement, at most 3 passes, then through the gate once), then the proposer's rounds, each pulling one lever, then one confirmation and the test flows. The default budget is 500000 converted tokens; going over it costs money.

Diagnosis layerLeverWho
ontologyadd a rejected field (a queue field, a field of a read relation), add a missing relation or verb, add a closing transition only when the diagnosis names it; never remove gatesno model
budgetthe Soot's execution.max_run_tokens/max_tool_calls/max_turns = measured peak ×1.5, inside that Soot's execution.daily_tokensno model
budget above daily_tokens, the team's or host's daily budget, a more expensive model tier, a paid tool, an improve budget above 500000, removing gatesmoney: never applied by itself, becomes a proposal for a person to decideperson
teamone fact in team memory (source improve, stating what is true and why, not a numbered procedure, at most 400 characters) or the team descriptionproposer (model)
Sootan eval draft from the first run that went off course, source flow:<flow id>, through the existing draft mechanism (soot improve <soot> --drafts); never measured before a person accepts itthe person accepting the draft
packonly a referral to the pack owner-
worldreport only-
noisedo nothing-

Team memory is runtime state, not a file, so a candidate's fact is seeded into each trial runtime through flows.Options.Memory; when applied, the fact enters the real runtime through POST /v1/workspace/memory (tag source:improve) and the receipt keeps its id so that a revert forgets exactly that fact. The receipt contains the Soot loop's file receipt and the facts taught.

The result is a diff, the facts, the money proposals and the drafts, waiting for review (--propose, --status). --apply writes everything except money, then runs val+pinned again; if the pass rate is lower than the baseline recorded at measurement, it reverts by the receipt on its own and records the reason (--history, --revert <receipt>). --auto is --propose then --apply. A source-owned workspace is never applied on the host for files that belong to the source: --pull --into <repo> writes that part into the checkout. API: POST /v1/improve/flows (mode propose|auto), GET /v1/improve/flows, POST /v1/improve/flows/apply, POST /v1/improve/flows/revert, GET /v1/improve/flows/history; the job is the CLI itself.

Review and revert

A proposal awaiting review appears in Cần bạn (Apply / Discard) and in the Improve tab, with the diff, before/after scores and cost. Only a person applies.

Every applied change (--apply, graduate, soot adapter deploy, soot adapter remove) leaves a receipt in .soot/improve/<soot>/history/<id>/: the before/after hash of each file and a copy of the previous content. If writing a file fails midway, every file is put back; if an adapter deploy fails validation, the old adapter is restored.

Terminal · review before running
soot improve ops --history          # newest first
soot improve ops --revert           # revert the newest change not yet reverted
soot improve ops --revert --id <id>

Revert restores, it does not merge, and it is all or nothing: if a file was edited after that change, the command refuses (improve_revert_drifted, HTTP 409) and writes nothing; after writing, the workspace must pass soot check, otherwise every file is put back as before (improve_revert_invalid). Reverting an adapter deploy returns to exactly the previous adapter, not just to "no adapter". Apply, revert and deploy/remove each request one reload on the running runtime (POST /v1/improve/{soot}/reload), so the next scheduled run measures the new config.

A Soot that belongs to a repo (agent as code)

A Soot from a source (agents.md) is defined in the repo and deployed by replacing the whole slice, so the server never writes into the slice directly (that would become drift). The rule: the repo always wins.

Sample text · not a command
GitHub repo ──release──▶ VM: sources.d/shop/…         (deploy replaces the whole slice)
                         VM: .soot/adapters/seller/   (tuned version learned on the VM)
       ◀──soot improve seller --pull──  proposal + tuned version, to review/commit

  • --apply is rejected (improve_source_owned): stage 1 proposals and stage 3 graduation go into the repo with soot improve <soot> --pull --into <checkout> (--export previews), then commit and release like any code change. Nest changes the Áp dụng button to Đưa vào repo ("Put into the repo").
  • soot adapter deploy writes to .soot/adapters/<soot>/ together with host.json, which records the adapter the repo was shipping at deploy time (over). This version applies only while the repo still ships exactly that adapter. The host_adapter.state values: active, in_repo, superseded, conflict; a version that cannot apply never breaks loading.
  • When a release carries the change to the server, everything closes up: a proposal that the tree already contains moves to in_repo.

Report: what Soot has improved on its own

The report is built purely from files that already exist (state*.json, scheduled.jsonl, receipts, pending proposals). GET /v1/improve returns for each Soot: pending, blocked (why the latest run changed nothing), applied / reverted, spent (weighted tokens over 30 days), runs (at most 20), schedule, next, cases. Each Soot also has signals (watch's latest reading), drafts (the number of waiting drafts) and auto_deploy. Each run has trigger (manual, scheduled, lead, watch with signal), stage, outcome (proposed, applied, discarded, in_repo, no_change, unchanged, healthy, waiting_review, budget, tuned, deployed with deployed, failed…), passed_before / passed_after, spent, levers. GET /v1/improve/{soot} adds report, history, watch, drafts and auto_deploy. Nest shows it at Kết quả › Tự cải thiện ("Results › Self-improvement") and in the Improve tab; Lead answers with the same numbers through improve.status (including the morning brief).

Input for coding agents: hillclimb

Hillclimb is the second form of stage 1 proposer: a coding agent (Claude Code, Cursor, Codex…) edits the source directly. soot spawn writes this loop as the soot-hillclimb skill in the workspace's .agents/skills/ and .claude/skills/:

  1. soot check --strict --json and soot info --json: the config is valid, and the Soot sees the right tools and skills.
  2. Write a case that reproduces the wrong behavior; soot eval <id> must see it fail.
  3. soot eval --json > base.json, then change one thing.
  4. soot eval --baseline base.json: the target case is improved, nothing regressed; then soot eval --tag smoke.
  5. Keep the case in soots/<id>/evals/ as evidence that the improvement still holds.

Compared with soot train: hillclimb can edit any authored part and starts from one specific failure, but it is manual and has no held-out set; soot train only adds guidance to the adapter, starts from the whole case set, and has val/test that the proposer does not see.

Boundaries and what is deliberately not done

WhatWhy
No separate model loop for evalsEvery rollout goes through the conversation API, the same path as a real run; what is measured is the real Soot, skills, tools and model
No automatic apply of authored changes; an adapter only when a person turns on auto_deployA person decides; every apply has a receipt and can be reverted, and an auto-used version is watched and reverted by itself when it gets worse
A watch draft is not measured before a person accepts itThe eval set is the exam; Soot does not write the exam that grades itself
No val, test or pinned in the proposer's materialSo training does not change policy or memorize the exam to get a score
No editing evals/ during improvementCases grade, they are not a lever
No editing vendored/Forge packs in placePropose a PR or issue to the pack owner
No final result from a rollout with a model other than the Soot'sAn adapter learned on a cheap model may not transfer to the real model
No letting tokens decide over correctnessA lower score is rejected even when cheaper
Lead does not apply, discard, schedule or accept draftsOnly an operator or a person with the permission does
WhoOwns
Core (runtime)The eval runner, trace store, the improve field, the pipeline and job lock, receipts, the /v1/improve, /v1/traces, /v1/usage APIs
PackTools with tool.preflight (so evals see what a tool would refuse without side effects); evals and skills that ship with a pack when Forge releases it
Host / projectThe project's eval cases, pinned policy, improve schedule and budget
NestDisplay and actions: the Improve tab, Không đúng ý, Cần bạn, Kết quả; it has no improvement logic of its own (surfaces.md)

Not yet available / limits

  • Outcomes are not yet an automatic input to the improvement loop (see above).
  • Watch compares the eval set on an auto-used version only once an eval has run on that version (the next scheduled or watch run, or soot eval); before that it relies on the broken-run rate alone.
  • Watch reads at most 5000 runs in each window.
  • Diagnosis only classifies by rules; a model only explains, optionally, and does not classify.
  • A small val set (under about 8 cases) gives noisy scores; this only warns and does not block.
  • An adapter learned with a cheap model is not guaranteed to transfer to the real model; the final test has to run on the model the Soot uses.
  • The self-measured pass rate over time is shown, but there is no gate that blocks a release automatically when the pass rate drops (the gate is the report reader's job, as with replay).

Content reviewed: 2026-11-12