02 / Self-improvement
Self-improvement: evals, traces and adapters
How Soot improves from evidence: evals, traces and tokens, soot improve, adapters, review and undo.
A Soot running for real will be wrong in ways nobody thought of when the mission was written. Soot improves itself on evidence, not on feel: every failure becomes an eval case, every run leaves a trace with tokens, a change is kept only when measuring again shows it is better, and every applied change can be reverted. This is the pillar "you stay in charge, Soot improves on evidence" in Soot's positioning (AGENTS.md, BRANDING.md §21).
Everything here already runs in code (except the "Not yet available" part at the end). The general principle: nothing applies itself. Soot only produces proposals with measurements, and a person decides. There is one exception, and a person has to switch it on: a tuned adapter is used automatically when improve.auto_deploy is in soot.json and the held-out check shows it is better. Soot then watches that version and reverts it by itself if the work gets worse (Auto-using a tuned version). Authored changes (stage 1, stage 3) never apply themselves.
Overall model
real run / eval run
│ writes
▼
┌─────────────────────┐ "Không đúng ý" ┌──────────────────┐
│ trace + tokens │ ───────────────────► │ eval case │
│ (traces.db) │ soot eval case from │ soots/<id>/evals │
└─────────────────────┘ └────────┬─────────┘
▲ │ runs through conversation API
│ score labels, tokens ▼
│ ┌──────────────────┐
└───────────────────────────────── │ soot eval │ score, tokens, adapter digest
└────────┬─────────┘
│ failing case
▼
0 Diagnose ─► 1 Fix authored ─► 2 Tune adapter ─► 3 Merge into authored
rules, free proposer + gate soot train (gate) durable rules → mission
diff awaiting deploy when a diff awaiting review
review person chooses
│ │ │
└── Áp dụng / Dùng bản này (a person) ─────┘
│
receipt ─► --revertThe two harness layers of a Soot, and who edits which:
| Layer | Lives in | Who writes | Edited by |
|---|---|---|---|
Authored: soul, mission, skill, packs, execution, tool descriptions | soots/<id>/, packs/ | A person, or the authored proposer (stages 1, 3) | Diff, reviewed like code |
| Adapter: learned guidance and skills | soots/<id>/adapter/ | soot train (stage 2) | Versions, deploy / remove |
| Eval case: the exam | soots/<id>/evals/, evals/ | A person, or soot eval case from <run> | Never by the proposer |
An adapter only adds to authored and never edits it. Authored fixes the root cause; the adapter only adds text. So the order is always authored first, adapter second.
Concepts
Eval case. A fixed input with hard gates (state, called, not_called, order, max_tool_calls, reply_includes…) and optionally a judge. It runs through the conversation API of the running runtime, the same path soot run takes. There is no separate model loop and no mocked tools. Full list of gates: soot schema eval.
Trace. The record of one run: a header (Soot, origin, model, adapter digest, config revision), each model turn with tokens, each tool call (ok, error, blocked, repeated), the context size of each turn, and the labels attached afterward. The trace is the evidence that evals, diagnosis and training all read.
Weighted tokens. The shared unit for comparing cost: input ×1 + cache write 5 minutes ×1.25 + cache write 1 hour ×2 + cache read ×0.1 + output ×5. Budgets, soot usage, eval reports and training all use this unit. For comparing candidates only, training uses comparable tokens (every prompt token ×1, output ×5) so a candidate's cold cache is not penalized unfairly.
Lever. Something that can be edited to fix a failure: wiring (groups, packs), the tool surface, a tool description, a skill description, the mission, execution, or the adapter. Diagnosis assigns each failing case to exactly one lever.
Adapter. The directory soots/<id>/adapter/ containing adapter.json (the operator's manifest: which parts may be learned, byte limits), ADAPTER.md (appended to the end of the mission under ## Learned guidance), skills/*/SKILL.md, and CARD.md (provenance). Every trace records the digest of the adapter that was running.
Proposer. Whatever proposes edits. Either a Soot on the same runtime (preferably with the strongest model), an operator's program ({"program": [...]}: material in on stdin, proposal out on stdout), or a coding agent following the soot-hillclimb skill.
Gate. The rule for accepting or rejecting a change after re-rolling out the cases. Correct first, cheap in tokens second: tokens never buy back correctness.
Held-out. The val, test and pinned cases. The proposer never sees them; they decide acceptance or rejection.
Evals: locking in behavior
Where cases live, how IDs work
Each Soot keeps its own cases in soots/<id>/evals/**/*.eval.json (soot spawn creates smoke.eval.json). Cases that span several Soots live in evals/ next to deployment.json. ID = Soot ID + the file path without .eval.json + the case id (notes/smoke/answers); index.eval.json takes the directory name. A duplicate ID is the error duplicate_eval. iterations: 3 multiplies one case into three runs to measure the stability of a non-deterministic model.
soot eval # every case
soot eval --tag smoke # by tag
soot eval --soot ops # every case of one Soot
soot eval notes/delete/refuses # one case or a whole branch
soot eval --json > base.json && soot eval --baseline base.jsonExit 0 when every case passes; 1 when there is a failed (a gate not met) or an error (no run to grade: refused, timed out, connection lost, broken judge). Flags: soot eval --help.
Evals do no harm
Each case runs in a conversation with origin `eval`:
- Isolated memory. The Soot neither reads nor writes user/workspace memory; evals teach the Soot nothing about you, and results do not depend on earlier runs. Knowledge gaps and feedback from evals do not reach the team.
- Dry run by default (
"effects": "dry_run"). A pack'swrite/repairtools run only thetool.preflighthook; the Soot receivesdry run: …with a preview or a refusal. Host tools with effects (write,bash,http_request…) also returndry run:. Read tools run for real. Socalled/ordercan still lock in the decision to call a write tool, and rerunning any number of times is safe."effects": "live"allows real execution for end-to-end checks on test data. - Not in Cần bạn ("Needs you"). An
evalconversation does not appear in Cần bạn (see it withsoot attention list --include-evals); clean up withsoot conversations close --origin eval. - Public events only (
tool.started,tool.failed): tool name and order, never arguments or output. - Needs an operator token and costs real model calls. Run it on a dev workspace.
Judge
For quality that cannot be written as a hard gate. A judge is an ordinary Soot (for example grader, no tools) running in its own conversation; the answer being graded is fenced with a random fence and sent as data, and the verdict is read only from the first line (PASS or FAIL: <reason>). A broken judge is not a failed case: the case becomes error and is not labeled as failing (so training and diagnosis data stay clean). Prefer hard gates and use judges sparingly.
Score, tokens, labels
A report has a score (the average, over cases, of the fraction of checks passed, finer than pass/fail) and weighted_tokens for the whole suite and for each case. With --label (the default) each run gets a label source: eval (name = case ID, score, pass, failed checks) on its trace, so soot traces export --labeled can pull a dataset and soot usage --origin eval shows the suite's cost. --baseline compares two reports: improved, regressed, new, missing. A hillclimb step passes when the target case is improved and nothing regressed.
--adapter <dir|none> runs the cases with a candidate adapter without deploying it (the overlay is fixed when the conversation is created, and the runtime rechecks the digest). That is how an adapter is measured on the held-out set and it is the rollout mechanism of soot train.
Case store: cheaper than running training
Evals are the main asset; the algorithm is only as good as the case set. So creating cases has to be cheap.
- From a real run. In Nest, the Không đúng ý ("That's not right") button on an answer creates a case from that run. On the CLI:
soot eval case from <run> --not-called spawn_subagent --reply-excludes "sẽ sửa" --pinned. The command reads the messages from the trace (needstraces.content: full, or pass--send), appends the case tosoots/<soot>/evals/captured.eval.jsonwith the tagfrom-run. A run in the middle of a conversation becomes a multi-turn case (the early turns only check state). Review the file like code. - Automatic split. Priority order:
pinned>"split"in the case > tagtrain/val/test> hash of<soot>/<case id>(60% train, 20% val, 20% test, stable between runs; renaming or moving a file does not change the set). With fewer than 8 val cases,soot trainandsoot improve --diagnosewarn. - `pinned: true` marks a policy rule ("never promise refunds", "money that was charged must go to a person"). A pinned case always belongs to val and test, never enters the proposer's material, and a candidate that fails a pinned case its parent passed is rejected regardless of score.
- Do not leak the exam.
soot traces exportby default leaves out runs of val, test and pinned cases, and of deleted cases (fail closed: deleting a case does not turn it into training data).
Replay: measuring on recorded conversations
soot eval replay --soot le-tan --cases cases.jsonl --report replay.json runs a Soot on real recorded conversations (for example a customer-support export), each case being a dry-run eval conversation, and read tools with a matching fixture return the fixture. The report computes escalation_recall, over_escalation, forbidden_calls, unsourced_numbers, run_error_rate, p95_first_reply_ms, cost_usd; each run is also labeled onto its trace. Exit 0 means the replay finished running; the gate for accepting or rejecting is the report reader's job. Full contract and fixtures: contracts/eval-replay.md.
Traces and tokens: the evidence
Every run is recorded as a trace from the moment it starts. Recording a trace adds no model call, changes no behavior, and a write failure does not break the run. The data is in traces.db next to the runtime's state, read through the operator API.
soot traces list --since 24h
soot traces show <run>
soot traces label <run> --score 0.8 --passed --feedback "correct but long"
soot traces export --format chat --passed --out sft.jsonl
soot usage --since 7dWhat content to keep
traces.content in the deployment:
| Value | What is kept |
|---|---|
shape (default) | Tokens, timing, tool names, status, sizes, argument hashes. No text. |
full | Adds input, output, arguments and tool results, with secrets redacted and truncated. |
off | Only the daily token ledger. |
Two rules override the config: runs from public customers are always shape; runs with origin: eval are always full (unless off) because they are the operator's fixtures and training needs the content. Retention deletes old traces first, then shrinks large traces without dropping the numbers.
What a trace contains
Header (Soot, origin, model, adapter digest), tokens (input, output, cache_read, cache_write, reasoning, weighted), context per turn, each turn, each tool call (ok|error|blocked|invalid, repeat), compaction and labels. A long run split across several slices is still one trace. If the process dies mid-run, on the next start the runtime closes it as failed / runtime_interrupted, keeping the numbers already recorded; each token enters the ledger exactly once. Details: soot traces show <run>, internal/feature/traces/types.go.
Budget
execution.max_run_tokens and execution.daily_tokens (weighted tokens, set in the deployment or per Soot). A run about to hit the limit gets one final turn with no tools and then stops with the stop reason budget_exhausted; a new run of a Soot that has used up its daily budget is rejected with token_budget_exhausted. Runs with origin: eval go into the ledger (eval_weighted_tokens) but do not count against daily_tokens, so overnight improvement does not use up the real chat budget; they are limited by the train/improve budget.
soot usage: token audit
Reads traces over a window and prints a token-audit template: input / cache / output split, cache hit (first turn and later turns), per-turn background (instructions, skills, tool schemas, the part of the system prompt the runtime adds), carry (large tool results carried into later turns), friction (tool errors, blocked calls, repeated calls, stops from budget) and a list of proposals ranked by saving ÷ risk, each line naming the exact setting to change. --json for automation. The suggestions of soot usage feed stage 0 diagnosis as stage 1 levers. It is an input for measuring tokens; it changes nothing by itself.
Outcomes
Outcomes (units of finished work declared by a pack, counted with their cost) are not yet an input to the improvement loop in code: improve and train do not read them. Nest shows the self-improvement report next to outcomes (the Kết quả ("Results") section), but wiring "outcome drops, so open a case" is still a direction (workspace.md).
Improve: the four-stage pipeline
soot improve <soot> and the Cải thiện ("Improve") button on a Soot's tab in Nest run the same pipeline. Stage 0 is free, stages 1 and 3 produce diffs awaiting review, stage 2 produces an adapter version.
Stage 0: Diagnose (rules, no model)
soot improve ops --diagnose assigns each failing case to one lever, from signals in the trace and soot info. It counts only eval runs of the currently deployed adapter; a judge error does not count as failing; a multi-turn case fails when any turn fails.
| Signal in the trace | Lever | Stage |
|---|---|---|
| A tool returns "not found", group members cannot be called by name | Wiring: groups, default_groups, packs | 1 |
Calling tools outside the Soot's job (spawn_subagent, learn, memory_edit, ask), a large tool surface | Tool surface: packs (pack versus pack:read), execution | 1 |
| A tool fails on arguments, repeated calls | Tool description/arguments (third-party pack: propose to the pack owner) | 1 |
| Reads the wrong skill or reads many skills | Skill description | 1 |
Fails a pinned case, answers against a rule | Mission/skill | 1 |
| Stops on turn limit/budget, gets asked again | execution, runtime contract | 1 |
| Right tool, right rule; wrong detail or token-heavy | Adapter | 2 |
| The case's expectation contradicts itself or is wrong | Fix the case | Person |
Why this stage exists: the baseline for the Lohi customer-support eval set passed only 2/12 cases. Training an adapter "raised the score", but mostly by noise, because the root cause was wiring (one group made the pack lose all its tools). Removing that group with an authored fix took it to 13/14. An adapter cannot fix a wiring error, and a small val set accepts noise too. That is why diagnosis reads the tool result ("not found") and not just the tool name.
Stage 1: Fix authored (strong proposer, needs review)
soot improve ops --propose --proposer <strong-soot>:
- The proposer reads the whole authored part, the diagnosis result and the traces of the failing cases, then proposes one lever per round (at most 3 rounds) as a patch. The usual format is
LEVER:/RATIONALE:/=== EDIT <file>with--- FIND/--- REPLACE, or a JSON object;REFER <pack>: …for third-party packs. The proposer may not invent facts (opening hours, prices, names) that the files and cases do not state. - Each patch is built into a trial workspace and rolled out on every train + val case (with
iterations) on asoot serve --trialruntime: a temporary runtime that only answers the conversation API and does not run the copy's schedules or periodic improve. - Gate: a patch is rejected if it fails a
pinnedcase or does not raise the score. The decision is measured twice (the fix the gate is about to accept is rolled out again, since a single rollout of a non-deterministic model may just be luck);--no-confirmskips this step, cheaper but noisier. - A fix blocked by a
pinnedcase is held for one round: sometimes it exposes a bug that it was hiding, and the next round pulls the mission lever on top of that fix so the two changes are gated as one. - Finally the
testset (held back until now) is scored, and.soot/improve/<soot>/proposal/{proposal.json,changes.diff}is written. That is the diff awaiting review.
Edit scope: soots/<id>/ (mission, soul, packs, execution, the Soot's own skills) and local packs in packs/. It never edits evals/ (they grade, they are not a lever), adapter/ (stage 2's) or bin/. Vendored or Forge packs are never edited in place.
soot improve ops --status shows the result; --apply is a human action: it writes the approved proposal (or the Áp dụng ("Apply") button in Nest). The --budget / improve.budget budget is in weighted tokens.
Stage 2: Tune the adapter (soot train)
Opens only when diagnosis has no stage 1 cases left and there is no authored diff awaiting review (--tune, locked behind stage 1; an operator can force it, and the result carries a warning label). soot train also runs on its own:
soot adapter init ops # empty soots/ops/adapter/
soot train init nightly --soot ops # train/nightly.train.json
soot train nightly # evolve the adapter, checkpoint every step
soot adapter diff none .soot/train/nightly/versions/v0005
soot eval --adapter .soot/train/nightly/versions/v0005 --tag test
soot adapter deploy .soot/train/nightly/versions/v0005
soot adapter remove opsEach step: sample a minibatch from train, build the learning material (the current adapter and its limits, the authored mission excerpt, the list of authored skills, the failing train cases with the shape of their tool calls, at most 5 passing cases, token data per case and per tool, the last 5 steps), the proposer returns rationale + guidance + skills, the runtime checks (inside the manifest, an edit bounded by max_edit_bytes, repaired up to 2 times if wrong), rolls out the candidate on the same minibatch, and the algorithm decides.
algorithm | Keep the candidate when | Fits |
|---|---|---|
continual | the minibatch objective does not drop; no val needed | Cheapest, continuous learning |
gate (default) | the minibatch does not drop and the val objective rises by more than min_gain | Safe default |
pareto | the minibatch objective rises; the parent is drawn from the frontier by the number of val cases it is best on | Many kinds of cases |
compress | the score does not drop and tokens per case drop | After the score is reached: cutting tokens |
Two objectives, correct before cheap. If the score drops by more than score_tolerance the candidate is rejected, no matter how many tokens it saves. Otherwise it must raise objective = score − token_weight × (token/case ÷ initial token/case) (token_weight defaults to 0.1): the same score with fewer tokens counts as progress.
Guarding against overfitting and against learning wrong rules:
- The proposer sees only train; val decides; test is scored only once at the end (for
none, the initial adapter and the best). pinnedcases do not enter the material; a candidate that fails one is rejected. When there is no val (continual, compress), the runtime still scores the pinned cases separately for the parent and the candidate.- The step history the proposer sees contains only the kind of decision, no val scores and no pinned case IDs.
- The material states the runtime contract (the final answer must contain
STATUS: DONE; an adapter that teaches a different last line gets asked again and doubles the tokens), the list of authored skills, and the rule "an adapter never goes against authored". That rule exists because one adapter once learned to forbidread_skilland promised to "will fix" a faulty chapter, against the mission. Lock policy withpinnedcases, and always reviewsoot adapter diffbefore deploying.
State lives in .soot/train/<name>/ (state.json, index.jsonl, report.md, versions/vNNNN/, material/), not in the bundle. It stops at budget.steps, budget.tokens (rollout plus proposer) or budget.rollouts; Ctrl-C keeps the state, and --resume redoes the interrupted step. soot train status <name> prints the report.
Stage 3: Merge into authored
soot improve ops --graduate: rules of the adapter that hold up across many versions are rewritten into the mission/skill by the authored proposer (a diff awaiting review), and then the adapter starts over from empty. So the adapter stays short and proven rules become real config. Two lines count as one rule when they share at least half their words and negate the same things (không, chưa, cấm, never…), so "never promise refunds" is not treated as the same rule as "always promise refunds". The contradiction check step compares rules against the mission/skill; if there is a contradiction it stops at conflicts and creates no diff.
Which model for which role
| Role | Model |
|---|---|
| Diagnosis | No model (rules) |
| Authored proposer (stages 1, 3) and the contradiction check | The strongest the workspace has |
| Adapter proposer (stage 2) | Medium or better |
| Rollout (running cases) | Always the Soot's model: measuring with another model measures the wrong thing |
The proposer is only about a tenth of the cost (most of it is rollouts), so to make it cheaper reduce rollouts and do not downgrade the proposer. Choose the proposer with --proposer, or proposer.soot in the train file.
Who triggers what
| Who | How | Limits |
|---|---|---|
| Person (CLI) | soot improve …, soot train …, soot adapter deploy/remove, soot eval case from | The only party that applies and reverts |
| Person (Nest) | The Soot's Cải thiện tab: Chẩn đoán (Diagnose) → Sửa cấu hình (Fix config) → Tinh chỉnh (Tune) → Gộp (Merge), the Áp dụng / Bỏ (Discard) / Hoàn tác (Revert) buttons, Không đúng ý on an answer | Through POST /v1/improve (operator) |
| Lead (the workspace's coordinator) | soot manage has two commands: improve.status (read, with the signals, the number of drafts and the auto-used version) and improve.run (an automatic run, soot improve <soot> --auto, trigger lead, default budget 500000, maximum 2000000) | Does not apply, discard, schedule or accept drafts. Refuses when there are no cases (no_eval_cases), one is running (improve_running) or a proposal is already waiting (improve_pending) |
| Schedule | "improve" in soot.json; the runtime runs soot improve <soot> --scheduled | Only creates proposals; an adapter is used automatically only with auto_deploy |
| Watch (the runtime reads traces itself) | "improve": {"watch": {…}}; when a signal passes its threshold, the runtime runs soot improve <soot> --auto --trigger watch | Same limits as the schedule; a cooldown between two runs; does not run while the schedule is paused, a job is running, a proposal is waiting or an auto-used version is being watched |
The improve schedule (optional, one per Soot, set only by an operator):
"improve": {"every": "24h", "at": "03:00", "timezone": "Asia/Ho_Chi_Minh",
"budget": 500000, "min_failing": 1}Each run: run all evals; below min_failing it is healthy; if there is an unreviewed proposal it is waiting_review (no overwrite); if config errors remain it runs stage 1; otherwise stage 2. every is at least 6h; budget is required and covers both evals and fixes; a missed run is not made up; it does not overlap a running job. Dead ends are not rerun: each run records a fingerprint (the content of files in the edit scope, the eval cases, the proposer); if the last run ended no_change and the fingerprint has not changed, this run costs nothing.
A Soot has only one improve job at a time: propose, tune, graduate, scheduled, auto, apply, revert and adapter deploy/remove all hold the lock .soot/improve/<soot>/job.lock; a second command is rejected with improve_running (HTTP 409). Within one process the lock is re-entrant: an automatic run that holds the lock can still call soot adapter deploy.
soot improve <soot> --auto is exactly a scheduled run, but it starts now, called by a person (--trigger manual), Lead (lead) or watch (watch, with --signal). A Soot with no schedule uses --budget (default 500000). When a person or Lead asks, Soot skips the "nothing has changed" shortcut: asking means they want a fresh measurement.
Watching signals
A schedule runs by the clock; watch runs by evidence. On each check, the runtime reads the traces of real runs inside the window (skipping eval runs, dry runs and runs still in progress) with no model call and no tokens:
| Signal | What it measures | Default threshold |
|---|---|---|
failed_rate | Share of broken runs: failed, out of token budget, interrupted (a run cancelled by a person does not count) | above 20% |
tool_error_rate | Share of tool calls that errored or were blocked | above 20% |
memory_writes_per_run | Calls to learn / memory_edit per run. A Soot that keeps taking notes for itself is patching its way of working one run at a time, which is worth fixing at the root in the config | above 1 |
pass_rate | Share of cases that pass in the latest eval on the adapter in use | below 80% |
"improve": {"every": "24h", "budget": 500000,
"watch": {"window": "24h", "check": "1h", "cooldown": "24h", "min_runs": 20,
"failed_rate": 0.2, "tool_error_rate": 0.2,
"memory_writes_per_run": 1, "pass_rate": 0.8, "draft_cases": 3}}Every field is optional; a threshold of 0 turns that signal off. The first three rates are considered only when there are at least min_runs runs (a few stray runs are just noise). pass_rate is measured on the eval set, so it needs no run count. When a signal passes its threshold the runtime starts an automatic run, and signal records those signals as name=value/threshold joined by commas (for example failed_rate=0.35/0.2) so Nest can word it itself. That run still measures the eval set first: if the eval set does not catch what watch saw, it ends healthy and says so, with a reminder to accept the drafts below. On the next watch run, if nothing has changed (same fingerprint: authored files, cases, proposer), it does not measure again but records unchanged with basis: "healthy", at no token cost. Only watch skips this way; the schedule still measures on its own beat, and a person or Lead asking always measures again. Accepting a draft changes the case set, so the next watch run measures again. check is at least 15 minutes, cooldown at least 6 hours, and it counts the schedule's runs too.
soot improve <soot> --signals (or GET /v1/improve/{soot}/signals) prints exactly the numbers watch reads. The latest reading, with what watch did or why it did not, lives in .soot/improve/<soot>/watch.json and in signals of GET /v1/improve. The reason for not acting comes in two forms: held is an English sentence for the log, hold is a code for the screen (guarding, paused, cooldown, busy, review, no_cases, failed), with hold_until when the end time is known.
In Nest, the Improve tab has a Tự phát hiện vấn đề ("Detect problems automatically") block. The switch saves improve.watch through the same soot.improve as the schedule: turning it on when there is no schedule also turns on self-improvement (every day at 03:00, default budget); turning it off removes watch and auto_deploy but keeps the schedule. The block shows the four metrics against their thresholds (Vượt ngưỡng "Over", Ổn "OK", Chưa đủ dữ liệu "Not enough data", Đã tắt "Off"), the number of runs read, the last reading, the reason it did not run, the number of waiting drafts with a button that opens Cần bạn, and the Tự dùng bản tinh chỉnh ("Auto-use the tuned version") box (auto_deploy).
Draft tests
Watch also drafts cases from the broken runs, newest first, at most draft_cases per reading and no more than 20 waiting. A draft is built the same way as the Không đúng ý button: the run's message becomes send, the condition is that the run finishes, and if a tool errored then no tool may error. Drafts live in .soot/improve/<soot>/drafts/, show in Cần bạn (Nhận vào bộ kiểm tra "Add to the eval set" / Bỏ "Discard") and in soot improve <soot> --drafts [--accept <id> | --discard <id>].
The card in Cần bạn speaks in plain words: the message, the answer Soot gave (stored with the draft, at most 2000 characters), when the case passes, and a warning when the case only checks "finished, no error" without looking at the content of the answer. The raw data of the case sits in a collapsed section. These fields travel in the item's payload (send, earlier, reply, checks, loose, problem, run).
Until a person accepts it, a draft is not measured, not used for training and does not block any change. The eval set is the exam, and Soot does not write its own exam. Once accepted, the case goes into captured.eval.json like any other and is split into train/val/test by hash as usual. Watch does not draft a run it already drafted, and does not draft two cases for the same message. A run with no content (traces.content is not full, or an anonymous visitor) cannot be drafted.
Auto-using a tuned version
"improve": {"every": "24h", "budget": 500000,
"auto_deploy": {"min_gain": 0.05, "min_test_cases": 5, "hold": "72h"}}With auto_deploy, when an automatic run (from the schedule, watch or Lead) finishes training in stage 2, it deploys the best version by itself if it passes all of these gates:
- Training has finished and the version has a score on the held-out check (test). The best version is not
v0000and is not already in use. - The test part has at least
min_test_casescases, and the score rises by at leastmin_gainover the adapter in use. - Every
pinnedcase passes on the test part. - This version has never been reverted for making Soot worse, and no other auto-used version is still being watched.
The deploy goes through exactly soot adapter deploy, so it has a receipt and can be reverted. After that, within hold, watch compares Soot with how it was before the change: the eval set passes less than in the eval at deploy time, or the share of broken runs since deploy exceeds both the failed_rate threshold and the share before deploy. On either, watch reverts through the receipt (passing soot check like every revert) and records the digest in the list of versions that are not used automatically again. If hold ends and it is not worse, the version stays. During hold, watch starts no other improvement run, so only one change is measured at a time.
Cần bạn reports a version that was just auto-used (Giữ "Keep" / Hoàn tác "Revert"), a version that watch reverted, or a revert that could not be done because a file was edited afterwards (then a person reverts it in the Improve tab). If a person reverts it, that version is not auto-used again either. The card has the numbers to compare: the separate exam before and after, the eval pass rate and the broken-run rate at deploy time against watch's latest reading (suite_now, failed_now; "not measured yet" when there is no eval on the new version or not enough runs), and the watch deadline. When watch reverts, worse records which signal got worse, from what to what. The card calls the Soot by the name its owner gave it (not the id). A successful automatic revert is a notice (info): nothing broke and it has nothing to do with the budget; only revert_failed is an error. State lives in .soot/improve/<soot>/auto_deploy.json: watching, kept, reverted, undone, revert_failed. Watch and the person's answer (Giữ, Đã hiểu "Got it", Hoàn tác) write this file under the same lock and re-read it before writing: if a person kept it, watch does not overwrite it and does not revert it. Authored changes never go this way.
Improve the whole workspace from flows
The four-stage pipeline above improves one Soot from its evals. What breaks in a group of Soots is usually between teams or in the team config (a relation missing a field, a per-turn limit too small, a process with no way to finish, a team that does not know something everyone assumes it knows). A flow (plans/team-flows.md) is the test of the whole workspace, so there is a separate loop that improves the workspace from flows: soot improve --flows. The package is internal/feature/improve/wsimprove, and it reuses improve.Gate (the same gate as the Soot loop: val does not drop, train+val rises by more than MinGain, no pinned flow that was passing now fails), improve.Snapshot/Revert (receipts and drift-safe revert of files) and the job lock .soot/improve/workspace/.
The loop: measure train+val k times on a copy of the workspace (each try is a trial runtime), diagnose (soot flow diagnose), apply every no-model fix as one round (measure again; if it only raised a limit, plan again from the candidate's own measurement, at most 3 passes, then through the gate once), then the proposer's rounds, each pulling one lever, then one confirmation and the test flows. The default budget is 500000 converted tokens; going over it costs money.
| Diagnosis layer | Lever | Who |
|---|---|---|
| ontology | add a rejected field (a queue field, a field of a read relation), add a missing relation or verb, add a closing transition only when the diagnosis names it; never remove gates | no model |
| budget | the Soot's execution.max_run_tokens/max_tool_calls/max_turns = measured peak ×1.5, inside that Soot's execution.daily_tokens | no model |
budget above daily_tokens, the team's or host's daily budget, a more expensive model tier, a paid tool, an improve budget above 500000, removing gates | money: never applied by itself, becomes a proposal for a person to decide | person |
| team | one fact in team memory (source improve, stating what is true and why, not a numbered procedure, at most 400 characters) or the team description | proposer (model) |
| Soot | an eval draft from the first run that went off course, source flow:<flow id>, through the existing draft mechanism (soot improve <soot> --drafts); never measured before a person accepts it | the person accepting the draft |
| pack | only a referral to the pack owner | - |
| world | report only | - |
| noise | do nothing | - |
Team memory is runtime state, not a file, so a candidate's fact is seeded into each trial runtime through flows.Options.Memory; when applied, the fact enters the real runtime through POST /v1/workspace/memory (tag source:improve) and the receipt keeps its id so that a revert forgets exactly that fact. The receipt contains the Soot loop's file receipt and the facts taught.
The result is a diff, the facts, the money proposals and the drafts, waiting for review (--propose, --status). --apply writes everything except money, then runs val+pinned again; if the pass rate is lower than the baseline recorded at measurement, it reverts by the receipt on its own and records the reason (--history, --revert <receipt>). --auto is --propose then --apply. A source-owned workspace is never applied on the host for files that belong to the source: --pull --into <repo> writes that part into the checkout. API: POST /v1/improve/flows (mode propose|auto), GET /v1/improve/flows, POST /v1/improve/flows/apply, POST /v1/improve/flows/revert, GET /v1/improve/flows/history; the job is the CLI itself.
Review and revert
A proposal awaiting review appears in Cần bạn (Apply / Discard) and in the Improve tab, with the diff, before/after scores and cost. Only a person applies.
Every applied change (--apply, graduate, soot adapter deploy, soot adapter remove) leaves a receipt in .soot/improve/<soot>/history/<id>/: the before/after hash of each file and a copy of the previous content. If writing a file fails midway, every file is put back; if an adapter deploy fails validation, the old adapter is restored.
soot improve ops --history # newest first
soot improve ops --revert # revert the newest change not yet reverted
soot improve ops --revert --id <id>Revert restores, it does not merge, and it is all or nothing: if a file was edited after that change, the command refuses (improve_revert_drifted, HTTP 409) and writes nothing; after writing, the workspace must pass soot check, otherwise every file is put back as before (improve_revert_invalid). Reverting an adapter deploy returns to exactly the previous adapter, not just to "no adapter". Apply, revert and deploy/remove each request one reload on the running runtime (POST /v1/improve/{soot}/reload), so the next scheduled run measures the new config.
A Soot that belongs to a repo (agent as code)
A Soot from a source (agents.md) is defined in the repo and deployed by replacing the whole slice, so the server never writes into the slice directly (that would become drift). The rule: the repo always wins.
GitHub repo ──release──▶ VM: sources.d/shop/… (deploy replaces the whole slice)
VM: .soot/adapters/seller/ (tuned version learned on the VM)
◀──soot improve seller --pull── proposal + tuned version, to review/commit--applyis rejected (improve_source_owned): stage 1 proposals and stage 3 graduation go into the repo withsoot improve <soot> --pull --into <checkout>(--exportpreviews), then commit and release like any code change. Nest changes the Áp dụng button to Đưa vào repo ("Put into the repo").soot adapter deploywrites to.soot/adapters/<soot>/together withhost.json, which records the adapter the repo was shipping at deploy time (over). This version applies only while the repo still ships exactly that adapter. Thehost_adapter.statevalues:active,in_repo,superseded,conflict; a version that cannot apply never breaks loading.- When a release carries the change to the server, everything closes up: a proposal that the tree already contains moves to
in_repo.
Report: what Soot has improved on its own
The report is built purely from files that already exist (state*.json, scheduled.jsonl, receipts, pending proposals). GET /v1/improve returns for each Soot: pending, blocked (why the latest run changed nothing), applied / reverted, spent (weighted tokens over 30 days), runs (at most 20), schedule, next, cases. Each Soot also has signals (watch's latest reading), drafts (the number of waiting drafts) and auto_deploy. Each run has trigger (manual, scheduled, lead, watch with signal), stage, outcome (proposed, applied, discarded, in_repo, no_change, unchanged, healthy, waiting_review, budget, tuned, deployed with deployed, failed…), passed_before / passed_after, spent, levers. GET /v1/improve/{soot} adds report, history, watch, drafts and auto_deploy. Nest shows it at Kết quả › Tự cải thiện ("Results › Self-improvement") and in the Improve tab; Lead answers with the same numbers through improve.status (including the morning brief).
Input for coding agents: hillclimb
Hillclimb is the second form of stage 1 proposer: a coding agent (Claude Code, Cursor, Codex…) edits the source directly. soot spawn writes this loop as the soot-hillclimb skill in the workspace's .agents/skills/ and .claude/skills/:
soot check --strict --jsonandsoot info --json: the config is valid, and the Soot sees the right tools and skills.- Write a case that reproduces the wrong behavior;
soot eval <id>must see it fail. soot eval --json > base.json, then change one thing.soot eval --baseline base.json: the target case is improved, nothing regressed; thensoot eval --tag smoke.- Keep the case in
soots/<id>/evals/as evidence that the improvement still holds.
Compared with soot train: hillclimb can edit any authored part and starts from one specific failure, but it is manual and has no held-out set; soot train only adds guidance to the adapter, starts from the whole case set, and has val/test that the proposer does not see.
Boundaries and what is deliberately not done
| What | Why |
|---|---|
| No separate model loop for evals | Every rollout goes through the conversation API, the same path as a real run; what is measured is the real Soot, skills, tools and model |
No automatic apply of authored changes; an adapter only when a person turns on auto_deploy | A person decides; every apply has a receipt and can be reverted, and an auto-used version is watched and reverted by itself when it gets worse |
| A watch draft is not measured before a person accepts it | The eval set is the exam; Soot does not write the exam that grades itself |
| No val, test or pinned in the proposer's material | So training does not change policy or memorize the exam to get a score |
No editing evals/ during improvement | Cases grade, they are not a lever |
| No editing vendored/Forge packs in place | Propose a PR or issue to the pack owner |
| No final result from a rollout with a model other than the Soot's | An adapter learned on a cheap model may not transfer to the real model |
| No letting tokens decide over correctness | A lower score is rejected even when cheaper |
| Lead does not apply, discard, schedule or accept drafts | Only an operator or a person with the permission does |
| Who | Owns |
|---|---|
| Core (runtime) | The eval runner, trace store, the improve field, the pipeline and job lock, receipts, the /v1/improve, /v1/traces, /v1/usage APIs |
| Pack | Tools with tool.preflight (so evals see what a tool would refuse without side effects); evals and skills that ship with a pack when Forge releases it |
| Host / project | The project's eval cases, pinned policy, improve schedule and budget |
| Nest | Display and actions: the Improve tab, Không đúng ý, Cần bạn, Kết quả; it has no improvement logic of its own (surfaces.md) |
Not yet available / limits
- Outcomes are not yet an automatic input to the improvement loop (see above).
- Watch compares the eval set on an auto-used version only once an eval has run on that version (the next scheduled or watch run, or
soot eval); before that it relies on the broken-run rate alone. - Watch reads at most 5000 runs in each
window. - Diagnosis only classifies by rules; a model only explains, optionally, and does not classify.
- A small val set (under about 8 cases) gives noisy scores; this only warns and does not block.
- An adapter learned with a cheap model is not guaranteed to transfer to the real model; the final test has to run on the model the Soot uses.
- The self-measured pass rate over time is shown, but there is no gate that blocks a release automatically when the pass rate drops (the gate is the report reader's job, as with replay).
Content reviewed: 2026-11-12