The autoresearch loop, in one read

Autoresearch is the scored lane of Crucible. It is a keep/discard loop with a goal and a gate, in which an agent improves a codebase against a frozen objective. The loop does not depend on a domain. A domain is a problem packaged for the loop. For a graph that runs one time and ships, see Your first playbook.

This page describes the system in general concepts. Then it shows how the examples/counter domain maps to those concepts. ADR-0001 gives the reason that the judge is frozen.

Scope. The core is the engine, the World/Judge boundary, CommandWorld and CommandJudge, and the crucible.toml manifest. A composite manifest (ADR-0009) runs the engine across more than one component. Around the loop there are: provisioning and issue-tracker grounding through an MCP broker (ADR-0002), a profiler over MCP (ADR-0006), build and deploy in the engine (ADR-0005) with rendered deployment manifests (ADR-0011), and publish-on-keep with a draft PR for each fork. Authorized review comments on the PR steer the run again.

Summary

The loop is an optimization loop with these properties:

• The proposal step is an LLM agent.
• The objective is a frozen, expensive black-box evaluation.
• The world is state that the engine can change and restore.
• The loop has durable memory and human-in-the-loop control.

The loop

flowchart LR
    controls["budget / steer / stop / escalate"] -.-> propose["propose<br/>(agent)"]
    propose --> apply["apply<br/>(World)"]
    apply --> measure["measure<br/>(Judge)"]
    measure --> accept{"accept?<br/>keep or discard"}
    accept -->|"keep"| remember["remember<br/>(Git + session log)"]
    accept -->|"discard"| restore["restore<br/>(World)"]
    remember --> next["next iteration"]
    restore --> next
    next --> propose

• propose: the agent edits the world toward the goal. A backend supplies the proposal policy: local (the harness on the host), openshell (sandboxed), or command. [agent].backend selects it.
• apply: makes the candidate live. For a code repository, the edits are the apply step.
• measure: the frozen Judge scores the candidate.
• accept?: keeps the candidate if it is strictly better. Otherwise, restores the last good state.
• remember: kept states are Git commits. Each step is an NDJSON event.

By default, the loop has one candidate in each iteration. A wide round (ADR-0010) can first run N independent propose turns in parallel, each biased to a different [search].approaches entry. The same Judge ranks them, and the winner is the start point of the loop. With --wide 0 (the default), there is no wide round. The wide round is a step before the loop and does not change the engine.

The contract

Crucible defines a small contract. A repository implements it in any language. A domain is a manifest and some executables. It is not Rust code. Each command runs in the workspace:

What of the engine's environment measure, apply, snapshot, and restore get depends on the pack's edition, a top-level key of crucible.toml:

edition = 2

A manifest without edition is edition 1. Its commands inherit the engine's whole environment, credentials included, and crucible check and every run warn that the pack should move to edition 2. The capability disclosure says so per table: host-env [judge] commands inherit the engine's whole environment.

Under edition 2 the commands get a minimal environment, not the engine's: PATH, HOME, LANG, TMPDIR, the run's identity (CRUCIBLE_RUN_NAME, CRUCIBLE_RUN_DISPLAY_NAME, and CRUCIBLE_UI_BASE_URL, when the platform sets them), the variables the engine sets for that command (CRUCIBLE_BASELINE_*, CRUCIBLE_BEST_SCORE, CRUCIBLE_TOKEN), and the variables its table declares. The pod token and the controller URL it authenticates against are not part of the base. The agent can edit files a command runs, so nothing else of the engine's environment reaches it: no kubeconfig, registry authfile, broker token, or provider key unless you name it.

edition = 2

[judge]
measure_cmd = "./measure.sh"
direction = "lower"
env = ["KUBECONFIG", "EPP_ENDPOINT?"]        # measure_cmd only

[world]
apply_cmd = "./deploy.sh"
env = ["KUBECONFIG", "REGISTRY_AUTH_FILE"]   # apply, snapshot, and restore

[workflow].env does the same for a workflow's command and evaluate tasks. An env list in an edition-1 manifest fails the parse, and so does an edition this engine does not know. A name is a letter or underscore followed by letters, digits, and underscores. A required variable that the engine's environment does not set refuses the run before any of these commands runs, naming every missing one; crucible check reports it as a finding. A name with a trailing ? is optional: the command gets it when the engine's environment sets it and runs without it otherwise, so a script can keep a shell default (${EPP_ENDPOINT:-127.0.0.1:8000}) that a deployment overrides. Only the trailing ? is allowed (? alone, A??, and A?B fail the parse), and a name is either required or optional, not both. crucible check lists each table's declared variables in the capability disclosure as host-env, the optional ones marked (optional). setup runs with the engine's environment in every edition.

The accept policy is the same for all domains. The engine keeps a candidate if it is valid and its score is better than the best score in the direction. The run is solved only if measure reports it. A domain does not supply code for the keep logic.

Carrying pipeline artifacts across a discard

A discard resets the workspace and removes untracked files. Any files that an iteration generated for its candidate (code traces, a generated port tree) are gone in the next turn, and the agent must generate them again. [workspace] carry_forward names paths, relative to the workspace, that the discard keeps:

[workspace]
carry_forward = ["codegen-out/"]

The engine writes each entry to .git/info/exclude, and the clean step of the discard keeps it. Carried content does not go into a candidate diff, a snapshot commit, or a tree hash. If a turn only generates the carried content again, the engine sees no candidate change.

Limits:

• A path that the repository tracks is not protected. Git excludes apply only to untracked files, and git reset --hard in the discard reverts tracked content. Carry only pipeline output that the repository does not track.
• A path can be absent when the run starts. The exclude applies when the path appears.
• Nested paths (target/codegen-out) work. The clean step goes into the parent directory and does not delete it. Entries must be plain relative paths, without ., .., or a leading /.
• Wide-tournament rounds run in new worktrees without the carried directories, so the setting does not apply to them.
• Composite manifests reject the key. Carry-forward for each component is not supported.

If you omit the key, each iteration starts from a clean workspace. Use this default for a campaign whose method needs a clean start.

Declared artifacts: carried AND published

[[workspace.artifact]] is carry_forward with publishing. Use it for generated output that a reviewer must see, but that must not be in the candidate diff:

[[workspace.artifact]]
path   = "codegen-out/runtime"   # carried, same rules as carry_forward
embed  = "summary.txt"           # newest match → PR body (collapsed, 16 KiB cap) + record
upload = "summary.pptx"          # newest match → record only (binaries a PR can't inline)
title  = "Runtime investigation" # PR-body section heading (default: the path)

The newest match is used. If versioned pipeline directories (V1 … V7) each have a summary with the same name, the summary of the final iteration is published. The record destination accepts s3://bucket[/prefix] or file:///abs/path. The file form writes the same layout to a mounted PVC, for clusters that cannot reach S3. If two or more iterations have a measurement, the PR body also shows the score of each iteration as a mermaid xychart-beta chart.

Languages

The contract is JSON and exit codes, so you can write each command in any language:

• Small glue commands (apply, operations, steer, stop): we recommend nushell. These commands run kubectl or Prometheus, parse the output, and compute a value. Nushell works with structured data (... | from json | ... in, a record | to json out). Nushell is not a POSIX shell, so pin nu in the sandbox image.
• A large measure with real math (percentiles, replay): use a compiled binary (Rust or Go).
• Bash, Python, and other languages are also valid. Only the JSON and exit status format is fixed.

measure_cmd is any executable that prints one JSON object on stdout ({"valid": bool, "score": number, "note": string}). A plain POSIX shell script satisfies the full contract:

#!/bin/sh
p99=$(make bench | jq .p99_ms)
printf '{"valid": true, "score": %s, "note": "p99 %sms"}\n' "$p99" "$p99"

The engine reads only the JSON verdict, so all languages work the same way.

What the engine supplies

The control plane (crucible-controller) is optional. It finds candidate issues, scopes them into packs, launches runs, and shows run records as leaderboards. The controller links the engine. The engine binary does not link the controller, and everything on this page works without a controller.

Trust boundary (ADR-0001): the engine gives the agent a World but not a Judge. The objective is frozen, and the agent cannot reach it, so the agent cannot change the test that grades it. The same boundary applies to provisioning. The agent can only send requests to the broker over MCP. The privileges (GPU and Kueue, capture RBAC, the forge token, the issue-tracker credential) stay on the host and do not go into the sandbox of the agent.

[preflight]: prove the environment before spending an iteration

A rung that grades a candidate can fail for reasons that are not related to the candidate: a missing dev header, a pip install that hides the built wheel, a cache directory that is not writable. If the run finds these problems in iteration 1, it loses a full agent turn. [preflight] runs the same ladder against the unmodified tree first, with no agent cost:

[preflight]
commands = [
  "python3 tools/gate.py --only-rung 1 --mode {mode}",
  "python3 tools/gate.py --only-rung 2 --digest {digest} --limit 1",
]
baseline = "python3 tools/gate.py --only-rung 3 --digest {digest}"

• Each command runs with sh -c in the workspace. Its last non-empty stdout line must be a JSON object. The rung passes if the command exits with status 0 and pass is absent or true. The recognized keys are digest, score, tiebreak, note, and logs.
• {mode} runs the command once for each build mode in [measure.build].mutable_kwargs.mode, in declared order. A {mode} command without that list is a manifest load error. Without this rule, a preflight could pass in derive mode while full mode is broken.
• {digest} inserts the most recent digest that an earlier rung emitted, so the build rung gives its artifact to the correctness rung. If no rung produced a digest yet, the command fails.
• Any failure is an environment verdict. The run does not start. It logs the failed command, its note, and the end of its stderr as a preflight-failed row. The run uses no iteration.
• baseline is optional. Its score and tiebreak become segment.baseline_score. A skip_baseline (codegen) domain then decides iteration 1 against a real number, not against the worst-score sentinel of the direction. Without baseline, the sentinel is used.

The three extension points

• New domain: usually a manifest that points CommandWorld and CommandJudge at a repository, and a measure command. No Rust. Write a custom World or Judge only for in-process measurement or a world that is not files.
• Composite domain: combine N existing component domains into one run. It has a top-level [composite] table that references the component domain directories (used without change), a CompositeWorld (a tuple of Git overlays, one for each component, and the live deployment), and one combined gate over the full stack. The engine does not change. It runs a list of workspaces (ADR-0009).
• New frontend: implement Reporter.
• New agent transport: add an AgentSource.

The canonical example: the counter, mapped to the contract

examples/counter/ is the full system, with each large part replaced by the smallest part that satisfies the contract. Each of its five small files has a contract role:

There is no [world] block, so GitWorld supplies the restore: kept iterations are commits, and discards use git reset --hard. The backend is command, so the whole loop runs in milliseconds and costs nothing. It is still a real end-to-end run (manifest → setup → propose → measure → keep/discard → Git history).

A production domain fills the same roles with larger parts. A serving-stack domain has two reversible parts: the code tree and a live deployment. It adds snapshot_cmd and restore_cmd that capture the image and configuration together with the Git state. Its measure is a compiled benchmark binary. Its propose backend is local or openshell. Its glue commands (apply pipelines, deployment inspection, steer and stop) are on PATH as bare names, so the contract names are the same for every implementation. A normal repository needs none of this. It needs one measure command and the Git default.

Add a new domain

1. crucible init creates a starter crucible.toml and a measure script in the repository. You can also write the manifest by hand:

   [repo]      url|path, ref
   [workspace] setup_cmd          # optional; default: git clone + checkout
               carry_forward      # optional; untracked derived paths a discard keeps
               [[workspace.artifact]] path, embed?, title?   # carried + published to PR/S3
   [agent]     model, method_prompt, goal_file|goal, toolbox_dir, env
   [judge]     measure_cmd, direction = "lower"|"higher"
   [world]     apply_cmd?, snapshot_cmd?, restore_cmd?   # omitted → git
   

2. Write a measure command (any language) that prints {valid, score, solved?}.
3. crucible check --manifest crucible.toml validates the manifest without an agent turn. It checks that every referenced file resolves, and it runs measure_cmd one time to check the output format.
4. Run crucible --manifest crucible.toml. The engine supplies the loop, the budget, all frontends, steer, stop, and resume, the session log, and escalation.

Any other repository can run this way with no new Rust.

The task lane: no [judge]

Some work has no objective to score, for example "consolidate the open dependabot PRs" or "fix flaky tests every night". If you omit [judge], the run is a task. The loop runs as usual (sandbox, broker, session log, publish, resume), with the built-in TaskJudge, which keeps everything. There is no baseline measure and no score. The engine keeps and snapshots every completed turn. The run exits with status 0 when no iterations remain (ADR-0021).

[repo]
url = "https://github.com/you/repo"
[agent]
backend = "openshell"
goal = "Consolidate the open dependabot PRs into one branch; run the tests."
[publish]
pr_repo = "you/repo-fork"   # the deliverable: kept commits as one draft PR

On the wire, the run has the normal format, with gate: "task" as the discriminator: an iteration-0 baseline-skipped row, then keep rows with score: null, and Summary.best_score null. crucible check does not probe the measure. It requires a goal and prints a "task mode" notice. Notes:

• Exit status 0 means that the run completed. It does not mean that the task succeeded. Examine the rows or the PR.
• solved does not occur. Only the iteration limit ends the run (--no-early-stop has no effect).
• The trust boundary does not change. The agent has no credentials. The output is a draft PR. Privileged write actions (merge, close issues) stay in broker tools.
• Composites still need [judge]. [search], [workflow], and [preflight] also need it, and a task manifest rejects them at load. The scope pipeline rejects proposed packs without a judge, and a scored judge cannot use objective = "task".

examples/task/ is the runnable reference (deterministic command backend, no LLM). Tasks: the no-judge lane is the full guide.

Distress: the agent can page you

The broker has a distress tool. The agent uses it to report that a run cannot succeed. The required severity sets the result:

Use error for a run that cannot succeed without an operator: the same environment failure in many iterations, a missing capability, or a goal that is not possible as written. Do not use it for a task that is difficult but possible. The tool replies {"status":"suspending"}, and the agent finishes its turn. The suspend occurs at the start of the next iteration, after the engine records that turn, so no data is lost. More error calls during the suspend have no effect. If the engine cannot write the marker (a full or read-only volume), the tool returns an error, not suspending, and posts nothing. The run is not suspended, so the agent and the page must not report that it is.

The suspend is a pending approval. The loop opens the same approval_wait bracket that provisioning uses, so suspended wall-clock time goes to parked_total and does not count toward --max-time. The distressed iteration still counts, because its turn ran. The suspended time counts against no limit. --max-park limits the wait.

The signal is one file on the forge-storage volume of the loop pod: /var/lib/forge/distress. To resume the run, delete the file:

oc exec <pod> -n <ns> -- rm /var/lib/forge/distress   # resume in place

If the fix must be in place at pod start (image pins, the pack, resources), replace the pod. The marker is on an emptyDir, so the new pod starts without it. A resume that finds the open distress bracket does not park again. Info and warn notes go to /var/lib/forge/distress-notes.jsonl, and the engine adds them to the next row.

The render sets these variables on the loop pod. The broker inherits them, so the page can name the run: CRUCIBLE_RUN_NAME, CRUCIBLE_ITERATIONS, CRUCIBLE_LOOP_IMAGE, and the downward-API CRUCIBLE_POD_NAME and CRUCIBLE_POD_NAMESPACE. A pod that the controller dispatches also has CRUCIBLE_RUN_DISPLAY_NAME, the two-word name of the run (benevolent-monkey). The page and report cards show this name instead of the run id. Links still use the id. Slack supports only webhooks. Set SLACK_WEBHOOK_URL to an incoming webhook through the [[secret_env]] of the profile. Set DATADOG_BASE_URL if your Datadog site is not app.datadoghq.com. Without a webhook, the suspend still occurs. The engine sends the page one time and does not retry.

crucible flow: explain a finished run

crucible flow --session state/session.jsonl --out flow.html converts a finished run into a small flow-model IR and renders it. The --out extension selects the format: .json (the IR), .dot, .mmd, or .html. The HTML format is a self-contained page with a header, a swimlane of iterations (agent turn and check ladder in each column, scores against the baseline), detail for each iteration, and the draft-PR footer. The optional --spans <dd-export.json> (Datadog APM) adds real wall-clock time: columns in proportion to time, check durations, and the tool-call timeline of the agent.

Concept to code

• loop, budget, keep/discard: crucible/src/runloop/driver.rs, crucible/src/runloop/graph.rs
• the contract traits: crucible/src/crucible.rs
• World and Judge implementations: crucible/src/command_world.rs (GitWorld, CommandWorld, CompositeWorld), crucible/src/command_judge.rs (CommandJudge)
• composite domains: crucible/src/manifest/mod.rs (CompositeManifest), crucible/src/cli/scored.rs (run_composite)
• broker (provisioning, issue tracker, profiler, draft PR): crucible-broker/
• distress (suspend from the agent, and the Slack page): crucible-broker/src/distress.rs, crucible/src/control/distress.rs
• build and rendered deploy in the engine: forge/ (build, native-OCI derive, kube-rs), crucible/src/deploy/
• agent transport: crucible/src/agent/
• frontends: crucible/src/report/{console,stream}.rs
• session log wire format: crucible/src/report/session.rs
• the example domain: examples/counter/ (manifest, measure.nu, bump.nu)