crucible

Agentic autoresearch loop

An agent forms a hypothesis, changes the code or config, measures once, and is gated keep-or-discard against a frozen objective. Git is the memory.

Crucible is a goal-driven, gated, keep/discard loop for letting an agent improve a codebase against a frozen objective. Each iteration the agent reads the history, forms one hypothesis, changes the world, and is gated on a single measurement: keep if better, else revert. The loop itself is domain-neutral, it names nothing about any one problem.

A domain typically ships one or both of two gate shapes:

• bench (perf goals): replay a frozen workload against the candidate, minimize a latency or throughput metric. The first production domain measured an inference router GPU-free this way, via the vllm-vcr workload simulator.
• test (bug-fix goals): run the affected packages' test suite; a green suite with a new regression test wins.

One domain need not mean one component. A composite domain assembles several component domains into one deployment; a single agent turn can edit any component, and one combined gate scores the assembled stack (ADR 0009).

The pieces

The crucible engine is a Cargo workspace; domain packs live in their own repositories and are loaded via --manifest.

• crucible/: the domain-neutral engine binary: manifest load, the keep/discard loop, front-ends, resume, and crucible deploy (it renders its own loop/deployment manifests, ADR 0012). The World + Judge traits are the domain boundary; a top-level [composite] manifest assembles several components into one run.
• forge/: engine-side build + deploy: buildah for real source builds, a native-OCI layer-append path (no container runtime), and a kube-rs client for apply/rollout.
• crucible-broker/: the mediated provisioning broker (the agent asks, the host holds the keys): GPU/trace capture, issue-tracker grounding, the profiler, and draft-PR publishing, all over MCP (ADR 0002). A domain may layer a thin per-domain binary on top of it.
• tools/*.nu: general control-plane glue installed as bare names on PATH (steer, stop, escalate, session, goal-from-issue).

Where to go next

How it works

The whole system in one diagram: a goal goes in, an agent proposes a change, the change is applied to a reversible world, a frozen judge measures it once, and the loop keeps it only if it strictly beats the best so far. Git is the memory; the operator can steer; the privileged operations live host-side behind a mediated broker the agent can only ask.

flowchart TD
    issue["GitHub issue / Jira ticket"] -->|scope --issue| goal["Run goal<br/>(frozen objective)"]
    goal --> wide["wide round (optional)<br/>N parallel propose → rank → winner seeds"]
    wide --> propose

    subgraph control["Control plane (operator, human-in-the-loop)"]
        direction LR
        budget["budget"]
        steer["steer"]
        stoppark["stop / park"]
        resume["resume"]
        escalate["escalate"]
    end

    subgraph loop["The keep / discard loop"]
        direction TB
        propose["propose<br/>agent backend: local · openshell · command"]
        apply["apply<br/>World: edit tree, or build + set image"]
        measure["measure<br/>Judge: frozen, scores once"]
        accept{"accept?<br/>strictly better?"}
        remember["remember<br/>git commit + session NDJSON"]
        restore["restore<br/>World rollback to last good"]

        propose --> apply --> measure --> accept
        accept -->|keep| remember
        accept -->|discard| restore
        remember -->|next iteration| propose
        restore --> propose
    end

    control -. "steer / stop / resume / budget" .-> loop

    subgraph broker["Mediated MCP broker: host holds the keys, agent only asks"]
        direction TB
        prov["provisioning<br/>GPU / Kueue, capture RBAC"]
        jira["issue-tracker grounding"]
        prof["profiler<br/>pprof / GPU traces"]
        forge["forge build + deploy<br/>buildah / native-OCI derive-layer, digest-pinned"]
        draft["draft PR per fork"]
    end

    propose -. "MCP ask" .-> broker
    forge -. "image + set image" .-> apply

    remember -->|publish-on-keep| s3[("S3 run records<br/>session.jsonl · summary.json · diffs")]
    remember --> draft
    draft -->|review comments re-steer| steer
    remember -->|ingest API, optional| ctl["control plane<br/>(crucible-contract ingest)"]

The control shell around that loop, the gates between iterations and the ways a run ends, is its own generated diagram: Loop control states.

The stages

Glossary

The loop leans on a few overloaded words. The two that trip people up most are World and Judge: they're the trust boundary, and they mean something specific here.

For the prose version of the same model, see What crucible is.

Crucible: what it is, in one read

Crucible is a goal-driven, gated, keep/discard loop for letting an agent improve a codebase against a frozen objective. The loop itself is domain-agnostic; a domain is a problem packaged for it.

This doc explains the whole system in higher-order concepts and then shows exactly how the examples/counter domain maps onto them. For why the judge is frozen, see ADR 0001.

Scope. The engine, the World/Judge boundary, CommandWorld/CommandJudge, and the crucible.toml manifest form the core. A composite manifest (ADR 0009) exercises the engine across multiple components. Around the loop: mediated provisioning + issue-tracker grounding over an MCP broker (ADR 0002), a profiler over MCP (ADR 0006), engine-side build/deploy (ADR 0005) with rendered deployment manifests (ADR 0012), and publish-on-keep with a draft PR per fork whose authorized review comments re-steer the run.

The one sentence

An optimization loop where the proposal step is an LLM agent and the objective is a frozen, expensive black-box evaluation, over a reversibly-mutable world, with durable memory and human-in-the-loop control.

The loop

flowchart LR
    controls["budget / steer / stop / escalate"] -.-> propose["propose<br/>(agent)"]
    propose --> apply["apply<br/>(World)"]
    apply --> measure["measure<br/>(Judge)"]
    measure --> accept{"accept?<br/>keep or discard"}
    accept -->|"keep"| remember["remember<br/>(Git + session log)"]
    accept -->|"discard"| restore["restore<br/>(World)"]
    remember --> next["next iteration"]
    restore --> next
    next --> propose

• propose: the agent edits the world toward the goal. The proposal policy is a pluggable backend (local in-process Claude / openshell sandboxed / command), selected by [agent].backend, not the engine.
• apply: make the candidate live (for a code repo: the edits are the apply).
• measure: the frozen Judge scores the candidate.
• accept?: keep if it's strictly better; otherwise restore the last good state.
• remember: kept states are git commits; every step is an NDJSON event.

This single-candidate line of descent is the default, and it's not the only shape: a wide round (ADR 0010) can fan N independent propose turns out in parallel first, biased to distinct [search].approaches, rank them by the same Judge, and seed the loop above with the winner. --wide 0 (the default) skips it entirely, no engine change, just a pre-loop step.

The contract: framework owns it, the repo owns the implementations

Crucible defines a small contract; a repo fills it however it likes, in any language. A domain is a manifest + a few executables, not Rust code. Each command runs in the workspace with the domain's env:

The accept policy is generic: keep if valid and score beats the best per direction; solved iff measure says so. No per-domain code for the keep/win logic.

Carrying pipeline artifacts across a discard

A discard resets the workspace and cleans untracked files, so anything an iteration derived on the way to its candidate (code traces, a generated port tree) is gone by the next turn and gets re-derived. [workspace] carry_forward names workspace-relative paths that survive:

[workspace]
carry_forward = ["codegen-out/"]

Each entry is written to .git/info/exclude and spared by the discard's clean, so carried content never reaches a candidate diff, snapshot commit, or tree hash: a turn that only regenerates it still reads as no candidate change.

Limits worth knowing before you use it:

• A path the repo tracks is not protected. Git excludes only apply to untracked files, and the discard's git reset --hard reverts tracked content regardless. Carry pipeline output the repo doesn't track.
• A path that doesn't exist yet is fine; the exclude is prospective.
• Nested paths (target/codegen-out) work: the clean descends into the parent instead of deleting it. Entries must be plain relative paths, so no ., .., or leading /.
• Wide-tournament rounds run in fresh worktrees with no untracked carried dirs, so they don't benefit.
• Composite manifests reject the key; per-component carry-forward is a non-goal.

Omitting it is the default and keeps today's fresh-start behavior exactly, which is what a methodology-sensitive campaign wants.

Declared artifacts: carried AND published

[[workspace.artifact]] is carry_forward plus publishing, for derived output a reviewer should see without it riding in the candidate diff:

[[workspace.artifact]]
path   = "codegen-out/runtime"   # carried, same rules as carry_forward
embed  = "summary.txt"           # newest match → PR body (collapsed, 16 KiB cap) + record
upload = "summary.pptx"          # newest match → record only (binaries a PR can't inline)
title  = "Runtime investigation" # PR-body section heading (default: the path)

Newest match wins, so versioned pipeline dirs (V1 … V7, same-named summary in each) surface the final iteration's. The record destination accepts s3://bucket[/prefix] or file:///abs/path (the same layout written to a mounted PVC, for clusters with no S3 reach). The PR body also charts per-iteration scores as a mermaid xychart-beta when at least two iterations measured.

Languages: pick per command, the engine doesn't care

The contract is JSON + exit codes, so each command can be whatever fits:

• Thin glue (apply / ops / steer / stop): nushell is the recommended default. The work is "run kubectl/prometheus, parse, compute," and nushell is structured-data native (... | from json | ... in, a record | to json out), cleaner than bash, lighter than python. (It's its own shell, not posix; pin nu in the sandbox image.)
• Heavy measure with real math (percentiles, replay): a compiled binary (Rust/go) earns its keep.
• bash / python / anything else is equally valid: only the JSON/exit shape is fixed.

measure_cmd is any executable that prints one JSON object on stdout ({"valid": bool, "score": number, "note": string}). A plain POSIX shell script satisfies the whole contract:

#!/bin/sh
p99=$(make bench | jq .p99_ms)
printf '{"valid": true, "score": %s, "note": "p99 %sms"}\n' "$p99" "$p99"

Any language works the same way, because the engine reads only the JSON verdict.

What the engine gives you for free

An optional private control plane can sit above all of this: a controller that discovers candidate issues, scopes them into packs, launches runs, and renders run records into leaderboards. It consumes the public crucible-contract ingest API over HTTP; the engine and loop pods never link it, and everything on this page runs identically without one.

Trust boundary (ADR-0001): the engine hands the agent a World, never a Judge. The objective is frozen and out of the agent's reach: the agent can't tune the test it's graded on. The same boundary runs through provisioning: the agent can only ask the broker (over MCP); the privilege (GPU/Kueue, capture RBAC, the forge token, the issue-tracker credential) stays host-side, never in the agent's sandbox.

[preflight]: prove the environment before spending an iteration

Every rung a candidate is graded on can fail for reasons that have nothing to do with the candidate: a missing dev header, a pip install shadowing the built wheel, an unwritable cache dir. Discovering those at iteration 1 costs a full agent turn. [preflight] runs the same ladder against the unmodified tree first, at zero agent cost:

[preflight]
commands = [
  "python3 tools/gate.py --only-rung 1 --mode {mode}",
  "python3 tools/gate.py --only-rung 2 --digest {digest} --limit 1",
]
baseline = "python3 tools/gate.py --only-rung 3 --digest {digest}"

• Each command runs sh -c in the workspace. Its last non-empty stdout line must be a JSON object; the rung passed when it exited 0 and pass is absent or true. Recognized keys: digest, score, tiebreak, note, logs.
• {mode} fans a command out over every build mode declared in [measure.build].mutable_kwargs.mode, in declared order. A {mode} command without that list is a manifest load error: a derive-only preflight passes while full mode is broken.
• {digest} interpolates the most recent digest an earlier rung emitted, so the build rung hands its artifact to the correctness rung. Using it before any rung produced one fails.
• Any failure is an environment verdict: the run refuses to start, logging the failing command, its note, and a stderr tail, as a preflight-failed row. No iteration is burned.
• baseline is optional. Its score/tiebreak become segment.baseline_score, so a skip_baseline (codegen) domain decides iteration 1 against a real number instead of the direction's worst-score sentinel. Without it, that sentinel stands.

The three extension points

• New domain → common case: a manifest pointing CommandWorld + CommandJudge at a repo and a measure command. No Rust. Custom World/Judge impls only for in-process measurement or a non-file world.
• Composite domain → assemble N existing component domains into one run: a top-level [composite] table referencing component domain dirs (reused verbatim), a CompositeWorld (a tuple of per-component git overlays + the live deployment), and one combined gate over the assembled stack. No engine changes: the engine just runs a vector of workspaces (ADR 0009).
• New frontend → implement Reporter.
• New agent transport → add an AgentSource.

The canonical example: the counter, mapped to the contract

examples/counter/ is the whole system with every heavy part swapped for the smallest thing that satisfies the contract. Five small files, and each one fills a contract role. This table is the example:

Because there is no [world] block, GitWorld supplies reversibility for free: kept iterations are commits, discards are git reset --hard. Because the backend is command, the whole loop runs in milliseconds and costs nothing, yet it is a real end-to-end run (manifest → setup → propose → measure → keep/discard → git memory), not a mock.

A production domain fills the same roles with heavier pieces. A serving-stack domain has two reversible things (the code tree and a live deployment), so it adds snapshot_cmd/restore_cmd that capture image + config alongside the git half; its measure becomes a compiled benchmark binary; its propose backend becomes local or openshell; and its thin glue (apply pipelines, deployment introspection, steer/stop) lands on PATH as bare names so the contract vocabulary reads the same regardless of what's behind each one. A normal repo needs none of that: one measure command and the git default.

Adding a new domain (the target workflow)

1. crucible init scaffolds a starter crucible.toml + measure stub onto the repo (or write one by hand):

   [repo]      url|path, ref
   [workspace] setup_cmd          # optional; default: git clone + checkout
               carry_forward      # optional; untracked derived paths a discard keeps
               [[workspace.artifact]] path, embed?, title?   # carried + published to PR/S3
   [agent]     model, method_prompt, goal_file|goal, toolbox_dir, env
   [judge]     measure_cmd, direction = "lower"|"higher"
   [world]     apply_cmd?, snapshot_cmd?, restore_cmd?   # omitted → git
   

2. Write a measure command (any language) that prints {valid, score, solved?}.
3. crucible check --manifest crucible.toml validates it (every referenced file resolves, measure_cmd runs once and prints the contract shape) with no agent turn spent.
4. Run crucible --manifest crucible.toml. You get the loop, budget, all frontends, steer/stop/resume, the session log, and escalation, free.

The litmus test for "framework, not demo": any second repo runs this way with no new Rust.

The task lane: no [judge]

Some work has no objective to score: "consolidate the open dependabot PRs", "fix flaky tests nightly". Omit [judge] entirely and the run becomes a task: the loop runs as ever (sandbox, broker, session log, publish, resume), but with the built-in keep-everything TaskJudge — no baseline measure, no score, every completed turn is kept and snapshotted, and the run exits 0 when the iterations are spent (ADR-0026).

[repo]
url = "https://github.com/you/repo"
[agent]
backend = "openshell"
goal = "Consolidate the open dependabot PRs into one branch; run the tests."
[publish]
pr_repo = "you/repo-fork"   # the deliverable: kept commits as one draft PR

On the wire the run is the normal shape with gate: "task" as the discriminator: an iter-0 baseline-skipped row, then keep rows with score: null, Summary.best_score null. crucible check skips the measure probe and instead requires a goal, printing a loud "task mode" notice. Things to know:

• Exit 0 means the run completed, not that the chore succeeded — inspect the rows or the PR.
• solved never fires; the iteration budget is the only terminator (--no-early-stop is a no-op).
• The trust boundary is unchanged: the agent still holds no credentials. Output lands as a draft PR; privileged write actions (merging, closing issues) stay broker-tool material.
• Composites still require [judge]; so do [search], [workflow], and [preflight], which a task manifest rejects at load. The scope pipeline still rejects judge-less proposed packs, and a scored judge may not claim objective = "task".

examples/task/ is the runnable reference (deterministic command backend, no LLM), and Tasks: general-purpose orchestration is the full how-to.

Distress: the agent can page you

A doomed run used to burn its budget quietly. The broker exposes a distress tool so the agent can say so, with a required severity that decides what happens:

error is for runs that cannot succeed without you: the same environment failure recurring across iterations, a missing capability, a goal that is infeasible as specified. It is not for hard-but-possible tasks. The tool replies {"status":"suspending"} and the agent wraps up its turn; the suspend happens at the next iteration head, after that turn's bookkeeping, so nothing is lost. Repeat error calls while suspended are no-ops. If the marker cannot be written (a full or read-only volume) the tool returns an error instead of suspending and posts nothing: the run is not suspended, so neither the agent nor the page may claim it is.

Mechanically the suspend is a pending approval: the loop opens the same approval_wait bracket provisioning uses, so suspended wall-clock accrues to parked_total and is excluded from --max-time. The distressed iteration still counts (its turn ran); the suspended time counts against nothing. --max-park bounds the wait.

The handoff is one file on the loop pod's forge-storage volume, /var/lib/forge/distress. Deleting it is the resume grant:

oc exec <pod> -n <ns> -- rm /var/lib/forge/distress   # resume in place

For a fix that has to be baked at pod start (image pins, the pack, resources), re-roll the pod instead: the marker lives on an emptyDir, so the replacement starts clean, and a resume that finds the dangling distress bracket does not re-park. Info/warn notes ride /var/lib/forge/distress-notes.jsonl and are consumed onto the next row.

Env the render stamps on the loop pod (the broker inherits it, so the page can name the run): CRUCIBLE_RUN_NAME, CRUCIBLE_ITERATIONS, CRUCIBLE_LOOP_IMAGE, alongside the existing downward-API CRUCIBLE_POD_NAME / CRUCIBLE_POD_NAMESPACE. Slack is webhook-only: point SLACK_WEBHOOK_URL at an incoming webhook through the profile's [[secret_env]], and set DATADOG_BASE_URL if your Datadog site is not app.datadoghq.com. With no webhook configured the suspend still happens; delivery is fire-and-forget by design.

crucible flow: explain a finished run

crucible flow --session state/session.jsonl --out flow.html folds a finished run into a small flow-model IR and renders it; the --out extension picks .json (the IR itself), .dot, .mmd, or .html — a self-contained explainer page: header strip, a swimlane of iterations (agent turn + check ladder per column, scores vs baseline), per-iteration detail, and the draft-PR footer. An optional --spans <dd-export.json> (Datadog APM) adds real wall-clock: time-proportional columns, check durations, the agent's tool-call timeline.

Concept → code (where to look)

• loop / budget / keep-discard: crucible/src/run.rs + crucible/src/loop_driver.rs
• the contract traits: crucible/src/crucible.rs
• World/Judge batteries: crucible/src/command_world.rs (GitWorld/CommandWorld/CompositeWorld), crucible/src/command_judge.rs (CommandJudge)
• composite domains: crucible/src/manifest.rs (CompositeManifest) + crucible/src/run.rs (run_composite)
• mediated broker (provisioning / issue tracker / profiler / draft PR): crucible-broker/
• distress (agent-raised suspend + the Slack page): crucible-broker/src/distress.rs + crucible/src/distress.rs
• engine-side build + rendered deploy: forge/ (build, native-OCI derive, kube-rs) + crucible/src/deploy/
• agent transport: crucible/src/agent.rs
• frontends: crucible/src/{console,jsonl,stream}.rs
• session log wire format: crucible/src/session.rs
• the example domain: examples/counter/ (manifest + measure.nu + bump.nu)

Getting started: zero to a running loop

This is the hands-on path from nothing to a crucible loop. It's deliberately end-to-end, two acts:

1. Run a loop locally — the hello-world domain, no cluster, no LLM, runs in milliseconds.
2. Onboard your own domain — the manifest + the measure contract, the only two things you have to write.

For the why behind any of this, read How it works (the diagram + glossary) and What crucible is first. The manifest and command shapes below are the hands-on expansion of the contract, which is the normative reference. When this page and the contract disagree, the contract wins.

Before you start

Tooling (from the README): Rust + cargo, go, kubectl/oc, helm/helmfile, just, nu (nushell, the glue language), podman/buildah, gitleaks, and the claude CLI. Act 1 only needs Rust, just, and nu.

Credentials (Act 2 with a real agent backend): Claude access for the claude CLI. The command backend and crucible init/check/scope need no credentials at all.

Build the binary. Clone the repo (Act 1 uses the examples/counter files anyway) and build from source:

git clone https://github.com/neuralmagic/crucible.git
cd crucible
cargo build --release -p crucible
install -m 755 target/release/crucible ~/.local/bin/   # or anywhere on PATH

Or grab a prebuilt binary from the releases page when one is available for your platform. With just the binary you can already onboard your own repo (Act 2's crucible init → crucible check → crucible scope path needs nothing else).

One convention worth knowing before Act 2: a manifest refers to a pack's tools by bare name. The counter example sidesteps this entirely (its bump.nu and measure.nu live in the pack directory and are invoked by relative path); a pack that ships tool binaries is responsible for putting them on $PATH before the loop runs.

Act 1 — Run a loop locally

The fastest way to see the whole engine is examples/counter. The "repo under test" is a git tree whose entire source is an integer in value.txt; the score is that integer; a deterministic command backend (bump.nu) stands in for the LLM so it runs for free. It exercises the real generic path (manifest → setup → propose → measure → keep/discard → git memory) with no cluster and no model.

crucible --manifest examples/counter/crucible.toml --iterations 6
# or, from a source checkout: cargo run -p crucible -- --manifest examples/counter/crucible.toml --iterations 6

The manifest is the whole story (examples/counter/crucible.toml):

[repo]
path = "."

[workspace]
dir = "workspace"
setup_cmd = "mkdir -p workspace && cp value.txt measure.nu bump.nu workspace/ && git -C workspace init -q && git -C workspace add -A && git -C workspace -c user.email=crucible@local -c user.name=crucible -c commit.gpgsign=false commit -qm baseline"

[agent]
backend   = "command"
agent_cmd = "./bump.nu"   # deterministic proposer: value.txt += 1
goal      = "Raise the integer in value.txt as high as you can (target: 5)."

[judge]
measure_cmd = "./measure.nu"
direction   = "higher"
objective   = "value"

# No [world] block -> GitWorld: kept iterations are commits, discards reset --hard + clean.

What happens each iteration: the engine runs agent_cmd to mutate the workspace, runs measure_cmd to score it, and keeps the change (a git commit) iff it's strictly better per direction, else rolls the workspace back. What you get out:

• examples/counter/state/session.jsonl — the append-only NDJSON event log (the source of truth for a run).
• The workspace git history — kept iterations are commits. Git is the memory.

That's the engine. Everything else is swapping the deterministic proposer for a real agent and pointing measure at a real objective.

Act 2 — Onboard your own domain

A domain is a manifest plus a few executables, not Rust. The litmus test for "framework, not demo" is that a new domain runs with no engine changes. You write two things: a crucible.toml and a measure command.

crucible init scaffolds both onto the current directory (a minimal manifest + a measure stub that always reports the same score), so onboarding starts as editing two generated files rather than writing them from scratch. Once you've pointed [judge].measure_cmd at the real gate, crucible check --manifest crucible.toml validates the whole thing (every referenced file resolves, measure_cmd runs once and prints the contract shape, and it warns if the gate sits unprotected inside the workspace the agent edits) before you spend a real agent turn on it. crucible scope --pack <dir> chains ingest (resolve the goal) → check → render WORKFLOW.png → freeze (SCOPE.md with a RunIdentity digest) into one pipeline (ADR-0014 S0); add --issue owner/repo#N to ground the goal in a GitHub issue (native API fetch, honors GITHUB_TOKEN/GH_TOKEN).

The manifest

Sections (full schema + per-field semantics in the contract; the parser rejects unknown keys, so typos fail loudly):

The single most important rule: omit [world] entirely and you get GitWorld (git is the only thing that needs rolling back). Set any of the three *_cmds and you get CommandWorld (git plus your snapshot/restore, e.g. a live cluster).

The measure contract

measure_cmd is the judge. Its last stdout line beginning with { is parsed as JSON:

{ "valid": true, "score": 234.1, "solved": false, "note": "p99 234.1ms" }

• valid (required bool), score (number), solved (bool, default false), note/detail (optional).
• A nonzero exit forces valid=false regardless of what you printed.
• The engine injects CRUCIBLE_BASELINE_SCORE, CRUCIBLE_BASELINE_TOTAL, CRUCIBLE_BEST_SCORE so your gate can compare.
• Keep rule: keep iff valid and score strictly beats the best per direction (or solved is true, which forces a keep and ends the run, that's the bug-fix-gate case).

measure_cmd is any executable that prints one JSON object. A plain sh script is a complete judge:

#!/bin/sh
p99=$(make bench | jq .p99_ms)
printf '{"valid": true, "score": %s, "note": "p99 %sms"}\n' "$p99" "$p99"

Any language works; the engine only reads stdout.

Backends: who actually proposes

Set by [agent].backend:

• local (default) — spawns the claude CLI on this machine in the workspace. Needs the CLI + Claude creds.
• openshell — runs the turn inside a sandboxed pod (Landlock + deny-by-default egress). Needs sandbox_image + an [agent.openshell].endpoints egress allowlist. This is the in-cluster path.
• command — a fixed sh -c command as the "proposal," deterministic, no LLM, cost 0. The Act 1 counter case.

Composite domains

A manifest with a top-level [composite] table assembles ≥2 existing component domains (reused verbatim) into one run with a single combined gate. Picture a server domain and a router domain: each keeps its own workspace and repo, but one measure_cmd scores the assembled system, so a change to either component is judged by how the pair behaves together. No engine changes; the engine just runs a vector of workspaces. Beyond the manifest, the work of a composite is the shared plumbing: a combined gate that exercises both components, and per-component deploy targets if the domains rebuild images.

Where to go next

• How it works — the full diagram + glossary.
• What crucible is — the conceptual deep-dive and the contract table.
• Implementation contract — the normative manifest + command spec.
• JIRA tools (mediated) — grounding a goal in an issue.

Tasks: general-purpose orchestration

Crucible is an optimization loop, but not every job has an objective. The task lane runs the same loop — sandbox, broker mediation, session log, publish, resume — for work that just needs doing: consolidate the open dependabot PRs, fix flaky tests nightly, regenerate docs, triage an issue backlog. One manifest, no Rust, no gate script.

Opt in by omitting [judge]. That's the whole switch (ADR-0026).

The minimal manifest

[repo]
url = "https://github.com/you/repo"

[agent]
backend = "local"        # or "openshell" for the sandboxed path
model   = "claude-opus-4-6"
goal    = """
Consolidate the open dependabot PRs into a single branch.
Resolve conflicts, run the test suite, and write a summary to RESULTS.md.
"""

[publish]
pr_repo = "you/repo-fork"   # kept commits ship as one draft PR

Run it:

crucible check --manifest task.toml     # validates; prints the task-mode notice
crucible --manifest task.toml --iterations 3

Each iteration is one agent turn. Every completed turn is kept and committed to git memory; there is no baseline, no score, no discard. The run exits 0 when the iterations are spent.

What you get for free

• Reversible memory: each turn is a commit; an interrupted run resumes with --resume from the session log, rolling back any half-finished turn to the last kept state.
• The deliverable: kept commits pushed as a draft PR ([publish].pr_repo), with the run log and per-turn diffs recorded to S3 when a results bucket is configured.
• The trust model: with backend = "openshell" the agent runs under Landlock with a deny-by-default egress allowlist, and holds no credentials. Anything privileged goes through the broker's tools, never the agent's environment. Merging the PR stays with you.
• Forensics: state/session.jsonl is the source of truth; crucible flow --session state/session.jsonl --out flow.html renders a self-contained explainer of what the agent did each turn.
• Steering: STEER.md, the control bridge, distress paging, and budget caps (--max-cost, --max-time) all work exactly as in scored runs.

Semantics to know

The runnable reference

examples/task/ is the litmus domain: the deterministic command backend stands in for an LLM, so the full path (manifest → turns → keeps → session log → resume) runs in milliseconds:

crucible --manifest examples/task/crucible.toml --iterations 3

Use its manifest as the template for anything that authors task manifests programmatically.

Images for a new domain

A crucible run uses two pods, and a domain that needs a toolchain has to put it in both:

The runtime image is domain-neutral on purpose: it ships gcc, openssl-devel, python3, git, jq, and nothing else a measure might want. The sandbox ships the agent CLI and python. Everything a domain's measure_cmd or agent turn needs beyond that is the domain's image to add. examples/selfhost (Rust) is the worked example: Containerfile.runtime-rust and Containerfile.sandbox-rust.

What the two images must satisfy

Loop pod (Containerfile.runtime-<domain>)

• FROM ghcr.io/neuralmagic/crucible:<tag>; keep WORKDIR /opt/crucible and do not replace /usr/local/bin/{crucible,openshell,openshell-gateway}.
• Install the toolchain somewhere world-readable (/usr/local/...), not under /root: the pod runs as an arbitrary uid on OpenShift.
• Pre-fetch what measure_cmd builds against. The loop pod has egress (crates.io answers 200 from crucible-system on waldorf, checked 2026-08-25), but a cold dependency fetch inside the first measure is slow and is the first thing to flake.
• dnf is available (full UBI10).

Agent sandbox (Containerfile.sandbox-<domain>)

• FROM quay.io/aipcc/agentic-ci/claude-sandbox:<tag>. It is a stripped UBI10: rpm but no dnf, no compiler, uid sandbox (998), HOME=/sandbox, claude at /usr/local/bin/claude.

• To add RPMs, install them into a rootfs on a full UBI10 stage of the same release and overlay it:

  FROM registry.access.redhat.com/ubi10/ubi:10.2 AS pkgs
  RUN dnf install -y --nodocs --installroot /mnt/rootfs --releasever 10 \
          --setopt install_weak_deps=false gcc glibc-devel binutils ... \
      && dnf clean all --installroot /mnt/rootfs
  FROM quay.io/aipcc/agentic-ci/claude-sandbox:0.3.11
  COPY --from=pkgs /mnt/rootfs/ /
  

  perf is not in the UBI repos; sampling profilers that carry their own sampler (samply) work, cargo flamegraph does not.

• Toolchain under /usr/local with chmod -R a+rwX on its caches, and USER sandbox at the end. Any ENV PATH you set must include the toolchain's bin dir; the agent inherits it.

• Pre-fetch dependencies here too, and more aggressively: the sandbox is deny-by-default egress, so a cargo add/pip install the agent does mid-turn only works if the registry is on the manifest's [agent.openshell].endpoints allowlist and the binary doing the fetch is in [agent.openshell].binaries (e.g. /usr/local/cargo/bin/cargo). Anything already in the image's cache sidesteps both.

• Add the sandbox base to tools/base-image-allowlist.txt (the build-graph lint fails on an unknown base).

Build

Both images are amd64 and heavy (compiler + dep fetch). Build on the cluster, not under QEMU:

buildit build quay.io/<you>/crucible-sandbox-<domain>:v1 -n weaton-dev \
    --kubecontext coreweave-waldorf --mode job -f Containerfile.sandbox-<domain> \
    --request cpu=8 --request memory=16Gi
buildit wait <job> -n weaton-dev --kubecontext coreweave-waldorf   # last line: digest-pinned ref

.dockerignore keeps target/, .claude/, and example workspaces out of the context. Pin CRUCIBLE_TAG for the runtime derivative (--build-arg CRUCIBLE_TAG=<tag>), or the loop pod's engine can drift from the controller's.

A fresh quay repo is private: the namespace needs an imagePullSecrets entry (a kubernetes.io/dockerconfigjson secret from ~/.docker/config.json) before the pod can pull, or the pod sits in ImagePullBackOff with a 401.

Verify before pointing a manifest at it

Run each image as a uid it will not expect and check the toolchain, the cache, and an offline build:

kubectl run smoke --restart=Never --image=<ref> \
    --overrides='{"spec":{"imagePullSecrets":[{"name":"quay-pull"}],"securityContext":{"runAsUser":12345,"runAsGroup":0}}}' \
    --command -- sh -c 'id; cargo --version; cd /tmp && cargo new -q t && cd t && cargo build -q --offline && ./target/debug/t'

Then crucible check --manifest <pack>/crucible.toml locally (it runs measure_cmd and the gate self-test), set backend = "openshell" + sandbox_image, and deploy.

Crucible implementation contract (frozen interface)

This is the spec for the interface between the engine (crucible) and a domain. The engine implements it, a domain satisfies it, and a minimal fake domain (examples/counter/) tests it end-to-end with no EPP and no cluster.

Concept: What crucible is. Trust line (engine hands the agent a World, never a Judge): ADR 0001.

Contract status: NORMATIVE. Words like must / exactly one are binding. The engine and every domain (EPP included) conform to this; tests assert it.

1. The manifest (crucible.toml)

The engine reads exactly one manifest per run. Default path ./crucible.toml (override --manifest <path>). The manifest directory = dirname(manifest); it anchors all config-relative paths below.

[repo]
# Exactly one of url|path. The upstream the agent edits a checkout of. A playbook may omit the
# whole table: its workspace is then an empty dir the injects populate.
url  = "https://github.com/owner/name.git"   # OR
path = "."                                    # local path, relative to manifest dir
ref  = "main"                                 # optional; default: clone default branch

[workspace]
dir       = "workspace"                       # checkout dir, relative to manifest dir. default "workspace"
setup_cmd = "git clone ... && git checkout"   # optional; default: engine git clone+checkout of [repo]

# Frozen-judge / fixture injection (optional). After setup, the engine copies each `src` (relative
# to the manifest dir, a baked artifact outside the clone target, so the agent has no pre-clone
# copy) to `dst` (relative to the workspace), then commits the result as the baseline. A `frozen`
# inject (default true) is ALSO re-copied before every scored measure and before every plan task,
# so a candidate can't edit the gate (a T1 scoring harness, a seeded regression test) to game it;
# set `frozen = false` for a one-time fixture the agent may then edit. This is the generic
# alternative to hand-chaining an `install` into `setup_cmd`.
#
# A string entry is a frozen copy to the same path. A directory or a glob (`*`, `?`, `[`) expands
# to every regular file under it, sorted; it must stay under the manifest dir and must match
# something. Use the table form to rename, to unfreeze, or to reach outside the manifest dir.
inject = [
  "measure.sh",                                                                    # -> measure.sh, frozen
  "tools/*.py",                                                                    # every script, frozen
  { src = "judges/1489/judge_harness_test.go", dst = "pkg/.../judge_harness_test.go" },
  { src = "fixtures/seed.json", dst = "testdata/seed.json", frozen = false },
]

[agent]
model         = "claude-opus-4-6"             # default engine constant
method_prompt = "method.md"                   # manifest-relative file; template, {{GOAL}}/{{STATUS}}/{{STEER}}
goal          = "raise the score"             # OR goal_file = "goals/x.md" (manifest-relative)
toolbox_dir   = "commands"                    # optional; copied into <workspace>/.claude/skills
backend       = "local"                       # local | openshell | command   (see §6)
sandbox_image = "ghcr.io/<org>/<domain>-sandbox:<tag>"  # openshell backend only
allow_unverified_image = false            # launch on an image the controller's catalog cannot vouch for

[agent.requires]                          # what the sandbox image must provide; the controller refuses a launch otherwise
"toolchain.go" = ">=1.25"

[agent.prefers]                           # ranks compatible images in the controller's picker
"toolchain.go" = ">=1.26"
agent_cmd     = "..."                          # command backend only (§6)
[agent.env]                                   # injected into the agent process (creds, Vertex, etc.)
ANTHROPIC_VERTEX_PROJECT_ID = "my-gcp-project"

[judge]
measure_cmd = "./measure.sh"                  # REQUIRED. §3. Any executable, any language.
direction   = "lower"                         # lower | higher. REQUIRED.
tiebreak_direction = "lower"                  # optional; direction of the `tiebreak` scalar (§4). default: direction
objective   = "score"                         # display label (the old `gate` name). default "score"

[judge.selftest]                              # optional (ADR-0014 S1). Runs in `crucible check`, never in a loop iteration.
good_cmd = "..."                              # stages a known-good config in the workspace
bad_cmd  = "..."                              # stages a known-bad config in the workspace
runs     = 1                                  # measurements averaged per control. default 1

[world]
# All three omitted  →  GitWorld (tree-only reversibility; the 80% case).
apply_cmd    = "..."                          # optional. §3.
snapshot_cmd = "..."                          # optional. §3.
restore_cmd  = "..."                          # optional. §3.

[search]                                      # optional (ADR-0010). Absent or wide=0 → pure-deep, the default.
wide       = 0                                # u32. N parallel propose turns before the deep loop. default 0
approaches = ["...", "..."]                   # REQUIRED when wide > 0: one distinct approach string per candidate slot
policy     = "top-k"                          # only v1 policy. default "top-k"
policy_k   = 1                                # how many wide-round winners seed the deep loop. default 1, must be in 1..=wide

Rules:

• Required: [repo] (url xor path), [judge].measure_cmd, [judge].direction.
• [world] with no commands ⇒ GitWorld (§5). Any command given ⇒ CommandWorld (§5), which still owns git memory and layers the given commands on top.
• [judge.selftest], if present, requires both good_cmd and bad_cmd (a self-test that only stages one side isn't a control); runs must be >= 1.
• [search], if present with wide > 0, requires approaches.len() >= wide (hard error, diversity is engineered, not auto-generated) and policy_k in 1..=wide.
• Unknown keys are an error (typo protection), not silently ignored.
• Frozen loading (load_frozen). When the manifest file lives inside the workspace it targets (the BYO on-ramp: crucible init scaffolds [repo] path = "."), the engine parses it from the workspace's pristine base commit, not the current working tree (so an in-flight agent edit to crucible.toml can't retarget its own gate mid-run). Before any base commit exists (the very first run), it hard-warns and trusts the working tree for that run only; later runs freeze to the base commit. A manifest that lives outside the workspace (any out-of-workspace pack) is unaffected.

1.1 The gate self-test (crucible check, ADR-0014 S1)

[judge.selftest] declares two controls the gate must tell apart before it's trusted. crucible check (§9) runs it pre-loop, never inside a loop iteration:

1. snapshot the pristine workspace,
2. restore to pristine, stage good_cmd, measure runs times through the domain's own Judge, restore to pristine again,
3. same for bad_cmd,
4. pass iff both controls' readings are all valid and good's mean score is strictly better than bad's per [judge].direction.

The workspace is restored to pristine on every exit path (pass, fail, or error). A manifest with no [judge.selftest] isn't an error, crucible check warns instead, since the gate hasn't been proven to discriminate.

1.2 Wide-round search ([search], ADR-0010)

[search].wide > 0 (or --wide N on the CLI, which overrides the manifest) fans out N independent PROPOSE turns in per-candidate git worktrees under the state dir before the deep loop starts, one turn per approaches entry biased into its prompt. Each candidate's diff is applied (cherry-picked) into the shared main workspace and measured serially there (measurement never runs concurrently, only proposal does). The scored set is ranked by [search].policy (v1: "top-k"); the policy_k (or --wide-keep K) winner(s) seed the deep loop, which then runs as normal. Session rows from the wide round carry an additive phase: "wide" field (§7) so a consumer can tell a wide-round row from a deep-loop row without a wire-shape change.

1.3 Scope-authored workflows (workflow.star)

A scoped pack may include workflow.star beside crucible.toml. It is authoring syntax, not a runtime interpreter: scope compiles it to the existing [[workflow.task]] manifest IR before validation and again before freeze. The generated TOML is the runtime authority. This keeps plan authorship readable without moving the frozen judge or execution semantics into Starlark.

Topology is authorable; authority is not. A workflow declares type = "autoresearch" or type = "custom", and an engine or outer orchestrator admits it only when it advertises the matching workflow capability. Each engine task also requires its own capability. Crucible's loop advertises workflow.autoresearch plus propose/apply/measure/grade/decide, while its generic plan runner does not. It adds agent.session.persist only when the selected backend and harness can honor an opaque continuation. Serializing an engine task never grants access to the World or frozen Judge.

An autoresearch workflow is checked by semantics rather than reserved task names. Its selected result must be a decide task sourced from a frozen measure or authored grade, with apply and propose ancestors. Tasks may be inserted anywhere, operations may be renamed, and multiple candidate or measurement branches may exist. A custom workflow has no autoresearch-shape requirement; universal DAG, source-typing, and operation-capability rules still apply.

The basic autoresearch flow is explicit and editable:

candidate = propose(name = "invent", session = "solver")

critics = [
    agent(
        name = "correctness",
        prompt = prompt_file("prompts/correctness.md"),
        model = "claude-opus-4-6",
        effort = "high",
        isolated = True,
        depends_on = [candidate],
    ),
    agent(
        name = "novelty",
        prompt = prompt_file("prompts/novelty.md"),
        required = False,
        isolated = True,
        depends_on = [candidate],
    ),
]

synthesize = agent(
    name = "synthesize",
    prompt = prompt_file("prompts/synthesize.md"),
    session = "solver",
    depends_on = critics,
    join = "passed",
)
smoke = command(name = "smoke", run = "./smoke.sh", depends_on = [synthesize])
live = apply(name = "deploy-preview", depends_on = [smoke])
score = measure(name = "benchmark", depends_on = [live])
decision = decide(name = "keep-if-better", measurement = score)

workflow(
    type = "autoresearch",
    tasks = [candidate] + critics + [synthesize, smoke, live, score, decision],
    result = decision,
)

default_autoresearch(extra_tasks) is the compatibility convenience. It expands to the same ordinary propose/apply/measure/decide tasks, attaching unconnected extras after proposal and before apply. A missing workflow.star retains the historical default loop behavior. The old positional workflow(tasks) form remains accepted as a splice adapter while packs migrate.

Measurement can remain the historical opaque measure() call, or be authored as a visible DAG:

live = apply(name = "deploy", depends_on = [candidate])
correctness = evaluate(
    name = "correctness",
    run = "./correctness.sh",
    depends_on = [live],
    isolated = True,
)
latency = evaluate(
    name = "latency",
    run = "./latency.sh",
    depends_on = [correctness],
    threshold = 12.5,
    direction = "lower",
    isolated = True,
)
racecheck = evaluate(
    name = "racecheck",
    run = "./racecheck.sh",
    depends_on = [correctness],
    required = False,
    isolated = True,
)
measurement = grade(
    name = "grade",
    evidence = [correctness, latency, racecheck],
    score = latency,
)
decision = decide(name = "choose", measurement = measurement)

Dependencies define measurement rungs; ready isolated siblings run concurrently. evaluate() expects a JSON object on its last stdout line. pass = false vetoes a result, malformed pass fails closed, and paired threshold/direction fields grade numeric score. Without a threshold, omitted pass means success. grade() selects a passing score evaluator and folds passing evidence by default; set join = "all" for a strict join. The legacy measure() path and default workflow are unchanged.

session = "solver" binds agent-producing tasks to a durable logical conversation. The checkout may roll back after a discarded candidate while the solver session continues forward and retains what it learned. Tasks sharing a session must be dependency-ordered and cannot be isolated; parallel critics should stay fresh or use distinct sessions. Admission requires agent.session.persist. A missing session preserves the historical fresh-turn behavior.

Sessions can also be declared first-class with session(...) and bound by value:

solver = session(name = "solver", model = "claude-opus-4-6", effort = "high")
candidate = propose(name = "invent", session = solver)
refine = agent(name = "refine", prompt = prompt_file("prompts/refine.md"), session = solver,
               depends_on = [candidate])

Declarations are compile-time only; the generated manifest carries the same per-task session, harness, model, and effort fields as before. The rules:

• A declaration's harness / model / effort are defaults that materialize onto every agent task bound to it. A bound task may repeat a value but not contradict it: one session is one serial conversation under one agent config.
• A session carrying defaults cannot bind to propose(), whose agent config is owned by the manifest's [agent]. A default-free declaration binds to it exactly as a string does.
• Duplicate declarations of one name, and declarations never bound to a task, are compile errors.
• While a file declares no sessions, bare strings keep the historical pass-through behavior. Once any session() exists, every string session = "x" must name a declared session (declared before use), so a typo can no longer silently open a second fresh conversation.

Tasks may also declare their output contract: emits = ["score", "pass"] on agent(), command(), or evaluate() names fields the task's JSON output promises to include. Compilation rejects a top_k dependency, grade score source, or thresholded evaluate whose declared emits omits score; at runtime a passing attempt missing a declared field becomes a measured failure at the producing task instead of a mystery downstream. An absent emits declares nothing and changes nothing.

Compile errors carry file:line:col and a did-you-mean suggestion for unknown functions, kwargs, variables, and session names. A behavioral change from earlier releases: a task constructed in workflow.star but omitted from workflow(tasks = ...) is now a compile error naming the construction site, because a silently dropped task is a silently weakened measurement. Delete the assignment or include the task.

Crucible's private ledger contains only the logical name, an opaque harness cursor, and a completed-turn count. It never copies that cursor or Claude's native transcript into session.jsonl; the existing live harness event policy, including streamed thinking events, is unchanged. Claude Code implements native start/resume for both the local and Vertex/OpenShell paths. Because OpenShell sandboxes are per-turn-fresh, Crucible saves Claude's native transcript as mode-0600 private engine state and restores it at the exact pinned config path in the next sandbox; that file is never included in the published run record. Hermes fails closed for a persistent binding until it has an equivalent continuation store; it never silently degrades the binding to a fresh prompt.

The first turn receives the complete method and goal prompt. A resumed proposer receives only the new authoritative delta: current regime, current-best status, and new steering. Its retained session already holds the stable instructions and hypotheses, while the current checkout and RESULTS.md remain authoritative after any world rollback.

A custom orchestrator can admit a graph with no research lifecycle at all—for example, a creative studio that fans out three treatments, curates them, and publishes a contact sheet:

treatments = [
    agent(name = "surreal", prompt = prompt_file("prompts/surreal.md"), isolated = True),
    agent(name = "minimal", prompt = prompt_file("prompts/minimal.md"), isolated = True),
    agent(name = "documentary", prompt = prompt_file("prompts/documentary.md"), isolated = True),
]
curate = agent(
    name = "curate",
    prompt = prompt_file("prompts/curate.md"),
    depends_on = treatments,
    join = "passed",
)
publish = command(name = "contact-sheet", run = "./render.sh", depends_on = [curate])
workflow(type = "custom", tasks = treatments + [curate, publish], result = publish)

That graph requires workflow.custom; it does not pretend to satisfy autoresearch just because it uses agents and commands.

This is Starlark over a constructor-only global surface: assignments, strings, numbers, booleans, lists, def, if/else, for, comprehensions, and calls to the functions below, at the top level or inside a def. load() resolves against the pack directory only. Task, session, and workflow values are opaque and immutable, so they can be referenced directly in depends_on, measurement, and result without repeating names, and a library cannot mutate one after handing it back.

The declared lane decides which constructors exist. Which lane each belongs to, and what arguments each takes, is the DSL reference, generated from the compiler's own tables. A playbook does not have the scored lane's constructors in scope at all, so naming one is an unknown-name error where it was written, and a did-you-mean never offers one.

• propose(...), apply(...), measure(...), grade(...), and decide(...) create capability-owned engine tasks. decide(measurement = score) selects its measurement. Scored lanes only.
• agent(...) creates an agent task. isolated = True gives it a disposable worktree, ideal for concurrent read-only critics; leave it false for a synthesizer whose edits must survive. session = "name" opts into an engine-managed durable conversation.
• session(name = ..., harness = ?, model = ?, effort = ?) declares a durable conversation with optional agent defaults, bindable as the session = value on agent() and propose().
• command(...) creates a deterministic shell task in the candidate workspace.
• evaluate(...) creates a typed measurement command with optional threshold grading.
• top_k(...) creates a reducer for wider authored graphs. Scored lanes only.
• prompt_file(path) reads a regular UTF-8 file below the pack directory and embeds its contents in the generated manifest. Absolute paths, .., symlinks, non-files, and oversized inputs are rejected.
• load(path, name, ...) pulls symbols from another .star file under the pack, resolved by the same policy as prompt_file and refused for absolute paths, .., symlinks, non-files, and cycles. Loaded modules run before the root against the same globals and the same compile state: their prompt_file() calls resolve against the pack root and charge the same byte budget, their session() declarations precede every root reference, and a task they construct at module level must still appear in workflow(tasks = ...). Re-export is off, so a symbol a library loads is not visible through it.
• workflow(type = ..., tasks = ..., result = ...) is the explicit final expression. A list of tasks is accepted anywhere a list of task names is, in tasks and in depends_on alike, so a list never needs wrapping to be passed.
• default_autoresearch(extra_tasks) expands the historical loop into fully visible nodes. Scored lanes only.

Five kwargs govern how a task runs and what its failure costs. They are independent, and this is the whole of it:

required = False and join = "all" do not compose: a required task may not depend, through a path of "all"-join edges, on an advisory one. That graph says a failure is both tolerable and disqualifying, and no run of it yields an honest verdict. Validation rejects it before dispatch, naming both tasks. join = "passed" is the exemption, because it declares up front that the task runs on whatever survived.

Agent tasks receive upstream results in their prompt and write one JSON object to PLAN_TASK_RESULT.json. Required failures discard the candidate; advisory tasks use required = False. join = "passed" waits for all dependencies, then receives their non-empty set of successful results. No passing input blocks the task.

The two settings must agree: a required task may not depend on an advisory task with join = "all", and validation rejects the graph naming both tasks. An advisory task is allowed to fail, and a join = "all" dependent blocks on that failure, so the required = False would buy nothing. Consume advisory work through a join = "passed" task, which is exempt along with everything reachable only through it. The legacy positional workflow(tasks) splice has no lever for this: its tasks feed the loop's required apply, so a spliced sink cannot be advisory.

For local review, crucible plan compile-workflow --file workflow.star prints stable canonical JSON. Add --manifest crucible.toml to also replace the generated [workflow] block. Compilation applies source-size, loaded-module, task-count, constructed-task, evaluation-tick, heap, call-depth, and prompt-size ceilings. The compiler exposes no filesystem API except prompt_file and load, and no process, environment, network, clock, or randomness API. Scope validation renders the admitted graph to WORKFLOW.png for the scope PR, grouping evaluate and grade as Measurement.

1.9 [outputs] and [capabilities] — declared writes and disclosed reach

[outputs] bounds the run's mediated writes (RFC-0001:C-OUTPUTS). The kind vocabulary is closed and engine-defined; a kind outside it is a manifest error.

# A declaration carries a per-run count, and a target for a kind that addresses one.
[outputs.image-push]
count = 20
target = { fixed = "quay.io/aipcc" }

# An OPEN target names a scope narrower than the kind's whole address space, and may bind that
# scope to a workflow parameter. The target must then equal that parameter's run value, which is
# how a run fanned out per tracker item confines its writes to the item that parameterized it.
[outputs.tracker-comment]
count = 3
target = { open = { scope = "PROJ-*", param = "issue_key" } }

A scope that admits any target ("*", "**", "") is rejected at validation. gpu-capture addresses nothing, so it takes a count and no target; every other kind requires one.

A kind the pack does not declare resolves to the engine default table. Every default carries a count and none carries an open target, so a pack shipping no [outputs] gets the conservative posture. A default whose target does not resolve refuses every write of that kind.

Bounds are enforced in the broker, at one chokepoint every mutating tool routes through. They are projected there from the frozen manifest as BROKER_OUTPUTS (with BROKER_OUTPUT_PARAMS and BROKER_SESSION_LOG), handed to the broker child only, so nothing inside the sandbox can alter one. The two kinds the engine writes itself rather than through a broker tool — the draft PRs publish-on-keep opens and the workflow_dispatch a github-actions build fires — are mediated in the engine against the same resolved value. A write over a count or outside a scope fails that write naming the bound, writes an output_refused row on the session log, and does not terminate the run: publishing skips the remaining candidates and the run completes.

[[capabilities.secret]] states what a credential authorizes; a name alone does not.

[[capabilities.secret]]
name    = "JIRA_API_TOKEN"
context = "broker"            # or "agent": whether the value enters the sandbox
system  = "jira"
scope   = "read + comment on PROJ"

The rest of the disclosure is read from what the manifest already declares: the resolved egress allowlist (each entry labelled builtin or manifest), every [agent].env name, every [[agent.relay]] destination, a substituted [agent.broker].bin, and whether the pack runs commands outside the sandbox (workflow command/evaluate tasks, world and judge hooks). At run start, an agent-visible env value or a relay file the disclosure does not cover is refused, naming the grant and what is missing.

crucible check prints the resolved bounds and the disclosure; crucible plan exposure --manifest crucible.toml emits the same as JSON ({version, outputs, capabilities}) for the controller to extract. Both compute it without executing pack content.

2. Path resolution (portable, never CARGO_MANIFEST_DIR)

The binary's own install location is never used to resolve anything. A target repo is self-describing: drop a crucible.toml at its root and run crucible inside it.

3. The command protocol

Every command (measure_cmd, apply_cmd, snapshot_cmd, restore_cmd, setup_cmd) is a string executed via sh -c "<cmd>", with:

• cwd = the agent workspace (manifest_dir/[workspace].dir).
• PATH inherited, so a command may be a bare installed tool (bench) or a workspace-relative script (./measure.sh).
• env = the engine's env + [agent].env (where relevant) + the injected variables below.

measure_cmd (the Judge): REQUIRED

• Injected env: CRUCIBLE_BASELINE_SCORE, CRUCIBLE_BASELINE_TOTAL, CRUCIBLE_BEST_SCORE (absent on the baseline measurement; present thereafter). Values are the engine's current numbers as decimal strings.
• stdout: the engine reads the last line that starts with { and parses it as:

  { "valid": true, "score": 12.5, "solved": false, "note": "p99=12.5ms", "detail": { } }
  

  • valid (bool, REQUIRED): false ⇒ unscoreable candidate, always discarded.
  • score (number|null): the fitness. null/absent ⇒ treated as invalid.
  • tiebreak (number, optional): secondary fitness for functional gates whose score is effectively boolean; on an exact score tie, a strictly better tiebreak still keeps (§4).
  • solved (bool, optional, default false): the win condition was met (terminates the loop).
  • note (string, optional): one-line human summary.
  • detail (object, optional): free-form; surfaced in the row + session log. The domain stashes anything extra here (e.g. EPP cache_hit_rate; the test gate's total).
• exit code: nonzero ⇒ the reading is forced valid:false regardless of stdout.

apply_cmd (make the candidate live): optional

• Run after the agent turn, before measure, when present. Nonzero exit ⇒ the iteration is treated as an invalid candidate (discard). No stdout contract.
• Omit for code/agent-deploys domains (EPP: the agent deploys via skills during its turn; the counter: the edit is the candidate). Present for engine-driven build+push+set-image.

3.1 Build mode: how an edit becomes the thing measured

Between "the agent edited a file" and "measure_cmd read a score" sits a step whose shape you must choose when you design a pack. It differs per component, it is the dominant term in per-iteration wall-clock, and picking it wrong fails silently. Full rationale in ADR-0020.

Rules that are easy to get wrong, and that crucible check should catch before you spend a turn:

• derive-layer requires that the base image and the push target share a registry. The layer mounts server-side only when they match; across registries the same operation streams every base blob (8 seconds becomes 20+ minutes) and nothing warns you.
• derive-layer cannot carry compiled sources. Appending a .go file to an image changes nothing that runs. The loop measures the base image forever and reports every candidate as a no-op, the failure mode ADR-0007 exists to catch.
• A compile failure must be distinguishable from a bad score. In image mode the build is the loop's fastest feedback signal: a compile error is returned to the agent as a free retry with no candidate spent. An apply_cmd that collapses "did not compile" into "scored badly" throws that away.
• A Containerfile on the measured path is part of the judge, not part of the solution. If the build recipe lives in the agent's workspace, it must be a frozen = true inject (§1), re-copied before every scored measure. Otherwise the agent can edit the recipe that builds the artifact it is scored on (vendor a prebuilt binary, neuter the compile), which is the ADR-0001 trust line, broken.

snapshot_cmd / restore_cmd (domain reversibility): optional, must come as a pair

• snapshot_cmd: stdout last line = one opaque token (any non-empty string; base64 it if it contains newlines). Nonzero exit ⇒ snapshot failed (engine aborts the keep).
• restore_cmd: receives the token in env CRUCIBLE_TOKEN. Rolls external state back to that token. Nonzero exit ⇒ restore failed (engine surfaces it). (Env, not argv, so a multi-line base64 payload round-trips without shell-quoting.)
• The engine treats the token as opaque and never parses it. (§5 explains how the engine frames its own git ref alongside this token.)

setup_cmd (prepare the workspace): optional

• The one cwd exception: runs with cwd = manifest dir (the workspace does not exist yet, it's what setup creates). Every other command runs with cwd = workspace.
• Default when omitted: engine does git clone [repo] <workspace> && git checkout [ref]; with no [repo] (a playbook), it creates <workspace> empty. Either way the injects land next and the engine git inits the workspace and commits the result as the baseline when setup left no repo behind, so a setup_cmd only has to produce the tree, never the commit. A setup_cmd that commits before returning freezes a baseline without the injects.

4. The decide rule (universal, no per-domain code)

Given a Reading { valid, score, solved, note, detail }, the current best_score, and the manifest direction:

keep   = valid && score.is_some() && (better(score, best_score, direction)
                                      || (score == best_score && tiebreak_better)
                                      || solved)
solved = reading.solved
better(s, b, lower)  = s < b
better(s, b, higher) = s > b

tiebreak_better applies only when the reading carries a tiebreak: it is better(tiebreak, best_tiebreak, tiebreak_direction), where tiebreak_direction is [judge].tiebreak_direction (optional, defaults to direction) and a best with no recorded tiebreak counts as the worst value. A reading without a tiebreak ties exactly as before: discard.

• solved implies keep. A win is the whole point, so a candidate the measure command declares solved is kept (and terminates the loop) even if its score doesn't strictly beat best. This is load-bearing for any domain whose win lands at an equal score: EPP's test gate wins with a green suite (0 failures == the baseline's 0) plus a new regression test. Without || solved that win would be discarded and the loop would never finish. solved never rescues an invalid reading.
• The first valid reading sets the baseline (best_score, and baseline_total = detail.total if present) and is always kept.
• The loop terminates when a kept iteration is solved, or budget/iterations exhausted, or stop/escalate.
• No domain Rust decides anything. A win condition more complex than "better score" (e.g. EPP's "green AND a new regression test") is computed inside the measure command, which reads CRUCIBLE_BASELINE_TOTAL and emits solved.

5. World = reversibility (engine never names git)

World::Snapshot = String, opaque to the engine (run_loop only round-trips it back to restore).

• GitWorld (default): snapshot() = stage+commit the workspace, return the commit SHA; restore(sha) = git reset --hard <sha> + git clean (excluding .claude/, RESULTS.md). The kept-commit chain is the memory. Works for any git repo, zero domain code.
• CommandWorld (any [world] command given): always owns git memory as above, and layers the domain commands. The snapshot token it stores is the composite "<git-sha>\t<domain-token>":
  • keep: commit (git half) → run snapshot_cmd, capture its token (domain half) → join.
  • discard: split → git reset --hard <sha> + clean → run restore_cmd with CRUCIBLE_TOKEN=<domain-token>.
  • The engine exposes last_commit_sha() (the git half) for kept_shas/publish; the domain half is never inspected.

The engine's loop body calls only world.snapshot() / world.restore(&snap). It contains no git/vcs:: calls and no kubectl.

6. Agent transport (AgentSource)

The engine renders a prompt (method_prompt with {{GOAL}}/{{STATUS}}/{{STEER}} filled), hands it + the workspace to an agent that edits the workspace, and never hands it the Judge (ADR-0001). Backends, selected by [agent].backend:

• local: direct claude --output-format stream-json with [agent].env (real Claude/Vertex turn).
• openshell: sandboxed pod turn driven by the in-Rust OpenShell driver (backend = "openshell", sandbox_image).
• command: run [agent].agent_cmd via sh -c in the workspace as the proposal. A deterministic, free proposer (no LLM). This is a real transport, not a mock: it makes the minimal example a fast, deterministic e2e (e.g. agent_cmd = "./bump.nu" increments a counter). Use it to test the engine's loop/protocol without burning tokens.

One vs two execution environments (the only backend fact a domain must know)

The engine and its measure/apply/snapshot/restore/setup commands always run where crucible runs. Only the agent turn is sandboxed under openshell. So:

• local / command: one environment. Agent and engine share the host PATH + filesystem; the agent edits the workspace in place. Provide one toolbox on PATH. Nothing else.
• openshell: two environments. (1) the engine/loop-image PATH runs the contract commands; (2) a separate sandbox image runs the agent's skills. The OpenShell driver uploads the workspace into the sandbox and syncs edits back, so anything the agent needs to reach the outside (kubeconfig, tokens) must be relayed via [[agent.relay]] or provided in [agent].env (the sandbox is network/fs-isolated). A domain therefore ships two tool surfaces under openshell: contract commands on the loop PATH, agent skills baked into sandbox_image.

The manifest [agent] block (backend, sandbox_image, env, relay, openshell) selects and configures the backend; flipping local↔openshell is config, not code. What the manifest cannot do is build the sandbox image or inject creds for you, that's the inherent cost of sandboxing, called out here so it's a known rule, not a surprise. Reversibility commands (snapshot/restore) run engine-side, so they reach the live system directly regardless of backend; only the agent is boxed.

6.1 Sandbox egress ([agent.openshell])

The sandbox is deny-by-default. Two lists open it back up, and a flag decides whether they extend the engine's built-ins or replace them:

[agent.openshell]
endpoints = ["api.example.com:443:full"]  # host:port:access[:proto[:enforcement]]
binaries  = ["/usr/local/bin/claude"]     # only these may open a socket
inherit_defaults = true                   # default

With inherit_defaults = true (the default) the lists are appended, de-duplicated, to the built-ins: the public forges, PyPI, Vertex, Anthropic, and the agent CLIs. Appending can never remove a built-in.

Set inherit_defaults = false and the resolved allowlist is exactly what the manifest names, binaries included. This is the only way to subtract a default, and it is required for two cases:

• Air-gapped or private-registry runs, where the public internet is not reachable (or not permitted) and the agent must talk to an in-cluster model endpoint and registry instead.
• Contamination control. An agent scored on an upstream issue can, with the default allowlist, read the upstream fix for that issue on github.com. A measurement that must not be polluted by the open web has to drop that endpoint, and dropping it means opting out.

The broker endpoint is auto-appended by the engine when [agent.broker].enabled is true (ADR-0019 P2). The engine first resolves the broker URL (an explicit [agent.broker].url override, or derived from the active compute driver's hostname), then derives the egress host:port:full entry from that URL's authority. When no explicit port is present, the scheme default applies (http = 80, https = 443). Because both are derived from the same resolved URL, the allowlist entry and the address the sandbox contacts cannot disagree. The broker endpoint is appended regardless of inherit_defaults, because the broker is engine plumbing the domain opted into, not a built-in the domain can subtract. A broker-less opt-out with both lists empty is still a legal total air-gap: nothing resolves, and no binary may open a socket.

7. Session wire format (compatibility)

The NDJSON session log (state/session.jsonl) keeps its existing event kinds (start/phase/row/budget/summary/finished) and field names unchanged. In particular the objective label is still written under the JSON key gate (now carrying a free-text label like "score"/"bench", not the deleted enum). This keeps --resume, the remote viewer, and already-published S3 runs loading. Do not rename the wire key.

A row event's wire record (RowWire) carries an additive, optional phase field ("wide" for a wide-round candidate row, absent for a deep-loop row). It's skip_serializing_if = "Option::is_none", so a deep-only run's wire bytes are unchanged from before wide rounds existed.

A phase event's phase is one of starting/preflight/baseline/wide/iteration/ paused/parked/distressed/escalated/epilogue/finished, the same token the control bridge's status reply carries, emitted once each time the loop's (phase, iter) pair changes. The three tokens logs carried before contract 1.3.0 are a subset, so old logs still decode.

A task_result event whose status is blocked carries an additive blocked object: { reason, task? } with reason one of required_task_failed/budget_ceiling/ wall_clock_ceiling/dependency_did_not_pass/staging_refused and task naming the required task whose failure short-circuited the plan (present only for required_task_failed). note stays the rendered form of the same reason. report.json's per-task entries carry the same object.

A task_result event whose status is transport carries an additive transport token (contract 1.4.0): what the last attempt died on, one of gateway/sandbox/agent/provider/ workspace/command/other. note stays the retry summary with the last attempt's detail. report.json's per-task entries carry the same token.

Additive event kinds beyond the compat set include:

• identity: the run's RunIdentity (below), emitted once at setup and again on --resume (the freshly recomputed identity). A mismatch against the original run's identity is a hard-warning note event, never an abort.
• shutdown: { outcome, reason }, emitted exactly once, as the last line of every run (after finished/summary). outcome is one of finished/solved/budget/stopped/ escalated/stalled/error. Session-log consumers key a run's terminal state off this line; a dead stream with no shutdown line means the pod likely died mid-run, not a clean exit. --resume consumes this invariant, not just documents it: a resumed run classifies the log tail (see recovery below) and a trailing shutdown is the "exited on purpose" signal. In a resumed (appended) log, only the trailing shutdown counts; one followed by more events belongs to an earlier process.
• agent_session: { session, action, turn }, emitted before a persistent agent turn so a viewer can draw continuation lanes and distinguish started from resumed. It deliberately contains neither the provider cursor nor native transcript content.
• approval_wait: { handle, trace_id, mode }, emitted when the loop reads the agent's pending-provisioning marker. mode is block (the loop parks idle) or continue (it keeps iterating in the frozen regime). Bracket invariant: every approval_wait is closed by an approval_resolved except on stop-while-parked and process death, so a dangling wait in the log tail means the run ended with the approval outstanding, and a resume re-parks a block-mode one and re-registers the approval key so an operator approve still resolves it.
• approval_resolved: { outcome, reason } with outcome one of granted/denied/ timeout. A grant is emitted at the iteration-head rescope drain (the single re-baseline site); a stop deliberately emits nothing (a stop doesn't resolve the ask).
• recovery: { class, iter, detail }, emitted once per --resume right after the resume note: how the resumed process classified its predecessor's end. class is one of clean_exit/died_in_baseline/died_in_wide_round/died_mid_turn/died_deciding/ died_in_plan_task/died_awaiting_approval/died_between_iterations; iter is the iteration the interruption touched (0 when not iteration-scoped); detail is a human-readable evidence summary. Purely a record: the loop acts on the in-process classification, never by re-reading this line.

RunIdentity (crucible/src/identity.rs) is the comparability key: two runs' scores are comparable only if it matches. It's a hash-of-hashes (v1:<hex>) over, per component (one unnamed entry for a single-domain run, one per [[component]] for a composite): repo ([repo].url/path) and the workspace's pristine base commit SHA; plus, once per run: the frozen manifest text's hash, a hash over every [[workspace.inject]]'s source content plus destination path, [judge].measure_cmd, and [judge].direction. It's computed once at run setup and doesn't change within a run (a re-scope moves the loop's own Segment fingerprint, a different hash over goal/objective/regime; the two are deliberately independent).

7.1 Admission ledger (state/admissions.jsonl) and the control-bridge id

Every external input into a run (steer, approve, deny, rescope, set-budget, pause, resume, stop, abort) is recorded in a second NDJSON file, state/admissions.jsonl, before it takes effect. Same envelope shape as the session log ({"v":1,"kind":…}, blank/torn lines skipped), two kinds:

• admitted: { key, seq, ts, input, …payload } — input is the command token and the payload is flattened alongside it ({"input":"rescope","regime":"c=48"}).
• settled: { key, outcome, ts, note } with outcome one of applied/superseded/ rejected.

Contract, per idempotency key: exactly one admitted, then at most one settled, and the first terminal outcome wins. A key with no settled line is an input the run still owes; --resume re-arms exactly those (an un-delivered steer, a granted-but-undrained re-scope, the live budget cap, the pause level) and closes out the ones a resume overrides (stop/abort become superseded, as do approvals that died before their grant was recorded).

Precedence: admissions.jsonl is authoritative for what an operator asked for; the session log is authoritative for what the loop was waiting on. Where they disagree about an outstanding approval, the ledger wins: a re-scope recorded under the key derived from the ask suppresses the session log's re-park.

Control-bridge commands gain an optional id (string, non-empty, ≤256 bytes) on every mutating object-form command:

{"cmd":"steer","text":"…","id":"pr-comment:owner/repo#7:12345"}

Redelivering the same id with the same payload converges on the original admission ({"ok":true,"cmd":"steer","key":"…","dup":true}, plus "outcome" when it already settled) rather than acting twice; the same id with a different payload is refused ({"ok":false,…,"error":"idempotency conflict: …"}) and nothing is written. Omitting id is exactly the old behavior: the server generates a key and every delivery is a fresh input, so old clients and old servers interoperate unchanged. A stop/abort whose record cannot be written still stops the run and says so with "unrecorded":true; every other command fails closed (no effect) if its admission can't be recorded.

Two consequences worth knowing: the bridge no longer writes STEER.md (a steer command goes straight into the ledger, and the loop's drain reads both the ledger and whatever the file channel accumulated), and "applied" for a steer means delivered into a turn's prompt, not heeded, and not that its iteration was kept.

8. What a domain author writes (the whole surface)

1. a crucible.toml,
2. a measure command emitting {valid, score, solved?} (any language),
3. optionally apply/snapshot/restore commands,
4. a method prompt + goal,
5. agent creds in [agent].env.

Everything else (loop, budget, keep/discard, all reporters + remote viewer, steer/stop/resume, session log, escalation, git memory) is the engine's, for free. The litmus test (examples/counter/) exercises items 1, 2, 4 with GitWorld and the command backend, no Rust.

9. CLI surface (non-normative pointer)

The manifest/protocol above is the contract; these subcommands are mechanical consumers of it, noted here tersely so this doc stays the map of what's authoritative:

• crucible init [--dir <path>]: scaffolds a minimal crucible.toml + a measure stub that always reports the same score. Refuses to overwrite existing files.
• crucible check --manifest <path>: validates a manifest with no agent turn (parses it, unknown keys are a parse error, resolves every referenced file, runs measure_cmd once to prove the measure contract, runs [judge.selftest] if declared (§1.1), and warns if the gate is reachable by the agent's own edits: a measure_cmd token pointing inside the workspace with no matching frozen inject). Exit nonzero with findings on failure; warnings never fail it.
• crucible scope --pack <dir> [--issue owner/repo#N | --goal-file <f>] [--force] [--json] (ADR-0014 S0): pipeline over a hand-written domain pack. ingest the goal (from --issue, fetched natively from the GitHub REST API (honors GITHUB_TOKEN/GH_TOKEN and GITHUB_API_URL), or --goal-file, or the pack's own [agent].goal/goal_file), then validate (crucible check, as a library call), then freeze (writes SCOPE.md in the pack dir with the goal, the check outcome, and the pack's RunIdentity digest). Stops at the first failing stage. S2 (propose)/S3 (preflight)/S4 (approval) are listed in SCOPE.md as pending, see ADR 0014. A --goal-file's text is goal framing under the same de-prescription rules as a --issue's title/body: the problem for the pack-designing agent to solve, never a solution. §8's _controls/self-test strip at freeze applies identically regardless of which arm sourced the goal.
• crucible ps [--namespace <ns>] [--json]: lists loop pods across the cluster, selecting on the app.kubernetes.io/managed-by=crucible label every rendered loop pod carries. ITER ships as - (reserved, see crucible/src/cli/ps.rs's module doc for why it isn't wired up yet).
• crucible deploy render|apply --manifest <path> --profile <path> [--iterations N] [--max-cost USD] [--no-pin] [--pack [--pack-configmap-name <name>]] [--pr-repo <owner/repo>] [--clusters <path>] [--harness <h>] [--model <m>] [--playbook --max-time <dur> [--param NAME=VALUE]…]: renders (or renders-then-applies) the loop pod + a cross-namespace RoleBinding from the manifest + a deploy profile, image tags resolved to @sha256:… digests. Works for a composite manifest or a plain single-domain one (the latter needs its own [deploy] block naming the build/deploy target, a single domain is a degenerate composite of one). --playbook renders the second mode: the pod runs crucible plan run over the manifest's [workflow] under the given ceilings and parameters instead of the agent loop. It is an explicit flag, never inferred from the manifest, and it conflicts with the loop-only knobs (--iterations, --controller, --pr-repo, --harness, --model). It requires a positive --max-cost and --max-time, and needs neither a [deploy] block nor [agent].sandbox_image (both describe a deployment a playbook never performs). --pack is orthogonal: it controls delivery, --playbook controls the command, and a controller-dispatched playbook passes both.
• crucible watch-pr --pr <url> [--pr <url> ...] (--control-addr <host:port> | --reseed <path>) [--once] [--poll-secs N] [--bot-user <login>] [--allow-user <login> ...]: watches one or more draft PRs' review comments (--pr repeatable: a kept composite candidate opens one linked PR per component fork) and either steers a live run over its control bridge or appends to a reseed file the next run's first turn reads. --once fetches and exits instead of polling.

Work graphs

A plan is a versioned DAG of tasks that a deterministic executor runs. Tasks are agent turns, plan-authored commands, or engine-owned reducers. The executor owns advancement: a task never decides what runs next.

Today, the engine supplies a default loop graph and a wide-tournament template, and a human or pack can supply TOML or JSON through the plan CLI. Workflow admission separates authorable topology from authority: an orchestrator must advertise the workflow type and engine operations it can safely execute.

Running a plan

# compile and print it, without executing
crucible plan show --file plan.toml
crucible plan show --file plan.toml --mermaid      # flowchart source
crucible plan show --file plan.toml --render       # PNG, inline if the terminal supports it

# execute: command tasks run as subprocesses, agent tasks through the real harness
crucible plan run --file plan.toml --manifest crucible.toml

# execute without a manifest: agent tasks run a stand-in command instead
crucible plan run --file plan.toml --agent-cmd ./role.sh

# replace the manifest's [agent] harness and model for this run; a task that pins its own keeps it
crucible plan run --manifest crucible.toml --harness codex --model gpt-5.6-luna

--cap <name> (repeatable) declares what the substrate can do; see needs below.

plan run exits nonzero when the plan does not reach a valid verdict.

File format

version = 1                 # the only format version accepted today
# reason = "..."            # reserved for a future replan protocol

[budget]
usd = 5.0                   # required, positive; execution fails closed on overrun

[[task]]
name = "propose"            # unique within the plan
kind = "agent"
prompt = "..."
model = "claude-opus-4-6"   # optional per-task overrides of the manifest's [agent]
harness = "claude"
effort = "high"
session = "solver"         # optional durable logical conversation

[[task]]
name = "measure"
kind = "command"
command = "./bench.sh"
depends_on = ["propose"]
needs = "gpu"               # default "any"
required = true             # default true
isolation = "worktree"      # optional
join = "all"                # default "all"
emits = ["score"]           # optional declared output fields; absent = undeclared

[[task]]
name = "pick"
kind = "top_k"
k = 1
direction = "lower"         # or "higher"
depends_on = ["measure"]

Task kinds

Serialization is not authority. A workflow may author and sequence engine nodes, but admission requires matching capabilities such as workflow.autoresearch, engine.apply, and engine.measure. The generic plan runner rejects them because it owns neither a World nor a frozen Judge.

A command string is not a trust boundary by itself. If it invokes a script that must remain trusted after an agent task edits the workspace, declare that script as a frozen [[workspace.inject]] in the manifest. The plan runner restores frozen injects in the task's actual workspace before every task, including isolated worktrees.

A logical session is serial state, so every pair of tasks sharing one must have a dependency path between them. A session task cannot use disposable worktree isolation. The private ledger keeps only an opaque harness cursor and completed-turn count; neither that cursor nor Claude's native transcript is copied into the plan or public session log. Normal streamed harness events retain their existing visibility. Claude Code's native transcript remains in its private local store or a mode-0600 engine store restored into each fresh OpenShell sandbox. Omit the field for the historical fresh-turn behavior.

Task output

A task's output is JSON and becomes its dependents' input.

• command: the last non-empty stdout line. Nonzero exit is a measured failure; a spawn failure is a transport failure. Upstream outputs arrive as CRUCIBLE_INPUTS (a JSON object keyed by task name), plus CRUCIBLE_TASK.
• agent under --manifest: the turn writes a single JSON object to PLAN_TASK_RESULT.json in the workspace root. A missing file after a normal turn is a measured failure; an explicit spawn, harness, or stream error is a transport failure and follows the retry policy.
• agent under --agent-cmd: the stand-in receives CRUCIBLE_PROMPT, CRUCIBLE_HARNESS, CRUCIBLE_MODEL, CRUCIBLE_EFFORT, and returns JSON on its last stdout line.
• evaluate: requires a JSON object. pass = false fails and malformed pass fails closed. Paired threshold and direction compare numeric score (lower is <=; higher is >=). Without a threshold, a successful command passes unless pass is false.

top_k reads a finite numeric score from each input, so an upstream task that wants to rank must emit one. That contract is declarable: emits = ["score"] on an agent, command, or evaluate task names fields its JSON output promises to include. Validation rejects a top_k dependency, grade score source, or thresholded evaluate whose declared emits omits score, before anything runs; at runtime a passing attempt missing a declared field is converted to a measured failure at the producing task (never retried, blocks dependents), so output drift fails where it happened instead of downstream. An empty or absent emits declares nothing and is never checked. top_k and engine tasks cannot declare emits; their outputs are engine-defined.

Execution semantics

How the executor walks a graph, what one task goes through and how the plan as a whole ends, is its own generated diagram: Plan execution states.

Readiness. A task dispatches when its dependencies are terminal and its join is satisfied. Dispatch order is declaration-stable, so the event stream is deterministic.

Truncation is fail-closed. If a required task can never run on this substrate (its needs is not in the declared caps, or a dependency cannot run), the whole plan is truncated and nothing is dispatched. A truncated DAG cannot produce an honest pass. An advisory task in the same position is skipped, along with its dependents, and validity is unaffected.

Failure. A required task that fails short-circuits the plan; everything undispatched is blocked. An advisory failure blocks only its dependents.

Retry is not recheck. Transport failures retry, bounded (2 by default). A measured failure never reruns: a task that failed, failed.

Budget. Cost is known only after an attempt completes, so an in-flight attempt may report a total above budget.usd. Any overrun invalidates the plan and blocks all further dispatch and retries. Reaching the budget exactly is valid only when no further retry or task is needed.

needs

The substrate capability a task requires. "any" runs everywhere. Anything else must be declared with --cap, otherwise the task is unrunnable, and plan show reports the truncation verdict before you spend anything.

isolation

isolation = "worktree" gives the task a private clone of the workspace, including its uncommitted state. Two effects:

• Tasks isolated this way and ready at the same time run concurrently. Without isolation they would collide on the single PLAN_TASK_RESULT.json in the shared workspace.
• The task's edits are discarded. What leaves is its declared output, so this is for review and analysis work, not for a task whose diff has to survive.

A runner that cannot isolate refuses the task rather than silently running it in the shared workspace.

join

join = "all" (default) requires every dependency to pass. join = "passed" waits for every dependency, then folds the non-empty passing set. It fails closed if none can run or pass.

The loop as a plan

Each loop iteration runs as a capability-admitted autoresearch workflow. With no authored workflow, the default expands to ordinary tasks:

flowchart LR
    propose["propose (engine)"] --> apply["apply"] --> measure["measure"] --> decide["decide"]

Task names and intervening topology are author-defined. An autoresearch result must be a decision fed by a frozen measurement with apply and proposal ancestors. A custom workflow has no such shape requirement; the outer orchestrator only admits it when it advertises workflow.custom.

[workflow]
type = "autoresearch"
result = "keep-if-better"

[[workflow.task]]
name = "invent"
kind = "engine"
op = "propose"
session = "solver"

[[workflow.task]]
name = "review"
kind = "command"
command = "./review.sh"
depends_on = ["invent"]

[[workflow.task]]
name = "deploy-preview"
kind = "engine"
op = "apply"
depends_on = ["review"]

[[workflow.task]]
name = "benchmark"
kind = "engine"
op = "measure"
depends_on = ["deploy-preview"]

[[workflow.task]]
name = "keep-if-better"
kind = "engine"
op = "decide"
source = "benchmark"
depends_on = ["benchmark"]

The corresponding admission needs workflow.autoresearch, engine.propose, engine.apply, engine.measure, engine.decide, and—because it binds solver—agent.session.persist. A custom orchestrator can instead admit type = "custom" and any subset of operations it implements. Task-level needs still controls where an admitted task can run; workflow capabilities control what the orchestrator is authorized to mean.

Same decisions and same session events as the default path, plus additive plan_admitted and task_result lines. Cross-round state, keep/discard, and every between-round control (parking, steering, re-scoping, budget) stay with the driver. The wide round runs as a template compiled from [search] on both paths.

The templates carry no budget of their own: the run budget is the driver's, checked between rounds, so a turn that overruns the cap is still measured and decided.

Authored measurement subgraphs

The compatible measure() path remains available. For visible measurement, use evaluate() and grade(): dependencies define rungs, isolated peers can run concurrently, and grade() selects the score source for decide().

candidate = propose(name = "invent")
live = apply(name = "deploy", depends_on = [candidate])

refcheck = evaluate(
    name = "cpu-reference",
    run = "./refcheck.sh",
    depends_on = [live],
)
diff = evaluate(
    name = "single-gpu-diff",
    run = "./diff.sh",
    depends_on = [refcheck],
    threshold = 0.001,
    direction = "lower",
    needs = "gpu",
    isolated = True,
)
latency = evaluate(
    name = "latency",
    run = "./latency.sh",
    depends_on = [diff],
    direction = "lower",
    threshold = 12.5,
    isolated = True,
)
racecheck = evaluate(
    name = "racecheck",
    run = "./racecheck.sh",
    depends_on = [diff],
    required = False,
    isolated = True,
)
measurement = grade(
    name = "final-grade",
    evidence = [diff, latency, racecheck],
    score = latency,
)
decision = decide(name = "choose", measurement = measurement)
workflow(
    type = "autoresearch",
    tasks = [candidate, live, refcheck, diff, latency, racecheck, measurement, decision],
    result = decision,
)

The renderer groups evaluate and grade as Measurement while preserving edges. During crucible scope, validation writes the admitted graph to WORKFLOW.png; agents cannot replace it.

Default off while it soaks.

Run-scoped epilogue tasks

stage = "epilogue" on a [[workflow.task]] (Starlark: stage = "epilogue" on agent(), command(), or evaluate()) removes the task from the per-iteration graph. The epilogue subgraph runs once, after the loop concludes cleanly (finished, budget, or solved), against the final kept candidate, and only if the run kept something. This is where a 90-minute compute-sanitizer racecheck or a slow perf benchmark belongs.

Each task's CRUCIBLE_INPUTS carries the kept candidate under the reserved kept key: {"iter", "score", "tiebreak", "sha", "snapshot", "note"}. Dependencies may not cross stages, engine ops cannot be epilogue, and the workflow result must iterate.

Epilogue results are advisory: they cannot un-keep the candidate. Rows land in the session log and RESULTS.md (epilogue / epilogue-skip / epilogue-fail), and the PR body gets an "Epilogue checks (advisory)" section with failures marked FAILED.

report(name = "publish-report", destination = {"kind": "slack"}, template = "reports/slack.md.j2", result = roundup, required = True) is the engine-owned publication epilogue. Destinations are engine-known keys, not URLs or secret names; slack is the only destination in the first version. The optional result selector projects only that main-graph task's declared JSON fields into an engine-built Block Kit card. It does not expose prompts, stdout, workspaces, undeclared fields, raw Slack blocks, channels, or credentials. Selected output defaults to a 16 KiB encoded limit; an operator may lower or raise it up to 64 KiB with CRUCIBLE_REPORT_RESULT_MAX_BYTES. Oversize data fails without truncation. A required report makes rendering or delivery failure fail the workflow; it does not rely on an agent remembering to call a tool.

Worked example

examples/adversarial-review puts a review task between a code node and the gate below it, in single-reviewer and two-reviewer panel shapes. The panel runs isolated reviewers concurrently and joins them on a policy gate: correctness blocks, copy-edit is advisory. It runs free against a stand-in manifest or against real models with the live one.

Loop control states

Generated from crucible/src/runloop/machine.rs by crucible loop-states; scripts/state-docs.sh --check keeps it current. The driver advances through this table at every gate, so an edge missing here is a transition the loop cannot take.

Each Turn is one iteration's work graph (propose → apply → measure → decide), rendered in Work graphs. Everything else is the control shell around it: the gates at the Head, the parks, and how a run ends. Dashed states are idle; the colored edges are the ways out.

The source is docs/img/loop-states.dot (crucible loop-states --format dot).

How a run ends

The edge label after the arrow is the shutdown token on the session log.

An error inside the loop reports error and takes none of these edges.

Plan execution states

Generated from crucible/src/plan/machine.rs by crucible plan states; scripts/state-docs.sh --check keeps it current. The executor walks both tables at every decision, so an edge missing here is a path it cannot take. The graph itself, what the tasks are and how they depend on each other, is described in Work graphs; this page is how the executor walks one.

One task

A task is pending until its dependencies settle, runs (retrying transport-class failures up to the configured count), and settles on one of the six statuses the session log reports. Everything that reaches a settled state without an attempt is a blocked task, and the edge names why.

The plan

The plan dispatches ready tasks in topological order until every task has settled or a required task fails or a ceiling is reached. After a halt the remaining tasks settle as blocked; epilogue tasks still run after a required task fails so the failure is reported.

The sources are docs/img/plan-task-states.dot and docs/img/plan-states.dot (crucible plan states --format dot).

Why a task is blocked

The note on a blocked task's result is one of these.

How a plan ends

Workflow DSL reference

The dialect is Starlark with assignments, if, for, comprehensions, def, lambda, load(), at the top level or inside a def. Every one of them runs at compile time: the compiled plan is a static graph, so a loop in the source unrolls into tasks rather than becoming a cycle.

prompt_file() and load() are the only file access, both confined below the pack directory, and a loaded module cannot re-export what it loaded. The surface has no processes, network access, clock, or randomness.

For what the engine does with the compiled graph, see Work graphs; for the normative rules, see the implementation contract.

Every lane

Available in every workflow type, playbooks included.

agent()

An agent turn driven by a prompt.

skill()

An agent turn whose prompt is a skill's instructions plus its arguments.

command()

A deterministic shell task in the candidate workspace.

evaluate()

A measurement command. Its last non-empty stdout line is a JSON object; pass = false vetoes the result and numeric score feeds grade() and top_k().

report()

Publish a rendered template to a controller-configured destination. The workflow selects a destination key, never an endpoint or a credential.

session()

Declare a durable agent conversation. Tasks that share one run serially under one agent config, across dependency order and loop iterations.

param()

Read a launch parameter. The params block must be the source's first statement, and a source that declares one compiles per run.

Takes one positional argument, name.

prompt_file()

Embed a UTF-8 file below the pack directory. Absolute paths, .., symlinks, non-files, and oversized inputs are refused.

Takes one positional argument, path.

workflow()

The source's final expression: the lane, the tasks that ship, and the result. A task constructed but not listed is a compile error.

Takes one positional argument, tasks.

Scored lanes only

Available to type = "autoresearch" and type = "custom". A playbook does not have these in scope at all, so naming one is an unknown-name error and a did-you-mean never offers one.

propose()

The loop's candidate-producing agent turn.

apply()

Make the candidate live through the configured world. A failure means unscoreable, not worse.

measure()

Run the manifest's frozen judge as one opaque measurement task.

grade()

Fold evaluation evidence into a measurement. Evidence includes tasks that failed or never ran, which is what the score source alone cannot see.

decide()

Apply the engine's keep-or-discard rule to a measurement. An autoresearch workflow must end here.

top_k()

Engine-owned reducer: the best k dependency outputs by numeric score.

default_autoresearch()

Expand the built-in propose/apply/measure/decide loop into visible nodes, plus the tasks passed to it.

Takes one positional argument, extra_tasks.

Reserved fields

Names the engine reads and writes for itself. They are not constructor arguments; they appear in a task's own JSON output and in the inputs it receives.

A task's own JSON output

Read out of the object the task returns.

Inputs the engine writes

Present alongside the dependency entries, never wrapped in one.

Hand-rolled codegen pipelines

This document describes how to run Crucible's brokered code-build and GPU-measurement tools without the controller or optimization loop. Use this path to validate a new domain's tool contract, test a GPU substrate, or diagnose build and measurement failures.

The normal deployment path remains a manifest-driven loop rendered by crucible deploy.

Current architecture

The broker owns registry credentials and Kubernetes access. Its caller supplies source edits and a restricted set of declared arguments; it does not supply build credentials, arbitrary job specifications, or measurement commands.

flowchart LR
    caller["MCP client or agent"] -->|"codegen_build / measure calls"| broker["crucible-broker"]
    source["Git candidate checkout"] -->|"exact tree"| broker
    broker -->|"buildah build and push"| registry["OCI registry"]
    registry -->|"digest-pinned candidate"| broker
    broker -->|"suspended Job with Kueue label"| cluster["GPU Kubernetes cluster"]
    cluster -->|"result JSON, logs, or trace"| broker
    broker -->|"typed MCP result"| caller

The data path has four invariants:

1. codegen_build hashes the candidate tree and builds that exact tree.
2. The registry result is resolved to an immutable digest.
3. Measurement tools accept only digests produced by codegen_build during the current broker lifetime.
4. Benchmark, evaluation, and profile commands are loaded from trusted configuration rather than accepted from the caller.

GPU measurement jobs run the candidate image directly. Do not mount the source workspace over the candidate's installed tree: that would make the measurement depend on mutable files instead of the recorded image digest.

Prerequisites

The machine or pod running crucible-broker needs:

• the crucible-broker binary;
• buildah, git, and tar on PATH;
• a writable build and log directory;
• containers-auth JSON credentials for any private base and candidate registries;
• network access to the OCI registry;
• Kubernetes credentials for the measurement cluster.

The Kubernetes target needs:

• Kueue and a configured LocalQueue;
• nodes that provide the requested nvidia.com/gpu resource;
• permission for the broker identity to create, inspect, and delete Jobs, read Pods and pod logs, and inspect Kueue Workloads;
• an image-pull Secret in the measurement namespace when the candidate registry is private;
• any model-cache or trace-transport PVCs referenced by the broker configuration.

Build the standalone broker from this repository:

cargo build --release -p crucible-broker

For a remote or delegated cluster, validate submission and Kueue admission with the CPU-only sentinel before using a GPU:

target/release/crucible-broker spoke-smoke gpu-east \
  --kubeconfig /path/to/kubeconfig \
  --context gpu-east \
  --namespace crucible-measure \
  --queue crucible-measure

Use --image when the cluster cannot pull the default public sentinel image.

Configuration model

Rendered deployments separate domain configuration from cluster configuration:

For a hand-rolled deployment, provide the equivalent environment variables directly.

Frozen tool contract

BROKER_CODEGEN_TOOLS_DEFAULTS is a JSON object. The following example is validated by the broker's own configuration test:

BROKER_CODEGEN_TOOLS_DEFAULTS = {
  "gpus": 2,
  "build": {
    "base_image": "registry.example.com/project/base@sha256:0123456789abcdef",
    "src_dir": "/workspace/project",
    "install_cmd": "python -m pip install -e . --no-deps",
    "full_install_cmd": "MAX_JOBS=4 python -m pip install -e . --no-deps",
    "copy_chown": "1000:1000",
    "mutable_kwargs": {
      "mode": ["derive", "full"]
    }
  },
  "benchmark": {
    "command": "python /opt/crucible/benchmark.py --out \"$OUT\"",
    "output_len": 1024,
    "num_prompts": 4,
    "objective": {
      "key": "tpot_ms",
      "direction": "lower"
    },
    "mutable_kwargs": {
      "toggles": {
        "PROJECT_FEATURE": ["0", "1"]
      },
      "reps": {
        "min": 1,
        "max": 3,
        "default": 1
      }
    }
  },
  "lm_eval": {
    "command": "python /opt/crucible/evaluate.py --out \"$OUT\"",
    "objective": {
      "key": "score",
      "direction": "higher"
    },
    "mutable_kwargs": {
      "limit": {
        "min": 32,
        "max": 500,
        "default": 500
      }
    }
  },
  "profile": {
    "command": "python /opt/crucible/profile.py --out \"$OUT\"",
    "trace_ext": "json.gz"
  }
}

Required fields are gpus, build.base_image, build.src_dir, build.install_cmd, benchmark.command, and lm_eval.command. The profile section is optional. Omitting it makes codegen_profile return unconfigured.

Build modes have distinct purposes:

Only modes listed in build.mutable_kwargs.mode are accepted. derive is the only default mode when no list is declared. Every other optional caller argument is also rejected unless its domain is declared under mutable_kwargs.

BROKER_CODEGEN_TOOLS_OVERLAY, when set, uses the same JSON shape. Its fields override the defaults and unspecified fields continue to come from BROKER_CODEGEN_TOOLS_DEFAULTS.

Standalone substrate environment

Set these values before starting the broker:

Optional Job sizing variables are BROKER_CODEGEN_CPU (default 16), BROKER_CODEGEN_MEM_REQUEST (128Gi), BROKER_CODEGEN_MEM_LIMIT (200Gi), BROKER_CODEGEN_SHM_GI (16), BROKER_CODEGEN_DEADLINE_SECONDS (5400), and BROKER_CODEGEN_TTL_SECONDS (86400). Queue wait time is allowed in addition to the Job's active deadline.

Optional volume variables are:

• BROKER_CODEGEN_MODEL_PVC and BROKER_CODEGEN_MODEL_MOUNT for a read-only model cache;
• BROKER_CODEGEN_ARTIFACTS_PVC, BROKER_CODEGEN_ARTIFACTS_MOUNT, and BROKER_CODEGEN_ARTIFACTS_DIR for profile trace transport;
• BROKER_CODEGEN_WORKSPACE_PVC and BROKER_CODEGEN_WORKSPACE_MOUNT for legacy workloads that explicitly require a workspace mount.

Do not set BROKER_CODEGEN_WORKSPACE_PVC for normal measurements. The rendered deployment intentionally omits it so a workspace volume cannot shadow files baked into the candidate image.

To delegate Jobs to a separate cluster, set BROKER_CODEGEN_KUBECONFIG. Optional companion variables are BROKER_CODEGEN_KUBE_CONTEXT, BROKER_CODEGEN_PROXY_URL, BROKER_CODEGEN_CLUSTER, and BROKER_CODEGEN_CLUSTER_TIER. Without a delegated kubeconfig, the broker uses its ambient in-cluster client or the standard local kubeconfig resolution.

Example startup

Store the JSON object from the tool-contract example in tools.json, then start the broker in the foreground:

export BROKER_CODEGEN=1
export BROKER_CODEGEN_TOOLS_DEFAULTS="$(jq -c . tools.json)"
export BROKER_CODEGEN_SANDBOX_WORKDIR=/path/to/project
export BROKER_CODEGEN_NAMESPACE=crucible-measure
export BROKER_CODEGEN_QUEUE=crucible-measure
export BROKER_CODEGEN_MAX_GPUS=2
export FORGE_REGISTRY=registry.example.com/project/candidates
export FORGE_AUTHFILE=/path/to/containers-auth.json
export REGISTRY_AUTH_FILE=/path/to/containers-auth.json
export FORGE_STORAGE_ROOT=/path/to/writable/forge-state
export KUBECONFIG=/path/to/kubeconfig
export BROKER_CODEGEN_PULL_SECRET=registry-pull
export BROKER_BIND=127.0.0.1:8849
export BROKER_TOKEN="replace-with-a-random-token"
export STORAGE_DRIVER=vfs

target/release/crucible-broker

Run the broker as the foreground process in a container or pod. A background broker tied to a short-lived shell or sleep process will disappear while calls are still in flight.

Connect an MCP client

The endpoint is streamable HTTP at /mcp. A Claude-compatible client configuration for a broker running on the same host is:

{
  "mcpServers": {
    "crucible": {
      "type": "http",
      "url": "http://127.0.0.1:8849/mcp",
      "headers": {
        "Authorization": "Bearer <random-bearer-token>"
      }
    }
  }
}

For a sandboxed caller, use the address reachable from that sandbox. Crucible-generated configurations use host.containers.internal for the Podman driver and host.openshell.internal for the Kubernetes driver. A custom hostname must also be added to BROKER_ALLOWED_HOSTS on the broker.

The MCP server name controls the agent-visible prefix. The example above exposes tools as mcp__crucible__codegen_build, mcp__crucible__codegen_benchmark, and so on.

Tool reference

The normal call sequence is:

1. Call codegen_build and retain the returned digest.
2. Pass that digest to codegen_benchmark and codegen_lm_eval.
3. Call codegen_profile when the contract includes a profile command.
4. While a GPU call is blocked, use a concurrent MCP request to call codegen_jobs, then tail its log handle with fetch_log.
5. Read profile artifacts with fetch_trace, base64-decode each window, and concatenate the decoded bytes in offset order.

Benchmark, lm-eval, and profile calls block until their Jobs finish. codegen_jobs keeps an in-memory ring of at most 20 Jobs and reports newest first. Cluster-derived lifecycle data is best-effort; an unavailable live lookup produces unknown rather than a false queued or running state.

Frozen command output

The broker runs configured measurement commands through /bin/sh -c inside the candidate image. It sets OUT to the result path and HF_HUB_OFFLINE=1. Each command must create the file named by $OUT and exit successfully.

Benchmark

The benchmark output must be a JSON object with at least one top-level numeric or boolean field. All such fields are returned as metrics. When the object contains elapsed_time, the broker also derives tpot_ms and tokens_per_s using the configured output_len and num_prompts; a JSON num_requests field overrides the configured prompt count.

Declared benchmark toggles are passed as environment variables. Declared reps is passed as CRUCIBLE_BENCH_REPS.

Language-model evaluation

The lm-eval output must contain a numeric value under the configured objective key, either at the top level or one object level below it. A declared limit is passed as CRUCIBLE_LM_EVAL_LIMIT.

Profile

The profile command writes its binary artifact to $OUT. With an artifacts PVC, the Job writes directly to that volume and the broker imports and removes the transport file. Without the PVC, the broker transfers the trace through Job logs and rejects artifacts larger than 64 KiB rather than returning truncated data.

Use fetch_trace, not fetch_log, for the returned trace handle.

Provenance, caching, and budgets

For a Git checkout, codegen_build creates a temporary index, stages all non-ignored changes, and hashes the resulting tree. The build context is exported from that exact tree, so files excluded by Git do not enter the image under the recorded hash. The response identifies whether the source was a live sandbox, local checkout, or legacy non-Git path.

The broker memoizes:

• builds by tree hash, build mode, and complete build configuration;
• benchmarks by digest, toggles, and repetition count;
• lm-eval runs by digest and limit;
• profiles by digest.

The memo and the set of broker-built digests are in memory. Restarting the broker preserves log files under FORGE_STORAGE_ROOT but forgets cache entries and digest provenance; call codegen_build again before measuring.

BROKER_CODEGEN_MAX_CALLS_PER_TURN and BROKER_CODEGEN_MAX_GPU_MINUTES_PER_TURN limit GPU work when Crucible supplies a turn token. Direct standalone calls have no turn token and therefore do not use the per-turn budget.

Result statuses

Codegen tools return a tagged JSON object. Handle these statuses explicitly:

A delegated-cluster reachability error also includes the configured spoke name and tier.

Troubleshooting

The workflow DSL and generic work-graph runner are documented separately in Work graphs. The current broker surface still exposes the fixed codegen_benchmark, codegen_lm_eval, and codegen_profile tools; the named-job registry in ADR-0022 is only partially implemented.

The codex harness

crucible --harness codex (or [agent].harness = "codex") runs the turn with OpenAI's Codex CLI instead of Claude Code. Everything downstream of the decoder is unchanged: the turn still emits AgentEvent NDJSON, and keep/discard still reads the same Result.

[agent]
harness = "codex"

[agent.codex]
# The shared `[agent].model` names a Claude model, so a codex domain overrides it here.
model = "gpt-5.6-sol"
# auto (default), api, or chatgpt
auth = "api"
# Which of the deployment's OpenAI keys to use; unset = the unnamed default key.
api_key = "WORK"

What differs from a claude turn:

• No session resume. codex exec resume is not wired; a logical session's second turn errors rather than silently starting fresh.
• No OTEL. codex exec exports no metrics, so cost is the pricing-table estimate over the token usage the live --json stream reports, not an otel_summary.
• Egress. The codex arm adds chatgpt.com, auth.openai.com, api.openai.com, and ab.chatgpt.com to the sandbox allowlist. Those hosts are per-harness: a claude turn's allowlist is byte-identical to what it was before codex existed.

Auth selection

Crucible supports both Codex login methods. [agent.codex].auth controls selection:

• auto (default) uses the selected non-empty API key and otherwise falls back to ChatGPT OAuth.
• api requires the selected API key and never silently falls back.
• chatgpt uses the OAuth flow even when API keys are present.

[agent.codex].api_key names one of the deployment's OpenAI keys (uppercase letters, digits, and underscore); unset selects the unnamed default. The manifest names a key, never the variable carrying it: the host resolves api_key = "WORK" to OPENAI_API_KEY_WORK and an unset api_key to OPENAI_API_KEY, so a manifest cannot reach any other host credential, and an openshell provider that can deliver the key later replaces the resolution without a manifest change.

A Kubernetes deploy profile injects independently rotatable keys under those names:

[[secret_env]]
name = "OPENAI_API_KEY_WORK"
secret = "crucible-openai-work"
key = "OPENAI_API_KEY"

[[secret_env]]
name = "OPENAI_API_KEY_PERSONAL"
secret = "crucible-openai-personal"
key = "OPENAI_API_KEY"

Switch api_key in the manifest (or select a deployment profile carrying that manifest) to choose a key for newly created turns. Use dedicated project-scoped keys and rotate their Kubernetes Secrets independently. The selected key is not exported into the sandbox environment, but Codex can read it from its seeded auth file.

ChatGPT OAuth: CODEX_CREDENTIALS and the single-refresher rule

Codex authenticates against the ChatGPT backend with a personal subscription, not Vertex. The credential is an OAuth pair produced by codex login on a host with a browser, stored at ~/.codex/auth.json.

Setup:

1. codex login on your machine, then confirm ~/.codex/auth.json exists.

2. Ship its verbatim contents to the loop process as the CODEX_CREDENTIALS env var. In cluster that is a secretKeyRef, exactly how GCLOUD_CREDENTIALS is delivered:

   oc create secret generic codex-auth \
     --from-file=auth.json="$HOME/.codex/auth.json"
   

3. Reference it from the deploy profile's [[secret_env]].

At the top of every turn the loop process performs the OAuth refresh_token grant against https://auth.openai.com/oauth/token and seeds the result into the sandbox as $CODEX_HOME/auth.json: access token, account id, id token, and a placeholder refresh token. Provider-delivered env cannot carry it, because the sandbox sees only an openshell:resolve:env: placeholder that the L7 egress proxy would have to resolve, and codex reaches the backend over a WebSocket through an L4 tunnel. All four tokens fields have to be present, or codex drops the object and runs unauthenticated into a 401 loop.

The single-refresher rule: the refresh token never leaves the loop process. Exactly one thing performs the grant, so there is no rotation race between the host's copy and a sandbox's copy of auth.json (OpenAI rotates the refresh token on each grant, and a stale copy is dead). The consequence is that a sandbox holds a fixed short-lived access token: the seeded auth.json is read at exec, so a turn that outlives the access token fails loudly rather than silently reauthenticating. That is accepted for now.

Each grant's rotated refresh token is persisted to $HOME/.config/crucible/codex-credentials.json, and that file takes precedence over CODEX_CREDENTIALS on the next mint. The env secret is seed material for the first mint only: the first rotation spends it, so a fresh process with a fresh $HOME needs a freshly minted secret (re-run codex login and replace it as part of any pod restart). Two loop processes must never share one credential; each needs its own codex login.

Independently, an unused refresh token goes stale after roughly a week. When mints fail the grant with refresh_token_expired, re-run codex login on the host, replace the secret, and delete the state file if the process persists a home directory.

In auto mode this OAuth machinery remains the fallback when the selected API key is absent or empty; chatgpt selects it explicitly.

The opencode and pi harnesses

crucible --harness opencode or --harness pi (or [agent].harness = "opencode" | "pi") runs the turn with OpenCode or Pi instead of Claude Code. Both exist for one reason: an inference endpoint that speaks only OpenAI Chat Completions. Claude Code speaks the Anthropic Messages API, and Codex dropped its chat wire API in early 2026 and speaks only the Responses API, so neither can drive such an endpoint; these two can. Everything downstream of the decoder is unchanged: the turn still emits AgentEvent NDJSON, and keep/discard still reads the same Result.

[agent]
harness = "pi"
model = "qwen-3-8-27b"

Auth and the endpoint

Both harnesses authenticate with a direct API key from the loop's environment, relayed into the sandbox. Which API the turn speaks follows the environment:

The controller sets these from the provider a launch pinned: a custom provider with the chat_completions protocol lands as OPENAI_BASE_URL + OPENAI_API_KEY. A turn with neither key is refused before the sandbox starts.

The seeded config registers the endpoint as a provider named crucible and the model under it, so the CLI never consults its own model catalog: opencode gets an opencode.json with the crucible provider on @ai-sdk/openai-compatible (or @ai-sdk/anthropic) reading the key back through {env:OPENAI_API_KEY}, pi gets a models.json with the crucible provider on openai-completions (or openai-responses, anthropic-messages) reading $OPENAI_API_KEY.

What differs from a claude turn

• No session resume on either harness; a logical session's second turn errors rather than silently starting fresh.
• No OTEL. Cost is the endpoint's own number when the CLI priced the model (it never does for the crucible provider) and otherwise the pricing-table estimate over the token usage the stream reports. OpenCode also spends one extra model request per turn generating the session's title, which its export counts in the turn's usage.
• Egress. Both add api.openai.com to the sandbox allowlist; a custom base URL's host is added per turn as it is for codex. A claude turn's allowlist is unchanged.
• Pi has no MCP client. The provisioning broker is unreachable from a pi turn, so a pack whose agent needs the broker's tools cannot run on pi. OpenCode takes the broker as a remote MCP server in its config, like codex.
• Skills. OpenCode discovers Claude Code's .claude/skills, so the toolbox lands there. Pi discovers .agents/skills, and the turn passes --approve so the workspace's project-local files are trusted without a prompt.
• Reasoning effort. Pi takes the shared [agent].reasoning_effort as --thinking (max maps to xhigh). OpenCode's equivalent (--variant) is provider-specific, so it is not passed.

OpenCode's transcript is the session export

opencode run --format json mirrors its server's event stream to stdout and exits on the session's idle signal. In a container that signal can overtake the last text/step_finish events, so the stream is treated as display only. The sandbox invocation is a bash wrapper that runs opencode run, then opencode export <session> into /sandbox/.local/share/opencode/crucible-export.json; that export is the turn's transcript and its only source of the result, token usage, and tool spans (backfill_required, the hermes posture). A missing export is a loud transcript error, never a $0 success.

Pi's --mode json stream comes from its own process and is complete, so pi closes the turn from the live stream and its session file under $PI_CODING_AGENT_DIR/sessions is trace garnish, like claude's and codex's.

Sandbox images

The sandbox needs the process that opens the socket on the egress binary allowlist, which OpenShell matches by the kernel-resolved binary: /usr/local/bin/opencode must resolve to the native opencode binary (npm's launcher never opens the socket itself), and pi is a node script, so its allowlist carries /usr/bin/node and /usr/local/bin/node beside /usr/local/bin/pi. The opencode and pi features of the controller's image feedstock install exactly that.

The OpenShell fork

Crucible pins openshell-core to a fork — wseaton/OpenShell, branch crucible/grpc-base — instead of upstream nvidia/OpenShell. This page is the running ledger of why, so nobody has to re-derive it from git archaeology.

Why a fork exists at all

The branch policy is trivially rebasable: crucible/grpc-base is upstream main plus a small stack of crucible-needed commits, rebased forward rather than diverging. The module docs in crucible/src/openshell/mod.rs state the consequence: because the fork is trivially rebasable, version-locking crucible to openshell-core "is no longer the risk it was when this shelled the CLI" — the control-plane boundary is the gateway's native openshell.v1 gRPC API, and the pinned rev is the exact rev the shipped gateway binary is built from.

Two distinct gaps keep the fork alive:

1. Fork-only commits — features upstream doesn't have yet (the ledger below).
2. Upstream-main commits ahead of the last release — the pin rides upstream main, not the release tag, because crucible needs post-release work (see "Why the pin is ahead of v0.0.81").

Pin mechanics (where the rev lives)

One rev, four consumers, all derived from Cargo.lock:

The divergence ledger

Fork-only commits on crucible/grpc-base, relative to upstream main (merge-base 40194f9, checked 2026-07-13):

That's the whole fork today: one commit. Earlier gRPC-boundary work that lived on this branch has already landed upstream (which is why the branch is named crucible/grpc-base but no gRPC commits remain fork-only).

Why the pin is ahead of v0.0.81

The pin is 11 commits ahead of the last upstream release tag (v0.0.81); 10 of those are upstream main commits, not fork work. The ones crucible actually depends on:

• 8eacb47 feat(kubernetes): add sidecar supervisor topology (#2076) — the supervisor runs as a sidecar container in the sandbox pod, which is where crucible's client-cert trust model lives (see below).
• 1070213 fix(core): pin supervisor image tag to gateway version for all drivers (#2070) — keeps the sha-<rev> gateway/supervisor image pair coherent.
• 614c8c1 feat(kubernetes): support PVC subPath driver config (#2034) — workspace layout under the k8s driver.
• 40194f9 fix(network): fail closed when credential placeholders cannot be rewritten (#2162) — credential-safety fix on the egress path.

This is why MIN_GATEWAY_VERSION in grpc.rs is 0.0.82: the RPC surface crucible calls does not exist in any released gateway yet.

Pending upstream contributions

Candidates to shrink the gap to zero:

• The AWS web-identity provider (bc8342bb, the only fork commit). Upstreaming it makes the fork pure upstream main @ <rev> and the "fork" becomes just a pin.
• Upload/download RPCs. File transfer still goes through the openshell CLI (SSH-tar over the gateway's CreateSshSession relay) because no RPC covers it — one of the two CLI remnants named in crucible/src/openshell/mod.rs. A native upload/download RPC would let crucible drop the CLI from the turn path entirely.
• mTLS user auth under the kubernetes driver. The gateway hard-rejects --enable-mtls-auth with the kubernetes compute driver (openshell-server/src/cli.rs: "mTLS user authentication is not supported with the Kubernetes compute driver"), so an earlier fork PR had to render allow_unauthenticated_users = true for k8s-driver gateways (crucible/src/openshell/gateway.rs). Trust model while the escape hatch exists: transport mTLS with a per-pod CA still gates every connection (require_client_auth), and the client cert lives only in crucible's turn pod and the supervisor sidecar container — never the agent container — so possession of the cert is the authorization. Acceptance criterion for the upstream change: crucible deletes the allow_unauthenticated_users escape hatch.

Maintenance rule

Every pin bump and every new fork commit adds or updates a ledger row above, in the same PR that moves Cargo.lock. A rev in the lockfile that this page cannot explain is a bug.

How to bump the pin

1. Rebase the fork branch: rebase crucible/grpc-base onto upstream main (it must stay trivially rebasable — if a commit stops rebasing cleanly, that is the signal to upstream it or drop it), push to wseaton/OpenShell.
2. Update the dependency: bump rev in crucible/Cargo.toml and let Cargo.lock re-resolve.
3. Images build themselves: openshell-gateway.yml triggers on Cargo.lock changes and builds openshell-gateway:sha-<rev> + openshell-supervisor:sha-<rev>. Note the first-run ordering: docker.yml fails fast until the gateway image exists, then retries clean.
4. Version stamp needs upstream tags: the gateway stamps its version via git describe --tags --long, and the fork's own tags are stale — the workflow fetches refs/tags/v* from nvidia/OpenShell before building. Nothing to do manually, but if the stamp ever reads 0.0.0, this is where to look.
5. Update this ledger (see the rule above), and bump MIN_GATEWAY_VERSION in crucible/src/openshell/grpc.rs if the bump starts using RPCs older gateways lack.

JIRA tools (mediated)

The broker embeds a native JIRA Cloud client (the shared jira-mcp crate) and exposes a read+comment slice of it to the sandboxed agent over the broker's existing MCP wire. It's the ADR-0002 mediation pattern, JIRA edition: the agent gets JIRA tools without ever holding an Atlassian credential. There is no upstream MCP child process; the broker calls JIRA's REST API directly, server-side.

What the agent sees

Three tools on the broker wire (prefixed by the broker name, e.g. mcp__epp-broker__jira_search):

jira_add_comment is the only write. The agent has no tool for create/transition/edit/delete, and the client itself is pinned to the read+comment ceiling at construction (crucible-broker/src/jira.rs), so scope is enforced by construction, never by trusting the agent. When JIRA isn't configured the tools return {"status":"disabled"} (same as build_epp/profile when their feature is off), so they never vanish.

Context hint. The deployment can name its projects of interest with BROKER_JIRA_PROJECTS (comma-separated, e.g. PROJ,PROJ2); the jira_search tool description then tells the agent to scope JQL to those boards instead of searching the whole instance. BROKER_DESC_JIRA_JQL_EXAMPLE and BROKER_DESC_JIRA_KEY_EXAMPLE flavor the inline examples the descriptions show. Unset, the descriptions stay fully generic.

Trust boundary

The Atlassian credentials live in the broker pod's env. They are never in [agent.env], so the sandbox never sees a token, exactly like the build/deploy creds. The agent reaches only the broker endpoint (already egress-allowlisted); the REST calls to Atlassian Cloud ride the loop pod's egress, not the sandbox's, so no sandbox egress change is needed.

Enabling it (broker-pod env)

All three set = the jira_* tools go live; anything missing = they answer disabled. The broker overrides any configured access level to read+comment regardless of what the env or the shared crate's config would allow.

RFC-0001: Crucible implementation contract

Version: 0.2.0 | Status: normative | Phase: spec Owners: @Will Eaton

1. Summary

2. Specification

[RFC-0001:C-MANIFEST] Manifest schema and validation (Normative)

The engine must read exactly one manifest per run. Default path is ./crucible.toml, overridable via --manifest.

The manifest directory (dirname of the manifest file) must anchor all config-relative paths.

Required fields: [repo], when present, must contain exactly one of url or path. [repo] may be omitted only by a manifest whose [workflow].type is playbook; every other manifest must carry it. When [judge] is present, [judge].measure_cmd and [judge].direction are required. When [judge] is absent, the run is a task lane.

[workspace].inject entries take two forms. A table names src (manifest-relative), dst (workspace-relative), and frozen (default true). A string is shorthand for a table whose dst equals its src and whose frozen is true. A string that names a directory or contains a glob metacharacter must expand to every regular file it matches under the manifest directory, in sorted path order, each becoming its own frozen entry with dst equal to its manifest-relative path. Expansion must not escape the manifest directory, and a string that matches no file must be rejected as a manifest error naming the string.

[world] with no commands must produce GitWorld. Any command given must produce CommandWorld, which layers domain commands on top of git memory.

[judge.selftest], if present, must require both good_cmd and bad_cmd; runs must be >= 1.

[search], if present with wide > 0, must require approaches.len() >= wide and policy_k in 1..=wide.

Unknown manifest keys must be rejected as errors.

Frozen loading: when the manifest lives inside the workspace it targets, the engine must parse it from the workspace's pristine base commit, not the current working tree. Before any base commit exists (the very first run), the engine must hard-warn and trust the working tree for that run only.

Since: v0.1.0

[RFC-0001:C-SELFTEST] Gate self-test protocol (Normative)

[judge.selftest] declares two controls the gate must tell apart before it is trusted. crucible check must run it pre-loop, never inside a loop iteration.

The protocol must:

1. Snapshot the pristine workspace.
2. Restore to pristine, stage good_cmd, measure runs times through the domain's Judge, restore to pristine again.
3. Same for bad_cmd.
4. Pass if and only if both controls' readings are all valid and good's mean score is strictly better than bad's per [judge].direction.

The workspace must be restored to pristine on every exit path (pass, fail, or error).

A manifest with no [judge.selftest] must not be an error; crucible check must warn instead.

Since: v0.1.0

[RFC-0001:C-SEARCH] Wide-round search protocol (Normative)

[search].wide > 0 (or --wide N on the CLI, which overrides the manifest) must fan out N independent PROPOSE turns in per-candidate git worktrees before the deep loop starts, one turn per approaches entry biased into its prompt.

Each candidate's diff must be applied (cherry-picked) into the shared main workspace and measured serially (measurement must never run concurrently; only proposal does).

The scored set must be ranked by [search].policy (v1: top-k); the policy_k winners must seed the deep loop.

Session rows from the wide round must carry an additive phase: "wide" field so a consumer can distinguish a wide-round row from a deep-loop row without a wire-shape change. The field must be skip_serializing_if None so a deep-only run's wire bytes are unchanged.

Since: v0.1.0

[RFC-0001:C-WORKFLOW] Scope-authored workflow DSL (Normative)

A scoped pack may include workflow.star beside crucible.toml. It is authoring syntax; scope must compile it to [[workflow.task]] manifest IR before validation and again before freeze. The generated TOML is the runtime authority.

The DSL must accept: assignments, scalar values, lists, dictionaries, list concatenation, conditional expressions, comprehensions, iteration, user-defined functions, and direct calls to the constructors its lane defines. Every lane has agent, command, evaluate, skill, session, prompt_file, param, and workflow. A skill task is an agent task whose prompt is assembled rather than written: the instructions come from a SKILL.md the pack ships under a directory the task names, and the arguments the invocation supplies are rendered after them. It is a naming and reuse construct only, and must not widen what a task may reach. The scored lanes add propose, apply, measure, grade, decide, top_k, and default_autoresearch. A constructor outside the declared lane must not be in scope: naming one must be an unknown-name error at its own location rather than a validation failure after the graph compiles, and a did-you-mean must never offer one. The lane must therefore be readable from the source before the source is evaluated. Starlark's own pure builtins may be reached; a builtin that reaches the filesystem, a process, the network, the clock, or a source of randomness must be absent. The DSL must not provide mutation of frozen values, filesystem access other than prompt_file() and load(), processes, network, time, or randomness.

An author-supplied source is untrusted input. Compilation must terminate, and no source may abort the process, exhaust the native stack, or panic: a source that exceeds a bound must fail compilation with a diagnostic the scope pipeline can hand back to its author, because that channel is what turns a bad pack into a recoverable round rather than a dead run. The obligation covers the compiler's own traversal of the source as well as the evaluation of it. Parsing, module resolution, argument marshalling, and value allocation must each be bounded in depth and in size, iteration and recursion must execute under a bound enforced during evaluation, and every bound must be checked before the work it bounds is done rather than after.

load() must resolve only within the pack directory and must refuse absolute paths, parent traversal, and any path reaching a file outside the pack through a link. The number of modules loaded and their total size must be bounded, and each bound must be counted as a module is admitted rather than once it returns. Loaded content must reach the generated TOML, which remains the runtime authority and the hashed artifact.

Topology is authorable; authority is not. A workflow must declare type = "autoresearch", "custom", or "playbook". An autoresearch workflow's result must be a decide task sourced from a frozen measure or authored grade, with apply and propose ancestors. A custom workflow has no autoresearch-shape requirement. A playbook workflow runs its graph once and carries no judge; RFC-0002 states the rules particular to it. Universal DAG, source-typing, and operation-capability rules apply to all three.

Tasks marked isolated = True must run concurrently in disposable worktrees; their workspace state must be discarded. Only a task's declared output continues through the graph: its JSON output, and any files it declared under emits_files.

session = "name" must preserve one logical agent conversation across dependency-ordered tasks and loop iterations. Tasks sharing a session must not be isolated and must be dependency-ordered.

A required task must not depend, through a path of "all"-join edges, on a task declared advisory. Such a graph asserts both that a failure is tolerable and that the work it gates must pass, and no execution of it can produce an honest verdict. Validation must reject it before dispatch, naming both tasks. A required task joining "passed" or "settled" is exempt: it declares that it runs on whatever settled.

A list of tasks must be accepted wherever a list of task names is, so an author never wraps one to pass it. A task constructed in workflow.star but omitted from workflow(tasks = ...) must be a compile error.

Compile errors must carry file:line:col and a did-you-mean suggestion for unknown functions, kwargs, variables, and session names. A diagnostic about a named argument must locate that argument rather than the whole call, and a suggestion must draw on the names the source itself binds as well as on the DSL's own.

Since: v0.1.0

[RFC-0001:C-PATHS] Path resolution (Normative)

method_prompt, goal_file, and toolbox_dir must resolve relative to the manifest directory.

The agent workspace must resolve to manifest_dir / [workspace].dir.

Runtime state (session.jsonl, admissions.jsonl, control.json) must resolve to --state-dir, default manifest_dir/state.

STEER.md must resolve to --steer, default manifest_dir/STEER.md.

ESCALATION.json must resolve to /ESCALATION.json: written by escalate, consumed by the engine post-turn.

The binary's own install location must never be used to resolve anything.

Since: v0.1.0

[RFC-0001:C-COMMANDS] Command protocol (Normative)

Every command (measure_cmd, apply_cmd, snapshot_cmd, restore_cmd, setup_cmd) must be a string executed via sh -c with cwd = the agent workspace, except setup_cmd which must run with cwd = manifest dir (the workspace does not exist yet).

PATH must be inherited so a command may be a bare installed tool or a workspace-relative script.

measure_cmd is required when [judge] is present. It must inject CRUCIBLE_BASELINE_SCORE, CRUCIBLE_BASELINE_TOTAL, and CRUCIBLE_BEST_SCORE into env when those values are available (absent on baseline measurement). The engine must read the last stdout line that starts with { and parse it as JSON with fields: valid (bool, required), score (number or null), tiebreak (number, optional), solved (bool, optional, default false), note (string, optional), detail (object, optional). A nonzero exit code must force the reading to valid:false regardless of stdout.

apply_cmd, if present, must run after the agent turn and before measure. Nonzero exit must treat the iteration as an invalid candidate (discard).

snapshot_cmd must emit one opaque token on its last stdout line. restore_cmd must receive the token in CRUCIBLE_TOKEN env. snapshot_cmd and restore_cmd must come as a pair.

setup_cmd default when omitted: with [repo] present the engine must git clone [repo] into [workspace].dir and git checkout [ref]; with [repo] absent the engine must create [workspace].dir empty. In every case the engine must then apply [workspace].inject, and the workspace must end up a git repo whose base commit holds the injected files.

Since: v0.1.0

[RFC-0001:C-BUILD-MODE] Build modes (Normative)

Between "the agent edited a file" and "measure_cmd read a score" sits a build step whose shape must be declared per component.

Four modes: no artifact (compile+run in place), no rebuild (config tuning), derive-layer (interpreted sources appended as OCI layer), image (full container build).

derive-layer must require that the base image and push target share a registry. derive-layer must not carry compiled sources.

A compile failure must be distinguishable from a bad score. In image mode, compile error must be returned to the agent as a free retry with no candidate spent.

A Containerfile on the measured path is part of the judge, not part of the solution. If the build recipe lives in the agent's workspace, it must be a frozen = true inject, re-copied before every scored measure.

crucible check should enforce these preconditions before a turn is spent.

Since: v0.1.0

[RFC-0001:C-DECIDE] Decide rule (Normative)

Given a Reading { valid, score, solved, note, detail }, the current best_score, and the manifest direction, the decide rule must be:

keep = valid AND score.is_some() AND (better(score, best_score, direction) OR (score == best_score AND tiebreak_better) OR solved)

better(s, b, lower) = s < b better(s, b, higher) = s > b

tiebreak_better applies only when the reading carries a tiebreak field: better(tiebreak, best_tiebreak, tiebreak_direction). A best with no recorded tiebreak must count as the worst value.

solved must imply keep. A candidate the measure command declares solved must be kept and must terminate the loop even if its score does not strictly beat best. solved must never rescue an invalid reading.

The first valid reading must set the baseline (best_score, and baseline_total = detail.total if present) and must always be kept.

The loop must terminate when a kept iteration is solved, or budget/iterations exhausted, or stop/escalate.

No domain Rust may decide anything. Complex win conditions must be computed inside the measure command using CRUCIBLE_BASELINE_TOTAL and emitting solved.

Since: v0.1.0

[RFC-0001:C-WORLD] World reversibility (Normative)

World::Snapshot = String, opaque to the engine. The engine must only round-trip it back to restore.

GitWorld (default, no [world] commands): snapshot() must stage+commit the workspace and return the commit SHA. restore(sha) must git reset --hard + git clean (excluding .claude/, RESULTS.md). The kept-commit chain is the memory.

CommandWorld (any [world] command given): must always own git memory as above and layer the domain commands. The snapshot token must be the composite "\t". Keep: commit (git half), then run snapshot_cmd and capture its token (domain half), then join. Discard: split, git reset --hard + clean, then run restore_cmd with CRUCIBLE_TOKEN=.

The engine must expose last_commit_sha() (the git half) for kept_shas/publish; the domain half must never be inspected.

The engine's loop body must call only world.snapshot() / world.restore(&snap). It must contain no git/vcs calls and no kubectl.

Since: v0.1.0

[RFC-0001:C-TRANSPORT] Agent transport backends (Normative)

The engine must render a prompt (method_prompt with {{GOAL}}/{{STATUS}}/{{STEER}} filled), hand it + the workspace to an agent that edits the workspace, and must never hand it the Judge.

Three backends, selected by [agent].backend:

local: must run the selected harness directly on the host in the workspace with [agent].env.

openshell: must run the selected harness in a sandboxed pod driven by the OpenShell driver. Two execution environments: the engine and its contract commands always run where crucible runs; only the agent turn is sandboxed. The OpenShell driver must upload the workspace into the sandbox and sync edits back.

command: must run [agent].agent_cmd via sh -c in the workspace as the proposal. This is a deterministic, free proposer (no LLM). It makes the minimal example a fast, deterministic e2e test.

Flipping local to openshell must be config, not code. Reversibility commands (snapshot/restore) must run engine-side regardless of backend.

Since: v0.1.0

[RFC-0001:C-EGRESS] Sandbox egress policy (Normative)

The sandbox must be deny-by-default. Two lists open it: endpoints (host:port:access) and binaries (only these may open a socket).

With inherit_defaults = true (the default), the lists must be appended and de-duplicated to the built-ins (public forges, PyPI, Vertex, Anthropic, agent CLIs). Appending must never remove a built-in.

With inherit_defaults = false, the resolved allowlist must be exactly what the manifest names, binaries included. This is the only way to subtract a default and is required for air-gapped/private-registry runs and contamination control.

The broker endpoint must be auto-appended by the engine when [agent.broker].enabled is true. The engine must first resolve the broker URL, then derive the egress host:port:full entry from that URL's authority. The broker endpoint must be appended regardless of inherit_defaults because it is engine plumbing, not a built-in the domain can subtract.

Since: v0.1.0

[RFC-0001:C-WIRE] Session wire format (Normative)

The NDJSON session log (state/session.jsonl) must keep its existing event kinds (start/phase/row/budget/summary/finished) and field names unchanged. The objective label must be written under the JSON key "gate" (now carrying a free-text label). This must not be renamed (keeps --resume, the remote viewer, and published S3 runs loading).

Additive event kinds:

• identity: the run's RunIdentity, emitted once at setup and again on --resume.
• shutdown: { outcome, reason }, emitted exactly once as the last line of every run (after finished/summary). outcome must be one of finished/solved/budget/complete/stopped/escalated/stalled/error. finished means the graph or the loop ran out of work; complete means a task declared there was no work left to do and dispatch stopped short of that, the unscored counterpart of solved. The outcome says how dispatch stopped, not whether the run succeeded. A dead stream with no shutdown line means the pod died mid-run.
• agent_session: { session, action, turn }, emitted before a persistent agent turn. Must not contain the provider cursor or native transcript content.
• approval_wait: { handle, trace_id, mode }, emitted when the loop reads a pending-provisioning marker. Every approval_wait must be closed by an approval_resolved except on stop-while-parked and process death.
• approval_resolved: { outcome, reason } with outcome one of granted/denied/timeout.
• plan_admitted: { plan_version, reason, budget_usd, tasks }, emitted once after the graph is admitted and before any dispatch. Each task carries its name, kind, dependencies, session, needs, required flag, join, and stage. This is the consumer's route to the graph, so no gate label has to encode it. A consumer computing a verdict must skip tasks whose stage is epilogue, which is why the field is on the wire rather than inferable.
• asks_emitted: { task, asks }, emitted once per task that emitted any, as that task settles. Each ask carries an emitter-supplied key, the workflow it names, and that workflow's parameter values. The key must be rejected on decode, not only on construction, so the wire is where the key rules are enforced rather than a way around them.
• recovery: { class, iter, detail }, emitted once per --resume.

RunIdentity is the comparability key: two runs' scores are comparable only if it matches. It must be a hash-of-hashes over: repo URL/path + pristine base commit SHA, frozen manifest text hash, inject content+destination hashes, measure_cmd, and direction.

Since: v0.1.0

[RFC-0001:C-LEDGER] Admission ledger (Normative)

Every external input into a run (steer, approve, deny, rescope, set-budget, pause, resume, stop, abort) must be recorded in state/admissions.jsonl before it takes effect.

Two event kinds: admitted { key, seq, ts, input, ...payload } and settled { key, outcome, ts, note } with outcome one of applied/superseded/rejected.

Per idempotency key: exactly one admitted, then at most one settled. The first terminal outcome wins.

A key with no settled line is an input the run still owes. --resume must re-arm exactly those and close out the ones a resume overrides.

admissions.jsonl is authoritative for what an operator asked for; the session log is authoritative for what the loop was waiting on. Where they disagree, the ledger wins.

Control-bridge commands must accept an optional id (string, non-empty, <= 256 bytes) on every mutating command. Redelivering the same id with the same payload must converge on the original admission. Same id with different payload must be refused. Omitting id must generate a key and every delivery is a fresh input.

A stop/abort whose record cannot be written must still stop the run. Every other command must fail closed if its admission cannot be recorded.

Since: v0.1.0

[RFC-0001:C-SURFACE] Domain author surface (Normative)

A domain author must write:

1. A crucible.toml manifest.
2. A measure command emitting { valid, score, solved? } (any language), when [judge] is present.
3. Optionally apply/snapshot/restore commands.
4. A method prompt + goal.
5. Agent credentials in [agent].env.

Everything else (loop, budget, keep/discard, all reporters + remote viewer, steer/stop/resume, session log, escalation, git memory) is the engine's, provided for free.

The litmus test (examples/counter/) must exercise items 1, 2, 4 with GitWorld and the command backend, requiring no Rust.

Since: v0.1.0

[RFC-0001:C-OUTPUTS] Declared output kinds (Normative)

A run's effects on systems outside its workspace divide into mediated writes, performed by the engine or a broker on the agent's request, and open-ended capabilities the agent exercises directly (RFC-0001:C-CAPABILITY-DISCLOSURE). This clause governs the mediated writes.

Every mediated write MUST belong to an output kind drawn from a closed, engine-defined vocabulary. The vocabulary is part of the engine's versioned contract: a kind name, once retired, MUST NOT be reused with a different meaning. A write requesting a kind outside the vocabulary MUST be refused. A vocabulary kind the pack does not declare MUST resolve to a documented engine default, and an engine default MUST carry a count and MUST NOT contain an open target: a pack that ships no [outputs] section gets the conservative posture, not the permissive one. The resolved bounds for a frozen pack MUST be computable without executing pack content, and crucible check MUST print them.

A pack MAY declare bounds for a kind in an [outputs] manifest section. A declaration, once present, MUST carry a per-run count, and, for a kind that addresses a target (a repository, a tracker item, a chat destination, an image registry, a deployment), the target itself. Bounds MUST load from the frozen manifest and MUST be enforced at the mediation point; the agent MUST NOT be able to alter a bound from inside the sandbox.

Where a kind addresses a target, the target MUST come from the resolved declaration, never from an agent-supplied value, unless the declaration explicitly marks the target open. An open-target declaration MUST name a scope narrower than the kind's whole address space; a declaration whose scope admits any target MUST be rejected at manifest validation. A scope MAY bind to a named workflow parameter, and when it does, the target MUST equal that parameter's value for the run; this is how a run fanned out per tracker item confines its writes to the item that parameterized it.

A write that would exceed a count or address a target outside its scope MUST be refused at the mediation point. Every refusal under this clause MUST fail the requesting call naming the violated bound, MUST be recorded on the session log, and MUST NOT by itself terminate the run.

Bounding governs where mediated writes land and how many, not what they say. Payloads remain agent-authored; the reader on the addressed target is the payload's review.

Since: v0.2.0

[RFC-0001:C-CAPABILITY-DISCLOSURE] Opaque capability disclosure (Normative)

Some channels cannot be typed as output kinds because they hand the run open-ended reach: a credential whose value enters the sandbox, egress beyond what the engine's built-ins grant (RFC-0001:C-EGRESS), a relay that materializes host-side secrets into sandbox files, a substitute broker binary, or a mounted credential authorizing writes to an external system such as a cluster namespace. These are capabilities, not outputs, and bounding them by effect is not possible; the contract is disclosure.

A frozen pack MUST disclose every such capability, and the resolved disclosure MUST be computable from the pack without executing pack content. For each capability the disclosure MUST state its reach: for egress, the host, port, and access class in C-EGRESS's terms; for a credential, whether its value enters agent context or remains broker-held, which external system it authorizes, and at what scope; for a relay or a broker substitution, what it draws from and what it exposes. The agent's own credentials, [agent].env included, are credentials under this clause and MUST appear in the disclosure. The built-in egress allowlist is standing disclosed reach: it need not be re-declared, but the resolved disclosure MUST include it, and with inherit_defaults = false every entry the manifest names is disclosed manifest reach, built-in lookalikes included.

The engine MUST derive what it provisions and what it discloses from the same resolved declaration, so the two cannot diverge, and crucible check MUST print the resolved disclosure.

A capability granted from outside the pack, such as a secret binding supplied at launch, is covered when a disclosed capability of the same kind has reach equal to or broader than the grant's. A grant that is not covered MUST be refused at run start, and the refusal MUST name the grant and the missing disclosure.

Sandbox-resident pack content, skills included, holds no reach of its own: what it can touch is the union of the resolved disclosure and the declared output kinds. Pack-authored commands that execute outside the sandbox (workflow command and evaluate tasks, world and judge hooks) hold their executor's reach instead, and the disclosure MUST state that the pack runs commands outside the sandbox whenever it does. Narrowing that executor's reach is outside this clause; disclosing that it exists is not.

Since: v0.2.0

Changelog

v0.2.0 (2026-08-22)

Admit the playbook lane

Changed

• DSL grammar admits dictionaries, conditionals, comprehensions, iteration, user-defined functions, load() and the pure standard library
• compilation must be bounded in depth and size and must not abort the process on an author-supplied source
• workflow type may be playbook
• a task's declared files continue through the graph alongside its JSON output, isolation notwithstanding
• a required task may not depend on an advisory one through all-join edges
• param joins the enumerated constructors
• shutdown outcome gains complete for a run a task ended with no work left
• a named-argument diagnostic must locate the argument, and suggestions draw on source bindings
• session log gains the plan_admitted event, so a consumer reads the graph instead of inferring it from a gate label
• deps() is removed: depends_on already accepts task values, so the wrapper was ceremony
• the declared lane scopes the constructor namespace, so a playbook cannot name or be offered a scored constructor
• session log gains asks_emitted, so what a run proposed is auditable apart from what an orchestrator admitted
• skill joins the enumerated constructors: an agent task whose prompt is assembled from a shipped SKILL.md and the invocation's arguments

v0.1.0 (2026-08-19)

Initial draft

RFC-0002: Playbook workflows

Version: 0.1.0 | Status: normative | Phase: spec Owners: @Will Eaton

1. Summary

[RFC-0002:C-SCOPE] Scope (Informative)

A playbook is a workflow that runs exactly once and produces no score. It exists for work whose value is the work itself: a skill pipeline, a scheduled chore, a discovery sweep. The optimization loop's propose, apply, measure, and decide protocol does not participate, and no frozen judge is constructed.

RFC-0001 governs the engine's manifest, command, world, transport, wire, and workflow-authoring contracts. This RFC adds the playbook lane on top of them and does not restate their universal rules. The two changes this lane needed there landed in version 0.2.0 of that RFC: RFC-0001:C-WORKFLOW admits "playbook" as a workflow type and treats a task's declared files as output rather than as workspace state that isolation discards, and RFC-0001:C-WIRE carries the shutdown outcome an early-completing run reports.

Fan-out over an item set that is not known until a task has run is expressed as one run per item rather than as graph expansion. The graph stays static and fully renderable before any spend; RFC-0002:C-ASKS defines the boundary where new work leaves a run.

Since: v0.1.0

2. Specification

[RFC-0002:C-PLAYBOOK-SURFACE] Playbook author surface (Normative)

A playbook author must write:

1. A crucible.toml manifest with no [judge] and a [workflow] whose type is "playbook".
2. The graph: a workflow.star, or the [[workflow.task]] tables it would compile to.
3. A goal, and a prompt for each agent task.
4. The commands its command and evaluate tasks run, where it has any (any language).
5. Agent credentials in [agent].env.

A playbook author must not be required to write a measure command, a judge, a baseline, an apply command, a snapshot or restore command, or a result task. No obligation in this RFC may be conditioned on one of them, and the manifest check must not demand one of a playbook. This is the whole difference from RFC-0001:C-SURFACE: the scored lane's author writes a gate and gets a loop, and the playbook author writes a graph and gets one pass.

Everything else is the engine's, provided for free: dispatch and concurrency, ceilings and per-task deadlines, git memory, the session log and every reporter, steer, stop, resume, publish, and ask emission.

The litmus test (examples/playbook/) must exercise items 1 through 4 on the command backend, requiring no Rust, no judge, and no credentials.

Since: v0.1.0

[RFC-0002:C-PLAYBOOK-LANE] Playbook lane admission and execution (Normative)

A manifest with no [judge] MUST accept a [workflow] whose type is "playbook". A manifest with no [judge] MUST reject every other workflow type, and a manifest with [judge] MUST reject type = "playbook".

A playbook graph MUST NOT contain a task naming an engine operation: an operation the scored loop's orchestrator owns rather than the plan author, namely producing a candidate, applying one, measuring one, grading evidence, deciding keep or discard, and measuring one differentially against another. A task that runs an author-supplied command and grades its own result is not an engine operation and MUST be permitted, because it asserts something about the work rather than advancing a scored loop.

A playbook MUST execute its graph exactly once per run. The engine MUST reject a requested iteration count greater than one rather than silently ignoring it.

A playbook's verdict MUST be invalid when any required task settled failing, was truncated, or was left blocked, and when a cost or wall-clock ceiling was exhausted, whatever else the run reports. Otherwise the verdict MUST be valid when every required task settled passing, and when a task declared early completion. A required task left undispatched by early completion MUST NOT affect the verdict; a required task that had already failed MUST invalidate the run whatever a later early-completion signal says, so that a concurrent task cannot launder a failure into a success. The verdict MUST NOT depend on the outcome of any advisory task or of any epilogue task.

The run MUST exit zero if and only if its verdict is valid, and MUST exit nonzero otherwise. This is a stronger obligation than the task lane's, where exit zero means only that the run completed. The exit code alone does not say why a run ended, so the shutdown outcome MUST distinguish an exhausted graph, an early completion, a failure, an operator stop, and an exhausted ceiling. That outcome MUST be drawn from the vocabulary RFC-0001:C-WIRE defines for the shutdown event; the value that vocabulary carries for early completion is "complete".

Any task MAY declare early completion by returning the boolean field "complete" as true in its output, optionally alongside a "reason" string. Both names are reserved, and a task MUST NOT name either in its declared outputs: a declared output must be present on every passing attempt, whereas these appear only on the attempt that ends the run, so declaring them would fail every ordinary run. The engine MUST reject a graph that declares either. The engine MUST then stop dispatching further tasks, MUST record the reason where one was given, and MUST NOT treat undispatched tasks as blocked. Determining that there is nothing to do is a successful outcome, and a playbook that cannot say so has no way to end a scheduled run quietly. The shutdown outcome says how dispatch stopped and the verdict says whether the run succeeded; they are independent axes, so a run whose dispatch stopped on early completion after a required task had already failed records the early-completion outcome and an invalid verdict.

A task's workspace changes MUST be committed to the run's git memory when, and only when, it settles with a passing outcome. A task that failed, was cut short by a deadline, or never settled MUST NOT contribute commits. This is what lets a resumed run rebuild from settled work alone, and what keeps a run that stopped early from publishing a half-finished tree: publication keys on commits beyond the base, and unsettled work never became one.

Every settled task MUST produce exactly one session row identifying the task by name. Rows MUST be written as tasks settle, so a reader tailing the log sees progress live; row order is therefore completion order, which concurrent tasks make distinct from graph order. A row's score MUST be null.

A playbook MUST NOT be required to declare a result task. Its verdict is defined above, so there is no single task whose output stands for the run.

A task declared required = false MUST NOT affect the run's verdict, and its failure MUST block only its dependents. The choice of which tasks are advisory is the author's; the engine MUST NOT infer it.

A task declared stage = "epilogue" MUST be excluded from the main graph, MUST run once after the main graph settles, and MUST NOT change the verdict. It MUST run when the main graph completed, when it failed, and when a task declared early completion. It MUST NOT run when the run was stopped by an operator or ended by an exhausted ceiling, evaluated at the moment the main graph settles: continuing to spend after either would defeat the control that ended the run.

An epilogue task MUST NOT depend on a main-graph task. It MUST instead receive the main graph's outcome as the reserved input RFC-0002:C-SETTLED-JOIN defines, which carries the run's shutdown outcome and one entry per settled main-graph task. This is what makes an epilogue task reachable on the failure path, where dependencies would not have passed.

A playbook MUST carry the task lane's gate label, written under the key RFC-0001:C-WIRE reserves for it. The engine MUST NOT introduce a third gate label; a consumer needing the graph MUST read the plan_admitted event RFC-0001:C-WIRE defines, which a playbook always emits.

Since: v0.1.0

[RFC-0002:C-PLAYBOOK-SHAPE] What the graph expresses (Informative)

A playbook's graph says what runs before what, and what must pass before what runs. It does not say how many times anything runs. The graph is dispatched once, so a task that must retry, poll, or converge does that inside its own execution, where the author's command or the agent's own turn loop owns the repetition. A repair loop is therefore one task, not an edge back into the graph, and the engine never sees the iteration at all.

This is also why an item set discovered at run time cannot become new nodes. The only two things a run can do with such a set are handle it inside one task or emit it as asks under RFC-0002:C-ASKS, and which one is right turns on whether the items deserve their own budget, isolation, and verdict.

The payoff is that the compiled graph is fully renderable before any spend and identical across runs of the same pack at the same parameters: the node set and the edges between them do not depend on what a task finds. Which of those nodes are dispatched still varies. RFC-0002:C-PLAYBOOK-LANE leaves tasks undispatched after an early completion, and RFC-0002:C-PLAYBOOK-CAPS truncates the plan or skips an advisory subtree when a capability is unavailable. Dispatch order among concurrent tasks is not fixed either. What is fixed is the shape. RFC-0002:C-PLAYBOOK-LANE states the once-per-run obligation; this clause records what follows from it for an author deciding where to put a loop.

Since: v0.1.0

[RFC-0002:C-PLAYBOOK-PARAMS] Workflow parameters and launch schema (Normative)

A workflow source MAY declare its parameters in a params block. The block MUST be the source's first statement and MUST be a literal: it MUST NOT contain a call, an interpolation, or a reference to a variable. The engine MUST be able to read the block without evaluating the source, so that an unevaluated or untrusted source can still be introspected. Only the source a run names may declare parameters; a loaded library MUST NOT.

A declared parameter MUST have one of these types: string, integer, number, boolean, or list of strings. A declaration MAY carry a human-readable description, a default, a required flag, and a value constraint: a pattern for a string, a minimum and maximum for a numeric type, or an enumerated choice set. A parameter declared required MUST NOT also declare a default.

Parameter values MUST be bound during compilation. The compiled graph MUST contain no unresolved parameter reference, so that the rendered graph shows exactly what will run and the frozen artifact remains fully static. Where a manifest field admits a parameter, it MUST be substituted at the same point, so that no parameter survives into execution unresolved.

A required parameter with no supplied value MUST be a compile error, and the engine MUST NOT dispatch any task of that plan. A supplied value violating its declared constraint MUST be rejected before compilation.

The engine MUST emit the declared block as a JSON Schema document on request, so that one declaration serves command-line validation, ask validation, and a generated launch form.

A parameter value that did not originate inside the pack MUST reach an agent prompt only inside a region the prompt marks as external input not to be followed as instruction. An inspector reading the rendered prompt MUST be able to tell which spans came from outside the pack.

Since: v0.1.0

[RFC-0002:C-TASK-FILES] Declared file outputs between tasks (Normative)

A task MAY declare emits_files: the workspace-relative paths its output includes as files. Each declared path MUST be relative, MUST NOT traverse outside the workspace, and MUST NOT reach a file outside the workspace through a link of any kind. A symbolic link is refused per path component; a hard link is not distinguishable by resolving the path, so the engine MUST establish confinement by a means that does not rely on resolution alone.

When a task passes, the engine MUST capture each declared file. A declared file absent after an otherwise-passing attempt MUST convert that attempt to a measured failure at the producing task, MUST NOT be retried, and MUST block its dependents. This mirrors the rule for declared JSON output fields: output drift fails where it happened. A task that settles failing is captured too, under the additional provenance and withholding rules of RFC-0002:C-SETTLED-JOIN, and its set is staged only along the edges that clause names.

Captured files MUST be staged into a task's workspace before it is dispatched, for every ancestor on its dependency paths and not only its direct dependencies. A pipeline whose later stages need an earlier stage's artifact is the ordinary case, and requiring each hop to re-emit what it received would reproduce by hand the state files this replaces. JSON outputs keep the narrower direct-dependency rule.

Staged files MUST be namespaced by the producing task's name, so that two producers declaring the same path cannot collide. Staged files MUST be read-only to the consuming task.

Captured files MUST be staged for a dependent even when the producing task ran in disposable isolation. A declared file is part of a task's output, not part of the workspace state that isolation discards.

The total size of files captured in one run MUST be bounded, the bound MUST be operator-configurable and discoverable before a run starts, and exceeding it MUST fail the producing task naming the bound rather than truncating silently.

emits_files MUST NOT be used as a channel between runs. Material that must outlive its run MUST be referenced by published artifact location instead.

Since: v0.1.0

[RFC-0002:C-PLAYBOOK-CAPS] Task capability requirements (Normative)

A playbook MUST execute on one substrate for its whole life. A task therefore cannot be moved to reach a capability, and privileged or specialized work MUST be reached by asking a mediated broker instead.

A needs declaration on a task MUST name a capability that the run can verify is available before dispatch. Whether a substrate provides that capability directly or a broker provides it on request is not the task's concern.

A required task whose needs is not available MUST truncate the whole plan before dispatch, and the engine MUST report that verdict before any spend. An advisory task in the same position MUST be skipped along with its dependents, leaving the verdict unaffected.

Since: v0.1.0

[RFC-0002:C-PLAYBOOK-COMPOSITE] Judgeless composite domains (Normative)

A composite domain MUST be permitted without [judge] when its workflow type is playbook. A playbook over several repositories combines work, not scores, so a composite MUST NOT be required to carry a judge merely because it has more than one component.

A judgeless composite MUST NOT produce a combined score, and the engine MUST NOT construct one.

A component MAY carry its own publish target. A component whose workspace holds commits beyond the base it started from, and that carries a target, MUST publish one draft pull request; a playbook has no propose-and-apply cycle, so this commit comparison is what stands in for the scored loop's notion of a change. A component with no target MUST publish nothing, which is the ordinary case for a playbook whose output is a review, a report, or a filed ask rather than a diff.

Since: v0.1.0

[RFC-0002:C-ASKS] Work emission (Normative)

A playbook MAY emit asks: descriptions of work for another run to perform. An ask MUST carry a key, the workflow it names, and that workflow's parameter values. An ask's parameter values MUST satisfy the named workflow's declared schema, and the receiving orchestrator MUST reject an ask that does not.

The key MUST be supplied by the emitter and MUST be stable across runs for the same underlying item, so that a receiving orchestrator can recognize a repeat without interpreting the ask's contents. What counts as the same item is the emitter's judgment and cannot be checked by the engine; the obligation is verified by auditing emitted keys on the session log against what they referred to.

A run MUST NOT dispatch an ask it emitted. Admission belongs to the receiving orchestrator, which owns deduplication, exclusion policy, and rate limits; an emitting run therefore cannot widen its own blast radius.

The number of asks one run may emit MUST be bounded, the bound MUST be operator-configurable and discoverable before a run starts, and reaching it MUST fail the emitting task rather than silently dropping the remainder.

Emitted asks MUST be recorded on the session log, so that what a run proposed is auditable independently of what was admitted.

An ask MUST NOT carry file content. Material a receiving run needs MUST be referenced by published artifact location.

Since: v0.1.0

[RFC-0002:C-PLAYBOOK-BUDGET] Cost and time ceilings (Normative)

A playbook MUST run under two ceilings supplied by whatever launched it: a total cost ceiling and a total wall-clock ceiling. A playbook source MUST NOT declare either, so that a pack cannot raise a limit its operator set.

The engine MUST refuse to dispatch any task of a playbook for which either ceiling is missing. Unsupervised scheduled work with no stated limit is the failure this rule exists to prevent, so the refusal MUST come before any spend rather than at the first overrun. A run that ends because a ceiling was exhausted MUST name which one in its shutdown outcome.

The engine MUST NOT terminate an attempt already in flight on cost grounds, so a completed attempt MAY carry the total past the cost ceiling. Any cost overrun MUST invalidate the run and MUST block all further dispatch and retries.

The engine MUST terminate every attempt in flight when the wall-clock ceiling is reached, MUST invalidate the run, and MUST block all further dispatch and retries. This is deliberately the opposite of the cost rule, and a reader who pattern-matches from one to the other will get it wrong: a cost total is known only once an attempt finishes, so killing that attempt would spend the money and learn nothing, whereas elapsed time is known continuously and a run past its window is over no matter what any task is still doing.

Separately from the run's ceilings, every task MUST execute under a bounded wall-clock deadline. Exceeding it MUST settle that task as a failure, which propagates like any other failure rather than ending the run directly: a hung task spends nothing, so the cost ceiling cannot bound it. The operator MUST set a default deadline and a maximum; a task MAY declare a shorter one, and a declared deadline exceeding the operator's maximum MUST be rejected before dispatch. A deadline MUST NOT extend a task past the run's wall-clock ceiling; whichever bound falls first settles the task.

A ceiling reached while an epilogue task is in flight MUST terminate that attempt and MUST block further dispatch, and MUST NOT change a verdict already determined by the main graph. The epilogue exists to report on the run, so letting its own spend rewrite the run's verdict would defeat it.

Every ceiling and deadline in force MUST be discoverable by the pack author and the operator before a run starts.

Since: v0.1.0

[RFC-0002:C-PLAYBOOK-RESUME] Resumption (Normative)

A resumed playbook MUST NOT re-dispatch a task that already settled. The engine MUST fold prior task results from the session log and MUST treat them as terminal, feeding their outputs to dependents exactly as a live execution would.

A resumed playbook MUST reconstruct its workspace from the pristine checkout plus the declared file outputs of every folded task that settled passing, staged under the rules in RFC-0002:C-TASK-FILES. Files captured from a task that settled failing MUST NOT be restored into the workspace; they reach a dependent joining "settled" through its staged inputs only. It MUST NOT attempt to reconstruct the workspace as some particular task left it: when concurrent tasks were in flight at the interruption, no such state is well defined, and a task's declared output is the only part of its work the contract ever promised would survive.

Spend and elapsed time recorded before the interruption MUST count against the ceilings on resumption. A crash MUST NOT reset either.

A resumed run MUST NOT re-dispatch the main graph when the folded results cover every main-graph task, and equally when any folded result declared early completion. Early completion leaves tasks undispatched by design, so a resume that only checked for full coverage would dispatch them again on every restart. It MUST still dispatch any epilogue task that has no folded result, under the conditions in RFC-0002:C-PLAYBOOK-LANE. The run's summary, shutdown, and publication MUST each happen exactly once across the original run and all of its resumptions.

Since: v0.1.0

[RFC-0002:C-REPORTS] Typed workflow reports (Normative)

A playbook MAY declare an engine-owned report task with a destination kind and an optional result task selector. The selected task MUST be a task in the same graph. Selection MUST NOT create a dependency from the report epilogue to the main graph. After the main graph settles, the report task MUST receive only the selected task's declared JSON output when that task passed, never its prompt, stdout, workspace, undeclared files, or credentials. When the selected task did not pass, the report MUST retain the selected task's terminal status without projecting partial output.

A selected report result MUST be bounded before delivery. The operator MUST configure a maximum encoded size, and exceeding it MUST fail the report task without truncation. Values MUST retain their JSON types through the report snapshot. A report renderer MUST escape text for its destination and MUST NOT interpret result values as destination-native payload fragments.

The Slack destination MUST support a structured card projection containing an engine-authored header, run verdict and cost, the selected result fields, bounded task status context, and a controller-owned run link. The engine MUST construct the destination-native payload; a pack MUST NOT supply raw Slack blocks, attachments, webhook URLs, channels, actions, or identifiers. Delivery failure MUST settle a required report task as failing.

The persisted session result for a report task MUST record whether delivery succeeded but MUST NOT persist the webhook credential. An observer MUST be able to distinguish a delivered report, a rendering failure, an oversized result, and a destination failure.

Since: v0.1.0

[RFC-0002:C-SETTLED-JOIN] Joining on a settled dependency (Normative)

A task MAY declare join = "settled". The engine MUST accept "all", "passed", and "settled", and MUST reject every other value. A task declaring join = "settled" MUST declare at least one dependency. The engine MUST reject join = "settled" on a reducer task, on the engine-owned report task, and on any task naming an engine operation, whose contracts are defined over the passing set or over a fixed typed context.

The engine MUST dispatch a task joining "settled" once every one of its dependencies has reached a terminal status, whatever that status is: passed, failed, skipped whether the task declared the skip or the substrate did, transport-failed after its retries, or blocked. A settled join MUST NOT make a dependency's runnability a condition of the dependent's; the engine MUST treat such a task as runnable wherever the substrate satisfies its own capability requirement, whatever its dependencies require.

A settled join governs dependency status and nothing else. The engine MUST NOT dispatch such a task once dispatch has stopped for any other reason, and a task left undispatched by a required task settling other than passing, by an exhausted cost ceiling, by an exhausted wall-clock ceiling, or by a truncated graph MUST settle blocked whatever its join. An epilogue task is exempt from the required failure alone, because RFC-0002:C-PLAYBOOK-LANE requires it to run when the main graph failed: a required failure MUST NOT block it whatever its join, and an exhausted ceiling or a truncated graph MUST block it as it blocks any other task. Where dispatch stopped because a task declared early completion, RFC-0002:C-PLAYBOOK-LANE governs: the engine MUST NOT treat an undispatched task as blocked, and a settled join MUST NOT cause it to dispatch.

The engine MUST give a task joining "settled" one entry per declared dependency, under that dependency's name and present for every dependency whatever its status, carrying that dependency's terminal status as the token RFC-0001:C-WIRE defines for the task-result event, the engine's note or null where it recorded none, the dependency's JSON output or null, and whether a file set captured in this run was staged for this consumer. The entry MUST carry the output rather than be the output. For a mapped dependency the entry MUST additionally carry, per instance, that instance's status, note, output, and staged-file flag, and that per-instance mapping MUST be present and empty where the node produced no instances; a consumer distinguishes an empty fan-out from a node that never expanded by the entry's status. The engine MUST NOT wrap a reserved input, and MUST reject a plan that names a dependency after one: the reserved key is written after the envelope is built, so it would overwrite that dependency's entry. The engine MUST give every epilogue task a reserved input named "outcome", built when that task is dispatched, carrying under "exit" the token RFC-0001:C-WIRE defines for the shutdown event's outcome and under "tasks" one entry per settled main-graph task, under that task's name, each entry being this entry with its output and staged-file flag omitted. This is the input RFC-0002:C-PLAYBOOK-LANE requires an epilogue task to receive.

The engine MUST give a task joining "all" or "passed" each contributing dependency's JSON output directly, and MUST NOT give it an entry for a non-mapped dependency that settled failing, transport-failed, or was blocked. A mapped dependency's fold MUST continue to reach such a task on exactly the terms it reaches it today, whatever the fold's status: a reducer over a lossy fan-out is the case join = "passed" exists for.

A task that ran and measured a failure MUST retain that attempt's structured output. The engine MUST retain an evaluate task's own graded object, MUST retain a command task's final stdout line where it parses as a JSON object although the process exited nonzero, and MUST NOT retain output from an attempt that ended in transport failure. The engine MUST NOT treat a retained failure output as a mapped task's source list: a task fanning out over another task's field MUST fail, naming the absent source, unless that source settled passing.

A task MAY settle itself failing by returning the string field "status" as "fail" in its output. The engine MUST record such a task as failed, MUST retain the returned object as that task's output, and MUST NOT retry it. This is the third engine-meaningful value of the field that already carries "skipped"; the engine MUST ignore every other value.

Declared JSON output fields and declared file paths are owed by a passing attempt only. The engine MUST NOT check declared fields on a failing attempt, and MUST NOT guarantee a task joining "settled" that a failed dependency's output carries them.

The engine MUST capture a task's declared files when it settles passing and when it settles failing, under the confinement, atomic-publication, file-mode, and run-total-size obligations of RFC-0002:C-TASK-FILES. Capture MUST happen before the task's workspace changes are discarded, and the engine MUST NOT require isolated execution to capture on failure. On a failing attempt the engine MUST capture only paths that attempt itself wrote, determined by comparing each declared path's content in the root the attempt ran in against what that path held there before the attempt started, and MUST treat a declared path whose content did not change as absent: a file an earlier task left in the workspace is that task's evidence, not this one's, whether or not the workspace's version control can see it. A declared path that is absent, is not a regular file, or was not written by a failing attempt MUST NOT change that task's status; the engine MUST publish none of the set, MUST remove any set published for that task by an earlier attempt or run, and MUST record the capture problem with the failure note. Where an absent declared path converts an otherwise-passing attempt into the measured failure RFC-0002:C-TASK-FILES requires, the engine MUST retain that attempt's structured output as the failure's. Where the run-total size bound is exceeded by an attempt that has already failed, the engine MUST enforce it by publishing none of the set and recording the bound with the failure note. The engine MUST NOT capture declared files from a task that skipped, that transport-failed, or that was never dispatched, and MUST remove any set such a task published in an earlier run.

The engine MUST stage the declared files of a task that settled failing only into a task joining "settled" that declares it as a dependency, and MUST NOT stage them into any other task. The engine MUST stage a passing ancestor's declared files under the rules of RFC-0002:C-TASK-FILES unchanged. A staged set MUST be complete; the engine MUST NOT stage a partial set. One instance of a mapped producer MUST be staged under the instance's own name.

Capture is not commit. A task that settled other than passing MUST NOT contribute a commit to the run's git memory, and its undeclared workspace changes MUST be discarded, as RFC-0002:C-PLAYBOOK-LANE requires. A captured failure set MUST NOT make its producer a passing dependency, MUST NOT satisfy a join of "all" or "passed", and MUST NOT affect the run's verdict.

A resumed run MUST treat a folded non-passing result as satisfying a settled join exactly as a live result does, and MUST NOT re-dispatch the task to regenerate its evidence. Captured sets MUST remain in engine-owned run state, and the engine MUST NOT restore a non-passing task's captured files into the workspace. Where a captured set did not survive the interruption the engine MUST still dispatch the consumer, giving it that dependency's status and output, reporting no staged files, and MUST NOT re-dispatch the producer.

The admitted-plan event MUST carry "settled" under the key it already carries a task's join, and the engine MUST NOT add a companion field for it; a consumer MUST treat an unrecognised join token as opaque rather than inferring readiness from it. The task-result event MUST NOT change shape: a failing task's retained output MUST travel in the field that already carries a passing task's.

Since: v0.1.0

Changelog

v0.1.0 (2026-08-21)

Initial draft

RFC-0004: Linked controller/engine boundary

Version: 0.1.0 | Status: draft | Phase: spec Owners: @Will Eaton

1. Summary

[RFC-0004:C-SCOPE] Scope (Informative)

This RFC governs how the controller obtains work from the engine: rendered Kubernetes objects for turn and run pods, compiled workflows and their parameter schemas, run explainability documents, and the launching of an engine process for an agent turn or a local run. It applies to every deployment shape (pod dispatch, the local executor, the registry) and to both directions of version drift between a controller build and the engine it is paired with.

It does not govern what runs inside a pod once launched (the wrapper script, the loop, the judge), nor the human API the controller serves. Those keep their own contracts (RFC-0001, RFC-0002).

The motivating failure: the controller began sending a flag for a scenario's git ref weeks before any engine build accepted it. Nothing failed until a scenario with a ref was adopted in production, and then every one of them failed at argument parsing, ledgered only as a turn that produced no result.

[RFC-0004:C-BOUNDARY-INVENTORY] The controller-engine boundary (Informative)

The controller reaches the engine in two ways today, and this RFC treats them differently.

Renders and compilations are pure: given a profile, a manifest or workflow source, and a set of options, they produce a document (pod objects, a pack ConfigMap, a workflow graph, a parameter schema, a flow document). Today the controller spawns the engine executable for each of these and parses its stdout: turn-pod render, run-pod render, workflow compile, parameter schema, and flow.

Process launches are not pure: an agent turn or a plan run holds a checkout, spends money, streams a session, and must be killable. Today the controller spawns the engine for these too: scope, grounded rank, a local playbook run, and artifact fetch.

Both paths share one contract surface, the engine's command line, and one enforcement point, the engine's argument parser at dispatch time.

Three version-bearing things take part, and the clauses below name them consistently:

• the contract definition: the single shared description of every document that crosses the boundary, compiled into both sides, carrying the contract version;
• the linked engine revision: the engine source revision a controller build compiles its renders from;
• a dispatch image: an engine image the running controller launches pods or processes from. A controller may be configured with more than one (loop images, sandbox images, per-pack images), and a dispatch image can differ from the linked engine revision because images are pinned in deployment configuration, not at build time.

This RFC replaces the render path with in-process renders checked at build time, and constrains launches to a typed request whose contract version is checked before any work happens.

2. Specification

[RFC-0004:C-LINKED-RENDER] Renders are linked, not spawned (Normative)

A render or compilation the controller needs (a turn pod, a run pod and its pack ConfigMap, a compiled workflow, a workflow's parameter schema, a flow document) MUST be produced in the controller's own process from the linked engine revision. The controller MUST NOT obtain any of these by spawning an engine executable.

A controller build that requests a render option the linked engine revision does not define MUST fail to build; it MUST NOT be possible for such a controller to reach dispatch.

For the same inputs, a render produced in-process by the controller and one produced by the engine's own command line MUST be byte-identical. The existing rendered fixtures are the acceptance test.

A render MUST NOT depend on anything beyond the inputs it is handed (the profile, the pack, the workflow source, its options). A render MUST NOT resolve image tags to digests, contact a cluster, or read the process environment unless the caller passes that capability in explicitly; the controller decides when a render may reach the network.

This clause changes how a render is delivered, not what it contains: the parameter schema stays as RFC-0002:C-PLAYBOOK-PARAMS defines it and the compiled workflow stays as RFC-0001:C-WORKFLOW defines it.

Rationale: a render is a function of its inputs. A process boundary and an argument parser between the caller and that function add a contract nobody checks and remove build-time checking, which was the only tool able to check it.

[RFC-0004:C-TYPED-INVOCATION] Typed requests across a process boundary (Normative)

Where the engine MUST run as a separate process (an agent turn, a local plan run, work inside a pod), the request MUST be a single typed document described by the contract definition, carrying the contract version it was written under. The engine MUST reject a request whose document has an unknown field, or whose contract version is not equal to the engine's own, before it clones, spends, or writes anything, with an error that names the field or both versions.

The command line of such an invocation MUST carry only the request kind and the location of the request document; it MUST NOT carry request fields as flags. An environment variable MAY carry the document location where a file cannot (a pod wrapper), and the same rejection rules apply.

The engine's reply for an invocation MUST be a typed document under the same contract version, so that the controller parses a result the way it built the request, and a reply the controller cannot decode is a boundary failure per RFC-0004:C-BOUNDARY-FAILURE, not a missing result.

Rationale: a process boundary is unavoidable for work that must be isolated or killed, but the boundary can still be a document with a schema and a version instead of a flag list whose only validator is the parser on the far side.

[RFC-0004:C-CONTRACT-VERSION] One contract version, checked before dispatch (Normative)

The contract definition MUST carry exactly one contract version. Two contract versions match if and only if they are equal; there is no compatibility range. A controller build MUST record the contract version of the contract definition it was compiled against, and an engine build MUST record the one it was compiled against; both MUST be readable from the running executable and, for an image, from the image's labels.

A controller build whose linked engine revision was compiled against a different contract version than the controller MUST fail its build, so the mismatch is found in CI and not in production.

Because a dispatch image is pinned in deployment configuration and may differ from the linked engine revision, the running controller MUST, at startup and whenever a dispatch image pin changes, read the contract version from every dispatch image it is configured with and compare each against its own. It MUST expose the result per dispatch image (the image reference, both versions, and whether they match) on the same read endpoint that reports its effective configuration, so an operator can see a mismatch without reading logs.

A mismatch MUST NOT stop the controller from serving. The controller MUST refuse to launch a pod or a process from a mismatched dispatch image, and each refusal MUST be ledgered per RFC-0004:C-BOUNDARY-FAILURE. Pods already running when a pin changes are out of scope; they carry the request they were launched with.

Rationale: build-time checking removes drift for renders and for the linked revision; the runtime comparison covers the one place a link cannot reach, an image chosen by configuration, and turns a runtime surprise into a visible, explainable refusal.

[RFC-0004:C-BOUNDARY-FAILURE] Boundary failures are ledgered by name (Normative)

A failure at the controller-engine boundary MUST be recorded on the issue or run ledger with the request kind, the dispatch image or linked engine revision involved, the controller and engine contract versions, and the engine's own error text. It MUST NOT be reduced to a generic no-result outcome; the ledger entry alone MUST be enough to tell what was sent and why it was refused.

Boundary failures are of two kinds, and the ledger entry MUST say which:

• contract rejections are deterministic: a rejected contract version, an unknown field, a render option the engine does not define, a reply the controller cannot decode because its shape is wrong. The same request against the same engine cannot succeed, so a contract rejection MUST NOT be retried automatically.
• transport failures are not: a process killed before it replied, a truncated reply, an I/O error reading or writing the request document. These MAY be retried under the retry policy that already governs the launch in question, and the retry MUST be ledgered as a retry, not as a fresh attempt.

Rationale: the motivating failure was visible only to someone who read the raw ledger event; the summary said nothing, and the item stayed in its initial state as if never attempted. Retrying a deterministic rejection would have hidden the same fault behind a moving timestamp.

3. Compatibility

[RFC-0004:C-MIGRATION] Migration and flag ordering (Normative)

Until a render is delivered in-process, adding or changing an option on an engine command the controller invokes MUST land in the engine first, and the controller change MUST NOT be released before a dispatch image carrying that engine is deployed. A controller change that would send an option the linked engine revision does not accept MUST fail the controller's build.

Moving a render in-process MUST NOT change the rendered document for the same inputs; the existing rendered fixtures are the acceptance test.

Existing process-launch kinds MAY keep their current command-line form during migration, but a new launch kind MUST be introduced only as a typed request per RFC-0004:C-TYPED-INVOCATION.

The contract version starts at the version the contract definition carries when this RFC is finalized. A controller or engine that cannot report a contract version MUST be treated as mismatched per RFC-0004:C-CONTRACT-VERSION.

Changelog

v0.1.0 (2026-08-25)

Initial draft

Architecture decision records

The ADRs capture the load-bearing design calls behind crucible: why the loop is shaped the way it is, what was traded off, and what the alternatives were. They are ordered, append-only, and meant to be read when you want the why rather than the how.

Expand this section in the sidebar to browse the full list.

ADR 0001: Adaptive harness — adapt the setup, freeze the judge

Status: Accepted; implemented (the setup/solution tool split, fingerprinted segments, and the escalate hatch are live in the engine) Date: 2026-06-23 Context owner: agentic-epp-autoresearch

Context

The loop's integrity rests on one thing: the gate (fitness) must be trustworthy and outside the agent's reach. A kept "win" is only as honest as the harness that judged it.

We've found it valuable to tailor the harness to each issue — e.g. for a heterogeneous routing workload we hand-shaped the gate, chose a baseline, and picked the perf gate. Automating that tailoring ("an adaptive harness") is attractive: the harness would meet each issue where it is. But adaptivity aimed at the wrong layer turns the loop into a reward-hacking machine — if the harness can reshape itself toward the thing being optimized, the agent (or the adaptation itself) will make the workload easy, move the baseline, or pick a metric it already passes.

Decision

Adapt the setup. Freeze the judge. The agent may change its solution; it must never change its evaluation. Concretely, split the system into two phases with a hard wall between them:

1. Scoping (adaptive, once, then frozen)

Analyze the issue and decide the workload, baseline, and gate. This is the part worth automating (an agent may propose the harness config), but its output is:

• pinned — written to an immutable harness manifest,
• content-hashed — a fingerprint of (workload + gate + baseline),
• validated — passes the gate self-test (below) before it's trusted,
• human-approved — a person signs off before the loop spends budget.

2. The loop (immutable gate)

The optimizing agent runs against the frozen harness. It gets only solution-space tools and a read-only gate. It cannot touch the workload, the baseline, or the metric. The harness manifest is mounted read-only, outside the agent's editable workspace.

Tool classification (capability boundary)

Every tool is one of two kinds. The optimizing agent's toolbox contains only the second column. The first is used during scoping and then frozen.

The loop agent's toolbox must exclude rig-config and order-capture, since they can change MOCK_* (the latency model) and the workload — i.e. the evaluation surface. They are exposed only in scoping, never in the loop driver.

Safeguards (make adaptivity safe rather than scary)

1. Gate self-test (negative controls). Before a gate is trusted, prove it discriminates: a known-bad config must score worse and a known-good better (e.g. prefix-affinity p99 292ms vs prefix-blind 506ms). Automate it as a precondition — a gate that can't tell good from bad is the real danger, not the agent.
2. Harness fingerprint on every result. Stamp each kept result with the manifest hash so wins are reproducible and scope drift is detectable at a glance.
3. Fixed baseline. Derive the baseline once during scoping; never re-derive it inside the loop (a moving baseline makes "improvement" meaningless).
4. Multi-seed gating. Average a few seeds so a lucky single run isn't kept (esp. for noisy dynamics like load-scorer herding).
5. Cost/blast-radius gating. Anything in the setup column that spends GPU or mutates shared infra (e.g. order-capture) keeps an explicit human-gated --execute, as today.

Escape hatch: the agent may declare the judge inadequate (not change it)

Freezing the judge only stays honest if the agent has a sanctioned way to say "this judge cannot evaluate what I need." Without it, a blocked agent either thrashes out discarded changes or tries to game the gate. So the complement to "freeze the judge" is a structured escalation:

• The agent calls escalate --category {harness-limitation|infeasible|needs-info} --reason … --evidence … (writes ESCALATION.json). The loop detects it, stops, restores the rig, and surfaces the report for human review.
• This is distinct from a discard ("this change didn't help") — escalation means no change here can be fairly evaluated, so the harness itself needs a human fix.
• Anti-abuse: the tool requires a substantive reason and asks for evidence (e.g. inspect-rig output across configs); the escalation goes to a human, it is not a silent success. It is the agent's only sanctioned move against the evaluation — it can flag it, never edit it.

This is the path when, for example, tokenload reads zero in-flight load everywhere (the producer signal is dead, so the gate cannot see length-aware routing): rather than discard forever, the agent escalates "the gate can't differentiate length-aware routing" with the inspect-rig evidence, and a human fixes the producer wiring or the workload.

Consequences

• Positive: we keep the appealing part — the harness meets each issue where it is — while the optimization loop stays trustworthy and its results stay comparable and reproducible.
• Positive: clear capability boundary; the loop agent literally cannot reward-hack its evaluation because it has no tool to reach it.
• Negative / cost: scoping becomes a real, gated step (validation + approval), not a side-effect of running the loop. That's deliberate — a wrong harness is wasted or misleading research, so it deserves a checkpoint.
• The loop's tool wiring splits the toolbox into setup vs solution, emits a harness manifest + fingerprint, and runs the gate self-test as a scoping precondition. None of this requires an adaptive loop — only an adaptive scoping step.

Principle, restated

In ML you adapt the experimental setup once, carefully, and then you never let the model edit the test set. Same here: automate the setup, never the judge.

ADR 0002: Mediated provisioning — the agent asks, a server-side MCP admits

Status: Accepted; implemented (the broker now lives in the extracted crucible-broker crate; the draft-PR approval path was proven end-to-end against a fork) Date: 2026-06-25 Context owner: agentic-epp-autoresearch Related: ADR-0001 (freeze the judge), the request-trace gap, the control bridge (crucible/src/control.rs), the escalate hatch (tools/escalate.nu

• crucible/src/escalation.rs), the session MCP server.

Context

ADR-0001 drew the trust wall: the agent may change its solution, never its evaluation. But a blocked agent has a legitimate need the loop can't yet serve cleanly — it discovers it must measure a regime it has no calibrated trace for (a new model, a higher request concurrency, a novel workload), and the only honest GPU-free answer is capture/calibrate a new trace. Today that means either:

• the agent reaches the cluster directly (it holds a rig edit kubeconfig) and could, in principle, submit a GPU capture Job or dispatch CI itself, or
• it writes ESCALATION.json and the whole run halts for a human.

The first is unacceptable: we will not give the sandboxed agent direct authority to spend GPU, submit a Kueue Workload, or dispatch a GitHub Actions workflow. Those are privileged, shared-blast-radius, money-spending operations. The second is too coarse for a need that is often grantable (the trace just needs capturing) — it ends the run and waits on a human even when policy could have admitted it.

The missing piece is a mediation layer: a place that holds the privilege and the admission logic, exposes only a narrow "ask" to the agent, and decides whether/how to grant. An MCP server already sits on the loop pod (wrapping the read-only session surface). This ADR makes that server the provisioning admission broker too.

Decision

The agent asks; a server-side MCP on the loop pod admits. Provisioning privilege and the admission logic live in the MCP, never in the sandbox. Concretely:

1. Privilege separation by identity.

   • Solution-space cluster ops the agent already does (apply EPP config, bench, inspect-rig) stay direct, under the agent's scoped rig credential (ADR-0001's solution column). Unchanged.
   • Provisioning ops (submit a GPU capture Job / Kueue Workload, dispatch a GHA workflow, anything that spends GPU or mutates the evaluation) are removed from the agent's reach. The agent's credential must not carry these rights (RBAC: the agent SA can edit the rig namespace but cannot create Workloads/Jobs in the capture namespace; no forge token in the sandbox). Only the MCP's own identity can, and it gates every such call.

2. The MCP is the only privileged endpoint the sandbox can reach. OpenShell egress is deny-by-default; we allowlist exactly the loop pod's MCP endpoint and nothing else provisioning-capable. The agent speaks a small typed tool surface (below); it never sees a kubeconfig for the capture namespace, a Kueue API, or a forge token. Same mediated-capability pattern OpenShell already uses to serve the Vertex token through the metadata emulator without ever handing claude the token.

3. Requests are classified by whether they touch the judge (this is the integrity-critical axis, not the transport):

   • In-scope (a trace missing for the already-frozen regime, a pure cache fill): the MCP resolves it and returns MOCK_* knobs synchronously; the turn continues. No judge change, no re-baseline.
   • Judge-changing (a new concurrency / workload / regime the frozen baseline was never measured at): the MCP must not silently continue — that would make every later "keep" incomparable (the ADR-0001 reward-hack surface). It triggers a governed re-scope: re-derive the baseline at the new regime, bump the harness fingerprint, run the gate self-test, and only then start a new comparable segment. Here the coarseness is protective: the agent never measures its own change against a goalpost it just moved inside one turn.

4. Admission gating is server-side and opportunistic: on a cache MISS the MCP runs the gpu-check headroom logic (allocatable − Σrequested, with a reserve + a concurrent-capture cap); if there is headroom it auto-submits the capture under Kueue; else it defers/queues and the loop continues on fallback. The agent sees only a trace_id + a pollable handle.

5. Human-in-the-loop is pluggable, and headless-first. When a grant genuinely needs a human (beyond budget, a judge-changing re-scope under a stricter policy, a starving queue), the MCP raises the ask through one of two interchangeable approval backends behind a single request object — and keeps the request async (the loop records the ask and continues / parks; the human approves a batch out-of-band):

   • Forge-native (the headless default) — a draft PR. The MCP (which holds a repo-scoped forge token server-side — the agent does not) pushes an agentic/<run_id>-prefixed branch and opens a draft PR describing the request: params, trace_id, GPU estimate, the rendered capture manifest. Approval is a maintainer slash-command comment (/approve-capture <trace_id>) or marking the PR ready/merging — gated by CODEOWNERS + branch protection, audited by the forge itself. The MCP polls for the signal, then proceeds. A draft PR (over a bare issue) gives diff-review of the manifest, CODEOWNERS, and doubles as the state-persistence record (the run's branch outlives the pod). It builds on the existing publish.rs branch push.
   • Control-channel (attended). The MCP surfaces the same request as an event over the control bridge we built; an operator watching crucible view --pod approves with a keystroke (the steer/stop path). Immediate, for attended runs. The kept result records which path approved it (provenance, like the steer audit log), so a forge-approved capture is as traceable as an operator-approved one.

6. Degraded mode is escalate-halt, never auto-admit. When a judge-changing request arrives in a pure-headless run with no approval surface reachable (no forge token, no attended operator), the MCP does not silently auto-admit a goalpost move. It falls back to the escalate hatch: record the ask, restore the world, halt for a human. Unattended autonomy never extends to changing the evaluation without a recorded human (or policy) yes. In-scope cache fills still return synchronously in degraded mode — only judge-changing grants halt.

What this is NOT

• Not giving the agent a Kueue client, a forge token, or GPU-namespace RBAC. The agent's blast radius stays exactly the rig solution surface.
• Not a replacement for the frozen judge. A judge-changing grant re-scopes (new baseline + fingerprint); it never edits the gate in place.
• Not retiring escalation. ESCALATION.json (tools/escalate.nu → the engine hook) remains the honest valve for the unservable cases no tool can grant — a dead signal (tokenload=0), a missing rig capability, "infeasible" — and the degraded path when the MCP is unreachable (offline / pure-headless-no-forge). Mediated provisioning handles the grantable asks; escalation handles the un-grantable ones.

MCP tool surface (initial)

One server, narrow tools, all privileged work behind them:

request_trace content-hashes its params into a trace_id shared with the EPP CI cache, so a trace captured by the loop warms CI and vice-versa.

Architecture sketch

flowchart LR
    sandbox["Claude sandbox<br/>only MCP endpoint allowlisted"] -->|"MCP"| broker["loop-pod MCP server<br/>trusted rig, capture RBAC, and forge identity"]
    broker --> cache["resolve cache<br/>local traces + S3 by trace_id"]
    cache -->|"hit"| knobs["return knobs synchronously"]
    broker --> admission["admission<br/>GPU headroom + caps"]
    admission -->|"admitted"| kueue["submit Kueue job"]
    broker --> changed{"judge-changing request?"}
    changed -->|"yes"| rescope["re-scope<br/>baseline + fingerprint"]
    broker --> human{"human approval required?"}
    human -->|"headless"| forge["open issue or PR<br/>poll for approval command"]
    human -->|"attended"| control["surface control event<br/>operator decision"]

Consequences

• Positive: the agent literally cannot provision — no GPU spend, no Kueue, no CI dispatch from the sandbox — yet it is no longer dead-ended on a grantable need. The choke point is one auditable server-side process.
• Positive: admission + approval are policy, not code scattered across tools; headless runs get a real (forge-native, async, CODEOWNERS-gated) approval path, attended runs keep the keystroke path, both record provenance.
• Positive: reuses what exists — the control bridge, gpu-check, the request-trace cache design, the autoresearch/<run_id> branch push, the session MCP.
• Negative / cost: a real MCP server on the loop pod (lifecycle, the loop pod's identity now also carries capture/forge rights — concentrate and guard it), MCP plumbing into the sandbox claude config, and an egress allowlist entry for the MCP endpoint. RBAC must be tightened so the agent identity loses any latent provisioning rights.
• Negative / cost: the judge-changing re-scope (auto re-baseline + fingerprint + gate self-test) is real engine work, not just a knob hand-back.

Testing & rollout

The approval + state-persistence path is validated against a throwaway fork before targeting any real repo; the forge token is held server-side by the MCP, never relayed into the sandbox.

• The loop's [repo].url points at the fork; the MCP opens its draft PRs there.
• Branches are agentic/<run_id>-prefixed, so the agent's pushes/approval PRs are self-labelling and the fork's history is the durable run record (state persistence survives pod teardown).
• The rig forwards a repo-scoped fine-grained PAT (scoped to the fork only) into the loop pod — mounted server-side and held by the MCP exactly like the kubeconfig / AWS token, never relayed into the sandbox. The agent gets no forge credential.

This validates the full headless approval loop (open draft PR → /approve-capture → proceed) and state persistence before anything touches a real repo.

Design notes

• Re-scope unit: a judge-changing grant starts a new fingerprinted segment within the run (not a new run_id), so one session log tells the whole story across the boundary.
• Forge surface: a draft PR (not a bare issue) — diff-review + CODEOWNERS + it doubles as the state record.
• Degraded mode: escalate-halt (never auto-admit a judge change unattended) — decision 6.
• MCP transport is HTTP/SSE at the loop pod IP:port, allowlisted in the OpenShell egress policy as full for just that host:port, with the sandbox claude MCP config seeded (env / .mcp.json).
• Agent RBAC keeps edit on the rig namespace only, with no path to Workload/Job creation in a capture namespace; gpu-check's admission logic lives in a shared lib the MCP calls.

Principle, restated

The agent reaches the cluster only through doors the host holds the keys to. Solution-space doors are already open to it; every provisioning door is locked, and the MCP is the only one who can open one — after admission, and (when it must) after a human says yes through a channel that fits a headless run.

ADR 0003: Async approval waits — the agent decides block vs continue

Status: Accepted; implemented (park/rescope/deny + ParkOutcome live in loop_driver.rs; the budget clock pauses while parked) Date: 2026-06-25 Context owner: agentic-epp-autoresearch Related: ADR-0001 (freeze the judge; the solution/evaluation split), ADR-0002 (the broker, pending_approval, the re-scope), the control bridge (crucible/src/control.rs), the engine re-scope path (run_loop's take_rescope → run_baseline → new fingerprint segment), the escalate marker hook (crucible/src/escalation.rs).

Context

ADR-0002 decision 5 says a judge-changing request can open an async approval (a draft PR or a control-bridge ask) and "the loop records the ask and continues / parks; the human approves a batch out-of-band." It specified the approval — open a PR, poll for /approve-capture — but not what the run actually does between the ask and the answer, nor how it resumes. Today the only built behavior is the degraded floor: BROKER_DEGRADED=1 → judge-changing misses escalate-halt (Resolution::Escalate). That is correct when there is no approval surface, but it is not the wait-and-resume we actually want when there is one.

Two things need pinning:

1. Where the wait lives. The agent turn is ephemeral — a per-turn OpenShell sandbox, torn down when the turn ends. It cannot "sit and wait"; making it spin-poll check_capture inside a turn burns tokens on a paid sandbox to do nothing. The long-lived thing is the loop (crucible, the pod's entrypoint process). The wait is the loop's.
2. Whether to block the whole run. Parking the run idles a pod on a human. Sometimes that's right (the pending regime is the agent's only path); sometimes it's waste (the agent has other frozen-regime hypotheses to test meanwhile). Only the agent has the context to know which.

Decision

The broker decides whether a regime change is allowed (privilege + evaluation, server-side, ADR-0002). The agent decides how to spend the wait (solution-space, ADR-0001). The loop owns the wait and the resume. Concretely:

1. The wait is the loop's, not the turn's. On pending_approval, the agent records the ask and ends its turn — it never spin-waits in the sandbox (the request-trace skill already says "async by design; do not spin-wait a whole turn"). The loop reads the ask after the turn and acts.

2. The agent chooses the wait mode, written into the marker it leaves:

   • block — "this regime is my only path." The loop parks until the approval resolves.
   • continue — "I have other frozen-regime work." The loop keeps iterating; the re-scope lands whenever approval arrives. This is a solution-space call (how to use time), not an evaluation one — so it is the agent's, exactly as ADR-0001 draws the line. The broker's yes/no is unchanged and still server-side.

3. One wake mechanism for both modes. Whichever mode, the broker (it holds the PR + the forge token; its process is long-lived even though the HTTP transport is stateless — ADR-0002 note) watches the approval. On /approve-capture it sends a rescope{regime} command to crucible's control bridge (localhost:<control-port>), becoming a control-bridge client. crucible's existing re-scope path drains the pending re-scope at the next iteration head: re-baseline at the new regime, bump the harness fingerprint, open a new comparable segment, resume. The mode only gates whether the loop idles or keeps iterating while the watch runs — it is not a second system. crucible polls nothing; it parks (or not) until a rescope arrives.

   Two async phases, one terminal signal. There are actually two waits after pending_approval: (1) the human (approval) and (2) the provisioning — the GPU capture that produces the calibrated trace + MOCK_* knobs. The loop's park waits for a single terminal outcome; the watcher must only send the rescope once provisioning is ready, not merely approved (else the re-baseline measures an uncalibrated regime). So the correct watcher flow is poll approval → submit capture → poll capture until ready → send rescope. The capture submit/poll is the capture-wait boundary in watch.rs, backed by the Kueue/trace backend. Because the loop already parks for the terminal signal, only that boundary changes as the backend evolves; the simplest form has the watcher send rescope on approval. The terminal outcomes the park waits on: rescope (ready → resume), deny (rejected → resume-frozen or escalate), or a --max-park timeout (→ treated as deny).

4. The decision rides on a marker (mirror of the escalate hatch): the agent writes PROVISIONING_PENDING.json in its workspace; the loop reads + consumes it post-turn with a take-style hook beside escalation::take. Schema below.

5. Budget treatment follows the mode. block pauses the cost/time clock — an idle wait on a human must not count against --max-cost / --max-time. continue keeps spending, because the agent is doing real work. This falls out of which loop state we're in.

6. Two interchangeable human channels, same rescope (ADR-0002's two approval backends):

   • Forge (headless): comment /approve-capture <trace_id> on the draft PR. The broker's DraftPr poll catches it. Approve from anywhere; the parked pod resumes minutes later.
   • Control bridge (attended): the ControlBridge channel surfaces the ask as a viewer event; the operator watching crucible view --pod approves with a keystroke (a), the same exec/control path s/x already use (crucible/src/tui/remote.rs). Immediate.

7. Deny / timeout is not a harness failure. On denied or a max-park timeout, the loop discards the regime change and resumes in the frozen regime with a note — a human declining a goalpost move is a normal outcome, not an escalation. The exception: if the agent declared block because it had no fallback, a denial leaves nothing to do, so it escalate-halts (ESCALATION.json, exit 2) — the honest "I am genuinely stuck" signal.

What this is NOT

• Not the agent polling inside a turn. Turns are ephemeral and paid; the loop waits, the agent does not.
• Not the broker (or the harness) deciding how to spend the wait. That's the agent's solution-space call — the whole point of letting it choose block vs continue.
• Not a new evaluation path. Resume is the existing re-scope (new baseline + fingerprint + segment); the approval just feeds its pending slot. The judge is never edited in place.
• Not a replacement for escalate. Escalation stays the floor for the un-grantable and the no-approval-surface degraded case (ADR-0002 decision 6). This ADR is the grantable-but-pending case: a human will answer, the run just has to wait for it.

The marker

PROVISIONING_PENDING.json, written by the agent in its workspace (consumed + deleted by the loop, so a stale file can't re-trigger on resume):

{
  "trace_id": "model=Qwen/Qwen3-0.6B;c=48;p=8;mt=8;lt=0;lf=0.0000",
  "regime":   { "model": "Qwen/Qwen3-0.6B", "concurrency": 48, "prefixes": 8,
                "max_tokens": 8, "long_tokens": 0, "long_frac": 0.0 },
  "handle":   "https://github.com/<org>/<repo>/pull/NNN",
  "mode":     "block"
}

mode defaults from behavior when the agent omits it: a turn that produced a measurable CANDIDATE.md (a frozen-regime fallback change) implies continue; a turn that produced only the ask implies block. Explicit beats implicit — the skill asks the agent to declare it.

Architecture sketch

flowchart TD
    request["agent calls request_trace"] --> pending["pending_approval with handle"]
    pending --> marker["write PROVISIONING_PENDING.json<br/>with mode, then end turn"]
    marker --> take["loop calls take_pending after turn"]
    take -->|"mode = block"| park["park<br/>pause budget clock and emit Parked event"]
    take -->|"mode = continue"| iterate["continue in frozen regime"]

    pending --> watch["long-lived broker watches approval"]
    watch --> approved["approval command received"]
    approved --> rescope["send rescope with new regime"]
    rescope --> bridge["Crucible control bridge"]
    bridge --> head["take_rescope at next iteration head"]
    park --> head
    iterate --> head
    head --> baseline["re-baseline, bump fingerprint,<br/>and start a new segment"]
    baseline --> resume["resume loop"]

Consequences

• Positive: the run can genuinely wait for a human and resume, headless or attended, without the agent burning paid sandbox time spinning. The honest "is this worth blocking the whole run?" call sits with the only actor that has the context — the agent.
• Positive: almost entirely reuse. The wake is the existing re-scope path; the transport is the control bridge; the marker is the escalate pattern; the channels are ADR-0002's two backends; the viewer keystroke is the s/x path. The genuinely new code is small: the marker + take hook, a parked loop state (a budget-paused pause that a rescope wakes), the broker's PR-watcher → control-bridge sender, and the viewer a key.
• Negative / cost: a parked pod idles on a human (cheap — no tokens, paused budget — but a real pod-hour). A max-park timeout bounds it. The broker now also runs a background watcher with state across requests (fine: its process is long-lived; only the HTTP transport is stateless).
• Negative / cost: continue mode interleaves a frozen-regime segment with a later re-scoped one in the same run; the session log already models segments, so the story stays readable, but a reader must respect segment boundaries (scores across one are not comparable — ADR-0001 safeguard 2).

Design notes

• Max-park timeout + denial default: --max-park <dur> (empty = wait indefinitely) bounds the wait; a timeout is treated as a denial. A block denial escalate-halts (the agent had no fallback by definition of block); a continue denial just stays in the frozen regime with a note. This lives in loop_driver.rs (ParkOutcome) + the deny control command.
• Broker → control-bridge address: crucible injects BROKER_CONTROL_ADDR=127.0.0.1:<port> when it spawns the broker; same-pod loopback is the trust boundary (broker.rs / watch.rs).
• Control-bridge grammar at scale: the approve/deny commands handle a single ask in flight; a queue of asks plus viewer keys (a/d) and dedupe on re-press extend the same channel.
• Resume fidelity in block: a long park may outlive the rig's last-good snapshot or the sandbox gateway's warm state, so re-baseline after a wake must re-establish both cleanly.

Principle, restated

The host decides what may be granted; the agent decides what to do while it waits; the loop holds the door open and walks through it when the human says yes. Waiting is solution-space — the agent's to spend — but the gate it waits on, and the re-scope it resumes into, stay the host's.

ADR 0004: Core-loop state model — context struct + typed iteration, not whole-loop typestate

Status: Accepted (implemented 2026-06-25) Date: 2026-06-25 Context owner: agentic-epp-autoresearch Related: ADR-0001 (freeze the judge), ADR-0003 (park/rescope/deny gates), the loop in crucible/src/loop_driver.rs (run_loop), and the run/loop_driver split.

Context

run_loop has crossed a complexity inflection. It now threads ~13 mutable bindings across iterations — rows, spent, best_score, baseline_score, baseline_total, best_snap, regime, fp, kept_shas, solved_any, escalated, pending_block, parked_total — and exits through five breaks (stop, escalate, denied-escalate, budget, solved) plus two continues (parked-block, apply-failed). ADR-0003 added park / rescope / deny / provisioning gates on top of the existing apply → measure → decide → keep/discard core. The next addition gets risky: the invariants (re-baseline mutates a coherent set of fields together; you can't measure before apply, can't keep without a Reading, can't measure on top of an escalation) live only in the reader's head.

A typestate pattern was proposed. This ADR decides how much typestate, and where — because the naive answer (model every loop phase as a type) is the wrong shape for this loop.

Decision

Type the linear sub-protocol; keep a plain state machine with an explicit context for the cyclic shell. Concretely, three changes, in risk order:

1. A Run context struct bundling the per-run threaded state, with the segment-scoped fields factored into a Segment (because the re-scope mutates exactly those, together):

   #![allow(unused)]
   fn main() {
   struct Run {
       rows: Vec<Row>,
       spent: f64,
       kept_shas: Vec<String>,
       solved_any: bool,
       parked_total: Duration,         // ADR-0003 budget-pause accumulator
       pending_block: Option<PendingProvisioning>,  // ADR-0003
       segment: Segment,
   }
   /// Everything a re-scope replaces atomically (ADR-0001 safeguard 2 / ADR-0003).
   struct Segment {
       regime: String,
       fingerprint: String,
       baseline_score: f64,
       best_score: f64,
       baseline_total: u64,
       best_snap: Snapshot,
   }
   }

   A re-scope becomes run.segment = Segment::baseline(world, judge, regime)? — one assignment that can't half-update the goalpost. This is the single biggest readability win and is independent of the rest.

2. An explicit LoopExit enum replacing the escalated: Option<_> flag + scattered breaks, so there is one place that enumerates how a run can end, mapped to Outcome:

   #![allow(unused)]
   fn main() {
   enum LoopExit { Finished, Solved, Budget, Stopped, Escalated(Escalation) }
   }

3. Typestate the inner iteration — the one genuinely linear protocol with real ordering invariants. The candidate moves Proposed → Applied → Measured, and only a Measured can be decided/kept (so "keep without a reading" and "measure before apply" stop compiling):

   #![allow(unused)]
   fn main() {
   struct Iteration<S> { it: u32, state: S }
   struct Proposed;                       // agent staged a candidate in the world
   struct Applied;                        // world.apply() succeeded
   struct Measured { reading: Reading, note: String, diff: String, diffstat: String }
   
   impl Iteration<Proposed> {
       fn apply(self, world: &dyn World) -> Result<Iteration<Applied>, ApplyFailed>;
   }
   impl Iteration<Applied> {
       fn measure(self, judge: &dyn Judge, ctx: &MeasureCtx, p: &Paths) -> Result<Iteration<Measured>>;
   }
   impl Iteration<Measured> {
       fn decide(self, judge: &dyn Judge, best: f64) -> (Row, Verdict);  // keep path holds the Reading by construction
   }
   }

   The outer for it in … stays a loop over &mut Run; the gates (park / rescope / deny / escalation / provisioning) run before an iteration enters Proposed, and decide whether it does.

What this is NOT

• NOT whole-loop typestate (Loop<Parked>, Loop<Rescoping>, … with self-consuming transitions). Typestate pays off when each state owns distinct data + capabilities (a connection, a builder, a resource protocol). This loop is the opposite: nearly all state is shared and threaded through every phase, so a per-phase type would carry one big shared context + a marker — the ceremony of typestate with none of the safety. The cyclic shell wants a state machine with an explicit context (#1/#2), not types-as-states.
• NOT a behavior change. This is a structural refactor: identical control flow, identical session log, identical exit codes. The existing tests (park outcomes, resume, fingerprint, exit codes) are the regression net and must stay green untouched.
• NOT a new module boundary. It stays in loop_driver.rs (+ maybe an iteration.rs sibling for the typestate); the run/loop_driver split is preserved.

Consequences

• Positive: the re-scope can't half-update the goalpost (the Segment swap is atomic); the iteration protocol's ordering invariants are compile-enforced; run_loop's signature and body shrink to "drive gates, then run a typed iteration, fold the result into Run"; the ways a run can end are enumerated in one enum.
• Positive: the next gate (e.g. the ADR-0003 capture-wait, or a new domain's phase) plugs into a named context + a typed iteration instead of adding a 14th threaded binding.
• Negative / cost: more types (the Iteration<S> zoo, the marker structs) and the self-consuming transitions are slightly more verbose than inline statements; worth it for the protocol, which is why we scope typestate to only the iteration.
• Negative / cost: a refactor of a core file — it must land as one reviewed change with the test suite green, not a drip of edits.

Implementation notes

The change lands in three independently testable steps — the Run + Segment struct (threaded bindings moved in, re-scope becomes one assignment), the LoopExit enum (replacing the escalated flag + scattered breaks, mapped to Outcome), then the Iteration<S> typestate (apply/measure/ decide extracted into the typed protocol) — each keeping cargo test -p crucible green (notably the park/resume/exit-code tests).

• LoopExit::Escalated is a unit variant, not Escalated(Escalation). The escalation is reported and the world rolled back eagerly at each break site, and the two sites report differently (the agent path calls Reporter::escalation; the denied-block path emits a note). Centralizing the report post-loop to consume a carried payload would change the session log, which this refactor must not do, so the carried value would be dead. The variant only marks the run "needs human" for the exit code; Outcome.escalated = matches!(exit, LoopExit::Escalated).
• Segment::baseline(world, judge, goal, regime) owns the fingerprint (vs the ADR sketch's separate fp binding), so opening a segment is one call for both the initial baseline and a re-scope. The resume path constructs its Segment directly from the restored scores.
• Decided { row, verdict, reading } is what Iteration::<Measured>::decide returns (vs the sketch's (Row, Verdict)): the keep path needs the reading's score (→ best_score) and note (→ snapshot label), so the typed step hands it back by construction.

Open questions

1. Where the gates live. Do park / rescope / deny / escalation / provisioning stay inline in run_loop, or become a small enum Gate { Park, Rescope, … } step? Lean inline until a third gate makes the sequence unwieldy.
2. Snapshot type. best_snap is a String today (the world's opaque snapshot handle). The Segment struct is a good moment to newtype it (struct Snapshot(String)).
3. Resume path. run_loop's resume branch reconstructs the same state; it must build a Run from ResumeState cleanly — confirm the Segment factoring doesn't complicate the replay.

Principle, restated

Type the protocol, not the process. The iteration is a short linear contract — make its illegal orderings unrepresentable. The loop is a long cyclic process over shared state — give that state a name and its exits an enum, and leave the rest a plain, readable for.

ADR 0005: Engine-side EPP builds, brokered to the agent over MCP

Status: Accepted; implemented (builds run buildah-unified on the loop pod — kaniko was dropped; open questions 1–2 were resolved by ADR-0008, see the annotations below) Date: 2026-06-26 Related: ADR-0001 (freeze the judge), ADR-0002 (the agent asks, the host holds the keys), the World::apply trait, the epp-mcp broker, Containerfile.epp-sandbox, the #1109 TTFT domain (crucible.ttft.toml).

Context

#1109 is the first domain that needs in-loop EPP code rebuilds. The perf domain dodged this — it is config-tuning (apply scorer weights onto the live EPP, no rebuild). When the #1109 loop ran, the sandboxed agent could not make progress and doom-looped, because:

1. It can't build. build-image shells out to an out-of-sandbox remote-build script, which isn't in the sandbox; and a kaniko build needs build-namespace kubectl, a push credential for the candidate registry, and egress that the deny-by-default sandbox does not have. The agent correctly concluded it had no container builder available and gave up.
2. It can't see the measurement. epp-sandbox:spike-1 predates bench --ttft, so the agent's bench is the old prefix-cache bench — it can't reproduce the judge's TTFT metric, so it reverse-engineers the binary (strace/strings) and spirals.

Putting build creds in the sandbox (the "provision the sandbox" option) enlarges the attack surface and fights the World trait, which already says build+deploy is apply's job (engine-side).

Decision

Move build+deploy off the agent to the loop pod (engine-side), and expose it to the agent as broker (MCP) tools — the ADR-0002 pattern, the agent asks, the loop pod holds the keys.

Why MCP rather than only [world].apply_cmd: a code loop needs compile feedback within the turn. A purely post-turn apply build means a typo fails the whole iteration with no feedback. The agent must trigger a build, get errors, and fix them before its turn ends. So:

• Primary path — broker tools. Add build_epp (and deploy_candidate, or a combined build_and_deploy) to epp-mcp, alongside request_trace. The agent calls them over the existing bridge (host.containers.internal:8849); the loop pod runs the container build (buildah today; the original design said kaniko) with its own push credential for the candidate registry, and returns a Resolution: {ok, image_ref} | {compile_error, log}. The agent iterates on errors, then deploys.
• Backstop — [world].apply_cmd. Keep an engine-side apply_cmd that ensures the latest built candidate is the deployed one before the judge measures (idempotent; a no-op if deploy_candidate already ran). The manifest already supports [world].apply_cmd and CommandWorld::apply runs it.
• The agent's sandbox loses build creds entirely. build-epp/deploy-candidate direct tools are dropped; the agent only edits Go + calls the broker.

Components to build

1. Loop-pod build capability (in the crucible loop image): a self-contained buildah build + push (no dependency on an out-of-sandbox remote-build script) and a scoped push robot credential for the candidate registry (mounted secret).
2. epp-mcp build tools build_epp / deploy_candidate, mirroring request_trace's admit/return shape. They run the loop-pod build and surface the compile log on failure.
3. Workspace sync (the crux). The loop pod must build the agent's current edits, which live in the sandbox mid-turn (the driver copies the workdir in/out per turn, it is not shared). Proposed: the agent commits its branch and pushes it to the loop pod's workspace as a git remote over the relay; the broker builds from that ref. (Reuses git-memory; no new driver plumbing. Alternative: a driver "mid-turn checkpoint" that syncs /sandbox/epp → loop workspace on the build call.)
4. Rebuild epp-sandbox with current epp-tools (so the agent's read-only bench --ttft matches the judge) + bake bench.rs source (or a TTFT measurement skill) so the agent understands the metric instead of reverse-engineering it. Drop the direct build/deploy tools.
5. Manifest wiring: broker build tools enabled in [agent.broker]; [world].apply_cmd as the ensure-deployed backstop; the candidate registry + egress allowlisted.

Open questions

1. Build latency. Resolved by ADR-0008: the immutable compose bakes the toolchain and a warm GOMODCACHE/GOCACHE into the base, so the per-iteration build is an incremental build against a hot cache, not a cold from-scratch image build.
2. Workspace sync mechanism. Resolved by ADR-0008: the agent's branch push is the CoW overlay; the broker builds base@SHA + overlay.
3. Cred scoping — a push robot token limited to the candidate registry, not a broad registry cred.

Consequences

• Positive: build creds never enter the sandbox (ADR-0001/0002 aligned); the agent gets real compile feedback; the loop pod is the one trusted place for push creds; apply_cmd guarantees the measured candidate is the built one; code-change domains become first-class, not just config tuning.
• Negative / cost: real infra (push cred, RBAC, a builder in the loop image — kaniko as designed, buildah as shipped — the workspace-sync mechanism, a sandbox rebuild) and per-iteration build latency.

ADR 0006: Profiler support over MCP — a generic capability, pprof for the EPP

Status: Accepted; implemented (generic profiler in crucible-broker; pprof backend for the EPP, torch-trace for vLLM) Date: 2026-06-26 Related: ADR-0002 (the agent asks, the host holds the keys), ADR-0005 (engine-side build + the broker measure tool), the epp-mcp broker, the #1109 TTFT domain (crucible.ttft.toml).

Context

#1109 asks the agent to cut the EPP's contribution to TTFT. But the gate — end-to-end bench --ttft through the live service — cannot isolate the EPP, and cannot resolve small EPP wins:

• The measured path is client → envoy-proxy → (ext_proc) → epp → envoy → backend pool → first token, all three in one pod behind svc:80. The MOCK_PREFILL_TIME_PER_TOKEN_MS=0 scoping zeroes the backend compute, but the ~2.2 s TTFT still includes Envoy's 220 KB body handling, the data-plane transfer (220 KB × c=150 ≈ 33 MB/hop), backend ingest of that body, and queueing at c=150.
• Result: every micro-optimization the agent tried (alloc, JSON, GC, gRPC windows) came back "within noise." The agent was guessing at the hot path and optimizing a low-single-digit-% sliver that the c=150 measurement noise swamps. The TTFT gate is right for scoring a candidate; it is the wrong instrument for finding what to change.

A profile fixes both problems at once: pprof attributes CPU / allocations / contention to specific EPP functions and source lines — isolated to EPP code by construction, and far more sensitive than a noisy end-to-end delta. The EPP already ships pprof: --enable-pprof defaults to true (pkg/epp/server/options.go), handlers are registered on the metrics server (runner.go, pkg/common/observability/profiling/pprof.go → net/http/pprof), served on :9090. The catch: that port runs controller-runtime secure metrics, so /debug/pprof/ is token-gated (a bare GET returns Unauthorized), and the sandboxed agent can reach neither the in-cluster ClusterIP (egress wall) nor has go tool pprof.

Decision

Add a generic Profiler capability to the broker — mediated "observe the candidate's internals" — with the concrete profiler as a domain-configured backend, exactly the way measure shells the domain's BROKER_MEASURE_CMD. The agent asks; the loop pod (cluster reach + token + tooling) captures and analyzes and hands back text. The agent never touches the endpoint or knows the backend.

Generic (broker core) — two tools that shell domain-provided commands:

• profile{kind, seconds} — capture a profile of kind (backend-defined) over seconds. The broker shells BROKER_PROFILE_CAPTURE_CMD (env: $KIND, $SECONDS, $OUT path), stores the result on the loop pod under the build storage root, returns a handle + short summary.
• profile_query{query} — run query against the stored profile. The broker shells BROKER_PROFILE_QUERY_CMD (env: $PROFILE, $QUERY), returns text. The query vocabulary is backend-defined. Read-only by construction (the backend command is a fixed analyzer; the broker applies a deny-list so a query can't start a server or write files).

The broker knows nothing about pprof, Go, or the EPP — only "capture a profile, then query it," delegated to configured commands. This is the same split as forge (generic build engine) vs the EPP build wiring, and measure (generic) vs BROKER_MEASURE_CMD (domain).

Domain implementation (#1109 = pprof against the Go EPP): the EPP domain sets the two commands — CAPTURE_CMD = an authed GET of epp:9090/debug/pprof/<kind>?seconds=$SECONDS; QUERY_CMD = go tool pprof -<query> $PROFILE. So for EPP, kind ∈ {cpu,heap,mutex,block} and query is pprof's command surface: top20, list HandleRequestBody (per-line ns/alloc in the real source — the killer), peek json.Unmarshal, traces, tree. Another domain (a Python service, a native binary) swaps in py-spy / perf / eBPF via the same two env commands; the broker tools and the agent-facing surface are unchanged.

The loop becomes profile → read the hot lines → fix them → re-profile to confirm the hotspot shrank → measure to score. Profile to find (isolated, sensitive); the gate to score.

Implementation

• Broker tools (epp-mcp/src/profile.rs, registered in server.rs, exported in lib.rs): generic profile{kind,seconds} + profile_query{query,handle?}, shelling BROKER_PROFILE_CAPTURE_CMD (env $KIND/$SECONDS/$OUT) / BROKER_PROFILE_QUERY_CMD (env $PROFILE/$QUERY). Captures stored under <FORGE_STORAGE_ROOT>/profiles with a latest marker; unconfigured → disabled (mirrors measure). Read-only deny-list on the query (no $(), no pipes/redirects, no -http/-web/-output/-svg/...), kind sanitized, handle path-traversal-guarded. No pprof/Go/EPP knowledge in the tool code. Unit-tested (deny-list, kind validation, traversal).
• EPP/pprof backend (swappable domain wiring): capture adapter domains/<domain>/tools/epp-pprof.nu (port-forwards :9090, mints the reader-SA token, authed-GETs /debug/pprof/$KIND → $OUT, same pattern as epp-metrics.nu); the two BROKER_PROFILE_* commands in the loop's deployment manifest (epp-pprof ... / pprof -$QUERY $PROFILE); the standalone pprof analyzer baked into Containerfile.crucible via a CGO-free Go stage.
• RBAC (the access path — chose RBAC over an unauth :6060): extended the epp-metrics-reader ClusterRole in rig-up.rs to add /debug/pprof + /debug/pprof/* nonResourceURLs, so the same reader token controller-runtime already authorizes for /metrics also covers pprof on that mux.
• Advertised: the profile skill (domains/<domain>/skills/profile/) + a "profile before you guess" note and the two tools in prompts/research.md.

Validated live:

• Access works. The reader token authorizes /debug/pprof end-to-end — /metrics, /debug/pprof/, and /debug/pprof/profile?seconds=1 all return 200 once the epp-metrics-reader ClusterRole carries the pprof nonResourceURLs (this ADR) and the EPP service account holds system:auth-delegator (required for :9090 auth to function at all — without it /metrics 500s too; rig-up creates the binding). controller-runtime authorizes /debug/pprof for a metrics-style nonResourceURL grant — same mux, per-path authz, so the URLs must be granted explicitly.
• Capture under load works and the signal is real. A 20 s CPU profile during a measure (TTFT 2275 ms, c=150) showed the EPP busy only ~24% of one core — so most of the 2.2 s is NOT EPP CPU (it's transfer/queue/envoy, exactly the isolation finding). Within the EPP's CPU the hot path is: encoding/json validation/parse of the 220 KB bodies (checkValid/unquoteBytes/appendString, ~15–20% cum), GC churn from those bodies (gcDrain 17% cum, memclrNoHeapPointers 9% flat, scanobject/memmove), and prometheus/common/expfmt text parsing (~15% cum — the EPP scraping backend metrics). None of this is the routing/scheduling logic the agent kept guessing at; the profile names the real cost. top / peek / traces / tree all work from the symbolized profile.

list=<Func> per-line cost is resolved by mapping the profile's source paths to the candidate's staged tree. The EPP is built without -trimpath (Dockerfile.epp WORKDIR /workspace), so the profile embeds /workspace/pkg/epp/... paths, while forge stages the candidate repo root at /var/lib/forge/ctx. BROKER_PROFILE_QUERY_CMD now carries -trim_path=/workspace -source_path=/var/lib/forge/ctx, so pprof strips the build prefix and finds the file under the staged ctx: list=<Func> shows real per-line ns/alloc for the EPP's own pkg/epp/... source once a build_epp has run this turn (ctx is the tree the deployed candidate was built from). top/peek/traces/tree work without a build. Deliberate non-goal: list on stdlib/vendored frames (encoding/json, prometheus/common/expfmt) doesn't resolve — only the EPP's own source is staged, and the agent edits EPP code, not the stdlib, so the actionable surface is covered. Verified the embedded paths against a live capture (/workspace/pkg/epp/datalayer/collector.go).

Generic multi-target profiler and the vLLM backend

Three changes make the broker genuinely component-agnostic and ready for composite (multi-domain) rigs, all in crucible-broker/src/profile.rs (the profiler lives in the extracted broker crate):

• Capture extension externalized (BROKER_PROFILE_EXT). The broker named handles <kind>-<nonce>.pprof — a pprof-ism in a supposedly generic core. The extension is a domain concern (the broker stores the bytes the capture writes and never interprets them); a format-sniffing analyzer wants the right name. Now domain-set, neutral default prof. The EPP sets pprof; the vLLM torch-trace backend sets json.gz.
• Multiple profile targets (BROKER_PROFILE_TARGETS). A composite deployment (vLLM + EPP) has more than one profileable component, each with its own image, way in, and profiler. Profiling is now keyed by a named target: BROKER_PROFILE_TARGETS="vllm epp" plus per-name BROKER_PROFILE_<T>_CAPTURE_CMD / _QUERY_CMD / _EXT. profile/profile_query take an optional target (omit it for a single-component rig; name it — e.g. vllm or epp — when there are several; an omitted target with several is an error that lists them). Captures nest under profiles/<target>/ with a per-target latest. A single-component rig still just sets the un-prefixed BROKER_PROFILE_* (one implicit default target) — fully backward-compatible, no EPP-run change.
• vLLM GPU-trace backend reconciled to the contract. domains/<domain>/tools/vllm-profile.nu (capture)
  • vllm-analyze.nu (kernel→source correlation, inverting neuralmagic/ai_auto_perf_analysis) now honor the same $KIND/$SECONDS/$OUT // $PROFILE/$QUERY surface as epp-pprof, so the fold-in is pure env wiring — no broker core change. $SECONDS is unreferenced for vLLM (a GPU window is prompt-count bounded). The gate-workload question is settled serve-path: both gate and profiler drive the same frozen VLLM_BENCH_* workload via vllm bench serve. Live GPU validation requires the analysis-layer image; the contract is tested at its boundaries.

The agent-facing tools and the per-component domain wiring are unchanged in shape; the broker just routes by target name. This is the same generic-core / domain-backend split as forge and measure.

Consequences

• Positive: the agent optimizes the measured hot path instead of theorizing in the noise; the isolation/sensitivity problem the TTFT gate cannot solve is solved by direct observation; composes with measure/build/deploy/cap; generic, so it strengthens the case for the broker extraction.
• Negative / cost: a small RBAC grant, a binary in the loop image, two more broker tools, and a rebuild. Profiling adds minor overhead to the EPP during capture (CPU pprof is sampled — low). The pprof pass-through needs a sane command deny-list so the agent can't start servers or write files.

ADR 0007: Verify the gate isolates the target before spending a loop — the metric that misframed #1109

Status: Accepted Date: 2026-06-26 Related: ADR-0001 (automate the setup, freeze the judge — world-scoping is setup), ADR-0006 (the Profiler — found the EPP CPU is incidental), the onboarding issue-ranking pipeline (ranks issues by measurability), the #1109 TTFT domain.

Context

#1109 ("long-context requests see >1s TTFT overhead in the EPP request path") ran as a loop and produced two valid-but-flat candidates (collector caching, body re-chunking; both within c=150 noise). ADR-0006's profiler showed the EPP is ~24% of one core busy with GC incidental. We then ran the cheap, decisive check: the EPP's own instrumentation, delta'd around a 600-request measure.

The EPP's actual work is ~20–30 microseconds per request. The 2.26 s is the ext_proc handler sitting blocked, waiting on Envoy's 220 KB body buffering + the data-plane transfer + the mock backend + c=150 queueing — none of which the EPP controls.

How #1109 got misframed. llm_d_router_epp_request_duration_seconds ≈ 2.26 s ≈ the end-to-end TTFT, because that metric measures the handler's wall-clock attachment time (mostly blocked-wait), not EPP compute. Reading it as "the EPP request path has >1s overhead" is the trap. The compute metric (scheduler_e2e = 17 µs) tells the truth: a 17 µs decision wrapped in 2.26 s of waiting on everything else. There is no >1s EPP overhead to remove.

Root cause is world-scoping (ADR-0001's setup column), not the EPP. The 220 KB-body mock scoping was chosen to stress the EPP path, but it drowns the EPP in Envoy transfer cost — the scoping defeats the isolation it was meant to create. The gate is end-to-end and cannot attribute a result to the EPP, so every EPP-side candidate ties in the noise by construction.

Decision

Onboarding must verify the gate actually isolates the target — and that the target's contribution is material — BEFORE spending a loop. Measurability (can a frozen judge decide success?) is necessary but not sufficient; the issue-ranking pipeline ranks for it but #1109 passed measurability while being unwinnable. Add an isolation pre-flight to the judge-validation gate:

1. Attribute the gate. Before the first scored loop iteration, measure the target's own contribution to the gated metric using the target's instrumentation (here: the EPP's scheduler_e2e / plugin_duration vs the end-to-end request_duration), or a direct isolated harness. If the target is a sub-percent sliver of the gated number, the loop cannot win — escalate the issue as mis-scoped, don't run it.
2. Distinguish work from wait. A *_request_duration that tracks the end-to-end number is a handler-attachment time, not compute. Prefer a compute/decision metric (or a profile, ADR-0006) for attribution; never infer "the component is slow" from a duration that includes downstream wait.
3. Re-scope or escalate. A real EPP-TTFT experiment needs a world where the EPP's contribution is material: an EPP-heavy workload (large prefix trees, many pods, expensive scoring) or smaller bodies so transfer isn't the whole budget. Absent that, the correct output is an escalation.

#1109 is therefore escalated (no EPP-side win exists as gated), and the planned from-scratch ext_proc isolation harness is shelved for #1109 — the pre-flight already answered what it would measure (~30 µs). The harness stays a valid general instrument for an issue where the EPP request path genuinely is the cost.

Consequences

• Positive: a ~10-minute instrumented pre-flight kills an unwinnable run before it spends a paid loop; the autoresearch stack surfaces a mis-scoped problem instead of grinding (the system working); the check is generic (any target with self-instrumentation) and slots into the judge-validation gate the controller needs; it converts "the loop keeps tying in noise" from a mystery into a one-number verdict.
• Negative / cost: another gate in onboarding (the pipeline must learn to read a target's attribution metric, or stand up the isolated harness, before greenlighting); a per-target choice of "which metric attributes the gate," which isn't always obvious.
• Carries forward: this is the empirical case for the world-scoping discipline — the judge can be perfectly frozen and still measure the wrong thing if the world doesn't isolate the target.

ADR 0008: Domains as immutable composes — the rpm-ostree model for the frozen world

Status: Accepted; partially implemented (the baked-workspace / no-runtime-clone discipline and per-domain images are in use; the compose-digest machinery and the compose reconciler are not built) Date: 2026-06-26 Related: ADR-0001 (freeze the judge AND the world — this is the world half made concrete), ADR-0005 (engine-side per-iteration rebuilds — this ADR composes the base those builds layer onto, see Design details), ADR-0006 ([[workspace.inject]] — the frozen-judge reset, i.e. the ostree "reset /usr" primitive), ADR-0007 (judge-validation pre-flight), the onboarding issue-ranking pipeline, the controller follow-up.

Context

Launching the first two local go-test domains (#1489 T1 harness, #1474 T0 bug) surfaced a cluster of failures that looked unrelated but share one root cause:

1. libgit2 has no TLS in the runtime image → setup can't clone.
2. image Go was 1.23 but upstream go.mod requires ≥ 1.25.11.
3. the #1489 harness was authored/validated against main HEAD but didn't match the signatures of the code the loop actually compiled (getBlockHashes shape, PerPromptHashes).
4. the engine skipped the clone because a stale baked workspace existed, so the loop ran against an old commit (go-redis v9.11.0 vs main's v9.20.0) — and the main-authored harness didn't compile against it.
5. the first measure does a cold go mod download at run time — slow, network-dependent, the cause of the fast "go test failed" baseline crashes.
6. the agent sandbox has no Go toolchain, so the agent bootstraps one mid-turn to test its own edits.

Every one is runtime mutation of the environment drifting from what we authored against. The judge was frozen (ADR-0001), but the world it runs in was not: the loop clones a moving target ([repo].ref = "main"), downloads deps on the fly, and we patch the running pod (emptyDir, configMap) to compensate. A frozen judge measuring an unfrozen world measures different code on every run — which also makes cross-run memory meaningless (two runs of "the same" domain compile different bytes).

Decision

Treat each autoresearch domain as an rpm-ostree-style immutable compose. The world — upstream source@SHA + the matched toolchain + pre-fetched dependencies + the frozen judge harness — is composed ahead of time into a single content-addressed image. The loop runs that image; it does not clone, fetch, or get patched at run time. To change the world you re-compose and rebase to a new digest, atomically. The agent's edits are the only mutable layer.

The key inversion: a baked workspace is not a bug — it's the point. The #4 failure wasn't that the engine skipped the clone; it was that the baked tree was a stale accident instead of the deliberate, pinned, harness-matched compose. Skip-clone is correct against a frozen compose. So the fix isn't "always clone fresh main" (that re-introduces drift) — it's "bake the matched commit and don't clone."

Pin point: the harness is validated against a specific upstream commit (the compose's source pin), whose go.mod fixes the toolchain (e.g. go 1.25.11). That commit is the compose input for #1489/#1474; the toolchain version is read from its go.mod, the deps are go mod download-ed into the image at compose time.

Why this kills the whole cluster: with no runtime clone, #1 (TLS) is moot; the toolchain is in the base, so #2 and #6 vanish; source and harness are composed together, so #3 and #4 can't drift; deps are pre-fetched, so #5 never happens. Four mismatch failures and two capability gaps collapse into one build-time discipline.

Implementation sketch (a follow-up, not this ADR)

• A per-domain compose step (Containerfile / buildah): FROM the matched-toolchain base → clone [repo]@SHA → go mod download (or vendor) → inject the frozen judge → emit a content-addressed image. Validated at build time (it compiles + emits a valid baseline JUDGE_CONTRACT), which is exactly the ADR-0007 isolation/validation pre-flight moved left into the compose.
• Layering, ostree-style: a shared repo@SHA + deps + toolchain base across same-repo domains, plus a thin per-domain layer (harness + goal + manifest), so onboarding the next issue is a small top layer.
• The loop runs with setup_cmd as a no-op (workspace pre-populated by the compose); the agent's sandbox is the composed image, edits as its writable container layer; measure resets the frozen judge.
• A planned compose reconciler (distinct from ADR-0011's PR-feedback controller) gains a "compose + validate domain image" reconcile step that replaces the live setup/clone dance — groomed → composed → validated → deployed.

Design details

Scope — what this does and doesn't freeze

This ADR freezes the build/world-definition layer, and only that. Be precise about the boundary:

• In scope (frozen by construction): upstream source@SHA, the toolchain (hard-frozen — see below), the baseline dependency set, and the judge harness — every input that was previously resolved at run time. Two runs of the same (compose input digest, overlay commit) compile identical bytes. This closes the entire launch-failure cluster (#1–#6 in Context).
• Out of scope (NOT frozen here):
  • Measurement-environment drift. The compose freezes the EPP build, not the cluster it runs against. For a go test domain the judge runs inside the composed image, so the measured thing is fully inside the frozen boundary. For a perf domain (#1109 TTFT) the score depends on live model-server pods, GPU, Envoy, and network — none of which this ADR touches. Isolating the EPP's contribution from that confound is ADR-0006 (profiling), not this ADR.
  • Judge nondeterminism. Identical bytes do not make a flaky test or a timing metric deterministic. "Reproducibility by construction" here means a reproducible world, not a reproducible score.
• Discipline requirements (the freeze is only real if these hold):
  • Pin the toolchain base by digest, not tag. FROM golang:1.25.11 moves; the floor is frozen only because the base image digest is a compose input (see the digest section). The recipe must pin it.
  • Dependencies are part of the overlay; the toolchain is not. This is the one asymmetry that matters. A fix may legitimately need a new dependency — that's normal code work, so the sandbox keeps Go module-proxy egress (the #1489 manifest already allowlists proxy.golang.org/sum.golang.org) and the agent can go get mid-turn. The new dep lands in the overlay (the go.mod/go.sum delta on top of repo@SHA, fetched into the candidate build), captured by the overlay commit in the run identity — no re-compose. The baked deps are the starting set and a warm cache, not a ceiling. The toolchain is hard-frozen: the agent must not raise go.mod's go directive past the baked version. The baked toolchain is authoritative — the engine pins it regardless of an agent edit to the go line — and a toolchain bump is a deliberate re-compose, never a turn. Rule of thumb: new deps yes, new toolchain no.

The mutable overlay is the agent's git history, not a scratch container layer

The "copy-on-write overlay" in the rpm-ostree analogy is concrete: it's the agent's git commits on top of repo@SHA, not a vague writable container layer. This is how the loop already persists edits — the agent commits per turn (see #1109), so its work is a chain of commits whose parent is the pinned compose SHA. That maps onto ostree exactly: the immutable compose is the content-addressed base commit; each turn appends a commit; the branch ref is the overlay.

This framing buys three things the container-layer framing doesn't:

• The overlay is durable and inspectable. Edits survive the per-turn workdir copy in/out (the driver doesn't share the sandbox FS across turns); they're a real ref, diffable against the base, replayable, and the unit cross-run memory reasons about ("this commit on this base scored X").
• Reset is git, not teardown. Discarding a rejected turn is reset --hard to the base or the last kept commit — the immutable compose stays pristine because nothing ever wrote to it. The [[workspace.inject]] frozen-judge reset (ADR-0006) re-overlays the harness on top, the ostree "reset /usr" move.
• "Same domain" has a precise identity. A run is (compose digest, base SHA, overlay commit). Two runs are comparable iff the first two match — which is the reproducibility guarantee that makes cross-run memory meaningful.

Relationship to ADR-0005 (engine-side builds)

The two ADRs build at different layers and cadences, and compose cleanly:

ADR-0008 does not replace ADR-0005 — it provides the base that 0005's per-iteration build layers onto. build_epp reads the overlay (the agent's current branch ref) and builds it against the composed base. That resolves two things ADR-0005 left open:

• Workspace sync (0005's "the crux"). 0005 proposed the agent commit and push its branch to the loop pod as a git remote, and build from that ref. Under 0008 that's not a workaround, it's the model: the pushed ref is the CoW overlay, and the broker builds base@SHA + overlay. No new driver plumbing, no mid-turn FS checkpoint — the overlay was always git.
• Build latency (0005 open question #1). Because the compose bakes the toolchain and pre-fetches deps (a warm GOMODCACHE/GOCACHE in the base), the per-iteration build is an incremental go build against a hot cache, not a cold kaniko-from-scratch. The expensive, network-bound work (clone, dep download, toolchain install) happens once at compose time, not on every paid iteration. The isolated registry 0005 wants becomes the place re-composed base digests live.

In short: 0008 freezes the floor, 0005 builds on it. The agent edits Go (overlay), build_epp compiles overlay-on-base for feedback, deploy_candidate/apply_cmd ensures the measured EPP is that candidate, and measure resets the frozen judge before scoring.

What content-addresses a compose (the digest)

A compose has two distinct addresses, the way ostree separates a commit checksum from the rootfs it produces:

• Compose input digest — the canonical identity. A hash over the normalized inputs that define the world, not over the resulting image bytes. This is what cross-run memory keys on and what the controller reconciles against.
• Image digest — the OCI manifest digest of the artifact that actually gets pulled and run. One input digest can map to several image digests over time (a base-image security rebuild changes the bytes but not the world); they are attested equivalent. Never key memory on the image digest — it drifts on build noise (timestamps, layer order, base rebuilds) and would falsely split runs of an identical world.

The input digest is a hash over:

1. source — repo URL @ commit SHA. The SHA already content-addresses the tree, so we hash the SHA, not a tree scan.
2. toolchain — the Go version string from go.mod (e.g. go 1.25.11) plus the toolchain base image's digest. This is the hard-frozen input: the agent cannot move it without a re-compose (see Scope).
3. dependencies (baseline) — the go.sum content hash at the compose SHA. go.sum is already a content-addressed lockfile over every module, so we hash it, never the baked GOMODCACHE bytes. This fixes the starting dep set only; an overlay that adds a dep extends go.sum, and that delta is carried by the overlay commit in the run identity — it does not mint a new base digest.
4. frozen harness — a git tree hash over the injected set (judge/harness sources + the goal/manifest + the expected JUDGE_CONTRACT).
5. compose recipe — a hash of the Containerfile / buildah steps, so changing the build logic itself invalidates the digest.

The elegant part: every input is already content-addressed (git SHA, go.sum, tree hashes), so the compose digest is a cheap, deterministic hash-of-hashes — no need to checksum gigabytes of baked image. A digest is only minted after build-time validation passes (compiles + emits a valid baseline JUDGE_CONTRACT, ADR-0007), so a compose digest always denotes a validated world. The full run identity from the overlay section is therefore (compose input digest, overlay commit) — the base SHA is already folded into the input digest.

When a re-compose triggers (auto-detect, deliberately apply)

The trigger is always one thing: the desired compose input digest no longer equals the deployed one. The controller recomputes the digest from current pinned inputs each reconcile; a mismatch means the domain has drifted and a re-compose is proposed. The sources of a mismatch:

• upstream moved — you want to pull a newer SHA. This is not auto-on-every-upstream-commit (that re-introduces the moving-target drift this ADR exists to kill); it's a deliberate pin bump that changes input (1).
• harness changed — the judge/measurement evolved (input 4). This is a memory-epoch boundary: the new world is not comparable to the old, so prior cross-run memory for the domain is retired, not carried.
• toolchain or deps changed — a go.mod Go-version bump or a go.sum change (inputs 2/3), e.g. the agent's own dependency edit.
• recipe changed — the compose logic itself (input 5).

The discipline is detect continuously, apply deliberately. The reconcile loop is Kubernetes-shaped:

observe inputs → compute desired digest
  → if desired == deployed: no-op
  → else: compose + validate candidate
      → on pass:  rebase the agent's overlay onto the new base, then atomic digest swap
      → on fail:  hold on the current digest, report the failing compose (never deploy unvalidated)

Two things stay gated rather than automatic, because they can't be silently correct: the atomic swap (it changes which world the loop measures) and the overlay rebase (replaying the agent's commits onto a new base SHA is a real git rebase that can conflict — surfaced for review, never force-resolved). So the controller detects drift and prepares a validated candidate on its own, but promoting it is a deliberate act — which is exactly the "re-pin to a new digest, never patch in place" rule from the Decision, now with a concrete trigger and gate.

Consequences

• Positive: the build/world-definition mismatch class is designed out, not patched around (cross-run memory finally compares identical bytes — of the world; see Scope for what reproducibility does and doesn't cover); measure is offline and fast (no per-run dep fetch); validation moves to build time (fail the compose, not the paid loop); it's the natural unit for the controller to manage and version.
• Negative / cost: a per-domain compose build and a re-compose to pick up upstream changes (a deliberate, atomic act — which is the point); larger images (deps baked in); the agent-edits-as-overlay relies on git history on top of repo@SHA (see Design details) rather than the immutable base ever being written, so the base stays pristine by construction. Pinning means the kept patch is against SHA, not live main, so upstreaming a win is a separate rebase — correct separation (the loop optimizes a frozen world; the PR rebases it).
• Supersedes the runtime patching (emptyDir over the stale workspace, configMap over the adapter) used to limp the first launch — those are the anti-pattern this ADR removes.

ADR 0009: Composite domains — combined multi-component autoresearch

Status: Accepted; implemented (composite manifests, the multi-workspace world, and the combined gate are live) Date: 2026-06-27 Related: ADR-0008 (each component is an immutable compose; a composite is a tuple of composes plus the assembled rig), ADR-0006 (the multi-backend broker is what lets one run profile two runtimes — pprof for EPP, torch-trace for vLLM), ADR-0001 (freeze the judge AND the world — here extended to a cross-component workload), ADR-0002 (the combined rig is heavier to provision; the broker still mediates).

Context

Every domain so far is single-component: one upstream repo, one judge, one world (EPP config, or vLLM throughput). That shape can't express the features that actually matter most in llm-d, the cross-cut ones, where a behavior is split across the router (EPP) and the engine (vLLM) and neither side's gate can see the payoff. Two real, in-flight examples:

1. LoRA-aware routing. vLLM only emits vllm:lora_requests_info, which shows an adapter only while it has requests in flight — an idle-but-loaded adapter vanishes from metrics, so routers (llm-d, gateway-api-inference-extension) guess residency from request recency and can't tell a warm adapter (in a GPU slot, ~0 load cost) from a cold one (needs a load, a TTFT spike, worst at low concurrency). The vLLM half is prototyped as the upstream RFC vllm#45411 — adding a LoRALoadEvent engine notification and vllm:lora_adapter_loaded{adapter_name,level="gpu"|"cpu",pinned} + vllm:num_{gpu,cpu}_loaded_lora_adapters gauges, so a replica advertises its residency tiers before any traffic. The EPP half (tracked under llm-d-router #1500/#709/#926) extends the existing loraaffinity scorer + loraspec metric extractor to prefer GPU-slot > CPU-cache > cold endpoints. The win, avoided cold-load TTFT, is only measurable end-to-end: across the EPP→vLLM path on a workload with mixed adapter residency. vLLM's gate can't see routing; EPP's mock-rig gate can't see real adapter-load cost.

2. P/D rollout protection. In prefill/decode disaggregation the NIXL KV connector transfers KV from a prefill engine to a decode engine, which is only valid if the two agree on their KV-transfer config (kv_cache_layout, tp_size, block_size, dtype, model, NIXL backend/version — the NixlConnectorWorker/handshake inputs). During a rolling update some endpoints carry the new config and some the old; if the EPP pairs a prefill from one generation with a decode from another, the transfer fails or corrupts. The EPP has no way to block that pairing today, because vLLM exposes no P/D config hash for a routing filter to compare. The combined feature: vLLM exposes a fingerprint of the KV-transfer-relevant config; the EPP adds a filter plugin that excludes any P/D pairing whose hashes disagree (rollout protection). Its fitness, zero cross-incompatible transfers while preserving throughput during a rollout, again exists only in the live EPP→vLLM-P/D system under a rollout event.

Both share a shape the engine can't currently host: vLLM exposes a signal, the EPP routes/filters on it, and the fitness is a property of the assembled system, not of either repo. One is a soft scorer, one is a hard filter; both need two repos edited together and measured as one. A single-component manifest ([repo] is exactly one url|path, one [judge], one world) cannot represent this.

Decision

Introduce composite domains: a domain that composes N component domains into one autoresearch run. A composite is config, not new per-feature engine code — it reuses the component domains (each an ADR-0008 immutable compose) and adds three things the engine must learn to do generically:

1. A multi-workspace world — the agent may edit any component's checkout in a turn; keep/discard and git memory span all components (a kept iteration is a tuple of overlay commits, one per component).
2. A single combined gate — one measure_cmd that stands up the assembled system (EPP routing to vLLM backends) and runs the frozen cross-cut workload, emitting the usual contract.
3. The union of component surfaces — both components' PATH tools, skills, and profiler backends are available in the one run (the agent profiles vLLM with torch-trace and the EPP with pprof — ADR-0006).

The engine stays component-agnostic: "combined" is a composite manifest plus a combined gate command, not a lora-routing or pd-rollout special case baked into Rust. New cross-cut features are new composite manifests + new gate wrappers, the same way new single domains are new manifests today.

# a composite domain's crucible.toml (this one assembled the vllm + epp packs)
[composite]
name = "lora-routing"

[[component]]
domain = "vllm"            # reuses the vllm domain pack (its compose, tools, profiler)
[[component]]
domain = "epp"             # reuses the epp domain pack

[judge]
# Drives the ASSEMBLED system: EPP in front of vLLM replicas with a mixed-adapter workload, scores
# end-to-end serving (e.g. TTFT for requests whose adapter is cold). Frozen workload = the adapter mix +
# concurrency + (for P/D) the rollout schedule (ADR-0001 extended to a cross-component evaluation).
measure_cmd = "lora-routing-bench --json"
direction   = "higher"

[world]
# Composite reversibility: each component's git overlay PLUS the live assembled rig (snapshot/restore
# the deployed EPP config + the vLLM replica set). One opaque token bundles both (ADR-0008 §overlay).
snapshot_cmd = "lora-rig-snapshot"
restore_cmd  = "lora-rig-restore"

Implementation sketch (a follow-up, not this ADR)

• Manifest: a [composite] block with [[component]] entries referencing existing domain dirs (each contributing its [repo]/workspace, tools, skills, and profiler backend). A composite has no [repo] of its own; [judge]/[world] are the combined gate and reversibility. Single-component manifests are unchanged (a degenerate composite of one).
• Engine: generalize the World + run identity from one workspace to a vector of workspaces — the one real engine change. Run/Segment/Snapshot (ADR-0004) already model an iteration; extend the snapshot to a tuple of per-component overlay commits + the world token.
• Gate: the combined measure_cmd is a domain-owned wrapper (like vllm-throughput) that assembles the rig and runs the cross-cut workload. The engine treats it as the same opaque contract.
• Compose: each component is composed independently (ADR-0008); the composite pins a tuple of component digests. A re-compose of one component is a memory-epoch boundary for the composite.

Design details

Run identity is a tuple (cross-run memory still works)

ADR-0008 made a single run's identity (compose input digest, overlay commit). A composite extends this to a vector: ({(digest_i, overlay_i)} for each component, cross-cut workload hash). Two composite runs are comparable iff every component's compose digest matches and the frozen cross-cut workload matches — so cross-run memory ("this EPP-overlay + this vLLM-overlay on these bases scored X") stays meaningful, the same guarantee ADR-0008 gives per component, lifted to the tuple. The agent's keep/discard commits each component's overlay independently; a kept iteration is the tuple of commits that scored.

Why a composite, not a merged repo

We do not vendor EPP into vLLM or vice versa. Each stays its own upstream repo@SHA with its own toolchain (Go vs CUDA/Python), its own immutable compose, and its own profiler — because the kept patch must upstream to its repo as a normal PR (ADR-0008's "the loop optimizes a frozen world; the PR rebases it" applies per component). The composite is an assembly of frozen worlds, not a third repo. This is also why the engine must stay component-agnostic: the unit of reuse is the existing domain pack.

The combined gate is the only place the components meet

Amendment (2026-07-01). ADR-0013 (proposed) moves benchmark execution into a pluggable rig backend, which produces a metrics bundle. Scoring stays here: the engine runs the domain's measure_cmd (now over that bundle), so measure_cmd remains the single engine↔domain contract. The backend runs the workload; it never computes the score.

Everything component-specific stays in its component (tools name-prefixed — vllm-profile vs the EPP's profile; skills per-domain; profiler backends per-runtime). The single integration contract is the combined measure_cmd: it is the one artifact that knows both components exist, because it assembles them and runs the cross-cut workload. Keeping the meeting point to one command is what keeps the engine generic — mirrors the single-domain rule that measure_cmd is the only engine↔domain contract.

Merging surfaces without collisions (a real gotcha)

"Union of component surfaces" is not a free cp. Three things need care at assembly time:

• Tools are safe; skills are not. Component tools are already name-prefixed (vllm-profile, vllm-throughput vs the EPP's profile/bench), so PATH merges cleanly. But skills are not prefixed: two components router and engine both ship skills/profile, literally named profile, so merging both into one .claude/skills collides. The composite must namespace skills by component on assembly (e.g. router-profile / engine-profile), rather than each domain pre-prefixing (which would uglify the single-domain case). Namespacing is a merge-time concern, owned by the composite.
• The method prompt is composite-level. A component's method_prompt describes optimizing that component. A composite needs its own prompt that frames the cross-cut goal and both surfaces (when to edit the EPP scorer vs the vLLM signal it consumes). Components contribute their toolbox/skills; the composite owns the standing method.
• Frozen-judge injects stay per-component ([[workspace.inject]], ADR-0006) — each lands in its own component workspace; the combined gate is the cross-cut judge on top.

Where the combined gate runs (locus)

For a GPU composite the gate can't run in the agent's sandbox (no cluster reach, and measuring through a port-forward confounds the result — the EPP lesson, ADR-0006). It runs engine-side, the same locus as the EPP measure/build broker tools (ADR-0005/0006): the loop pod (or a broker-dispatched job) stands up the assembled rig and runs the cross-cut workload in-cluster. The composite therefore requires the broker on, with both runtimes' backends — which is why ADR-0006's multi-backend broker is load-bearing here.

What the two drivers exercise (and why both)

• LoRA-aware routing is a scorer (soft preference) — the gate rewards routing warm adapters; a bad policy costs TTFT but never breaks correctness. The signal is vLLM residency gauges; the agent edits the EPP scorer (and possibly the vLLM eviction policy), profiles both, and the gate scores end-to-end TTFT.
• P/D rollout protection is a filter (hard constraint) — the gate must show zero cross-incompatible KV transfers under a rollout while preserving throughput; too strict strands capacity, too loose corrupts. The signal is a vLLM P/D config hash; the agent edits the EPP filter and the vLLM hash it exposes. The frozen workload includes a rollout schedule (the cross-cut analogue of a request mix).

Carrying both in the ADR keeps the design honest: the engine change (multi-workspace world + tuple identity + combined gate) must serve a soft scorer and a hard filter, edits on the EPP side and the vLLM side, without either feature leaking into engine code.

Provisioning + cost (ADR-0002)

The combined rig is the heaviest yet: real vLLM replicas (GPU, and for P/D a prefill+decode pair with NIXL) behind a live EPP, plus a rollout/adapter-mix harness. The broker (ADR-0002) still mediates the expensive grants; the composite just needs two runtimes' worth of capacity and the multi-backend profiler from ADR-0006. This cost is the reason composites are opt-in, not the default shape.

Consequences

• Positive: the cross-cut feature class becomes optimizable at all — LoRA-aware routing and P/D rollout protection get a real fitness instead of hand-tuning; the win is measured where it actually exists (end-to-end), closing the neither-domain-can-see-it gap; the engine gains one general capability (multi-workspace runs) rather than per-feature code; component packs are reused verbatim, so a composite is mostly a manifest + a combined gate.
• Negative / cost: the heaviest rig to stand up and keep reversible (GPU P/D + EPP + a rollout/mix harness); multi-workspace git memory + tuple run-identity is new engine work (the one non-trivial change here); the combined gate is bespoke per cross-cut feature (authoring, not engine); a re-compose of any component retires the composite's cross-run memory epoch. Composites are opt-in for exactly these costs.
• Forward pointer: validates the component-agnostic discipline the recent work already follows — generic crucible-broker with injected per-domain backends, name-prefixed tools, per-domain skills, measure_cmd as the sole contract. Hold that line and a composite needs no engine special-casing beyond the multi-workspace world.

ADR 0010: Candidate portfolios — explore/exploit search over reviewable candidates

Status: Accepted; implemented (v1, 2026-07-02: parallel propose turns + serialized measurement + top-k ranking behind --wide N / [search]; since re-implemented as a work-graph template compiled from [search], with the same [search] surface; round-robin and successive-halving policies remain future impls) Date: 2026-06-28 Related: ADR-0004 (the core loop is a sequential refinement of one line; this generalizes it to a population), ADR-0009 (a composite candidate is a tuple of per-component edits; the portfolio is a set of such tuples), ADR-0001 (the frozen judge is what makes a portfolio rankable instead of a taste test), ADR-0002 / ADR-0003 (a candidate surfaces as a reviewable draft PR; a composite candidate as a set of linked PRs), ADR-0005 (deploy-to-rank: a candidate that changes the rig can only be scored once the broker builds+deploys it).

Context

The loop today (ADR-0004) is a single line of descent: iteration N refines iteration N−1, --iterations N turns deep. That is pure exploit — it makes one approach better. Running the composite P/D rollout-protection goal one-shot (--iterations 1, openshell backend) showed two things at once:

1. One-shotting the right shape works. Cold, in one turn, claude found that compute_nixl_compatibility_hash() already exists in vLLM, exposed a P/D config-hash, and added an EPP filter pairing only matching prefill↔decode — the correct cross-cut architecture, reasoned from scratch.
2. A single shot is one sample from a wide space. The same goal has architecturally distinct solutions with very different blast radius, and the model "knows" several of them. Betting the whole run on the first one it picks throws away the others — and "which is best" is not an eyeball question.

The competing approaches for this goal, concretely (each a legitimate, different candidate):

These are not refinements of each other — they are different families. And the frozen gate (pd-rollout-bench) can actually rank them, including behaviorally: a proactive filter jumps to ~100% success immediately; the reactive dead-peer ramps as it learns, so its score depends on how long the fixed workload runs. That difference is measured, not argued.

Decision

Generalize the loop from a single line of descent to a candidate portfolio with two explicitly different search modes, ranked by the frozen judge, and surfaced as reviewable PRs.

Design review amendments (2026-07-01, ahead of implementation). The approved v1 design pins: propose turns run in parallel (thread-per-candidate, worktree-per-candidate under the run's state dir) while measurement serializes on the shared rig by staging each candidate's patch through the main workspace (no World/Judge trait changes); the wide→deep hand-off is a pluggable SearchPolicy ([search].policy, v1 ships top-k only; round-robin / successive-halving are future impls — top-k-after-one-measurement picks winners from a single noisy reading, which those policies fix by re-measuring survivors each round); session-log rows gain an additive phase: "wide" field; and [search].approaches is required when wide > 0 — no auto-generated fallback, per §3's engineered-diversity rule.

1. Two investigation strategies, not one knob. Iteration count is not the axis; strategy is:

• Wide / explore. N independent candidates in parallel, each a different angle (method-prompt biased to a distinct approach from the table above), each with minimal context — "implement a minimal version of approach X." Each runs in its own sandbox, so the fan-out costs ~one turn of wall-time. Goal: discover which families clear the gate at all.
• Deep / exploit. Take the gate's top-k and iterate them sequentially with full context — the prior diff, its measured score, and what specifically failed — "make this actually solve it." This is ADR-0004's loop, unchanged, applied to a seeded starting point.

The method prompt differs by mode; the difference is the strategy.

2. Explore → exploit (a tournament with a refinement final). The default run is: a wide round → rank by the gate → a deep round on the winner(s). The wide round seeds the deep round. Today's engine only does the deep half; this ADR adds the wide half and the hand-off. (A pure-wide or pure-deep run remains valid for the degenerate cases.)

3. The judge is the ranker (ADR-0001). A portfolio is only better than guessing because the frozen gate gives an objective, reproducible score per candidate. No candidate is "picked" by review aesthetics; review chooses among gate-ranked options.

4. Candidates are reviewable artifacts (ADR-0002/0003), composite-aware (ADR-0009). Each candidate is published as a draft PR per edited component, cross-linked (the P/D candidate → one PR on the vLLM fork + one on the EPP fork). The PR is based on the exact sha the agent edited against (reachable in the fork's object network), so it is immune to the fork having diverged from upstream. A portfolio is therefore a set of PR-sets a human can diff side-by-side while the gate orders them.

5. Deploy-to-rank is the gate to real scores (ADR-0005). A candidate that changes the rig (vLLM + EPP) is only scored after the multi-backend broker builds+deploys it. Until then a candidate is captured and reviewable (its diff + PRs) but scores at baseline. Ranking a portfolio by true fitness requires the deploy path; reviewing it does not.

Consequences

• The loop gains a population and a (wide, deep) schedule; ADR-0004's Segment/Iteration typestate becomes the deep leg, and a new fan-out drives the wide leg over the same diff-capture + PR-publish plumbing (World::staged_diff, the native publish.rs single- and multi-fork draft-PR path).
• Cost scales with breadth: N wide candidates = N sandboxes (parallel) + N deploys to rank. Breadth is a budget knob, not free.
• Diversity must be engineered (distinct method prompts / forced approaches); N identical turns are not a portfolio. Prompt-diverse fan-out beats one-turn-self-enumerate because the latter shares the model's first framing.
• "Solved" is still the gate's call, now over the best of a population rather than the end of a single line.

Alternatives considered

• Single-line refinement only (status quo). Simplest, but bets the run on the first approach and has no way to compare families — exactly what the one-shot experiment exposed.
• One turn self-enumerates and implements K approaches. Cheaper than fan-out, but the K attempts share one context and one framing, so they cluster; weaker diversity for the same review/rank cost.
• Human picks the approach up front. Throws away the gate's whole reason to exist (objective ranking) and the model's ability to surface approaches a human wouldn't enumerate (e.g. #6 reactive dead-peer).

ADR 0012: Crucible-rendered deployments — generate the loop/broker/rig manifests from the manifest + a deploy profile

Status: Accepted (slice 1 implemented) Date: 2026-06-28 Related: ADR-0002 (the broker whose env the pod hand-duplicates), ADR-0005 (the FORGE_* build/deploy config the pod hand-duplicates), ADR-0008 (immutable composes — the rendered deploy must pin digests, not tags), ADR-0009 (the composite loop pod that drove this), ADR-0011 (the controller runs "deploy" as a step — it should call a renderer, not hand-edit yaml).

Context

A crucible run is a pod (the loop), usually a child broker process, the live rig it measures, and the RBAC

• secrets + volumes that wire those together. Today all of that is hand-written, per-domain YAML. Bringing a composite rig online requires hand-editing, in lockstep:

• loop-pod-openshell.yaml — ~250 lines: the wrapper script, ~20 env vars (FORGE_*, VLLM_*, BROKER_COMPOSITE, RIG_NAMESPACE, KUBECONFIG, Vertex creds), six volumes/mounts (kube token, kubeconfig, push authfile, forge storage, prior-candidate, quay auth), serviceAccountName, a digest-pinned image.
• loop-rbac.yaml — a cross-namespace RoleBinding.
• modelservers.yaml — the rig.
• ad-hoc configmaps (prior-candidate, loggers).

Three structural problems:

1. The pod env duplicates the manifest. FORGE_REGISTRY/FORGE_DOCKERFILE/FORGE_DEPLOY_*, VLLM_BASE_REF, BROKER_COMPOSITE, the broker bind/name — these are facts crucible already knows from crucible.pd-rollout.toml ([agent.broker], the components, the deploy targets). They are copied into the pod spec by hand and kept in sync by hand. The manifest is the source of truth; the pod is a stale shadow of it.

2. Manual digest pinning is a footgun, not a convenience. Every image (loop-openshell, the sandbox) is pinned to a sha256:… with a comment "re-pin after each rebuild — a mutable tag serves the stale cached layer." Re-pinning a milestone tag by hand after a rebuild is error-prone; forget it and the node silently runs the old binary. The digest is mechanically derivable from the tag at deploy time.

3. The pod spec is ~90% identical across domains. The EPP loop pod and the composite loop pod differ only in domain-specific env + which broker binary + which manifest. The shared structure (openshell privileged pod, projected kube token, IRSA, forge storage, broker child) is copy-pasted and diverges.

This is the same theme as ADR-0011: the engine is clean; the deployment of it is manual toil that should be the engine's job.

Decision

Crucible renders its own deployment. A crucible deploy render (emit YAML) / crucible deploy apply command generates the loop pod + broker wiring + RBAC + rig references from two inputs:

• the domain manifest (already the source of truth: components, [agent.broker], [agent.env], build /deploy targets), and
• a thin deploy profile — the only hand-written part: the environment-specific facts crucible can't know (cluster/namespace, secret names, creds source, GPU/resources, the backend = openshell vs local, the results bucket / IRSA role). One small file per cluster, not per run.

What the renderer owns:

• Project manifest config into pod env, instead of hand-duplicating it. The [agent.broker] + build /deploy targets become the broker child's FORGE_*/BROKER_* env; the components become VLLM_*/the apply-hook env. Change the manifest, re-render — no second copy to forget.
• Resolve + pin image digests at render time. The profile names a tag (or :latest of a build); the renderer looks up the digest and emits the @sha256:… pin. The stale-layer footgun disappears; the pin is always correct because it's computed, not typed.
• Share a template per backend pattern. One openshell-loop template (privileged pod, projected token, IRSA, forge storage, broker child) parameterized by manifest + profile; domains stop copy-pasting it.
• Emit the RBAC + secret references the run needs (the SA → edit binding in the rig namespace, the push-authfile mount), as part of the same render — not a separate hand-authored file that drifts.

What it deliberately does not own: the contents of secrets (it references them; a human/External Secrets provisions them), and the rig's domain YAML where the rig is a fixed artifact (ADR-0001 frozen workload) — the renderer references the rig, it doesn't invent model-server topology.

flowchart LR
    manifest["domain manifest<br/>crucible.toml"] --> render["crucible deploy render"]
    profile["deploy profile<br/>cluster facts"] --> render
    render --> output["loop-pod YAML + RBAC YAML<br/>optionally apply"]
    render --> env["project manifest values into environment"]
    render --> pins["resolve image references to sha256 digests"]
    render --> template["instantiate the shared OpenShell loop template"]

Consequences

• One source of truth. The manifest already defines the run; the deploy stops being a second, hand-synced copy of it. "Change the gate / the broker / a component" is a manifest edit + re-render, not a manifest edit + a yaml safari.
• Reproducible + footgun-free deploys. Digests are computed; a re-render after a rebuild is always correct. The "forgot to re-pin, ran the stale binary" class of bug is gone.
• Composes with the controller (ADR-0011). The autoresearch controller's "deploy" step becomes crucible deploy apply — it never hand-writes pod specs. And with ADR-0008, the rendered deploy pins the composed image digests, so the immutable-compose guarantee reaches all the way to the running pod.
• Lowers the bar for a new domain. Onboarding a domain (the vLLM/EPP/composite progression) stops requiring a 250-line pod spec; it's a manifest + a profile entry.
• Open questions.
  • Render engine. A native Rust renderer in crucible (typed, knows the manifest) vs emitting kustomize /helm (gitops-native, but another layer). Lean native: crucible already parses the manifest and resolves images; a typed render keeps the pod spec honest to the config types.
  • How much k8s to model. Just the loop pod + RBAC + broker wiring (the run), or the rig too? Start with the run (the part that's pure duplication); leave the frozen rig as a referenced artifact. Resolved by ADR-0013 (proposed): the rig half is deferred to a pluggable external renderer ([rig].backend / RigBackend); this renderer keeps owning the run.
  • Profile granularity. Per-cluster vs per-domain-per-cluster. Start per-cluster with per-domain overrides only where a domain genuinely needs more (GPU, extra secrets).
  • Apply vs emit. Emit YAML (review/gitops) by default; apply as a convenience that shells kubectl. The renderer stays declarative; applying is a thin wrapper, same split as ADR-0005's build-vs-deploy.
  • Secrets. Reference-only (the renderer never holds a credential); document the expected secret names in the profile so a missing secret is a clear render-time error, not a CreateContainerConfigError later.

Implementation (slice 1)

crucible deploy render|apply (crucible/src/deploy/) renders a composite's loop Pod + cross-namespace RoleBinding from the manifest + a per-cluster profile, as real k8s-openapi objects serialized to YAML. Decisions taken, resolving the open questions above:

• Native Rust renderer, real k8s-openapi types (the spec stays honest to the API). Emit YAML by default; apply pipes it through kubectl apply -f -.
• Build/deploy targets are typed manifest config ([deploy]): the generic forge build contract (buildah → FORGE_*, deploy_name → FORGE_DEPLOY_NAME) is typed; a per-component env map carries the domain hook's own env names verbatim. On a composite, [deploy.<component>] overrides the base.
• Perfectly generic engine. The renderer names only env it itself consumes (BROKER_*, FORGE_*, OPENSHELL_SUPERVISOR_IMAGE, mount-path consts). Every domain/vendor name (EPP_*, VLLM_*, GCLOUD_CREDENTIALS, BENCH_*) lives in the profile's generic env/secret_env maps or [deploy].env. The nested podman was originally logged into a single registry parsed from the sandbox image ref (forge::oci::registry_of); the wrapper now sets REGISTRY_AUTH_FILE at the mounted authfile instead, which podman honors ahead of its own lookup and which covers every registry a run pulls from.
• Digests resolved in-process via oci-client (forge::oci::pin_digest, no crane/skopeo), using the operator's existing registry login. Verified: the milestone tag resolves to the exact @sha256:… the loop pod previously hand-pinned.
• Profile (profile.<cluster>.toml) is the only hand-written part: namespaces, SA, secret names, resources, the loop image tag (not digest), and the generic hook/gate env.

Scope held to the run (loop pod + RBAC). Single-domain (non-composite) render landed 2026-07-02 (a plain manifest renders as a degenerate composite of one; it needs its own [deploy] block). Still deferred: a local-backend template, and modeling the rig (since answered by ADR-0013).

Second mode: a playbook pod

The renderer emits one of two wrapper commands. The default is the agent loop (crucible --manifest … --iterations …). With --playbook it is the plan runner (crucible plan run --manifest … --max-cost … --max-time … --param …), which is what a [workflow] type = "playbook" pack needs: no iterations, no agent-loop flags, and ceilings + parameters supplied per launch.

The mode is an explicit flag, consistent with this ADR's stance that the renderer is told what to render and never infers it. The renderer does not read [workflow].type; the launcher (the controller's playbook dispatch, or a human) states the mode, exactly as --pack states delivery. The two compose: --pack decides how the manifest reaches the pod, --playbook decides what the pod runs, and a controller-dispatched playbook passes both.

Both modes publish the run's session at <domain>/state/session.jsonl and deliver it the same two ways (the Tier 2 drop-box when the profile names one, the SESSION delimiter otherwise), and both end that log with a shutdown line, so completion ingest cannot tell them apart.

A playbook render relaxes two loop-template requirements, because both describe a deployment a playbook never performs: [deploy] is optional (it builds and deploys nothing) and [agent].sandbox_image is optional (its tasks pick their own backend; a pack with a sandboxed task still declares one). It also persists nothing. plan run has no --resume and opens the session log in append mode, so a run-state claim would leave a second launch publishing the first launch's events too; a playbook pod is one-shot and mounts no run-state volume even under a profile that names a claim.

kube-rs migration — the engine no longer depends on kubectl

The goal here is that the engine and the [world] hooks never depend on shelling kubectl — not to strip the binary. kubectl stays in the loop images as an operator/agent debugging affordance (too handy to drop); the engine just doesn't call it. forge::kube (typed kube-rs 4.0 + k8s-openapi 0.28) owns the client, patches, rollout-wait, pod exec/exec-streaming, GPU headroom, and dynamic server-side apply, behind a blocking facade (a block_on that uses block_in_place under the broker's runtime, else a private one). Migrated off kubectl:

• forge::deploy / current_image → typed patch + get (the jsonpath is gone); tag_of → oci-client Reference; rollout timeouts parse via jiff.
• the broker GPU admission (gpu_check) → typed Node/Pod sums + apply_yaml (no more serde_json::Value walking).
• crucible deploy apply → server-side apply_yaml.
• the pd-rollout [world] hooks (apply/snapshot/restore) → a generic forge-kube CLI (set-image/set-pull-secret/set-rolling-update/rollout-restart/rollout-status/current-image/ pod-name/exec/apply); the domain orchestration stays in the hook.
• the remote viewer (crucible view --pod) → a streaming exec (tail -F) + exec-with-stdin for steer/stop.

Both loop images ship forge-kube (the hooks' typed kube path) and keep kubectl (the debugging affordance). The EPP rig's engine-critical [world] hooks (rig-snapshot/restore/down) are migrated to forge-kube; the EPP debug/profiling tools (inspect-rig, epp-metrics, epp-pprof) deliberately stay on kubectl — they're interactive debugging, and epp-pprof is better migrated alongside ADR-0006.

ADR 0014: Scoping as a governed pipeline — crucible scope <issue>

Status: Proposed; S0+S1 implemented (the crucible scope pipeline skeleton and the [judge.selftest] primitive are live; S2, the propose turn, is in progress) Date: 2026-07-02 Related: ADR-0001 (the scoping phase this finally implements — "an agent may propose the harness config"), ADR-0007 (the preflight, here a pipeline stage), ADR-0008 (the freeze the pipeline ends in), ADR-0010 (the wide leg that makes a freshly scoped issue cheap to attack from several angles), the judge-tier issue ranking (onboarding triage), crucible check + crucible init (the BYO on-ramp), crucible::identity (the freeze fingerprint).

Context

The engine side is now rich — composites, mediated builds/deploys, profiling, portfolio search, identity stamping — but every run still starts from a hand-authored harness: a human writes the manifest, the gate, the world hooks, picks the workload and the baseline. That is roughly a day of expert work per issue, and it has become the throughput bottleneck: the loop can explore five approaches in parallel, and it still takes us a day to give it something to explore.

It is also the most dangerous manual step, not just the slowest. The two best incident stories in this repo are both harness failures: #1109 burned a loop on a gate that could not isolate its target (ADR-0007 exists because of it), and #1489's reward hack lived in a gate that under-counted the PreRequest path. ADR-0001 designed the answer — scoping is adaptive, once, then frozen, with the proposal automated and the output validated and human-approved — but only the freeze was ever built. This ADR builds the rest.

Decision

Introduce crucible scope <issue-url>: a five-stage pipeline that turns an upstream issue into a validated, human-approved, frozen domain pack. Every stage exists to kill a bad harness as early and as cheaply as possible.

1. Ingest + triage. goal-from-issue extracts the goal; the judge-tier ranking (T0/T1/T2/N) decides whether the issue is scopeable at all and biases the proposal (a T0 bug wants a test gate; a T1 perf issue wants a measured workload). An N-tier issue exits here with the reason.
2. Propose. One agent turn drafts the harness: the manifest, the gate script, the workload shape, the baseline choice, and the negative controls (a known-bad and a known-good configuration the gate must tell apart). This is ADR-0001's sanctioned adaptivity, in its sandbox: the proposer gets solution-space tools plus the issue, nothing else.
3. Validate mechanically. Three checks, all existing machinery, all unbypassable:
   • crucible check — the contract lint (manifest parses, files resolve, the measure contract emits {valid, score}, the gate is not agent-editable without a frozen inject);
   • the gate self-test (ADR-0001 safeguard 1) — the proposed known-bad must score strictly worse than the known-good, else the gate cannot discriminate and the proposal dies;
   • the isolation preflight (ADR-0007) — the target's own contribution to the gated metric must be material, measured from the target's instrumentation, else the issue is mis-scoped and the pipeline emits that finding instead of a harness. A failure at this stage kills the proposal before any human reads it.
4. Approve. The surviving domain pack opens as a draft PR (the ADR-0002 approval muscle): the manifest, gate, controls, and the validation evidence (self-test scores, preflight attribution) in one reviewable diff. A human signs off before the loop spends budget — ADR-0001's checkpoint, unchanged.
5. Freeze. On approval the manifest is pinned, the pack's RunIdentity digest is computed and recorded as its fingerprint, and the issue is runnable: crucible run (or a wide round, ADR-0010) against a world whose provenance is one content-addressed line.

Why an agent proposing the judge stays honest

This is the reward-hacking surface ADR-0001 fenced, so the trust story must be explicit:

• Proposer ≠ optimizer. The scoping turn and the optimizing loop are different runs with no shared context; the scoping agent never runs against — and never sees the results of — its own gate.
• Validation is mechanical. The self-test and preflight are computed, not judged; a gate that cannot discriminate, or a target that is a sub-percent sliver of the metric, fails arithmetic, not taste.
• A human approves the evaluation. Nothing the proposer emits becomes a judge until a person has read the gate and its evidence — the same rule as every other judge change in the system.
• The freeze is stamped. The approved pack's identity digest makes post-approval drift detectable (ADR-0008); the gate that runs is provably the gate that was approved.

What this is NOT

• Not an adaptive loop. The ADR-0001 wall stands: scoping adapts once, then freezes. The pipeline never runs concurrently with an optimizing loop on the same domain.
• Not auto-run. Stage 4 is mandatory. A wrong harness is wasted or misleading research; it keeps its human checkpoint.
• Not a replacement for author-m2-rig. v1 targets test-gate and local/GitWorld domains (the #1474/#1489 shape, and the BYO on-ramp). Composite GPU rigs need rig topology judgment that stays with a human until the ADR-0013 rig backends make "stand up the rig for the preflight" cheap.

Consequences

• Positive: onboarding drops from a day of expert work to review-of-a-draft-PR; the two known harness failure modes (#1109 mis-scoping, #1489 gate blind spots) are caught by construction — stage 3 runs exactly the checks that would have caught each; every prior wave's machinery becomes load-bearing (check, identity, publish, the wide leg).
• Negative / cost: the gate self-test becomes a first-class engine contract (the manifest needs a place to declare the controls — likely [judge.selftest] with good_cmd/bad_cmd), which is new surface; scope turns cost real tokens on issues that may die in stage 3 (bounded by a per-scope budget cap); the preflight needs per-target attribution metrics, which not every issue has (absent instrumentation, stage 3 degrades to self-test + check and says so in the PR).

Workstreams

Slicing: S0 + S1 first (they harden hand-written harnesses immediately, before any agent proposes one); S2 once the validation net is real; S3/S4 to close the loop.

ADR 0017: Turn result contract — structured state back from turn pods

Status: Proposed (supersedes the 2026-07-06 stub; amends one ADR-0016 ruling, see §"Amendment"; refined 2026-07-07: harness crate + OTLP telemetry restoration + MLflow export workstream) Date: 2026-07-06 Related: ADR-0015 (the WorkPod primitive whose results this carries), ADR-0016 (the ledger the results land in, and the truth model this must not break), the private control plane's workpod dispatcher (today's log-marker scrapers), a run of internal incident PRs (the fragility class that motivated this), opendatahub-io/agentic-ci (the vendored harness whose OTLP collector and library role this restores). External prior art: Tekton results + TEP-0086/TEP-0127, Argo Workflows output offloading, GitHub Actions runner auth, Buildkite job tokens, Temporal task tokens (condensed in §"Prior art"; full cited notes in the appendix).

Context

Turn pods (grounded rank, scope) and run pods return everything to the controller by printing to stdout, scraped from the kube log after the pod goes terminal. Six marker literals, each duplicated across crucible/ and crucible-controller/ with "keep in sync" comments as the only contract:

One night of production use (2026-07-05) surfaced the fragility class four ways:

• the openshell event stream glues messages without newlines, breaking a last-line parse contract (a validated $8 pack died on the adversary verdict parse — fixed by a reverse-scan fallback in an internal PR);
• every payload fights the kubelet's ~10 MiB log-rotation budget, forcing per-payload byte caps and truncation policies — and rotation can still eat a marker line entirely, which no scan direction survives;
• binary artifacts ride logs as giant base64 lines — the least log-shaped data imaginable;
• markers duplicate literals across crates because the crate DAG forbids sharing (crucible and crucible-controller deliberately do not depend on each other).

Two adjacent gaps compound it: startup adoption re-ingests grounded-rank verdicts but not scope reports (reconcile_turn_on_startup only calls parse_verdict_logs, workpod.rs:1506), so a scope turn that finishes while the controller is down is lost; and turn pods are invisible to the completion watch — they are polled in-band by KubePodDispatcher::await_terminal() (workpod.rs:1376), which discards the pod status it already holds.

Markers were the right zero-infrastructure bootstrap. They are the wrong permanent contract for control-flow-critical, sized, structured data.

The harness gap

Crucible began as a consumer of the vendored agentic-ci harness and re-implemented its stream decode + NDJSON event model natively when the vendored package was dropped. Two things fell on the floor in that move:

• The OTLP collector never made the jump. agentic-ci ran a small OTLP http/json receiver next to the agent, injected CLAUDE_CODE_ENABLE_TELEMETRY=1 + the OTEL_* exporter env into the sandbox, logged every export to a jsonl, derived a live token rate, and rolled it all up into the otel_summary NDJSON event at run end. Crucible kept the consumer half — AgentEvent::OtelSummary (event.rs:135), the TUI rendering, the prefer-OTEL-cost logic — but nothing produces the event anymore: cost is estimated from a hardcoded pricing table (agent.rs:283) that silently drifts with every model release, Tokens.rate is permanently None (stream_json.rs:214), and per-model splits, API request latency, and active time are simply gone from the evidence.
• The harness is trapped in a binary. Agent spawn, stream-json decode, the event pump, and session logging live inside crucible's bin target (agent.rs, stream_json.rs, event.rs, session.rs). Nothing outside this repo can drive an agent run the way agentic-ci's library consumers could; there is a standing requirement to expose that capability as a crate again.

The contract work below forces the crate question anyway (the marker literals need a home shared across a crate DAG that forbids crucible ↔ crucible-controller dependencies); this ADR answers it once for both gaps.

Prior art (survey, 2026-07-06)

The industry converged on exactly two mechanisms, and on abandoning everything else:

• Tekton returns task results via the container termination message and lives inside its limits: 4096 bytes per container, 12 KiB per pod divided equally among containers. Hitting that ceiling produced TEP-0086 (options: storage API service, sidecar, ConfigMaps, CRDs, PVCs, logs — logs rejected outright for having "no availability guarantee") and TEP-0127 (sidecar-logs escape hatch: ~3 s pod overhead, controller needs log RBAC, still capped by the 1.5 MiB CRD ceiling). Lesson: termination message for small verdicts only, and don't build the large tier on logs or CRDs.
• Argo Workflows started with output parameters in pod annotations (256 KiB limit), migrated to a WorkflowTaskResult CRD, and still fights the 1 MiB etcd ceiling with compression and optional database offload. Artifacts (the large tier) go to an artifact repository, not through the Kubernetes API. Lesson: the Kubernetes control plane is not a result store; two tiers are load-bearing, not an optimization.
• Kubernetes itself: terminationMessagePath (default /dev/termination-log) is read by the kubelet into containerStatuses[].state.terminated.message, atomically, as part of pod status — no scraping, no rotation exposure. Truncation at 4096 bytes is silent. FallbackToLogsOnError returns at most 2048 bytes / 80 log lines and only on error exits — useful as a debugging aid, not a contract.
• GitHub Actions / Buildkite runners push results and artifacts to the control plane over authenticated HTTP with a per-job token: minted by the control plane at dispatch, scoped to exactly one job, lifetime = job timeout plus a small grace (GitHub: +10 minutes), never persisted, scrubbed from logs. This is the standard shape for our Tier 2 — credential scoped to one unit of work — for which Kubernetes has a native carrier (bound service-account tokens, below).
• Temporal completes activities by RPC with a single-use task token and is explicit that delivery is at-least-once: the completer must be idempotent, keyed on the stable task identity, not the token. Lesson: the controller deduplicates; a retried POST is normal, not an error.
• Prow uploads artifacts to GCS from a sidecar and signals completion with marker objects an external watcher polls — the pattern we'd converge on if we ever outgrow a single controller; noted, not adopted.

Every migration in the survey moved away from logs, annotations, and CRDs and toward "tiny critical payload in pod status, everything else over authenticated push to a store." That is the design below.

Decision

A two-tier result contract, defined in one shared crate. Stdout demotes to a human-facing live/debug channel: progress beats stay, payloads leave.

Tier 1 — the verdict rides the termination message

The turn command (both rank-grounded and scope, behind the existing --marker-style flag, renamed in spirit to "result mode") writes its result JSON to /dev/termination-log as its final act. The controller reads it from pod.status.containerStatuses[].state.terminated.message — atomically, from the same pod status object every read point already holds. No scraping, no rotation exposure, no glue.

• Budget: turn pods are single-container, so the kubelet grants the full 4096 bytes. The engine self-caps the JSON at 3584 bytes (headroom against the silent kubelet truncation) with honest field-level truncation of rationale text, the same discipline the transcript cap uses. A truncated verdict is still valid JSON; a kubelet-truncated one is garbage — the self-cap is what keeps parse failures structural rather than probabilistic.
• Schema: a versioned envelope in the contract crate: {"v":1,"kind":"verdict"|"scope_report","payload":{…},"artifacts":[{"kind","digest","bytes","delivered"}],"usage":{…}}. usage is the optional OTEL rollup (§"The OTLP collector comes back"): {"cost_usd","total","models":[…],"api_requests","api_ms","active_seconds"} — a few hundred bytes that put authoritative cost on the ledger row instead of a pricing-table estimate. Under self-cap pressure the per-model rows drop first; the totals stay. The artifacts manifest is the load-bearing novelty: Tier 1 is the authoritative index of every Tier 2 upload the pod attempted, with content digests. The controller trusts the termination message (authenticated by the kubelet), then checks the drop-box against the manifest — a missing or digest-mismatched artifact is detected without trusting the POST path at all.
• terminationMessagePolicy stays File. FallbackToLogsOnError truncates to 2048 bytes/80 lines and only fires on error exits; it would hand us exactly the parse-a-maybe garbage contract this ADR exists to kill. During migration the marker scrape is the fallback instead (§Migration).
• Read points, all three: await_terminal() grows a return of the terminated container state (it polls api.get(name) already and throws the status away); the completion watch's pod_completion_key() path for run pods; and startup adoption, which reads the same pod status — closing the scope-turn adoption gap as a side effect of the mechanism rather than as a special case.

Tier 2 — artifacts POST to the controller's evidence drop-box

Packs, transcripts, and run sessions are files, and they move as files: an authenticated POST /api/pods/{pod}/artifacts/{kind} (kinds: scope-pack, scope-transcript, run-session, otel-log) with the raw gzipped bytes as the body. Typed, validated, size-checked at the door — no base64, no line discipline, no rotation budget.

• Storage is the state volume, not the database. The controller streams the body to state/evidence/<pod>/<kind> and records only pointers + digests in SQLite, preserving ADR-0016's rows-are-pointers rule. S3 can replace the directory later without touching the contract (the drop-box is an interface, not a place).
• Auth: a pod-bound projected ServiceAccount token, validated via TokenReview. The pod spec gains a serviceAccountToken projected volume with a dedicated audience (crucible-ingest) and a short TTL; render_turn() adds the volume and the CRUCIBLE_INGEST_URL env var (the default automountServiceAccountToken: false stays — this is one explicit, audience-locked file, not the kube-API passport). The ingest extractor sends the bearer to TokenReview and requires three things from the response: the audience matches, the service account is the turns' own, and the bound-pod claim (authentication.kubernetes.io/pod-name) equals the {pod} path segment. A turn is exactly one pod, so pod-binding is turn-scoping. Nothing is minted, stored, or expired by us: the kubelet rotates the token, the API server invalidates it when the pod dies, and a controller restart changes nothing about validation. The cost is a kube API call in the handler (cached briefly per token; the endpoint sees a handful of requests per turn, not a request stream), accepted because the controller is already a kube client everywhere else. Entirely separate from the human-facing oauth2-proxy/role stack.
• Delivery is at-least-once; the controller deduplicates. Uploads are content-addressed: a re-POST of a digest the drop-box already holds returns 200 and changes nothing. The engine retries a failed POST 3 times with backoff and then keeps going — it still writes Tier 1 with that artifact's manifest entry marked delivered:false, so the failure lands loudly on the work_pods row instead of silently truncating like the log path does today. Oversize at the door is a 413 with the limit in the body; the caps move from log-shaped guesses to real HTTP limits (pack 16 MiB, transcript 32 MiB, session 128 MiB, otel-log 32 MiB, all compressed — generous because they no longer fight the kubelet for log budget).
• Ordering: artifacts POST first, termination message last. The Tier 1 manifest can then state the truth about every upload, and the controller's fold has everything it needs the moment the pod goes terminal.

The fold stays pull-shaped (ADR-0016 amendment)

ADR-0016 ruled "a push endpoint is explicitly not v1 … never a correctness dependency." This ADR amends that ruling narrowly rather than reversing it. The ingest endpoint is a data-plane evidence drop — the network equivalent of the S3 bucket and the session log, just closer. Pods still never touch the database; no DB write happens in the request handler beyond recording the artifact pointer. The fold — outcomes into scopes/runs/candidates/ ledger rows — still happens in the controller's own reconcile transaction, triggered by the pod going terminal, reading evidence (now: pod status + drop-box instead of scraped logs). The database remains a rebuildable index; crucible db rebuild reads the evidence directory the same way it reads session logs. What actually changed from ADR-0016's assumption: logs turned out not to be a durable evidence store (the 2026-07-05 incidents, and Tekton's identical finding), so the evidence needed a real drop point. Push is the delivery mechanism for evidence, not the ingestion model.

One contract crate

A new crucible-contract crate (no async, no kube, serde only) that crucible, crucible-controller, and crucible-broker all depend on. It owns: the Tier 1 envelope and payload types, the artifact-kind enum + per-kind size caps, the ingest request/response types, the env-var names, the managed-by label, the AgentEvent NDJSON types (the session-log format both the engine and the controller's SSE relay read), and — until the migration completes — the six marker literals, deleting every "keep in sync" comment in the workspace. The termination message and the ingest bodies are serde round-trips of the same types on both sides; schema drift becomes a compile error instead of a 3 a.m. parse failure.

The harness crate — the agentic-ci replacement

A second crate, crucible-harness (library; async), extracted from the crucible bin target: agent spawn + env assembly, the stream-json decoder, the event pump + NDJSON session logging, and the OTLP collector below. It depends on crucible-contract and nothing controller-shaped. Consumers: the crucible binary (loop + turn commands), and external projects — the library role the vendored agentic-ci package used to play, restored as a Rust crate.

The boundary rule between the two: contract = wire types both sides serde-round-trip; harness = everything that runs next to an agent. The controller depends only on the contract; the crate DAG rule (crucible ↮ crucible-controller) survives intact.

The OTLP collector comes back, in-process

The harness embeds the collector agentic-ci ran as a subprocess, as an in-process task:

• Receiver: binds a configurable address, port 0 auto-assign; accepts /v1/metrics, /v1/logs, /v1/traces OTLP http/json POSTs; appends every export raw to otel.jsonl in the run dir (the otel-log Tier 2 artifact); answers {"partialSuccess":{}}.
• Live: a 60 s sliding-window token rate, fed to the event stream over a channel — Tokens.rate stops being permanently None. The rate/port files of the Python implementation were subprocess IPC; in-process they die.
• Rollup: at agent exit the collector builds the otel_summary event (authoritative cost, per-model input/output/cache tokens, api_requests, api_ms, active_seconds) — the event finally has a producer again — and the same rollup rides Tier 1 as the envelope's usage field.
• Env injection: the harness sets CLAUDE_CODE_ENABLE_TELEMETRY=1, OTEL_{METRICS,LOGS,TRACES}_EXPORTER=otlp, OTEL_EXPORTER_OTLP_PROTOCOL=http/json, the endpoint, OTEL_METRIC_EXPORT_INTERVAL=10000, plus full-fidelity spans exactly like agentic-ci did: CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 (tool nesting + subagent spans) and OTEL_LOG_USER_PROMPTS=1 / OTEL_LOG_TOOL_DETAILS=1 / OTEL_LOG_TOOL_CONTENT=1 — on the spawned agent (local exec) or into the sandbox env (openshell), following agentic-ci's per-harness matrix (docs/otel-configuration.md upstream). Consequence, stated plainly: the otel-log evidence contains prompt text and tool input/output, so it is confidential the same way session transcripts already are; anywhere it is exported to (§MLflow) must be as trusted as the state volume.
• Sandbox reach: local agents export to 127.0.0.1; an openshell sandbox reaches the collector exactly the way it reaches the broker — the collector binds 0.0.0.0 on the loop/turn pod and the sandbox gets a host.containers.internal:<port>:full egress rule (the proven 8849 pattern).
• Degradation: telemetry off or collector unreachable is graceful and identical to today — no otel_summary, no usage, pricing-table estimate as the cost fallback. The collector is evidence enrichment, never a run dependency.

MLflow export — a controller background task over the evidence

The MLflow exporter is a private control-plane component: it lives in the controller deployment and is a per-installation opt-in, no part of the public engine depends on it.

Capturing the raw OTLP jsonl as a first-class artifact is what makes an MLflow exporter a bolt-on instead of a rework. agentic-ci proved the shape (mlflow-push: a decoupled, allow-failure follow-up job that re-encodes the captured /v1/traces records OTLP-JSON → protobuf and POSTs them to MLflow), and MLflow grew the native half since. Crucible runs map onto the experiment-tracking ethos directly, so this is a workstream (R6), not a someday note.

• Locus: the controller, only. The exporter is a controller background task fed by evidence that already exists — the otel-log artifact in the drop-box and the folded ledger rows. It runs post-fold, never inline with a run or a fold, and failure marks the export row and retries later (the agentic-ci allow_failure: true ethos). Pods never see MLflow credentials; local crucible --manifest runs don't export.
• Traces: MLflow ≥ 3.6 ingests OTLP at POST /v1/traces — OTLP/HTTP + protobuf only (no gRPC), experiment selected by the x-mlflow-experiment-id header, bearer auth, gzip accepted from 3.7, SQL backend store required. Association is experiment-level only; there is no documented run-id header, so trace↔run correlation rides span/resource attributes we stamp ourselves (run id, turn, domain) plus matching tags on the MLflow run.
• Runs: plain tracking REST — experiments/get-by-name/create, runs/create, runs/log-batch (params: domain, model, effort, manifest knobs; metrics: gate score / cost / tokens, per-turn as steps), artifact upload via the proxied mlflow-artifacts route, runs/update to close. Mapping: domain (or rig) → experiment, crucible run → MLflow run, turn → metric step.
• Dependencies: no maintained Rust MLflow client exists; the exporter is a thin reqwest REST module plus opentelemetry-proto (serde feature) for the JSON→protobuf re-encode. Both already-idiomatic, well-maintained picks; nothing exotic.
• Confidentiality: with the content flags on (§env injection), exported traces carry prompt text and tool output — the MLflow instance must sit inside the same trust boundary as the state volume, and the exporter stays a per-deployment opt-in (tracking URI + token
  • experiment mapping in controller config, off when absent).

What stays on stdout

Live progress (CRUCIBLE_SCOPE_PROGRESS: beats, human-readable narration) stays on stdout by design: it is loss-tolerant, ordering-tolerant, and consumed live by the SSE relay (turn_live.rs) — logs are exactly the right transport for it. The boundary rule: stdout carries what a human tails; the contract carries what the controller acts on.

Migration

Three phases, each shippable alone, old/new compatible in both directions throughout:

1. Contract crate + Tier 1. Engine writes the termination message and keeps emitting markers; controller prefers the termination message and falls back to the marker scrape (which also covers pods launched by an old engine image). Startup adoption switches to pod status, fixing the scope-turn gap. Ships alone with zero coordination risk.
2. Tier 2. Controller mints tokens and injects CRUCIBLE_INGEST_*; engine POSTs artifacts when the env vars are present and falls back to marker emission when they are absent (old controller + new image keeps working). Run-session delivery moves onto the same endpoint family, retiring the === SESSION === delimiter and the 64 MiB scrape.
3. Demotion. After both sides are rolled and a soak, the engine stops emitting payload markers (pack/transcript base64 lines and the report/verdict lines disappear from logs), and the controller's scrapers + duplicated literals are deleted. Progress beats remain.

The compatibility matrix is the fallback chain itself: new controller + old image → markers; old controller + new image → markers (no ingest env, marker emission still on until phase 3); new + new → the contract. Phase 3 is the only step that burns a bridge, and it is gated on the fleet being past phases 1–2.

Failure modes, named

• Kubelet truncation: prevented by the 3584-byte self-cap; a manifest digest that doesn't parse is a structural bug, not an operational one.
• Pod dies before writing the message (OOM, eviction, wrapper crash): phase=Failed with an empty message — the same information content as today's no-marker log, handled by the same failed-turn path, now distinguishable from "succeeded but unparseable."
• Controller down during the turn: auth state lives in the API server, not the controller, so nothing to recover; the pod's POSTs retry through the outage window, and whatever didn't land is recovered by adoption reading the termination message + manifest and flagging undelivered artifacts. A turn is never lost to a controller restart again.
• Duplicate delivery (pod retry, network retry): content-addressed dedup; idempotent 200.
• Stolen token: audience-locked to the ingest endpoint (useless against the kube API or any other service), pod-bound (only impersonates the one turn pod, and the API server kills it when that pod dies), and the endpoint is write-only — blast radius is "attacker can upload a wrong artifact whose digest won't match the kubelet-authenticated manifest."
• Kube API unavailable at validation time: TokenReview fails closed (503 to the pod, which retries on its backoff budget); the termination-message manifest keeps the miss loud.

What this is NOT

• Not a message bus, not steering. One-shot result delivery at turn end. The ADR-0001 wall stands: nothing flows controller→pod mid-turn through this contract, and progress stays advisory.
• Not a CRD, not annotations, not a sidecar. The survey is unambiguous that all three are migrations waiting to happen (Argo's 256 KiB→1 MiB ladder, Tekton's TEP-0127 RBAC/overhead costs). We have a controller with a disk; we use it.
• Not the inner loop's state. A local crucible --manifest run still never needs a controller; result mode is opt-in exactly like --marker is today, and the session log + git remain a run's complete record (ADR-0016 unbroken).
• Not human-facing API surface. The ingest route lives outside the SPA/OpenAPI-typed surface's auth model, is write-only, and never appears in the UI's generated client.
• Not the controller's observability stack. The OTLP collector receives the agent's telemetry as run evidence; the controller's own prometheus /metrics + ServiceMonitor stack is a different concern and untouched by this ADR.

Consequences

• Positive: the verdict path stops depending on log rotation, newline discipline, and base64-in-logs; scope turns survive controller downtime (adoption gap closed by construction); artifact caps become honest HTTP limits instead of log-budget gymnastics; six keep-in-sync literal pairs collapse into one typed crate; the failure story upgrades from "silently truncated" to "row flagged with exactly which artifact didn't land"; authoritative cost and per-model usage return to the ledger (the pricing-table estimate demotes to a fallback), and the harness becomes a consumable crate again instead of a bin-shaped fossil.
• Negative / cost: the controller gains a pod-facing HTTP surface (new attack surface, mitigated by write-only + audience/pod-bound tokens) and its ingest handler couples to the kube API (TokenReview, cached, failing closed); the state volume now holds evidence payloads, so disk sizing and retention become real (pointer-rows stay small; the volume doesn't); a three-phase migration must actually be driven to phase 3 or we run both contracts forever; crucible-contract and crucible-harness are two more crates whose semver discipline matters from day one, and the harness extraction is a large mechanical refactor of the engine's core.

Workstreams

Slicing: R0+R1 ship together (pure win, no coordination). R2 behind them; R3 rides R2's plumbing; R4 only after a fleet soak proves both directions of the compat matrix were honored. R5 is independent of R2–R4 and can run in parallel once R0 lands: the extraction + collector are engine-local, usage piggybacks on R1's envelope, and the otel-log upload piggybacks on R2's endpoint (until R2, the otel jsonl just stays a run-dir file like every local artifact). R6 is last by construction — it consumes what R2 delivers and R5 produces.

Appendix: prior-art survey (full notes, 2026-07-06)

The condensed lessons live in §"Prior art"; these are the full survey notes with sources, kept so the numbers and rejections above stay auditable.

Tekton Pipelines

Task results ride the container termination message: 4096 bytes per container, 12 KiB per pod divided equally among all containers (init containers included — 12 containers means 1024 bytes each). Hitting the ceiling produced two design cycles:

• TEP-0086 "Changing the way result parameters are stored" (status: Proposed) surveyed the alternatives: a dedicated storage API service (OIDC tokens from pods, backend-agnostic), a result sidecar uploading to external storage, ConfigMaps per TaskRun (rejected: 3+ API requests per TaskRun, still 1.5 MiB-limited), a custom CRD (same ceiling), PVCs (complexity/ perf concerns), and logs — rejected outright for having "no availability guarantee". https://github.com/tektoncd/community/blob/main/teps/0086-changing-the-way-result-parameters-are-stored.md
• TEP-0127 "Larger Results via Sidecar Logs" (status: Implemented) shipped the escape hatch: a Tekton-injected sidecar watches /tekton/run, waits for all steps, prints results to its stdout in a parsable pattern, and the controller reads the sidecar's logs. Costs: ~3 s extra pod startup, the controller needs pod-log RBAC, per-result size still gated by the max-result-size flag (default 4096 bytes), total still capped by the 1.5 MiB CRD limit (TaskRun fails beyond it), and tasks assuming large results break on unconfigured installs. https://github.com/tektoncd/community/blob/main/teps/0127-larger-results-via-sidecar-logs.md

Motivating issues: https://github.com/tektoncd/pipeline/issues/4012, https://github.com/tektoncd/pipeline/issues/4060; docs: https://tekton.dev/docs/pipelines/tasks/.

Argo Workflows

Output parameters are capped at 256 KiB (the pod-annotation limit — parameters were originally reported by the wait sidecar patching pod annotations) (https://argo-workflows.readthedocs.io/en/latest/walk-through/output-parameters/); reporting later moved to the internal WorkflowTaskResult CRD (higher capacity, needs garbage collection) (https://github.com/argoproj/argo-workflows/blob/main/manifests/base/crds/full/argoproj.io_workflowtaskresults.yaml). Aggregate workflow status fights the 1 MiB etcd object limit with, in order: node-status compression (~20:1), opt-in SQL offload (nodeStatusOffLoad: true), and (v3.7+) container-args offload to ConfigMaps past 128 KiB (https://argo-workflows.readthedocs.io/en/latest/offloading-large-workflows/). The documented recommendation for anything larger is artifacts to an artifact repository (S3/GCS), never parameters — emit a small count parameter and index into the artifact (https://github.com/argoproj/argo-workflows/blob/main/examples/handle-large-output-results.yaml). The wait sidecar collects outputs and reports to the controller; it originally mounted docker.sock (bypassing RBAC) and was migrated to the Kubernetes API via service account.

Kubernetes primitives

terminationMessagePath (default /dev/termination-log) is read by the kubelet into containerStatuses[].state.terminated.message, atomically, as part of pod status, immediately on container termination. Limits: 4096 bytes per container, 12 KiB per pod total (divided equally), and truncation is silent. terminationMessagePolicy: FallbackToLogsOnError substitutes the log tail only when the file is empty and the exit was an error, capped at 2048 bytes or 80 lines, whichever is smaller. The path is immutable after pod creation, and the docs scope the mechanism to "brief final status, such as an assertion failure message". https://kubernetes.io/docs/tasks/debug/debug-application/determine-reason-pod-failure/

GitHub Actions runner

Per-job credential pattern: a job-scoped OAuth token is generated when the job is queued, lives for the job duration (default 6 h timeout) plus 10 minutes, is held only in memory, and is delivered inside a job message encrypted to the runner's RSA public key; each action subprocess receives it as an env var registered as a secret (scrubbed from logs). Artifacts upload through the authenticated API with that token. https://github.com/actions/runner/blob/main/docs/design/auth.md, https://docs.github.com/actions/security-guides/automatic-token-authentication

Buildkite agent

Same shape: an internal per-job access token generated at job start, exposed as BUILDKITE_AGENT_ACCESS_TOKEN, scoped to the single job, dead when the job finishes — distinct from the long-lived cluster-wide agent registration token. Used for artifact upload, annotations, metadata. https://buildkite.com/docs/agent/v3/tokens, https://buildkite.com/docs/apis/rest-api/artifacts

Temporal

Activity completion is an RPC carrying an opaque task token, single-use and scoped to one execution attempt — invalidated when the attempt retries, which is why Temporal recommends external services key off Workflow Run ID + Activity ID, never the token (https://docs.temporal.io/activities). Semantics: workflows are exactly-once, activities are at-least-once — the server does not deduplicate completions, so idempotency is the completer's job, enforced at the receiving service (https://temporal.io/blog/idempotency-and-durable-execution, https://community.temporal.io/t/is-a-system-generated-activity-id-suitable-to-use-as-an-idempotency-key/13181/2).

Prow, Airflow, Kueue (brief)

• Prow: pod utilities (initupload, sidecar) push artifacts to GCS under workload identity; finished.json marker objects signal completion and the crier watcher reports onward — sidecar-upload plus polled completion markers, the multi-controller shape we'd converge on past a single controller. https://docs.prow.k8s.io/docs/spyglass/
• Airflow: XCom results default into the metadata database, which bloats under artifact traffic; the documented fix is a custom XCom backend on object storage — the same pointers-not-payloads split as ADR-0016, by convention rather than contract. https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/xcoms.html, https://github.com/apache/airflow/issues/15761
• Kueue: admission-controller model over Job status; no custom result-return path — not relevant to this contract.