crucible

Crucible is a goal-driven, gated, keep/discard loop for letting an agent improve a codebase against a frozen objective. Each iteration the agent reads the history, forms one hypothesis, changes the world, and is gated on a single measurement: keep if better, else revert. The loop itself is domain-neutral, it names nothing about any one problem.

A domain typically ships one or both of two gate shapes:

• bench (perf goals): replay a frozen workload against the candidate, minimize a latency or throughput metric. The first production domain measured an inference router GPU-free this way, via the vllm-vcr workload simulator.
• test (bug-fix goals): run the affected packages' test suite; a green suite with a new regression test wins.

One domain need not mean one component. A composite domain assembles several component domains into one deployment; a single agent turn can edit any component, and one combined gate scores the assembled stack (ADR 0009).

The pieces

The crucible engine is a Cargo workspace; domain packs live in their own repositories and are loaded via --manifest.

• crucible/: the domain-neutral engine binary: manifest load, the keep/discard loop, front-ends, resume, and crucible deploy (it renders its own loop/deployment manifests, ADR 0012). The World + Judge traits are the domain boundary; a top-level [composite] manifest assembles several components into one run.
• forge/: engine-side build + deploy: buildah for real source builds, a native-OCI layer-append path (no container runtime), and a kube-rs client for apply/rollout.
• crucible-broker/: the mediated provisioning broker (the agent asks, the host holds the keys): GPU/trace capture, issue-tracker grounding, the profiler, and draft-PR publishing, all over MCP (ADR 0002). A domain may layer a thin per-domain binary on top of it.
• tools/*.nu: general control-plane glue installed as bare names on PATH (steer, stop, escalate, session, goal-from-issue).

Where to go next

How it works

The whole system in one diagram: a goal goes in, an agent proposes a change, the change is applied to a reversible world, a frozen judge measures it once, and the loop keeps it only if it strictly beats the best so far. Git is the memory; the operator can steer; the privileged operations live host-side behind a mediated broker the agent can only ask.

flowchart TD
    issue["GitHub issue / Jira ticket"] -->|scope --issue| goal["Run goal<br/>(frozen objective)"]
    goal --> wide["wide round (optional)<br/>N parallel propose → rank → winner seeds"]
    wide --> propose

    subgraph control["Control plane (operator, human-in-the-loop)"]
        direction LR
        budget["budget"]
        steer["steer"]
        stoppark["stop / park"]
        resume["resume"]
        escalate["escalate"]
    end

    subgraph loop["The keep / discard loop"]
        direction TB
        propose["propose<br/>agent backend: local · openshell · command"]
        apply["apply<br/>World: edit tree, or build + set image"]
        measure["measure<br/>Judge: frozen, scores once"]
        accept{"accept?<br/>strictly better?"}
        remember["remember<br/>git commit + session NDJSON"]
        restore["restore<br/>World rollback to last good"]

        propose --> apply --> measure --> accept
        accept -->|keep| remember
        accept -->|discard| restore
        remember -->|next iteration| propose
        restore --> propose
    end

    control -. "steer / stop / resume / budget" .-> loop

    subgraph broker["Mediated MCP broker: host holds the keys, agent only asks"]
        direction TB
        prov["provisioning<br/>GPU / Kueue, capture RBAC"]
        jira["issue-tracker grounding"]
        prof["profiler<br/>pprof / GPU traces"]
        forge["forge build + deploy<br/>buildah / native-OCI derive-layer, digest-pinned"]
        draft["draft PR per fork"]
    end

    propose -. "MCP ask" .-> broker
    forge -. "image + set image" .-> apply

    remember -->|publish-on-keep| s3[("S3 run records<br/>session.jsonl · summary.json · diffs")]
    remember --> draft
    draft -->|review comments re-steer| steer
    remember -->|ingest API, optional| ctl["control plane<br/>(crucible-contract ingest)"]

The stages

Glossary

The loop leans on a few overloaded words. The two that trip people up most are World and Judge: they're the trust boundary, and they mean something specific here.

For the prose version of the same model, see What crucible is.

Crucible: what it is, in one read

Crucible is a goal-driven, gated, keep/discard loop for letting an agent improve a codebase against a frozen objective. The loop itself is domain-agnostic; a domain is a problem packaged for it.

This doc explains the whole system in higher-order concepts and then shows exactly how the examples/counter domain maps onto them. For why the judge is frozen, see ADR 0001.

Scope. The engine, the World/Judge boundary, CommandWorld/CommandJudge, and the crucible.toml manifest form the core. A composite manifest (ADR 0009) exercises the engine across multiple components. Around the loop: mediated provisioning + issue-tracker grounding over an MCP broker (ADR 0002), a profiler over MCP (ADR 0006), engine-side build/deploy (ADR 0005) with rendered deployment manifests (ADR 0012), and publish-on-keep with a draft PR per fork whose authorized review comments re-steer the run.

The one sentence

An optimization loop where the proposal step is an LLM agent and the objective is a frozen, expensive black-box evaluation, over a reversibly-mutable world, with durable memory and human-in-the-loop control.

The loop

   ┌─────────────────────────── budget / steer / stop / escalate ──────────────────────┐
   │                                                                                    │
   ▼                                                                                    │
 propose ──▶ apply ──▶ measure ──▶ accept? ──▶ remember ──▶ (next iteration) ───────────┘
 (agent)    (World)    (Judge)    keep|discard  (git +
                                   │            session log)
                                   └─ discard ▶ restore (World) ▶ next

• propose: the agent edits the world toward the goal. The proposal policy is a pluggable backend (local in-process Claude / openshell sandboxed / command), selected by [agent].backend, not the engine.
• apply: make the candidate live (for a code repo: the edits are the apply).
• measure: the frozen Judge scores the candidate.
• accept?: keep if it's strictly better; otherwise restore the last good state.
• remember: kept states are git commits; every step is an NDJSON event.

This single-candidate line of descent is the default, and it's not the only shape: a wide round (ADR 0010) can fan N independent propose turns out in parallel first, biased to distinct [search].approaches, rank them by the same Judge, and seed the loop above with the winner. --wide 0 (the default) skips it entirely, no engine change, just a pre-loop step.

The contract: framework owns it, the repo owns the implementations

Crucible defines a small contract; a repo fills it however it likes, in any language. A domain is a manifest + a few executables, not Rust code. Each command runs in the workspace with the domain's env:

The accept policy is generic: keep if valid and score beats the best per direction; solved iff measure says so. No per-domain code for the keep/win logic.

Languages: pick per command, the engine doesn't care

The contract is JSON + exit codes, so each command can be whatever fits:

• Thin glue (apply / ops / steer / stop): nushell is the recommended default. The work is "run kubectl/prometheus, parse, compute," and nushell is structured-data native (... | from json | ... in, a record | to json out), cleaner than bash, lighter than python. (It's its own shell, not posix; pin nu in the sandbox image.)
• Heavy measure with real math (percentiles, replay): a compiled binary (Rust/go) earns its keep.
• bash / python / anything else is equally valid: only the JSON/exit shape is fixed.

measure_cmd is any executable that prints one JSON object on stdout ({"valid": bool, "score": number, "note": string}). A plain POSIX shell script satisfies the whole contract:

#!/bin/sh
p99=$(make bench | jq .p99_ms)
printf '{"valid": true, "score": %s, "note": "p99 %sms"}\n' "$p99" "$p99"

Any language works the same way, because the engine reads only the JSON verdict.

What the engine gives you for free

An optional private control plane can sit above all of this: a controller that discovers candidate issues, scopes them into packs, launches runs, and renders run records into leaderboards. It consumes the public crucible-contract ingest API over HTTP; the engine and loop pods never link it, and everything on this page runs identically without one.

Trust boundary (ADR-0001): the engine hands the agent a World, never a Judge. The objective is frozen and out of the agent's reach: the agent can't tune the test it's graded on. The same boundary runs through provisioning: the agent can only ask the broker (over MCP); the privilege (GPU/Kueue, capture RBAC, the forge token, the issue-tracker credential) stays host-side, never in the agent's sandbox.

The three extension points

• New domain → common case: a manifest pointing CommandWorld + CommandJudge at a repo and a measure command. No Rust. Custom World/Judge impls only for in-process measurement or a non-file world.
• Composite domain → assemble N existing component domains into one run: a top-level [composite] table referencing component domain dirs (reused verbatim), a CompositeWorld (a tuple of per-component git overlays + the live deployment), and one combined gate over the assembled stack. No engine changes: the engine just runs a vector of workspaces (ADR 0009).
• New frontend → implement Reporter.
• New agent transport → add an AgentSource.

The canonical example: the counter, mapped to the contract

examples/counter/ is the whole system with every heavy part swapped for the smallest thing that satisfies the contract. Five small files, and each one fills a contract role. This table is the example:

Because there is no [world] block, GitWorld supplies reversibility for free: kept iterations are commits, discards are git reset --hard. Because the backend is command, the whole loop runs in milliseconds and costs nothing, yet it is a real end-to-end run (manifest → setup → propose → measure → keep/discard → git memory), not a mock.

A production domain fills the same roles with heavier pieces. A serving-stack domain has two reversible things (the code tree and a live deployment), so it adds snapshot_cmd/restore_cmd that capture image + config alongside the git half; its measure becomes a compiled benchmark binary; its propose backend becomes local or openshell; and its thin glue (apply pipelines, deployment introspection, steer/stop) lands on PATH as bare names so the contract vocabulary reads the same regardless of what's behind each one. A normal repo needs none of that: one measure command and the git default.

Adding a new domain (the target workflow)

1. crucible init scaffolds a starter crucible.toml + measure stub onto the repo (or write one by hand):

   [repo]      url|path, ref
   [workspace] setup_cmd          # optional; default: git clone + checkout
   [agent]     model, method_prompt, goal_file|goal, toolbox_dir, env
   [judge]     measure_cmd, direction = "lower"|"higher"
   [world]     apply_cmd?, snapshot_cmd?, restore_cmd?   # omitted → git
   

2. Write a measure command (any language) that prints {valid, score, solved?}.
3. crucible check --manifest crucible.toml validates it (every referenced file resolves, measure_cmd runs once and prints the contract shape) with no agent turn spent.
4. Run crucible --manifest crucible.toml. You get the loop, budget, all frontends, steer/stop/resume, the session log, and escalation, free.

The litmus test for "framework, not demo": any second repo runs this way with no new Rust.

Concept → code (where to look)

• loop / budget / keep-discard: crucible/src/run.rs + crucible/src/loop_driver.rs
• the contract traits: crucible/src/crucible.rs
• World/Judge batteries: crucible/src/command_world.rs (GitWorld/CommandWorld/CompositeWorld), crucible/src/command_judge.rs (CommandJudge)
• composite domains: crucible/src/manifest.rs (CompositeManifest) + crucible/src/run.rs (run_composite)
• mediated broker (provisioning / issue tracker / profiler / draft PR): crucible-broker/
• engine-side build + rendered deploy: forge/ (build, native-OCI derive, kube-rs) + crucible/src/deploy/
• agent transport: crucible/src/agent.rs
• frontends: crucible/src/{console,jsonl,stream}.rs
• session log wire format: crucible/src/session.rs
• the example domain: examples/counter/ (manifest + measure.nu + bump.nu)

Getting started: zero to a running loop

This is the hands-on path from nothing to a crucible loop. It's deliberately end-to-end, two acts:

1. Run a loop locally — the hello-world domain, no cluster, no LLM, runs in milliseconds.
2. Onboard your own domain — the manifest + the measure contract, the only two things you have to write.

For the why behind any of this, read How it works (the diagram + glossary) and What crucible is first. The manifest and command shapes below are the hands-on expansion of the contract, which is the normative reference. When this page and the contract disagree, the contract wins.

Before you start

Tooling (from the README): Rust + cargo, go, kubectl/oc, helm/helmfile, just, nu (nushell, the glue language), podman/buildah, gitleaks, and the claude CLI. Act 1 only needs Rust, just, and nu.

Credentials (Act 2 with a real agent backend): Claude access for the claude CLI. The command backend and crucible init/check/scope need no credentials at all.

Build the binary. Clone the repo (Act 1 uses the examples/counter files anyway) and build from source:

git clone https://github.com/neuralmagic/crucible.git
cd crucible
cargo build --release -p crucible
install -m 755 target/release/crucible ~/.local/bin/   # or anywhere on PATH

Or grab a prebuilt binary from the releases page when one is available for your platform. With just the binary you can already onboard your own repo (Act 2's crucible init → crucible check → crucible scope path needs nothing else).

One convention worth knowing before Act 2: a manifest refers to a pack's tools by bare name. The counter example sidesteps this entirely (its bump.nu and measure.nu live in the pack directory and are invoked by relative path); a pack that ships tool binaries is responsible for putting them on $PATH before the loop runs.

Act 1 — Run a loop locally

The fastest way to see the whole engine is examples/counter. The "repo under test" is a git tree whose entire source is an integer in value.txt; the score is that integer; a deterministic command backend (bump.nu) stands in for the LLM so it runs for free. It exercises the real generic path (manifest → setup → propose → measure → keep/discard → git memory) with no cluster and no model.

crucible --manifest examples/counter/crucible.toml --iterations 6
# or, from a source checkout: cargo run -p crucible -- --manifest examples/counter/crucible.toml --iterations 6

The manifest is the whole story (examples/counter/crucible.toml):

[repo]
path = "."

[workspace]
dir = "workspace"
setup_cmd = "mkdir -p workspace && cp value.txt measure.nu bump.nu workspace/ && git -C workspace init -q && git -C workspace add -A && git -C workspace -c user.email=crucible@local -c user.name=crucible -c commit.gpgsign=false commit -qm baseline"

[agent]
backend   = "command"
agent_cmd = "./bump.nu"   # deterministic proposer: value.txt += 1
goal      = "Raise the integer in value.txt as high as you can (target: 5)."

[judge]
measure_cmd = "./measure.nu"
direction   = "higher"
objective   = "value"

# No [world] block -> GitWorld: kept iterations are commits, discards reset --hard + clean.

What happens each iteration: the engine runs agent_cmd to mutate the workspace, runs measure_cmd to score it, and keeps the change (a git commit) iff it's strictly better per direction, else rolls the workspace back. What you get out:

• examples/counter/state/session.jsonl — the append-only NDJSON event log (the source of truth for a run).
• The workspace git history — kept iterations are commits. Git is the memory.

That's the engine. Everything else is swapping the deterministic proposer for a real agent and pointing measure at a real objective.

Act 2 — Onboard your own domain

A domain is a manifest plus a few executables, not Rust. The litmus test for "framework, not demo" is that a new domain runs with no engine changes. You write two things: a crucible.toml and a measure command.

crucible init scaffolds both onto the current directory (a minimal manifest + a measure stub that always reports the same score), so onboarding starts as editing two generated files rather than writing them from scratch. Once you've pointed [judge].measure_cmd at the real gate, crucible check --manifest crucible.toml validates the whole thing (every referenced file resolves, measure_cmd runs once and prints the contract shape, and it warns if the gate sits unprotected inside the workspace the agent edits) before you spend a real agent turn on it. crucible scope --pack <dir> chains ingest (resolve the goal) → check → freeze (SCOPE.md with a RunIdentity digest) into one pipeline (ADR-0014 S0); add --issue owner/repo#N to ground the goal in a GitHub issue (native API fetch, honors GITHUB_TOKEN/GH_TOKEN).

The manifest

Sections (full schema + per-field semantics in the contract; the parser rejects unknown keys, so typos fail loudly):

The single most important rule: omit [world] entirely and you get GitWorld (git is the only thing that needs rolling back). Set any of the three *_cmds and you get CommandWorld (git plus your snapshot/restore, e.g. a live cluster).

The measure contract

measure_cmd is the judge. Its last stdout line beginning with { is parsed as JSON:

{ "valid": true, "score": 234.1, "solved": false, "note": "p99 234.1ms" }

• valid (required bool), score (number), solved (bool, default false), note/detail (optional).
• A nonzero exit forces valid=false regardless of what you printed.
• The engine injects CRUCIBLE_BASELINE_SCORE, CRUCIBLE_BASELINE_TOTAL, CRUCIBLE_BEST_SCORE so your gate can compare.
• Keep rule: keep iff valid and score strictly beats the best per direction (or solved is true, which forces a keep and ends the run, that's the bug-fix-gate case).

measure_cmd is any executable that prints one JSON object. A plain sh script is a complete judge:

#!/bin/sh
p99=$(make bench | jq .p99_ms)
printf '{"valid": true, "score": %s, "note": "p99 %sms"}\n' "$p99" "$p99"

Any language works; the engine only reads stdout.

Backends: who actually proposes

Set by [agent].backend:

• local (default) — spawns the claude CLI on this machine in the workspace. Needs the CLI + Claude creds.
• openshell — runs the turn inside a sandboxed pod (Landlock + deny-by-default egress). Needs sandbox_image + an [agent.openshell].endpoints egress allowlist. This is the in-cluster path.
• command — a fixed sh -c command as the "proposal," deterministic, no LLM, cost 0. The Act 1 counter case.

Composite domains

A manifest with a top-level [composite] table assembles ≥2 existing component domains (reused verbatim) into one run with a single combined gate. Picture a server domain and a router domain: each keeps its own workspace and repo, but one measure_cmd scores the assembled system, so a change to either component is judged by how the pair behaves together. No engine changes; the engine just runs a vector of workspaces. Beyond the manifest, the work of a composite is the shared plumbing: a combined gate that exercises both components, and per-component deploy targets if the domains rebuild images.

Where to go next

• How it works — the full diagram + glossary.
• What crucible is — the conceptual deep-dive and the contract table.
• Implementation contract — the normative manifest + command spec.
• JIRA tools (mediated) — grounding a goal in an issue.

Crucible implementation contract (frozen interface)

This is the spec for the interface between the engine (crucible) and a domain. The engine implements it, a domain satisfies it, and a minimal fake domain (examples/counter/) tests it end-to-end with no EPP and no cluster.

Concept: What crucible is. Trust line (engine hands the agent a World, never a Judge): ADR 0001.

Contract status: NORMATIVE. Words like must / exactly one are binding. The engine and every domain (EPP included) conform to this; tests assert it.

1. The manifest (crucible.toml)

The engine reads exactly one manifest per run. Default path ./crucible.toml (override --manifest <path>). The manifest directory = dirname(manifest); it anchors all config-relative paths below.

[repo]
# Exactly one of url|path. The upstream the agent edits a checkout of.
url  = "https://github.com/owner/name.git"   # OR
path = "."                                    # local path, relative to manifest dir
ref  = "main"                                 # optional; default: clone default branch

[workspace]
dir       = "workspace"                       # checkout dir, relative to manifest dir. default "workspace"
setup_cmd = "git clone ... && git checkout"   # optional; default: engine git clone+checkout of [repo]

# Frozen-judge / fixture injection (optional, repeatable). After setup, the engine copies each
# `src` (relative to the manifest dir, a baked artifact outside the clone target, so the agent has
# no pre-clone copy) to `dst` (relative to the workspace). A `frozen` inject (default true) is ALSO
# re-copied before every scored measure, so a candidate can't edit the gate (a T1 scoring harness, a
# seeded regression test) to game it; set `frozen = false` for a one-time fixture the agent may then
# edit. This is the generic alternative to hand-chaining an `install` into `setup_cmd`.
[[workspace.inject]]
src    = "judges/1489/judge_harness_test.go"  # baked judge, manifest-relative
dst    = "pkg/.../judge_harness_test.go"      # into the clone, workspace-relative
frozen = true                                 # re-copied before each measure (default true)

[agent]
model         = "claude-opus-4-6"             # default engine constant
method_prompt = "method.md"                   # manifest-relative file; template, {{GOAL}}/{{STATUS}}/{{STEER}}
goal          = "raise the score"             # OR goal_file = "goals/x.md" (manifest-relative)
toolbox_dir   = "commands"                    # optional; copied into <workspace>/.claude/skills
backend       = "local"                       # local | openshell | command   (see §6)
sandbox_image = "ghcr.io/<org>/<domain>-sandbox:<tag>"  # openshell backend only
agent_cmd     = "..."                          # command backend only (§6)
[agent.env]                                   # injected into the agent process (creds, Vertex, etc.)
ANTHROPIC_VERTEX_PROJECT_ID = "my-gcp-project"

[judge]
measure_cmd = "./measure.sh"                  # REQUIRED. §3. Any executable, any language.
direction   = "lower"                         # lower | higher. REQUIRED.
objective   = "score"                         # display label (the old `gate` name). default "score"

[judge.selftest]                              # optional (ADR-0014 S1). Runs in `crucible check`, never in a loop iteration.
good_cmd = "..."                              # stages a known-good config in the workspace
bad_cmd  = "..."                              # stages a known-bad config in the workspace
runs     = 1                                  # measurements averaged per control. default 1

[world]
# All three omitted  →  GitWorld (tree-only reversibility; the 80% case).
apply_cmd    = "..."                          # optional. §3.
snapshot_cmd = "..."                          # optional. §3.
restore_cmd  = "..."                          # optional. §3.

[search]                                      # optional (ADR-0010). Absent or wide=0 → pure-deep, the default.
wide       = 0                                # u32. N parallel propose turns before the deep loop. default 0
approaches = ["...", "..."]                   # REQUIRED when wide > 0: one distinct approach string per candidate slot
policy     = "top-k"                          # only v1 policy. default "top-k"
policy_k   = 1                                # how many wide-round winners seed the deep loop. default 1, must be in 1..=wide

Rules:

• Required: [repo] (url xor path), [judge].measure_cmd, [judge].direction.
• [world] with no commands ⇒ GitWorld (§5). Any command given ⇒ CommandWorld (§5), which still owns git memory and layers the given commands on top.
• [judge.selftest], if present, requires both good_cmd and bad_cmd (a self-test that only stages one side isn't a control); runs must be >= 1.
• [search], if present with wide > 0, requires approaches.len() >= wide (hard error, diversity is engineered, not auto-generated) and policy_k in 1..=wide.
• Unknown keys are an error (typo protection), not silently ignored.
• Frozen loading (load_frozen). When the manifest file lives inside the workspace it targets (the BYO on-ramp: crucible init scaffolds [repo] path = "."), the engine parses it from the workspace's pristine base commit, not the current working tree (so an in-flight agent edit to crucible.toml can't retarget its own gate mid-run). Before any base commit exists (the very first run), it hard-warns and trusts the working tree for that run only; later runs freeze to the base commit. A manifest that lives outside the workspace (any out-of-workspace pack) is unaffected.

1.1 The gate self-test (crucible check, ADR-0014 S1)

[judge.selftest] declares two controls the gate must tell apart before it's trusted. crucible check (§9) runs it pre-loop, never inside a loop iteration:

1. snapshot the pristine workspace,
2. restore to pristine, stage good_cmd, measure runs times through the domain's own Judge, restore to pristine again,
3. same for bad_cmd,
4. pass iff both controls' readings are all valid and good's mean score is strictly better than bad's per [judge].direction.

The workspace is restored to pristine on every exit path (pass, fail, or error). A manifest with no [judge.selftest] isn't an error, crucible check warns instead, since the gate hasn't been proven to discriminate.

1.2 Wide-round search ([search], ADR-0010)

[search].wide > 0 (or --wide N on the CLI, which overrides the manifest) fans out N independent PROPOSE turns in per-candidate git worktrees under the state dir before the deep loop starts, one turn per approaches entry biased into its prompt. Each candidate's diff is applied (cherry-picked) into the shared main workspace and measured serially there (measurement never runs concurrently, only proposal does). The scored set is ranked by [search].policy (v1: "top-k"); the policy_k (or --wide-keep K) winner(s) seed the deep loop, which then runs as normal. Session rows from the wide round carry an additive phase: "wide" field (§7) so a consumer can tell a wide-round row from a deep-loop row without a wire-shape change.

2. Path resolution (portable, never CARGO_MANIFEST_DIR)

The binary's own install location is never used to resolve anything. A target repo is self-describing: drop a crucible.toml at its root and run crucible inside it.

3. The command protocol

Every command (measure_cmd, apply_cmd, snapshot_cmd, restore_cmd, setup_cmd) is a string executed via sh -c "<cmd>", with:

• cwd = the agent workspace (manifest_dir/[workspace].dir).
• PATH inherited, so a command may be a bare installed tool (bench) or a workspace-relative script (./measure.sh).
• env = the engine's env + [agent].env (where relevant) + the injected variables below.

measure_cmd (the Judge): REQUIRED

• Injected env: CRUCIBLE_BASELINE_SCORE, CRUCIBLE_BASELINE_TOTAL, CRUCIBLE_BEST_SCORE (absent on the baseline measurement; present thereafter). Values are the engine's current numbers as decimal strings.
• stdout: the engine reads the last line that starts with { and parses it as:

  { "valid": true, "score": 12.5, "solved": false, "note": "p99=12.5ms", "detail": { } }
  

  • valid (bool, REQUIRED): false ⇒ unscoreable candidate, always discarded.
  • score (number|null): the fitness. null/absent ⇒ treated as invalid.
  • solved (bool, optional, default false): the win condition was met (terminates the loop).
  • note (string, optional): one-line human summary.
  • detail (object, optional): free-form; surfaced in the row + session log. The domain stashes anything extra here (e.g. EPP cache_hit_rate; the test gate's total).
• exit code: nonzero ⇒ the reading is forced valid:false regardless of stdout.

apply_cmd (make the candidate live): optional

• Run after the agent turn, before measure, when present. Nonzero exit ⇒ the iteration is treated as an invalid candidate (discard). No stdout contract.
• Omit for code/agent-deploys domains (EPP: the agent deploys via skills during its turn; the counter: the edit is the candidate). Present for engine-driven build+push+set-image.

3.1 Build mode: how an edit becomes the thing measured

Between "the agent edited a file" and "measure_cmd read a score" sits a step whose shape you must choose when you design a pack. It differs per component, it is the dominant term in per-iteration wall-clock, and picking it wrong fails silently. Full rationale in ADR-0020.

Rules that are easy to get wrong, and that crucible check should catch before you spend a turn:

• derive-layer requires that the base image and the push target share a registry. The layer mounts server-side only when they match; across registries the same operation streams every base blob (8 seconds becomes 20+ minutes) and nothing warns you.
• derive-layer cannot carry compiled sources. Appending a .go file to an image changes nothing that runs. The loop measures the base image forever and reports every candidate as a no-op, the failure mode ADR-0007 exists to catch.
• A compile failure must be distinguishable from a bad score. In image mode the build is the loop's fastest feedback signal: a compile error is returned to the agent as a free retry with no candidate spent. An apply_cmd that collapses "did not compile" into "scored badly" throws that away.
• A Containerfile on the measured path is part of the judge, not part of the solution. If the build recipe lives in the agent's workspace, it must be a frozen = true inject (§1), re-copied before every scored measure. Otherwise the agent can edit the recipe that builds the artifact it is scored on (vendor a prebuilt binary, neuter the compile), which is the ADR-0001 trust line, broken.

snapshot_cmd / restore_cmd (domain reversibility): optional, must come as a pair

• snapshot_cmd: stdout last line = one opaque token (any non-empty string; base64 it if it contains newlines). Nonzero exit ⇒ snapshot failed (engine aborts the keep).
• restore_cmd: receives the token in env CRUCIBLE_TOKEN. Rolls external state back to that token. Nonzero exit ⇒ restore failed (engine surfaces it). (Env, not argv, so a multi-line base64 payload round-trips without shell-quoting.)
• The engine treats the token as opaque and never parses it. (§5 explains how the engine frames its own git ref alongside this token.)

setup_cmd (prepare the workspace): optional

• The one cwd exception: runs with cwd = manifest dir (the workspace does not exist yet, it's what setup creates). Every other command runs with cwd = workspace.
• Default when omitted: engine does git clone [repo] <workspace> && git checkout [ref]. The workspace must end up a git repo (GitWorld commits into it); for a non-git [repo].path, give a setup_cmd that copies the tree in and git inits it (see examples/counter/).

4. The decide rule (universal, no per-domain code)

Given a Reading { valid, score, solved, note, detail }, the current best_score, and the manifest direction:

keep   = valid && score.is_some() && (better(score, best_score, direction) || solved)
solved = reading.solved
better(s, b, lower)  = s < b
better(s, b, higher) = s > b

• solved implies keep. A win is the whole point, so a candidate the measure command declares solved is kept (and terminates the loop) even if its score doesn't strictly beat best. This is load-bearing for any domain whose win lands at an equal score: EPP's test gate wins with a green suite (0 failures == the baseline's 0) plus a new regression test. Without || solved that win would be discarded and the loop would never finish. solved never rescues an invalid reading.
• The first valid reading sets the baseline (best_score, and baseline_total = detail.total if present) and is always kept.
• The loop terminates when a kept iteration is solved, or budget/iterations exhausted, or stop/escalate.
• No domain Rust decides anything. A win condition more complex than "better score" (e.g. EPP's "green AND a new regression test") is computed inside the measure command, which reads CRUCIBLE_BASELINE_TOTAL and emits solved.

5. World = reversibility (engine never names git)

World::Snapshot = String, opaque to the engine (run_loop only round-trips it back to restore).

• GitWorld (default): snapshot() = stage+commit the workspace, return the commit SHA; restore(sha) = git reset --hard <sha> + git clean (excluding .claude/, RESULTS.md). The kept-commit chain is the memory. Works for any git repo, zero domain code.
• CommandWorld (any [world] command given): always owns git memory as above, and layers the domain commands. The snapshot token it stores is the composite "<git-sha>\t<domain-token>":
  • keep: commit (git half) → run snapshot_cmd, capture its token (domain half) → join.
  • discard: split → git reset --hard <sha> + clean → run restore_cmd with CRUCIBLE_TOKEN=<domain-token>.
  • The engine exposes last_commit_sha() (the git half) for kept_shas/publish; the domain half is never inspected.

The engine's loop body calls only world.snapshot() / world.restore(&snap). It contains no git/vcs:: calls and no kubectl.

6. Agent transport (AgentSource)

The engine renders a prompt (method_prompt with {{GOAL}}/{{STATUS}}/{{STEER}} filled), hands it + the workspace to an agent that edits the workspace, and never hands it the Judge (ADR-0001). Backends, selected by [agent].backend:

• local: direct claude --output-format stream-json with [agent].env (real Claude/Vertex turn).
• openshell: sandboxed pod turn driven by the in-Rust OpenShell driver (backend = "openshell", sandbox_image).
• command: run [agent].agent_cmd via sh -c in the workspace as the proposal. A deterministic, free proposer (no LLM). This is a real transport, not a mock: it makes the minimal example a fast, deterministic e2e (e.g. agent_cmd = "./bump.nu" increments a counter). Use it to test the engine's loop/protocol without burning tokens.

One vs two execution environments (the only backend fact a domain must know)

The engine and its measure/apply/snapshot/restore/setup commands always run where crucible runs. Only the agent turn is sandboxed under openshell. So:

• local / command: one environment. Agent and engine share the host PATH + filesystem; the agent edits the workspace in place. Provide one toolbox on PATH. Nothing else.
• openshell: two environments. (1) the engine/loop-image PATH runs the contract commands; (2) a separate sandbox image runs the agent's skills. The OpenShell driver uploads the workspace into the sandbox and syncs edits back, so anything the agent needs to reach the outside (kubeconfig, tokens) must be relayed via [[agent.relay]] or provided in [agent].env (the sandbox is network/fs-isolated). A domain therefore ships two tool surfaces under openshell: contract commands on the loop PATH, agent skills baked into sandbox_image.

The manifest [agent] block (backend, sandbox_image, env, relay, openshell) selects and configures the backend; flipping local↔openshell is config, not code. What the manifest cannot do is build the sandbox image or inject creds for you, that's the inherent cost of sandboxing, called out here so it's a known rule, not a surprise. Reversibility commands (snapshot/restore) run engine-side, so they reach the live system directly regardless of backend; only the agent is boxed.

6.1 Sandbox egress ([agent.openshell])

The sandbox is deny-by-default. Two lists open it back up, and a flag decides whether they extend the engine's built-ins or replace them:

[agent.openshell]
endpoints = ["api.example.com:443:full"]  # host:port:access[:proto[:enforcement]]
binaries  = ["/usr/local/bin/claude"]     # only these may open a socket
inherit_defaults = true                   # default

With inherit_defaults = true (the default) the lists are appended, de-duplicated, to the built-ins: the public forges, PyPI, Vertex, Anthropic, and the agent CLIs. Appending can never remove a built-in.

Set inherit_defaults = false and the resolved allowlist is exactly what the manifest names, binaries included. This is the only way to subtract a default, and it is required for two cases:

• Air-gapped or private-registry runs, where the public internet is not reachable (or not permitted) and the agent must talk to an in-cluster model endpoint and registry instead.
• Contamination control. An agent scored on an upstream issue can, with the default allowlist, read the upstream fix for that issue on github.com. A measurement that must not be polluted by the open web has to drop that endpoint, and dropping it means opting out.

The broker endpoint is auto-appended by the engine when [agent.broker].enabled is true (ADR-0019 P2). The engine first resolves the broker URL (an explicit [agent.broker].url override, or derived from the active compute driver's hostname), then derives the egress host:port:full entry from that URL's authority. When no explicit port is present, the scheme default applies (http = 80, https = 443). Because both are derived from the same resolved URL, the allowlist entry and the address the sandbox contacts cannot disagree. The broker endpoint is appended regardless of inherit_defaults, because the broker is engine plumbing the domain opted into, not a built-in the domain can subtract. A broker-less opt-out with both lists empty is still a legal total air-gap: nothing resolves, and no binary may open a socket.

7. Session wire format (compatibility)

The NDJSON session log (state/session.jsonl) keeps its existing event kinds (start/phase/row/budget/summary/finished) and field names unchanged. In particular the objective label is still written under the JSON key gate (now carrying a free-text label like "score"/"bench", not the deleted enum). This keeps --resume, the remote viewer, and already-published S3 runs loading. Do not rename the wire key.

A row event's wire record (RowWire) carries an additive, optional phase field ("wide" for a wide-round candidate row, absent for a deep-loop row). It's skip_serializing_if = "Option::is_none", so a deep-only run's wire bytes are unchanged from before wide rounds existed.

Two event kinds beyond the compat set:

• identity: the run's RunIdentity (below), emitted once at setup and again on --resume (the freshly recomputed identity). A mismatch against the original run's identity is a hard-warning note event, never an abort.
• shutdown: { outcome, reason }, emitted exactly once, as the last line of every run (after finished/summary). outcome is one of finished/solved/budget/stopped/ escalated/error. Session-log consumers key a run's terminal state off this line; a dead stream with no shutdown line means the pod likely died mid-run, not a clean exit.

RunIdentity (crucible/src/identity.rs) is the comparability key: two runs' scores are comparable only if it matches. It's a hash-of-hashes (v1:<hex>) over, per component (one unnamed entry for a single-domain run, one per [[component]] for a composite): repo ([repo].url/path) and the workspace's pristine base commit SHA; plus, once per run: the frozen manifest text's hash, a hash over every [[workspace.inject]]'s source content plus destination path, [judge].measure_cmd, and [judge].direction. It's computed once at run setup and doesn't change within a run (a re-scope moves the loop's own Segment fingerprint, a different hash over goal/objective/regime; the two are deliberately independent).

8. What a domain author writes (the whole surface)

1. a crucible.toml,
2. a measure command emitting {valid, score, solved?} (any language),
3. optionally apply/snapshot/restore commands,
4. a method prompt + goal,
5. agent creds in [agent].env.

Everything else (loop, budget, keep/discard, all reporters + remote viewer, steer/stop/resume, session log, escalation, git memory) is the engine's, for free. The litmus test (examples/counter/) exercises items 1, 2, 4 with GitWorld and the command backend, no Rust.

9. CLI surface (non-normative pointer)

The manifest/protocol above is the contract; these subcommands are mechanical consumers of it, noted here tersely so this doc stays the map of what's authoritative:

• crucible init [--dir <path>]: scaffolds a minimal crucible.toml + a measure stub that always reports the same score. Refuses to overwrite existing files.
• crucible check --manifest <path>: validates a manifest with no agent turn (parses it, unknown keys are a parse error, resolves every referenced file, runs measure_cmd once to prove the measure contract, runs [judge.selftest] if declared (§1.1), and warns if the gate is reachable by the agent's own edits: a measure_cmd token pointing inside the workspace with no matching frozen inject). Exit nonzero with findings on failure; warnings never fail it.
• crucible scope --pack <dir> [--issue owner/repo#N | --goal-file <f>] [--force] [--json] (ADR-0014 S0): pipeline over a hand-written domain pack. ingest the goal (from --issue, fetched natively from the GitHub REST API (honors GITHUB_TOKEN/GH_TOKEN and GITHUB_API_URL), or --goal-file, or the pack's own [agent].goal/goal_file), then validate (crucible check, as a library call), then freeze (writes SCOPE.md in the pack dir with the goal, the check outcome, and the pack's RunIdentity digest). Stops at the first failing stage. S2 (propose)/S3 (preflight)/S4 (approval) are listed in SCOPE.md as pending, see ADR 0014. A --goal-file's text is goal framing under the same de-prescription rules as a --issue's title/body: the problem for the pack-designing agent to solve, never a solution. §8's _controls/self-test strip at freeze applies identically regardless of which arm sourced the goal.
• crucible ps [--namespace <ns>] [--json]: lists loop pods across the cluster, selecting on the app.kubernetes.io/managed-by=crucible label every rendered loop pod carries. ITER ships as - (reserved, see ps.rs's module doc for why it isn't wired up yet).
• crucible deploy render|apply --manifest <path> --profile <path> [--iterations N] [--no-pin]: renders (or renders-then-applies) the loop pod + a cross-namespace RoleBinding from the manifest + a deploy profile, image tags resolved to @sha256:… digests. Works for a composite manifest or a plain single-domain one (the latter needs its own [deploy] block naming the build/deploy target, a single domain is a degenerate composite of one).
• crucible watch-pr --pr <url> [--pr <url> ...] (--control-addr <host:port> | --reseed <path>) [--once] [--poll-secs N] [--bot-user <login>] [--allow-user <login> ...]: watches one or more draft PRs' review comments (--pr repeatable: a kept composite candidate opens one linked PR per component fork) and either steers a live run over its control bridge or appends to a reseed file the next run's first turn reads. --once fetches and exits instead of polling.

The OpenShell fork

Crucible pins openshell-core to a fork — wseaton/OpenShell, branch crucible/grpc-base — instead of upstream nvidia/OpenShell. This page is the running ledger of why, so nobody has to re-derive it from git archaeology.

Why a fork exists at all

The branch policy is trivially rebasable: crucible/grpc-base is upstream main plus a small stack of crucible-needed commits, rebased forward rather than diverging. The module docs in crucible/src/openshell/mod.rs state the consequence: because the fork is trivially rebasable, version-locking crucible to openshell-core "is no longer the risk it was when this shelled the CLI" — the control-plane boundary is the gateway's native openshell.v1 gRPC API, and the pinned rev is the exact rev the shipped gateway binary is built from.

Two distinct gaps keep the fork alive:

1. Fork-only commits — features upstream doesn't have yet (the ledger below).
2. Upstream-main commits ahead of the last release — the pin rides upstream main, not the release tag, because crucible needs post-release work (see "Why the pin is ahead of v0.0.81").

Pin mechanics (where the rev lives)

One rev, four consumers, all derived from Cargo.lock:

The divergence ledger

Fork-only commits on crucible/grpc-base, relative to upstream main (merge-base 40194f9, checked 2026-07-13):

That's the whole fork today: one commit. Earlier gRPC-boundary work that lived on this branch has already landed upstream (which is why the branch is named crucible/grpc-base but no gRPC commits remain fork-only).

Why the pin is ahead of v0.0.81

The pin is 11 commits ahead of the last upstream release tag (v0.0.81); 10 of those are upstream main commits, not fork work. The ones crucible actually depends on:

• 8eacb47 feat(kubernetes): add sidecar supervisor topology (#2076) — the supervisor runs as a sidecar container in the sandbox pod, which is where crucible's client-cert trust model lives (see below).
• 1070213 fix(core): pin supervisor image tag to gateway version for all drivers (#2070) — keeps the sha-<rev> gateway/supervisor image pair coherent.
• 614c8c1 feat(kubernetes): support PVC subPath driver config (#2034) — workspace layout under the k8s driver.
• 40194f9 fix(network): fail closed when credential placeholders cannot be rewritten (#2162) — credential-safety fix on the egress path.

This is why MIN_GATEWAY_VERSION in grpc.rs is 0.0.82: the RPC surface crucible calls does not exist in any released gateway yet.

Pending upstream contributions

Candidates to shrink the gap to zero:

• The AWS web-identity provider (bc8342bb, the only fork commit). Upstreaming it makes the fork pure upstream main @ <rev> and the "fork" becomes just a pin.
• Upload/download RPCs. File transfer still goes through the openshell CLI (SSH-tar over the gateway's CreateSshSession relay) because no RPC covers it — one of the two CLI remnants named in crucible/src/openshell/mod.rs. A native upload/download RPC would let crucible drop the CLI from the turn path entirely.
• mTLS user auth under the kubernetes driver. The gateway hard-rejects --enable-mtls-auth with the kubernetes compute driver (openshell-server/src/cli.rs: "mTLS user authentication is not supported with the Kubernetes compute driver"), so an earlier fork PR had to render allow_unauthenticated_users = true for k8s-driver gateways (crucible/src/openshell/gateway.rs). Trust model while the escape hatch exists: transport mTLS with a per-pod CA still gates every connection (require_client_auth), and the client cert lives only in crucible's turn pod and the supervisor sidecar container — never the agent container — so possession of the cert is the authorization. Acceptance criterion for the upstream change: crucible deletes the allow_unauthenticated_users escape hatch.

Maintenance rule

Every pin bump and every new fork commit adds or updates a ledger row above, in the same PR that moves Cargo.lock. A rev in the lockfile that this page cannot explain is a bug.

How to bump the pin

1. Rebase the fork branch: rebase crucible/grpc-base onto upstream main (it must stay trivially rebasable — if a commit stops rebasing cleanly, that is the signal to upstream it or drop it), push to wseaton/OpenShell.
2. Update the dependency: bump rev in crucible/Cargo.toml and let Cargo.lock re-resolve.
3. Images build themselves: openshell-gateway.yml triggers on Cargo.lock changes and builds openshell-gateway:sha-<rev> + openshell-supervisor:sha-<rev>. Note the first-run ordering: docker.yml fails fast until the gateway image exists, then retries clean.
4. Version stamp needs upstream tags: the gateway stamps its version via git describe --tags --long, and the fork's own tags are stale — the workflow fetches refs/tags/v* from nvidia/OpenShell before building. Nothing to do manually, but if the stamp ever reads 0.0.0, this is where to look.
5. Update this ledger (see the rule above), and bump MIN_GATEWAY_VERSION in crucible/src/openshell/grpc.rs if the bump starts using RPCs older gateways lack.

JIRA tools (mediated)

The broker embeds a native JIRA Cloud client (the shared jira-mcp crate) and exposes a read+comment slice of it to the sandboxed agent over the broker's existing MCP wire. It's the ADR-0002 mediation pattern, JIRA edition: the agent gets JIRA tools without ever holding an Atlassian credential. There is no upstream MCP child process; the broker calls JIRA's REST API directly, server-side.

What the agent sees

Three tools on the broker wire (prefixed by the broker name, e.g. mcp__epp-broker__jira_search):

jira_add_comment is the only write. The agent has no tool for create/transition/edit/delete, and the client itself is pinned to the read+comment ceiling at construction (crucible-broker/src/jira.rs), so scope is enforced by construction, never by trusting the agent. When JIRA isn't configured the tools return {"status":"disabled"} (same as build_epp/profile when their feature is off), so they never vanish.

Context hint. The deployment can name its projects of interest with BROKER_JIRA_PROJECTS (comma-separated, e.g. PROJ,PROJ2); the jira_search tool description then tells the agent to scope JQL to those boards instead of searching the whole instance. BROKER_DESC_JIRA_JQL_EXAMPLE and BROKER_DESC_JIRA_KEY_EXAMPLE flavor the inline examples the descriptions show. Unset, the descriptions stay fully generic.

Trust boundary

The Atlassian credentials live in the broker pod's env. They are never in [agent.env], so the sandbox never sees a token, exactly like the build/deploy creds. The agent reaches only the broker endpoint (already egress-allowlisted); the REST calls to Atlassian Cloud ride the loop pod's egress, not the sandbox's, so no sandbox egress change is needed.

Enabling it (broker-pod env)

All three set = the jira_* tools go live; anything missing = they answer disabled. The broker overrides any configured access level to read+comment regardless of what the env or the shared crate's config would allow.

Smoke-test your real instance

The jira_smoke example drives the exact client the broker uses (same read+comment ceiling, same compact renderings) against a live Atlassian Cloud instance. Configure it the same way the broker pod is configured, all through the environment (creds never land on a command line):

export JIRA_URL=https://your-org.atlassian.net
export JIRA_USERNAME=you@your-org.com
export JIRA_API_TOKEN="$(cat ~/.jiratoken)"

# read-only: a JQL search
cargo run -p crucible-broker --example jira_smoke -- 'project = PROJ ORDER BY created DESC'

# pass '' for the JQL plus an issue key and body to exercise get_issue and post one comment (the write path)
cargo run -p crucible-broker --example jira_smoke -- '' PROJ-123 'hello from the mediated broker'

Architecture decision records

The ADRs capture the load-bearing design calls behind crucible: why the loop is shaped the way it is, what was traded off, and what the alternatives were. They are ordered, append-only, and meant to be read when you want the why rather than the how.

Expand this section in the sidebar to browse the full list.

ADR 0001: Adaptive harness — adapt the setup, freeze the judge

Status: Accepted; implemented (the setup/solution tool split, fingerprinted segments, and the escalate hatch are live in the engine) Date: 2026-06-23 Context owner: agentic-epp-autoresearch

Context

The loop's integrity rests on one thing: the gate (fitness) must be trustworthy and outside the agent's reach. A kept "win" is only as honest as the harness that judged it.

We've found it valuable to tailor the harness to each issue — e.g. for a heterogeneous routing workload we hand-shaped the gate, chose a baseline, and picked the perf gate. Automating that tailoring ("an adaptive harness") is attractive: the harness would meet each issue where it is. But adaptivity aimed at the wrong layer turns the loop into a reward-hacking machine — if the harness can reshape itself toward the thing being optimized, the agent (or the adaptation itself) will make the workload easy, move the baseline, or pick a metric it already passes.

Decision

Adapt the setup. Freeze the judge. The agent may change its solution; it must never change its evaluation. Concretely, split the system into two phases with a hard wall between them:

1. Scoping (adaptive, once, then frozen)

Analyze the issue and decide the workload, baseline, and gate. This is the part worth automating (an agent may propose the harness config), but its output is:

• pinned — written to an immutable harness manifest,
• content-hashed — a fingerprint of (workload + gate + baseline),
• validated — passes the gate self-test (below) before it's trusted,
• human-approved — a person signs off before the loop spends budget.

2. The loop (immutable gate)

The optimizing agent runs against the frozen harness. It gets only solution-space tools and a read-only gate. It cannot touch the workload, the baseline, or the metric. The harness manifest is mounted read-only, outside the agent's editable workspace.

Tool classification (capability boundary)

Every tool is one of two kinds. The optimizing agent's toolbox contains only the second column. The first is used during scoping and then frozen.

The loop agent's toolbox must exclude rig-config and order-capture, since they can change MOCK_* (the latency model) and the workload — i.e. the evaluation surface. They are exposed only in scoping, never in the loop driver.

Safeguards (make adaptivity safe rather than scary)

1. Gate self-test (negative controls). Before a gate is trusted, prove it discriminates: a known-bad config must score worse and a known-good better (e.g. prefix-affinity p99 292ms vs prefix-blind 506ms). Automate it as a precondition — a gate that can't tell good from bad is the real danger, not the agent.
2. Harness fingerprint on every result. Stamp each kept result with the manifest hash so wins are reproducible and scope drift is detectable at a glance.
3. Fixed baseline. Derive the baseline once during scoping; never re-derive it inside the loop (a moving baseline makes "improvement" meaningless).
4. Multi-seed gating. Average a few seeds so a lucky single run isn't kept (esp. for noisy dynamics like load-scorer herding).
5. Cost/blast-radius gating. Anything in the setup column that spends GPU or mutates shared infra (e.g. order-capture) keeps an explicit human-gated --execute, as today.

Escape hatch: the agent may declare the judge inadequate (not change it)

Freezing the judge only stays honest if the agent has a sanctioned way to say "this judge cannot evaluate what I need." Without it, a blocked agent either thrashes out discarded changes or tries to game the gate. So the complement to "freeze the judge" is a structured escalation:

• The agent calls escalate --category {harness-limitation|infeasible|needs-info} --reason … --evidence … (writes ESCALATION.json). The loop detects it, stops, restores the rig, and surfaces the report for human review.
• This is distinct from a discard ("this change didn't help") — escalation means no change here can be fairly evaluated, so the harness itself needs a human fix.
• Anti-abuse: the tool requires a substantive reason and asks for evidence (e.g. inspect-rig output across configs); the escalation goes to a human, it is not a silent success. It is the agent's only sanctioned move against the evaluation — it can flag it, never edit it.

This is the path when, for example, tokenload reads zero in-flight load everywhere (the producer signal is dead, so the gate cannot see length-aware routing): rather than discard forever, the agent escalates "the gate can't differentiate length-aware routing" with the inspect-rig evidence, and a human fixes the producer wiring or the workload.

Consequences

• Positive: we keep the appealing part — the harness meets each issue where it is — while the optimization loop stays trustworthy and its results stay comparable and reproducible.
• Positive: clear capability boundary; the loop agent literally cannot reward-hack its evaluation because it has no tool to reach it.
• Negative / cost: scoping becomes a real, gated step (validation + approval), not a side-effect of running the loop. That's deliberate — a wrong harness is wasted or misleading research, so it deserves a checkpoint.
• The loop's tool wiring splits the toolbox into setup vs solution, emits a harness manifest + fingerprint, and runs the gate self-test as a scoping precondition. None of this requires an adaptive loop — only an adaptive scoping step.

Principle, restated

In ML you adapt the experimental setup once, carefully, and then you never let the model edit the test set. Same here: automate the setup, never the judge.

ADR 0002: Mediated provisioning — the agent asks, a server-side MCP admits

Status: Accepted; implemented (the broker now lives in the extracted crucible-broker crate; the draft-PR approval path was proven end-to-end against a fork) Date: 2026-06-25 Context owner: agentic-epp-autoresearch Related: ADR-0001 (freeze the judge), the request-trace gap, the control bridge (crucible/src/control.rs), the escalate hatch (tools/escalate.nu

• crucible/src/escalation.rs), the session MCP server.

Context

ADR-0001 drew the trust wall: the agent may change its solution, never its evaluation. But a blocked agent has a legitimate need the loop can't yet serve cleanly — it discovers it must measure a regime it has no calibrated trace for (a new model, a higher request concurrency, a novel workload), and the only honest GPU-free answer is capture/calibrate a new trace. Today that means either:

• the agent reaches the cluster directly (it holds a rig edit kubeconfig) and could, in principle, submit a GPU capture Job or dispatch CI itself, or
• it writes ESCALATION.json and the whole run halts for a human.

The first is unacceptable: we will not give the sandboxed agent direct authority to spend GPU, submit a Kueue Workload, or dispatch a GitHub Actions workflow. Those are privileged, shared-blast-radius, money-spending operations. The second is too coarse for a need that is often grantable (the trace just needs capturing) — it ends the run and waits on a human even when policy could have admitted it.

The missing piece is a mediation layer: a place that holds the privilege and the admission logic, exposes only a narrow "ask" to the agent, and decides whether/how to grant. An MCP server already sits on the loop pod (wrapping the read-only session surface). This ADR makes that server the provisioning admission broker too.

Decision

The agent asks; a server-side MCP on the loop pod admits. Provisioning privilege and the admission logic live in the MCP, never in the sandbox. Concretely:

1. Privilege separation by identity.

   • Solution-space cluster ops the agent already does (apply EPP config, bench, inspect-rig) stay direct, under the agent's scoped rig credential (ADR-0001's solution column). Unchanged.
   • Provisioning ops (submit a GPU capture Job / Kueue Workload, dispatch a GHA workflow, anything that spends GPU or mutates the evaluation) are removed from the agent's reach. The agent's credential must not carry these rights (RBAC: the agent SA can edit the rig namespace but cannot create Workloads/Jobs in the capture namespace; no forge token in the sandbox). Only the MCP's own identity can, and it gates every such call.

2. The MCP is the only privileged endpoint the sandbox can reach. OpenShell egress is deny-by-default; we allowlist exactly the loop pod's MCP endpoint and nothing else provisioning-capable. The agent speaks a small typed tool surface (below); it never sees a kubeconfig for the capture namespace, a Kueue API, or a forge token. Same mediated-capability pattern OpenShell already uses to serve the Vertex token through the metadata emulator without ever handing claude the token.

3. Requests are classified by whether they touch the judge (this is the integrity-critical axis, not the transport):

   • In-scope (a trace missing for the already-frozen regime, a pure cache fill): the MCP resolves it and returns MOCK_* knobs synchronously; the turn continues. No judge change, no re-baseline.
   • Judge-changing (a new concurrency / workload / regime the frozen baseline was never measured at): the MCP must not silently continue — that would make every later "keep" incomparable (the ADR-0001 reward-hack surface). It triggers a governed re-scope: re-derive the baseline at the new regime, bump the harness fingerprint, run the gate self-test, and only then start a new comparable segment. Here the coarseness is protective: the agent never measures its own change against a goalpost it just moved inside one turn.

4. Admission gating is server-side and opportunistic: on a cache MISS the MCP runs the gpu-check headroom logic (allocatable − Σrequested, with a reserve + a concurrent-capture cap); if there is headroom it auto-submits the capture under Kueue; else it defers/queues and the loop continues on fallback. The agent sees only a trace_id + a pollable handle.

5. Human-in-the-loop is pluggable, and headless-first. When a grant genuinely needs a human (beyond budget, a judge-changing re-scope under a stricter policy, a starving queue), the MCP raises the ask through one of two interchangeable approval backends behind a single request object — and keeps the request async (the loop records the ask and continues / parks; the human approves a batch out-of-band):

   • Forge-native (the headless default) — a draft PR. The MCP (which holds a repo-scoped forge token server-side — the agent does not) pushes an agentic/<run_id>-prefixed branch and opens a draft PR describing the request: params, trace_id, GPU estimate, the rendered capture manifest. Approval is a maintainer slash-command comment (/approve-capture <trace_id>) or marking the PR ready/merging — gated by CODEOWNERS + branch protection, audited by the forge itself. The MCP polls for the signal, then proceeds. A draft PR (over a bare issue) gives diff-review of the manifest, CODEOWNERS, and doubles as the state-persistence record (the run's branch outlives the pod). It builds on the existing publish.rs branch push.
   • Control-channel (attended). The MCP surfaces the same request as an event over the control bridge we built; an operator watching crucible view --pod approves with a keystroke (the steer/stop path). Immediate, for attended runs. The kept result records which path approved it (provenance, like the steer audit log), so a forge-approved capture is as traceable as an operator-approved one.

6. Degraded mode is escalate-halt, never auto-admit. When a judge-changing request arrives in a pure-headless run with no approval surface reachable (no forge token, no attended operator), the MCP does not silently auto-admit a goalpost move. It falls back to the escalate hatch: record the ask, restore the world, halt for a human. Unattended autonomy never extends to changing the evaluation without a recorded human (or policy) yes. In-scope cache fills still return synchronously in degraded mode — only judge-changing grants halt.

What this is NOT

• Not giving the agent a Kueue client, a forge token, or GPU-namespace RBAC. The agent's blast radius stays exactly the rig solution surface.
• Not a replacement for the frozen judge. A judge-changing grant re-scopes (new baseline + fingerprint); it never edits the gate in place.
• Not retiring escalation. ESCALATION.json (tools/escalate.nu → the engine hook) remains the honest valve for the unservable cases no tool can grant — a dead signal (tokenload=0), a missing rig capability, "infeasible" — and the degraded path when the MCP is unreachable (offline / pure-headless-no-forge). Mediated provisioning handles the grantable asks; escalation handles the un-grantable ones.

MCP tool surface (initial)

One server, narrow tools, all privileged work behind them:

request_trace content-hashes its params into a trace_id shared with the EPP CI cache, so a trace captured by the loop warms CI and vice-versa.

Architecture sketch

  sandbox (egress: ONLY the MCP endpoint allowlisted)
    claude ──MCP──> loop-pod MCP server  (trusted identity: rig + capture RBAC + forge token)
                          │
                          ├─ resolve cache (traces/ + S3 by trace_id)  ── HIT ─> knobs (sync)
                          ├─ admission: gpu-check headroom + caps  ── ok ─> Kueue submit
                          ├─ judge-changing? ─> re-scope (baseline + fingerprint, ADR-0001)
                          └─ needs human? ─> approval backend:
                                 ├─ forge: open issue/PR, poll for /approve-capture (headless)
                                 └─ control bridge: surface event, operator keystroke (attended)

Consequences

• Positive: the agent literally cannot provision — no GPU spend, no Kueue, no CI dispatch from the sandbox — yet it is no longer dead-ended on a grantable need. The choke point is one auditable server-side process.
• Positive: admission + approval are policy, not code scattered across tools; headless runs get a real (forge-native, async, CODEOWNERS-gated) approval path, attended runs keep the keystroke path, both record provenance.
• Positive: reuses what exists — the control bridge, gpu-check, the request-trace cache design, the autoresearch/<run_id> branch push, the session MCP.
• Negative / cost: a real MCP server on the loop pod (lifecycle, the loop pod's identity now also carries capture/forge rights — concentrate and guard it), MCP plumbing into the sandbox claude config, and an egress allowlist entry for the MCP endpoint. RBAC must be tightened so the agent identity loses any latent provisioning rights.
• Negative / cost: the judge-changing re-scope (auto re-baseline + fingerprint + gate self-test) is real engine work, not just a knob hand-back.

Testing & rollout

The approval + state-persistence path is validated against a throwaway fork before targeting any real repo; the forge token is held server-side by the MCP, never relayed into the sandbox.

• The loop's [repo].url points at the fork; the MCP opens its draft PRs there.
• Branches are agentic/<run_id>-prefixed, so the agent's pushes/approval PRs are self-labelling and the fork's history is the durable run record (state persistence survives pod teardown).
• The rig forwards a repo-scoped fine-grained PAT (scoped to the fork only) into the loop pod — mounted server-side and held by the MCP exactly like the kubeconfig / AWS token, never relayed into the sandbox. The agent gets no forge credential.

This validates the full headless approval loop (open draft PR → /approve-capture → proceed) and state persistence before anything touches a real repo.

Design notes

• Re-scope unit: a judge-changing grant starts a new fingerprinted segment within the run (not a new run_id), so one session log tells the whole story across the boundary.
• Forge surface: a draft PR (not a bare issue) — diff-review + CODEOWNERS + it doubles as the state record.
• Degraded mode: escalate-halt (never auto-admit a judge change unattended) — decision 6.
• MCP transport is HTTP/SSE at the loop pod IP:port, allowlisted in the OpenShell egress policy as full for just that host:port, with the sandbox claude MCP config seeded (env / .mcp.json).
• Agent RBAC keeps edit on the rig namespace only, with no path to Workload/Job creation in a capture namespace; gpu-check's admission logic lives in a shared lib the MCP calls.

Principle, restated

The agent reaches the cluster only through doors the host holds the keys to. Solution-space doors are already open to it; every provisioning door is locked, and the MCP is the only one who can open one — after admission, and (when it must) after a human says yes through a channel that fits a headless run.

ADR 0003: Async approval waits — the agent decides block vs continue

Status: Accepted; implemented (park/rescope/deny + ParkOutcome live in loop_driver.rs; the budget clock pauses while parked) Date: 2026-06-25 Context owner: agentic-epp-autoresearch Related: ADR-0001 (freeze the judge; the solution/evaluation split), ADR-0002 (the broker, pending_approval, the re-scope), the control bridge (crucible/src/control.rs), the engine re-scope path (run_loop's take_rescope → run_baseline → new fingerprint segment), the escalate marker hook (crucible/src/escalation.rs).

Context

ADR-0002 decision 5 says a judge-changing request can open an async approval (a draft PR or a control-bridge ask) and "the loop records the ask and continues / parks; the human approves a batch out-of-band." It specified the approval — open a PR, poll for /approve-capture — but not what the run actually does between the ask and the answer, nor how it resumes. Today the only built behavior is the degraded floor: BROKER_DEGRADED=1 → judge-changing misses escalate-halt (Resolution::Escalate). That is correct when there is no approval surface, but it is not the wait-and-resume we actually want when there is one.

Two things need pinning:

1. Where the wait lives. The agent turn is ephemeral — a per-turn OpenShell sandbox, torn down when the turn ends. It cannot "sit and wait"; making it spin-poll check_capture inside a turn burns tokens on a paid sandbox to do nothing. The long-lived thing is the loop (crucible, the pod's entrypoint process). The wait is the loop's.
2. Whether to block the whole run. Parking the run idles a pod on a human. Sometimes that's right (the pending regime is the agent's only path); sometimes it's waste (the agent has other frozen-regime hypotheses to test meanwhile). Only the agent has the context to know which.

Decision

The broker decides whether a regime change is allowed (privilege + evaluation, server-side, ADR-0002). The agent decides how to spend the wait (solution-space, ADR-0001). The loop owns the wait and the resume. Concretely:

1. The wait is the loop's, not the turn's. On pending_approval, the agent records the ask and ends its turn — it never spin-waits in the sandbox (the request-trace skill already says "async by design; do not spin-wait a whole turn"). The loop reads the ask after the turn and acts.

2. The agent chooses the wait mode, written into the marker it leaves:

   • block — "this regime is my only path." The loop parks until the approval resolves.
   • continue — "I have other frozen-regime work." The loop keeps iterating; the re-scope lands whenever approval arrives. This is a solution-space call (how to use time), not an evaluation one — so it is the agent's, exactly as ADR-0001 draws the line. The broker's yes/no is unchanged and still server-side.

3. One wake mechanism for both modes. Whichever mode, the broker (it holds the PR + the forge token; its process is long-lived even though the HTTP transport is stateless — ADR-0002 note) watches the approval. On /approve-capture it sends a rescope{regime} command to crucible's control bridge (localhost:<control-port>), becoming a control-bridge client. crucible's existing re-scope path drains the pending re-scope at the next iteration head: re-baseline at the new regime, bump the harness fingerprint, open a new comparable segment, resume. The mode only gates whether the loop idles or keeps iterating while the watch runs — it is not a second system. crucible polls nothing; it parks (or not) until a rescope arrives.

   Two async phases, one terminal signal. There are actually two waits after pending_approval: (1) the human (approval) and (2) the provisioning — the GPU capture that produces the calibrated trace + MOCK_* knobs. The loop's park waits for a single terminal outcome; the watcher must only send the rescope once provisioning is ready, not merely approved (else the re-baseline measures an uncalibrated regime). So the correct watcher flow is poll approval → submit capture → poll capture until ready → send rescope. The capture submit/poll is the capture-wait boundary in watch.rs, backed by the Kueue/trace backend. Because the loop already parks for the terminal signal, only that boundary changes as the backend evolves; the simplest form has the watcher send rescope on approval. The terminal outcomes the park waits on: rescope (ready → resume), deny (rejected → resume-frozen or escalate), or a --max-park timeout (→ treated as deny).

4. The decision rides on a marker (mirror of the escalate hatch): the agent writes PROVISIONING_PENDING.json in its workspace; the loop reads + consumes it post-turn with a take-style hook beside escalation::take. Schema below.

5. Budget treatment follows the mode. block pauses the cost/time clock — an idle wait on a human must not count against --max-cost / --max-time. continue keeps spending, because the agent is doing real work. This falls out of which loop state we're in.

6. Two interchangeable human channels, same rescope (ADR-0002's two approval backends):

   • Forge (headless): comment /approve-capture <trace_id> on the draft PR. The broker's DraftPr poll catches it. Approve from anywhere; the parked pod resumes minutes later.
   • Control bridge (attended): the ControlBridge channel surfaces the ask as a viewer event; the operator watching crucible view --pod approves with a keystroke (a), the same exec/control path s/x already use (crucible/src/tui/remote.rs). Immediate.

7. Deny / timeout is not a harness failure. On denied or a max-park timeout, the loop discards the regime change and resumes in the frozen regime with a note — a human declining a goalpost move is a normal outcome, not an escalation. The exception: if the agent declared block because it had no fallback, a denial leaves nothing to do, so it escalate-halts (ESCALATION.json, exit 2) — the honest "I am genuinely stuck" signal.

What this is NOT

• Not the agent polling inside a turn. Turns are ephemeral and paid; the loop waits, the agent does not.
• Not the broker (or the harness) deciding how to spend the wait. That's the agent's solution-space call — the whole point of letting it choose block vs continue.
• Not a new evaluation path. Resume is the existing re-scope (new baseline + fingerprint + segment); the approval just feeds its pending slot. The judge is never edited in place.
• Not a replacement for escalate. Escalation stays the floor for the un-grantable and the no-approval-surface degraded case (ADR-0002 decision 6). This ADR is the grantable-but-pending case: a human will answer, the run just has to wait for it.

The marker

PROVISIONING_PENDING.json, written by the agent in its workspace (consumed + deleted by the loop, so a stale file can't re-trigger on resume):

{
  "trace_id": "model=Qwen/Qwen3-0.6B;c=48;p=8;mt=8;lt=0;lf=0.0000",
  "regime":   { "model": "Qwen/Qwen3-0.6B", "concurrency": 48, "prefixes": 8,
                "max_tokens": 8, "long_tokens": 0, "long_frac": 0.0 },
  "handle":   "https://github.com/<org>/<repo>/pull/NNN",
  "mode":     "block"
}

mode defaults from behavior when the agent omits it: a turn that produced a measurable CANDIDATE.md (a frozen-regime fallback change) implies continue; a turn that produced only the ask implies block. Explicit beats implicit — the skill asks the agent to declare it.

Architecture sketch

  agent turn:  request_trace{c=48} ─▶ pending_approval{handle}
               write PROVISIONING_PENDING.json {mode}, END turn   (no spin-wait)

  loop (crucible, long-lived):  take_pending() post-turn
      ├─ mode=block    ─▶ PARK: budget clock paused, SessionEvent::Parked, idle
      └─ mode=continue ─▶ keep iterating in the frozen regime

  broker (long-lived process):  watch the approval
      └─ /approve-capture  ─▶  rescope{regime} ─▶ crucible control bridge (localhost:port)

  loop wakes (both modes, via the re-scope path):  take_rescope() at the next iteration head
      └─ re-baseline @ new regime + bump fingerprint + new segment ─▶ resume

Consequences

• Positive: the run can genuinely wait for a human and resume, headless or attended, without the agent burning paid sandbox time spinning. The honest "is this worth blocking the whole run?" call sits with the only actor that has the context — the agent.
• Positive: almost entirely reuse. The wake is the existing re-scope path; the transport is the control bridge; the marker is the escalate pattern; the channels are ADR-0002's two backends; the viewer keystroke is the s/x path. The genuinely new code is small: the marker + take hook, a parked loop state (a budget-paused pause that a rescope wakes), the broker's PR-watcher → control-bridge sender, and the viewer a key.
• Negative / cost: a parked pod idles on a human (cheap — no tokens, paused budget — but a real pod-hour). A max-park timeout bounds it. The broker now also runs a background watcher with state across requests (fine: its process is long-lived; only the HTTP transport is stateless).
• Negative / cost: continue mode interleaves a frozen-regime segment with a later re-scoped one in the same run; the session log already models segments, so the story stays readable, but a reader must respect segment boundaries (scores across one are not comparable — ADR-0001 safeguard 2).

Design notes

• Max-park timeout + denial default: --max-park <dur> (empty = wait indefinitely) bounds the wait; a timeout is treated as a denial. A block denial escalate-halts (the agent had no fallback by definition of block); a continue denial just stays in the frozen regime with a note. This lives in loop_driver.rs (ParkOutcome) + the deny control command.
• Broker → control-bridge address: crucible injects BROKER_CONTROL_ADDR=127.0.0.1:<port> when it spawns the broker; same-pod loopback is the trust boundary (broker.rs / watch.rs).
• Control-bridge grammar at scale: the approve/deny commands handle a single ask in flight; a queue of asks plus viewer keys (a/d) and dedupe on re-press extend the same channel.
• Resume fidelity in block: a long park may outlive the rig's last-good snapshot or the sandbox gateway's warm state, so re-baseline after a wake must re-establish both cleanly.

Principle, restated

The host decides what may be granted; the agent decides what to do while it waits; the loop holds the door open and walks through it when the human says yes. Waiting is solution-space — the agent's to spend — but the gate it waits on, and the re-scope it resumes into, stay the host's.

ADR 0004: Core-loop state model — context struct + typed iteration, not whole-loop typestate

Status: Accepted (implemented 2026-06-25) Date: 2026-06-25 Context owner: agentic-epp-autoresearch Related: ADR-0001 (freeze the judge), ADR-0003 (park/rescope/deny gates), the loop in crucible/src/loop_driver.rs (run_loop), and the run/loop_driver split.

Context

run_loop has crossed a complexity inflection. It now threads ~13 mutable bindings across iterations — rows, spent, best_score, baseline_score, baseline_total, best_snap, regime, fp, kept_shas, solved_any, escalated, pending_block, parked_total — and exits through five breaks (stop, escalate, denied-escalate, budget, solved) plus two continues (parked-block, apply-failed). ADR-0003 added park / rescope / deny / provisioning gates on top of the existing apply → measure → decide → keep/discard core. The next addition gets risky: the invariants (re-baseline mutates a coherent set of fields together; you can't measure before apply, can't keep without a Reading, can't measure on top of an escalation) live only in the reader's head.

A typestate pattern was proposed. This ADR decides how much typestate, and where — because the naive answer (model every loop phase as a type) is the wrong shape for this loop.

Decision

Type the linear sub-protocol; keep a plain state machine with an explicit context for the cyclic shell. Concretely, three changes, in risk order:

1. A Run context struct bundling the per-run threaded state, with the segment-scoped fields factored into a Segment (because the re-scope mutates exactly those, together):

   #![allow(unused)]
   fn main() {
   struct Run {
       rows: Vec<Row>,
       spent: f64,
       kept_shas: Vec<String>,
       solved_any: bool,
       parked_total: Duration,         // ADR-0003 budget-pause accumulator
       pending_block: Option<PendingProvisioning>,  // ADR-0003
       segment: Segment,
   }
   /// Everything a re-scope replaces atomically (ADR-0001 safeguard 2 / ADR-0003).
   struct Segment {
       regime: String,
       fingerprint: String,
       baseline_score: f64,
       best_score: f64,
       baseline_total: u64,
       best_snap: Snapshot,
   }
   }

   A re-scope becomes run.segment = Segment::baseline(world, judge, regime)? — one assignment that can't half-update the goalpost. This is the single biggest readability win and is independent of the rest.

2. An explicit LoopExit enum replacing the escalated: Option<_> flag + scattered breaks, so there is one place that enumerates how a run can end, mapped to Outcome:

   #![allow(unused)]
   fn main() {
   enum LoopExit { Finished, Solved, Budget, Stopped, Escalated(Escalation) }
   }

3. Typestate the inner iteration — the one genuinely linear protocol with real ordering invariants. The candidate moves Proposed → Applied → Measured, and only a Measured can be decided/kept (so "keep without a reading" and "measure before apply" stop compiling):

   #![allow(unused)]
   fn main() {
   struct Iteration<S> { it: u32, state: S }
   struct Proposed;                       // agent staged a candidate in the world
   struct Applied;                        // world.apply() succeeded
   struct Measured { reading: Reading, note: String, diff: String, diffstat: String }
   
   impl Iteration<Proposed> {
       fn apply(self, world: &dyn World) -> Result<Iteration<Applied>, ApplyFailed>;
   }
   impl Iteration<Applied> {
       fn measure(self, judge: &dyn Judge, ctx: &MeasureCtx, p: &Paths) -> Result<Iteration<Measured>>;
   }
   impl Iteration<Measured> {
       fn decide(self, judge: &dyn Judge, best: f64) -> (Row, Verdict);  // keep path holds the Reading by construction
   }
   }

   The outer for it in … stays a loop over &mut Run; the gates (park / rescope / deny / escalation / provisioning) run before an iteration enters Proposed, and decide whether it does.

What this is NOT

• NOT whole-loop typestate (Loop<Parked>, Loop<Rescoping>, … with self-consuming transitions). Typestate pays off when each state owns distinct data + capabilities (a connection, a builder, a resource protocol). This loop is the opposite: nearly all state is shared and threaded through every phase, so a per-phase type would carry one big shared context + a marker — the ceremony of typestate with none of the safety. The cyclic shell wants a state machine with an explicit context (#1/#2), not types-as-states.
• NOT a behavior change. This is a structural refactor: identical control flow, identical session log, identical exit codes. The existing tests (park outcomes, resume, fingerprint, exit codes) are the regression net and must stay green untouched.
• NOT a new module boundary. It stays in loop_driver.rs (+ maybe an iteration.rs sibling for the typestate); the run/loop_driver split is preserved.

Consequences

• Positive: the re-scope can't half-update the goalpost (the Segment swap is atomic); the iteration protocol's ordering invariants are compile-enforced; run_loop's signature and body shrink to "drive gates, then run a typed iteration, fold the result into Run"; the ways a run can end are enumerated in one enum.
• Positive: the next gate (e.g. the ADR-0003 capture-wait, or a new domain's phase) plugs into a named context + a typed iteration instead of adding a 14th threaded binding.
• Negative / cost: more types (the Iteration<S> zoo, the marker structs) and the self-consuming transitions are slightly more verbose than inline statements; worth it for the protocol, which is why we scope typestate to only the iteration.
• Negative / cost: a refactor of a core file — it must land as one reviewed change with the test suite green, not a drip of edits.

Implementation notes

The change lands in three independently testable steps — the Run + Segment struct (threaded bindings moved in, re-scope becomes one assignment), the LoopExit enum (replacing the escalated flag + scattered breaks, mapped to Outcome), then the Iteration<S> typestate (apply/measure/ decide extracted into the typed protocol) — each keeping cargo test -p crucible green (notably the park/resume/exit-code tests).

• LoopExit::Escalated is a unit variant, not Escalated(Escalation). The escalation is reported and the world rolled back eagerly at each break site, and the two sites report differently (the agent path calls Reporter::escalation; the denied-block path emits a note). Centralizing the report post-loop to consume a carried payload would change the session log, which this refactor must not do, so the carried value would be dead. The variant only marks the run "needs human" for the exit code; Outcome.escalated = matches!(exit, LoopExit::Escalated).
• Segment::baseline(world, judge, goal, regime) owns the fingerprint (vs the ADR sketch's separate fp binding), so opening a segment is one call for both the initial baseline and a re-scope. The resume path constructs its Segment directly from the restored scores.
• Decided { row, verdict, reading } is what Iteration::<Measured>::decide returns (vs the sketch's (Row, Verdict)): the keep path needs the reading's score (→ best_score) and note (→ snapshot label), so the typed step hands it back by construction.

Open questions

1. Where the gates live. Do park / rescope / deny / escalation / provisioning stay inline in run_loop, or become a small enum Gate { Park, Rescope, … } step? Lean inline until a third gate makes the sequence unwieldy.
2. Snapshot type. best_snap is a String today (the world's opaque snapshot handle). The Segment struct is a good moment to newtype it (struct Snapshot(String)).
3. Resume path. run_loop's resume branch reconstructs the same state; it must build a Run from ResumeState cleanly — confirm the Segment factoring doesn't complicate the replay.

Principle, restated

Type the protocol, not the process. The iteration is a short linear contract — make its illegal orderings unrepresentable. The loop is a long cyclic process over shared state — give that state a name and its exits an enum, and leave the rest a plain, readable for.

ADR 0005: Engine-side EPP builds, brokered to the agent over MCP

Status: Accepted; implemented (builds run buildah-unified on the loop pod — kaniko was dropped; open questions 1–2 were resolved by ADR-0008, see the annotations below) Date: 2026-06-26 Related: ADR-0001 (freeze the judge), ADR-0002 (the agent asks, the host holds the keys), the World::apply trait, the epp-mcp broker, Containerfile.epp-sandbox, the #1109 TTFT domain (crucible.ttft.toml).

Context

#1109 is the first domain that needs in-loop EPP code rebuilds. The perf domain dodged this — it is config-tuning (apply scorer weights onto the live EPP, no rebuild). When the #1109 loop ran, the sandboxed agent could not make progress and doom-looped, because:

1. It can't build. build-image shells out to an out-of-sandbox remote-build script, which isn't in the sandbox; and a kaniko build needs build-namespace kubectl, a push credential for the candidate registry, and egress that the deny-by-default sandbox does not have. The agent correctly concluded it had no container builder available and gave up.
2. It can't see the measurement. epp-sandbox:spike-1 predates bench --ttft, so the agent's bench is the old prefix-cache bench — it can't reproduce the judge's TTFT metric, so it reverse-engineers the binary (strace/strings) and spirals.

Putting build creds in the sandbox (the "provision the sandbox" option) enlarges the attack surface and fights the World trait, which already says build+deploy is apply's job (engine-side).

Decision

Move build+deploy off the agent to the loop pod (engine-side), and expose it to the agent as broker (MCP) tools — the ADR-0002 pattern, the agent asks, the loop pod holds the keys.

Why MCP rather than only [world].apply_cmd: a code loop needs compile feedback within the turn. A purely post-turn apply build means a typo fails the whole iteration with no feedback. The agent must trigger a build, get errors, and fix them before its turn ends. So:

• Primary path — broker tools. Add build_epp (and deploy_candidate, or a combined build_and_deploy) to epp-mcp, alongside request_trace. The agent calls them over the existing bridge (host.containers.internal:8849); the loop pod runs the container build (buildah today; the original design said kaniko) with its own push credential for the candidate registry, and returns a Resolution: {ok, image_ref} | {compile_error, log}. The agent iterates on errors, then deploys.
• Backstop — [world].apply_cmd. Keep an engine-side apply_cmd that ensures the latest built candidate is the deployed one before the judge measures (idempotent; a no-op if deploy_candidate already ran). The manifest already supports [world].apply_cmd and CommandWorld::apply runs it.
• The agent's sandbox loses build creds entirely. build-epp/deploy-candidate direct tools are dropped; the agent only edits Go + calls the broker.

Components to build

1. Loop-pod build capability (in the crucible loop image): a self-contained buildah build + push (no dependency on an out-of-sandbox remote-build script) and a scoped push robot credential for the candidate registry (mounted secret).
2. epp-mcp build tools build_epp / deploy_candidate, mirroring request_trace's admit/return shape. They run the loop-pod build and surface the compile log on failure.
3. Workspace sync (the crux). The loop pod must build the agent's current edits, which live in the sandbox mid-turn (the driver copies the workdir in/out per turn, it is not shared). Proposed: the agent commits its branch and pushes it to the loop pod's workspace as a git remote over the relay; the broker builds from that ref. (Reuses git-memory; no new driver plumbing. Alternative: a driver "mid-turn checkpoint" that syncs /sandbox/epp → loop workspace on the build call.)
4. Rebuild epp-sandbox with current epp-tools (so the agent's read-only bench --ttft matches the judge) + bake bench.rs source (or a TTFT measurement skill) so the agent understands the metric instead of reverse-engineering it. Drop the direct build/deploy tools.
5. Manifest wiring: broker build tools enabled in [agent.broker]; [world].apply_cmd as the ensure-deployed backstop; the candidate registry + egress allowlisted.

Open questions

1. Build latency. Resolved by ADR-0008: the immutable compose bakes the toolchain and a warm GOMODCACHE/GOCACHE into the base, so the per-iteration build is an incremental build against a hot cache, not a cold from-scratch image build.
2. Workspace sync mechanism. Resolved by ADR-0008: the agent's branch push is the CoW overlay; the broker builds base@SHA + overlay.
3. Cred scoping — a push robot token limited to the candidate registry, not a broad registry cred.

Consequences

• Positive: build creds never enter the sandbox (ADR-0001/0002 aligned); the agent gets real compile feedback; the loop pod is the one trusted place for push creds; apply_cmd guarantees the measured candidate is the built one; code-change domains become first-class, not just config tuning.
• Negative / cost: real infra (push cred, RBAC, a builder in the loop image — kaniko as designed, buildah as shipped — the workspace-sync mechanism, a sandbox rebuild) and per-iteration build latency.

ADR 0006: Profiler support over MCP — a generic capability, pprof for the EPP

Status: Accepted; implemented (generic profiler in crucible-broker; pprof backend for the EPP, torch-trace for vLLM) Date: 2026-06-26 Related: ADR-0002 (the agent asks, the host holds the keys), ADR-0005 (engine-side build + the broker measure tool), the epp-mcp broker, the #1109 TTFT domain (crucible.ttft.toml).

Context

#1109 asks the agent to cut the EPP's contribution to TTFT. But the gate — end-to-end bench --ttft through the live service — cannot isolate the EPP, and cannot resolve small EPP wins:

• The measured path is client → envoy-proxy → (ext_proc) → epp → envoy → backend pool → first token, all three in one pod behind svc:80. The MOCK_PREFILL_TIME_PER_TOKEN_MS=0 scoping zeroes the backend compute, but the ~2.2 s TTFT still includes Envoy's 220 KB body handling, the data-plane transfer (220 KB × c=150 ≈ 33 MB/hop), backend ingest of that body, and queueing at c=150.
• Result: every micro-optimization the agent tried (alloc, JSON, GC, gRPC windows) came back "within noise." The agent was guessing at the hot path and optimizing a low-single-digit-% sliver that the c=150 measurement noise swamps. The TTFT gate is right for scoring a candidate; it is the wrong instrument for finding what to change.

A profile fixes both problems at once: pprof attributes CPU / allocations / contention to specific EPP functions and source lines — isolated to EPP code by construction, and far more sensitive than a noisy end-to-end delta. The EPP already ships pprof: --enable-pprof defaults to true (pkg/epp/server/options.go), handlers are registered on the metrics server (runner.go, pkg/common/observability/profiling/pprof.go → net/http/pprof), served on :9090. The catch: that port runs controller-runtime secure metrics, so /debug/pprof/ is token-gated (a bare GET returns Unauthorized), and the sandboxed agent can reach neither the in-cluster ClusterIP (egress wall) nor has go tool pprof.

Decision

Add a generic Profiler capability to the broker — mediated "observe the candidate's internals" — with the concrete profiler as a domain-configured backend, exactly the way measure shells the domain's BROKER_MEASURE_CMD. The agent asks; the loop pod (cluster reach + token + tooling) captures and analyzes and hands back text. The agent never touches the endpoint or knows the backend.

Generic (broker core) — two tools that shell domain-provided commands:

• profile{kind, seconds} — capture a profile of kind (backend-defined) over seconds. The broker shells BROKER_PROFILE_CAPTURE_CMD (env: $KIND, $SECONDS, $OUT path), stores the result on the loop pod under the build storage root, returns a handle + short summary.
• profile_query{query} — run query against the stored profile. The broker shells BROKER_PROFILE_QUERY_CMD (env: $PROFILE, $QUERY), returns text. The query vocabulary is backend-defined. Read-only by construction (the backend command is a fixed analyzer; the broker applies a deny-list so a query can't start a server or write files).

The broker knows nothing about pprof, Go, or the EPP — only "capture a profile, then query it," delegated to configured commands. This is the same split as forge (generic build engine) vs the EPP build wiring, and measure (generic) vs BROKER_MEASURE_CMD (domain).

Domain implementation (#1109 = pprof against the Go EPP): the EPP domain sets the two commands — CAPTURE_CMD = an authed GET of epp:9090/debug/pprof/<kind>?seconds=$SECONDS; QUERY_CMD = go tool pprof -<query> $PROFILE. So for EPP, kind ∈ {cpu,heap,mutex,block} and query is pprof's command surface: top20, list HandleRequestBody (per-line ns/alloc in the real source — the killer), peek json.Unmarshal, traces, tree. Another domain (a Python service, a native binary) swaps in py-spy / perf / eBPF via the same two env commands; the broker tools and the agent-facing surface are unchanged.

The loop becomes profile → read the hot lines → fix them → re-profile to confirm the hotspot shrank → measure to score. Profile to find (isolated, sensitive); the gate to score.

Implementation

• Broker tools (epp-mcp/src/profile.rs, registered in server.rs, exported in lib.rs): generic profile{kind,seconds} + profile_query{query,handle?}, shelling BROKER_PROFILE_CAPTURE_CMD (env $KIND/$SECONDS/$OUT) / BROKER_PROFILE_QUERY_CMD (env $PROFILE/$QUERY). Captures stored under <FORGE_STORAGE_ROOT>/profiles with a latest marker; unconfigured → disabled (mirrors measure). Read-only deny-list on the query (no $(), no pipes/redirects, no -http/-web/-output/-svg/...), kind sanitized, handle path-traversal-guarded. No pprof/Go/EPP knowledge in the tool code. Unit-tested (deny-list, kind validation, traversal).
• EPP/pprof backend (swappable domain wiring): capture adapter domains/<domain>/tools/epp-pprof.nu (port-forwards :9090, mints the reader-SA token, authed-GETs /debug/pprof/$KIND → $OUT, same pattern as epp-metrics.nu); the two BROKER_PROFILE_* commands in the loop's deployment manifest (epp-pprof ... / pprof -$QUERY $PROFILE); the standalone pprof analyzer baked into Containerfile.crucible via a CGO-free Go stage.
• RBAC (the access path — chose RBAC over an unauth :6060): extended the epp-metrics-reader ClusterRole in rig-up.rs to add /debug/pprof + /debug/pprof/* nonResourceURLs, so the same reader token controller-runtime already authorizes for /metrics also covers pprof on that mux.
• Advertised: the profile skill (domains/<domain>/skills/profile/) + a "profile before you guess" note and the two tools in prompts/research.md.

Validated live:

• Access works. The reader token authorizes /debug/pprof end-to-end — /metrics, /debug/pprof/, and /debug/pprof/profile?seconds=1 all return 200 once the epp-metrics-reader ClusterRole carries the pprof nonResourceURLs (this ADR) and the EPP service account holds system:auth-delegator (required for :9090 auth to function at all — without it /metrics 500s too; rig-up creates the binding). controller-runtime authorizes /debug/pprof for a metrics-style nonResourceURL grant — same mux, per-path authz, so the URLs must be granted explicitly.
• Capture under load works and the signal is real. A 20 s CPU profile during a measure (TTFT 2275 ms, c=150) showed the EPP busy only ~24% of one core — so most of the 2.2 s is NOT EPP CPU (it's transfer/queue/envoy, exactly the isolation finding). Within the EPP's CPU the hot path is: encoding/json validation/parse of the 220 KB bodies (checkValid/unquoteBytes/appendString, ~15–20% cum), GC churn from those bodies (gcDrain 17% cum, memclrNoHeapPointers 9% flat, scanobject/memmove), and prometheus/common/expfmt text parsing (~15% cum — the EPP scraping backend metrics). None of this is the routing/scheduling logic the agent kept guessing at; the profile names the real cost. top / peek / traces / tree all work from the symbolized profile.

list=<Func> per-line cost is resolved by mapping the profile's source paths to the candidate's staged tree. The EPP is built without -trimpath (Dockerfile.epp WORKDIR /workspace), so the profile embeds /workspace/pkg/epp/... paths, while forge stages the candidate repo root at /var/lib/forge/ctx. BROKER_PROFILE_QUERY_CMD now carries -trim_path=/workspace -source_path=/var/lib/forge/ctx, so pprof strips the build prefix and finds the file under the staged ctx: list=<Func> shows real per-line ns/alloc for the EPP's own pkg/epp/... source once a build_epp has run this turn (ctx is the tree the deployed candidate was built from). top/peek/traces/tree work without a build. Deliberate non-goal: list on stdlib/vendored frames (encoding/json, prometheus/common/expfmt) doesn't resolve — only the EPP's own source is staged, and the agent edits EPP code, not the stdlib, so the actionable surface is covered. Verified the embedded paths against a live capture (/workspace/pkg/epp/datalayer/collector.go).

Generic multi-target profiler and the vLLM backend

Three changes make the broker genuinely component-agnostic and ready for composite (multi-domain) rigs, all in crucible-broker/src/profile.rs (the profiler lives in the extracted broker crate):

• Capture extension externalized (BROKER_PROFILE_EXT). The broker named handles <kind>-<nonce>.pprof — a pprof-ism in a supposedly generic core. The extension is a domain concern (the broker stores the bytes the capture writes and never interprets them); a format-sniffing analyzer wants the right name. Now domain-set, neutral default prof. The EPP sets pprof; the vLLM torch-trace backend sets json.gz.
• Multiple profile targets (BROKER_PROFILE_TARGETS). A composite deployment (vLLM + EPP) has more than one profileable component, each with its own image, way in, and profiler. Profiling is now keyed by a named target: BROKER_PROFILE_TARGETS="vllm epp" plus per-name BROKER_PROFILE_<T>_CAPTURE_CMD / _QUERY_CMD / _EXT. profile/profile_query take an optional target (omit it for a single-component rig; name it — e.g. vllm or epp — when there are several; an omitted target with several is an error that lists them). Captures nest under profiles/<target>/ with a per-target latest. A single-component rig still just sets the un-prefixed BROKER_PROFILE_* (one implicit default target) — fully backward-compatible, no EPP-run change.
• vLLM GPU-trace backend reconciled to the contract. domains/<domain>/tools/vllm-profile.nu (capture)
  • vllm-analyze.nu (kernel→source correlation, inverting neuralmagic/ai_auto_perf_analysis) now honor the same $KIND/$SECONDS/$OUT // $PROFILE/$QUERY surface as epp-pprof, so the fold-in is pure env wiring — no broker core change. $SECONDS is unreferenced for vLLM (a GPU window is prompt-count bounded). The gate-workload question is settled serve-path: both gate and profiler drive the same frozen VLLM_BENCH_* workload via vllm bench serve. Live GPU validation requires the analysis-layer image; the contract is tested at its boundaries.

The agent-facing tools and the per-component domain wiring are unchanged in shape; the broker just routes by target name. This is the same generic-core / domain-backend split as forge and measure.

Consequences

• Positive: the agent optimizes the measured hot path instead of theorizing in the noise; the isolation/sensitivity problem the TTFT gate cannot solve is solved by direct observation; composes with measure/build/deploy/cap; generic, so it strengthens the case for the broker extraction.
• Negative / cost: a small RBAC grant, a binary in the loop image, two more broker tools, and a rebuild. Profiling adds minor overhead to the EPP during capture (CPU pprof is sampled — low). The pprof pass-through needs a sane command deny-list so the agent can't start servers or write files.

ADR 0007: Verify the gate isolates the target before spending a loop — the metric that misframed #1109

Status: Accepted Date: 2026-06-26 Related: ADR-0001 (automate the setup, freeze the judge — world-scoping is setup), ADR-0006 (the Profiler — found the EPP CPU is incidental), the onboarding issue-ranking pipeline (ranks issues by measurability), the #1109 TTFT domain.

Context

#1109 ("long-context requests see >1s TTFT overhead in the EPP request path") ran as a loop and produced two valid-but-flat candidates (collector caching, body re-chunking; both within c=150 noise). ADR-0006's profiler showed the EPP is ~24% of one core busy with GC incidental. We then ran the cheap, decisive check: the EPP's own instrumentation, delta'd around a 600-request measure.

The EPP's actual work is ~20–30 microseconds per request. The 2.26 s is the ext_proc handler sitting blocked, waiting on Envoy's 220 KB body buffering + the data-plane transfer + the mock backend + c=150 queueing — none of which the EPP controls.

How #1109 got misframed. llm_d_router_epp_request_duration_seconds ≈ 2.26 s ≈ the end-to-end TTFT, because that metric measures the handler's wall-clock attachment time (mostly blocked-wait), not EPP compute. Reading it as "the EPP request path has >1s overhead" is the trap. The compute metric (scheduler_e2e = 17 µs) tells the truth: a 17 µs decision wrapped in 2.26 s of waiting on everything else. There is no >1s EPP overhead to remove.

Root cause is world-scoping (ADR-0001's setup column), not the EPP. The 220 KB-body mock scoping was chosen to stress the EPP path, but it drowns the EPP in Envoy transfer cost — the scoping defeats the isolation it was meant to create. The gate is end-to-end and cannot attribute a result to the EPP, so every EPP-side candidate ties in the noise by construction.

Decision

Onboarding must verify the gate actually isolates the target — and that the target's contribution is material — BEFORE spending a loop. Measurability (can a frozen judge decide success?) is necessary but not sufficient; the issue-ranking pipeline ranks for it but #1109 passed measurability while being unwinnable. Add an isolation pre-flight to the judge-validation gate:

1. Attribute the gate. Before the first scored loop iteration, measure the target's own contribution to the gated metric using the target's instrumentation (here: the EPP's scheduler_e2e / plugin_duration vs the end-to-end request_duration), or a direct isolated harness. If the target is a sub-percent sliver of the gated number, the loop cannot win — escalate the issue as mis-scoped, don't run it.
2. Distinguish work from wait. A *_request_duration that tracks the end-to-end number is a handler-attachment time, not compute. Prefer a compute/decision metric (or a profile, ADR-0006) for attribution; never infer "the component is slow" from a duration that includes downstream wait.
3. Re-scope or escalate. A real EPP-TTFT experiment needs a world where the EPP's contribution is material: an EPP-heavy workload (large prefix trees, many pods, expensive scoring) or smaller bodies so transfer isn't the whole budget. Absent that, the correct output is an escalation.

#1109 is therefore escalated (no EPP-side win exists as gated), and the planned from-scratch ext_proc isolation harness is shelved for #1109 — the pre-flight already answered what it would measure (~30 µs). The harness stays a valid general instrument for an issue where the EPP request path genuinely is the cost.

Consequences

• Positive: a ~10-minute instrumented pre-flight kills an unwinnable run before it spends a paid loop; the autoresearch stack surfaces a mis-scoped problem instead of grinding (the system working); the check is generic (any target with self-instrumentation) and slots into the judge-validation gate the controller needs; it converts "the loop keeps tying in noise" from a mystery into a one-number verdict.
• Negative / cost: another gate in onboarding (the pipeline must learn to read a target's attribution metric, or stand up the isolated harness, before greenlighting); a per-target choice of "which metric attributes the gate," which isn't always obvious.
• Carries forward: this is the empirical case for the world-scoping discipline — the judge can be perfectly frozen and still measure the wrong thing if the world doesn't isolate the target.

ADR 0008: Domains as immutable composes — the rpm-ostree model for the frozen world

Status: Accepted; partially implemented (the baked-workspace / no-runtime-clone discipline and per-domain images are in use; the compose-digest machinery and the compose reconciler are not built) Date: 2026-06-26 Related: ADR-0001 (freeze the judge AND the world — this is the world half made concrete), ADR-0005 (engine-side per-iteration rebuilds — this ADR composes the base those builds layer onto, see Design details), ADR-0006 ([[workspace.inject]] — the frozen-judge reset, i.e. the ostree "reset /usr" primitive), ADR-0007 (judge-validation pre-flight), the onboarding issue-ranking pipeline, the controller follow-up.

Context

Launching the first two local go-test domains (#1489 T1 harness, #1474 T0 bug) surfaced a cluster of failures that looked unrelated but share one root cause:

1. libgit2 has no TLS in the runtime image → setup can't clone.
2. image Go was 1.23 but upstream go.mod requires ≥ 1.25.11.
3. the #1489 harness was authored/validated against main HEAD but didn't match the signatures of the code the loop actually compiled (getBlockHashes shape, PerPromptHashes).
4. the engine skipped the clone because a stale baked workspace existed, so the loop ran against an old commit (go-redis v9.11.0 vs main's v9.20.0) — and the main-authored harness didn't compile against it.
5. the first measure does a cold go mod download at run time — slow, network-dependent, the cause of the fast "go test failed" baseline crashes.
6. the agent sandbox has no Go toolchain, so the agent bootstraps one mid-turn to test its own edits.

Every one is runtime mutation of the environment drifting from what we authored against. The judge was frozen (ADR-0001), but the world it runs in was not: the loop clones a moving target ([repo].ref = "main"), downloads deps on the fly, and we patch the running pod (emptyDir, configMap) to compensate. A frozen judge measuring an unfrozen world measures different code on every run — which also makes cross-run memory meaningless (two runs of "the same" domain compile different bytes).

Decision

Treat each autoresearch domain as an rpm-ostree-style immutable compose. The world — upstream source@SHA + the matched toolchain + pre-fetched dependencies + the frozen judge harness — is composed ahead of time into a single content-addressed image. The loop runs that image; it does not clone, fetch, or get patched at run time. To change the world you re-compose and rebase to a new digest, atomically. The agent's edits are the only mutable layer.

The key inversion: a baked workspace is not a bug — it's the point. The #4 failure wasn't that the engine skipped the clone; it was that the baked tree was a stale accident instead of the deliberate, pinned, harness-matched compose. Skip-clone is correct against a frozen compose. So the fix isn't "always clone fresh main" (that re-introduces drift) — it's "bake the matched commit and don't clone."

Pin point: the harness is validated against a specific upstream commit (the compose's source pin), whose go.mod fixes the toolchain (e.g. go 1.25.11). That commit is the compose input for #1489/#1474; the toolchain version is read from its go.mod, the deps are go mod download-ed into the image at compose time.

Why this kills the whole cluster: with no runtime clone, #1 (TLS) is moot; the toolchain is in the base, so #2 and #6 vanish; source and harness are composed together, so #3 and #4 can't drift; deps are pre-fetched, so #5 never happens. Four mismatch failures and two capability gaps collapse into one build-time discipline.

Implementation sketch (a follow-up, not this ADR)

• A per-domain compose step (Containerfile / buildah): FROM the matched-toolchain base → clone [repo]@SHA → go mod download (or vendor) → inject the frozen judge → emit a content-addressed image. Validated at build time (it compiles + emits a valid baseline JUDGE_CONTRACT), which is exactly the ADR-0007 isolation/validation pre-flight moved left into the compose.
• Layering, ostree-style: a shared repo@SHA + deps + toolchain base across same-repo domains, plus a thin per-domain layer (harness + goal + manifest), so onboarding the next issue is a small top layer.
• The loop runs with setup_cmd as a no-op (workspace pre-populated by the compose); the agent's sandbox is the composed image, edits as its writable container layer; measure resets the frozen judge.
• A planned compose reconciler (distinct from ADR-0011's PR-feedback controller) gains a "compose + validate domain image" reconcile step that replaces the live setup/clone dance — groomed → composed → validated → deployed.

Design details

Scope — what this does and doesn't freeze

This ADR freezes the build/world-definition layer, and only that. Be precise about the boundary:

• In scope (frozen by construction): upstream source@SHA, the toolchain (hard-frozen — see below), the baseline dependency set, and the judge harness — every input that was previously resolved at run time. Two runs of the same (compose input digest, overlay commit) compile identical bytes. This closes the entire launch-failure cluster (#1–#6 in Context).
• Out of scope (NOT frozen here):
  • Measurement-environment drift. The compose freezes the EPP build, not the cluster it runs against. For a go test domain the judge runs inside the composed image, so the measured thing is fully inside the frozen boundary. For a perf domain (#1109 TTFT) the score depends on live model-server pods, GPU, Envoy, and network — none of which this ADR touches. Isolating the EPP's contribution from that confound is ADR-0006 (profiling), not this ADR.
  • Judge nondeterminism. Identical bytes do not make a flaky test or a timing metric deterministic. "Reproducibility by construction" here means a reproducible world, not a reproducible score.
• Discipline requirements (the freeze is only real if these hold):
  • Pin the toolchain base by digest, not tag. FROM golang:1.25.11 moves; the floor is frozen only because the base image digest is a compose input (see the digest section). The recipe must pin it.
  • Dependencies are part of the overlay; the toolchain is not. This is the one asymmetry that matters. A fix may legitimately need a new dependency — that's normal code work, so the sandbox keeps Go module-proxy egress (the #1489 manifest already allowlists proxy.golang.org/sum.golang.org) and the agent can go get mid-turn. The new dep lands in the overlay (the go.mod/go.sum delta on top of repo@SHA, fetched into the candidate build), captured by the overlay commit in the run identity — no re-compose. The baked deps are the starting set and a warm cache, not a ceiling. The toolchain is hard-frozen: the agent must not raise go.mod's go directive past the baked version. The baked toolchain is authoritative — the engine pins it regardless of an agent edit to the go line — and a toolchain bump is a deliberate re-compose, never a turn. Rule of thumb: new deps yes, new toolchain no.

The mutable overlay is the agent's git history, not a scratch container layer

The "copy-on-write overlay" in the rpm-ostree analogy is concrete: it's the agent's git commits on top of repo@SHA, not a vague writable container layer. This is how the loop already persists edits — the agent commits per turn (see #1109), so its work is a chain of commits whose parent is the pinned compose SHA. That maps onto ostree exactly: the immutable compose is the content-addressed base commit; each turn appends a commit; the branch ref is the overlay.

This framing buys three things the container-layer framing doesn't:

• The overlay is durable and inspectable. Edits survive the per-turn workdir copy in/out (the driver doesn't share the sandbox FS across turns); they're a real ref, diffable against the base, replayable, and the unit cross-run memory reasons about ("this commit on this base scored X").
• Reset is git, not teardown. Discarding a rejected turn is reset --hard to the base or the last kept commit — the immutable compose stays pristine because nothing ever wrote to it. The [[workspace.inject]] frozen-judge reset (ADR-0006) re-overlays the harness on top, the ostree "reset /usr" move.
• "Same domain" has a precise identity. A run is (compose digest, base SHA, overlay commit). Two runs are comparable iff the first two match — which is the reproducibility guarantee that makes cross-run memory meaningful.

Relationship to ADR-0005 (engine-side builds)

The two ADRs build at different layers and cadences, and compose cleanly:

ADR-0008 does not replace ADR-0005 — it provides the base that 0005's per-iteration build layers onto. build_epp reads the overlay (the agent's current branch ref) and builds it against the composed base. That resolves two things ADR-0005 left open:

• Workspace sync (0005's "the crux"). 0005 proposed the agent commit and push its branch to the loop pod as a git remote, and build from that ref. Under 0008 that's not a workaround, it's the model: the pushed ref is the CoW overlay, and the broker builds base@SHA + overlay. No new driver plumbing, no mid-turn FS checkpoint — the overlay was always git.
• Build latency (0005 open question #1). Because the compose bakes the toolchain and pre-fetches deps (a warm GOMODCACHE/GOCACHE in the base), the per-iteration build is an incremental go build against a hot cache, not a cold kaniko-from-scratch. The expensive, network-bound work (clone, dep download, toolchain install) happens once at compose time, not on every paid iteration. The isolated registry 0005 wants becomes the place re-composed base digests live.

In short: 0008 freezes the floor, 0005 builds on it. The agent edits Go (overlay), build_epp compiles overlay-on-base for feedback, deploy_candidate/apply_cmd ensures the measured EPP is that candidate, and measure resets the frozen judge before scoring.

What content-addresses a compose (the digest)

A compose has two distinct addresses, the way ostree separates a commit checksum from the rootfs it produces:

• Compose input digest — the canonical identity. A hash over the normalized inputs that define the world, not over the resulting image bytes. This is what cross-run memory keys on and what the controller reconciles against.
• Image digest — the OCI manifest digest of the artifact that actually gets pulled and run. One input digest can map to several image digests over time (a base-image security rebuild changes the bytes but not the world); they are attested equivalent. Never key memory on the image digest — it drifts on build noise (timestamps, layer order, base rebuilds) and would falsely split runs of an identical world.

The input digest is a hash over:

1. source — repo URL @ commit SHA. The SHA already content-addresses the tree, so we hash the SHA, not a tree scan.
2. toolchain — the Go version string from go.mod (e.g. go 1.25.11) plus the toolchain base image's digest. This is the hard-frozen input: the agent cannot move it without a re-compose (see Scope).
3. dependencies (baseline) — the go.sum content hash at the compose SHA. go.sum is already a content-addressed lockfile over every module, so we hash it, never the baked GOMODCACHE bytes. This fixes the starting dep set only; an overlay that adds a dep extends go.sum, and that delta is carried by the overlay commit in the run identity — it does not mint a new base digest.
4. frozen harness — a git tree hash over the injected set (judge/harness sources + the goal/manifest + the expected JUDGE_CONTRACT).
5. compose recipe — a hash of the Containerfile / buildah steps, so changing the build logic itself invalidates the digest.

The elegant part: every input is already content-addressed (git SHA, go.sum, tree hashes), so the compose digest is a cheap, deterministic hash-of-hashes — no need to checksum gigabytes of baked image. A digest is only minted after build-time validation passes (compiles + emits a valid baseline JUDGE_CONTRACT, ADR-0007), so a compose digest always denotes a validated world. The full run identity from the overlay section is therefore (compose input digest, overlay commit) — the base SHA is already folded into the input digest.

When a re-compose triggers (auto-detect, deliberately apply)

The trigger is always one thing: the desired compose input digest no longer equals the deployed one. The controller recomputes the digest from current pinned inputs each reconcile; a mismatch means the domain has drifted and a re-compose is proposed. The sources of a mismatch:

• upstream moved — you want to pull a newer SHA. This is not auto-on-every-upstream-commit (that re-introduces the moving-target drift this ADR exists to kill); it's a deliberate pin bump that changes input (1).
• harness changed — the judge/measurement evolved (input 4). This is a memory-epoch boundary: the new world is not comparable to the old, so prior cross-run memory for the domain is retired, not carried.
• toolchain or deps changed — a go.mod Go-version bump or a go.sum change (inputs 2/3), e.g. the agent's own dependency edit.
• recipe changed — the compose logic itself (input 5).

The discipline is detect continuously, apply deliberately. The reconcile loop is Kubernetes-shaped:

observe inputs → compute desired digest
  → if desired == deployed: no-op
  → else: compose + validate candidate
      → on pass:  rebase the agent's overlay onto the new base, then atomic digest swap
      → on fail:  hold on the current digest, report the failing compose (never deploy unvalidated)

Two things stay gated rather than automatic, because they can't be silently correct: the atomic swap (it changes which world the loop measures) and the overlay rebase (replaying the agent's commits onto a new base SHA is a real git rebase that can conflict — surfaced for review, never force-resolved). So the controller detects drift and prepares a validated candidate on its own, but promoting it is a deliberate act — which is exactly the "re-pin to a new digest, never patch in place" rule from the Decision, now with a concrete trigger and gate.

Consequences

• Positive: the build/world-definition mismatch class is designed out, not patched around (cross-run memory finally compares identical bytes — of the world; see Scope for what reproducibility does and doesn't cover); measure is offline and fast (no per-run dep fetch); validation moves to build time (fail the compose, not the paid loop); it's the natural unit for the controller to manage and version.
• Negative / cost: a per-domain compose build and a re-compose to pick up upstream changes (a deliberate, atomic act — which is the point); larger images (deps baked in); the agent-edits-as-overlay relies on git history on top of repo@SHA (see Design details) rather than the immutable base ever being written, so the base stays pristine by construction. Pinning means the kept patch is against SHA, not live main, so upstreaming a win is a separate rebase — correct separation (the loop optimizes a frozen world; the PR rebases it).
• Supersedes the runtime patching (emptyDir over the stale workspace, configMap over the adapter) used to limp the first launch — those are the anti-pattern this ADR removes.

ADR 0009: Composite domains — combined multi-component autoresearch

Status: Accepted; implemented (composite manifests, the multi-workspace world, and the combined gate are live) Date: 2026-06-27 Related: ADR-0008 (each component is an immutable compose; a composite is a tuple of composes plus the assembled rig), ADR-0006 (the multi-backend broker is what lets one run profile two runtimes — pprof for EPP, torch-trace for vLLM), ADR-0001 (freeze the judge AND the world — here extended to a cross-component workload), ADR-0002 (the combined rig is heavier to provision; the broker still mediates).

Context

Every domain so far is single-component: one upstream repo, one judge, one world (EPP config, or vLLM throughput). That shape can't express the features that actually matter most in llm-d, the cross-cut ones, where a behavior is split across the router (EPP) and the engine (vLLM) and neither side's gate can see the payoff. Two real, in-flight examples:

1. LoRA-aware routing. vLLM only emits vllm:lora_requests_info, which shows an adapter only while it has requests in flight — an idle-but-loaded adapter vanishes from metrics, so routers (llm-d, gateway-api-inference-extension) guess residency from request recency and can't tell a warm adapter (in a GPU slot, ~0 load cost) from a cold one (needs a load, a TTFT spike, worst at low concurrency). The vLLM half is prototyped as the upstream RFC vllm#45411 — adding a LoRALoadEvent engine notification and vllm:lora_adapter_loaded{adapter_name,level="gpu"|"cpu",pinned} + vllm:num_{gpu,cpu}_loaded_lora_adapters gauges, so a replica advertises its residency tiers before any traffic. The EPP half (tracked under llm-d-router #1500/#709/#926) extends the existing loraaffinity scorer + loraspec metric extractor to prefer GPU-slot > CPU-cache > cold endpoints. The win, avoided cold-load TTFT, is only measurable end-to-end: across the EPP→vLLM path on a workload with mixed adapter residency. vLLM's gate can't see routing; EPP's mock-rig gate can't see real adapter-load cost.

2. P/D rollout protection. In prefill/decode disaggregation the NIXL KV connector transfers KV from a prefill engine to a decode engine, which is only valid if the two agree on their KV-transfer config (kv_cache_layout, tp_size, block_size, dtype, model, NIXL backend/version — the NixlConnectorWorker/handshake inputs). During a rolling update some endpoints carry the new config and some the old; if the EPP pairs a prefill from one generation with a decode from another, the transfer fails or corrupts. The EPP has no way to block that pairing today, because vLLM exposes no P/D config hash for a routing filter to compare. The combined feature: vLLM exposes a fingerprint of the KV-transfer-relevant config; the EPP adds a filter plugin that excludes any P/D pairing whose hashes disagree (rollout protection). Its fitness, zero cross-incompatible transfers while preserving throughput during a rollout, again exists only in the live EPP→vLLM-P/D system under a rollout event.

Both share a shape the engine can't currently host: vLLM exposes a signal, the EPP routes/filters on it, and the fitness is a property of the assembled system, not of either repo. One is a soft scorer, one is a hard filter; both need two repos edited together and measured as one. A single-component manifest ([repo] is exactly one url|path, one [judge], one world) cannot represent this.

Decision

Introduce composite domains: a domain that composes N component domains into one autoresearch run. A composite is config, not new per-feature engine code — it reuses the component domains (each an ADR-0008 immutable compose) and adds three things the engine must learn to do generically:

1. A multi-workspace world — the agent may edit any component's checkout in a turn; keep/discard and git memory span all components (a kept iteration is a tuple of overlay commits, one per component).
2. A single combined gate — one measure_cmd that stands up the assembled system (EPP routing to vLLM backends) and runs the frozen cross-cut workload, emitting the usual contract.
3. The union of component surfaces — both components' PATH tools, skills, and profiler backends are available in the one run (the agent profiles vLLM with torch-trace and the EPP with pprof — ADR-0006).

The engine stays component-agnostic: "combined" is a composite manifest plus a combined gate command, not a lora-routing or pd-rollout special case baked into Rust. New cross-cut features are new composite manifests + new gate wrappers, the same way new single domains are new manifests today.

# a composite domain's crucible.toml (this one assembled the vllm + epp packs)
[composite]
name = "lora-routing"

[[component]]
domain = "vllm"            # reuses the vllm domain pack (its compose, tools, profiler)
[[component]]
domain = "epp"             # reuses the epp domain pack

[judge]
# Drives the ASSEMBLED system: EPP in front of vLLM replicas with a mixed-adapter workload, scores
# end-to-end serving (e.g. TTFT for requests whose adapter is cold). Frozen workload = the adapter mix +
# concurrency + (for P/D) the rollout schedule (ADR-0001 extended to a cross-component evaluation).
measure_cmd = "lora-routing-bench --json"
direction   = "higher"

[world]
# Composite reversibility: each component's git overlay PLUS the live assembled rig (snapshot/restore
# the deployed EPP config + the vLLM replica set). One opaque token bundles both (ADR-0008 §overlay).
snapshot_cmd = "lora-rig-snapshot"
restore_cmd  = "lora-rig-restore"

Implementation sketch (a follow-up, not this ADR)

• Manifest: a [composite] block with [[component]] entries referencing existing domain dirs (each contributing its [repo]/workspace, tools, skills, and profiler backend). A composite has no [repo] of its own; [judge]/[world] are the combined gate and reversibility. Single-component manifests are unchanged (a degenerate composite of one).
• Engine: generalize the World + run identity from one workspace to a vector of workspaces — the one real engine change. Run/Segment/Snapshot (ADR-0004) already model an iteration; extend the snapshot to a tuple of per-component overlay commits + the world token.
• Gate: the combined measure_cmd is a domain-owned wrapper (like vllm-throughput) that assembles the rig and runs the cross-cut workload. The engine treats it as the same opaque contract.
• Compose: each component is composed independently (ADR-0008); the composite pins a tuple of component digests. A re-compose of one component is a memory-epoch boundary for the composite.

Design details

Run identity is a tuple (cross-run memory still works)

ADR-0008 made a single run's identity (compose input digest, overlay commit). A composite extends this to a vector: ({(digest_i, overlay_i)} for each component, cross-cut workload hash). Two composite runs are comparable iff every component's compose digest matches and the frozen cross-cut workload matches — so cross-run memory ("this EPP-overlay + this vLLM-overlay on these bases scored X") stays meaningful, the same guarantee ADR-0008 gives per component, lifted to the tuple. The agent's keep/discard commits each component's overlay independently; a kept iteration is the tuple of commits that scored.

Why a composite, not a merged repo

We do not vendor EPP into vLLM or vice versa. Each stays its own upstream repo@SHA with its own toolchain (Go vs CUDA/Python), its own immutable compose, and its own profiler — because the kept patch must upstream to its repo as a normal PR (ADR-0008's "the loop optimizes a frozen world; the PR rebases it" applies per component). The composite is an assembly of frozen worlds, not a third repo. This is also why the engine must stay component-agnostic: the unit of reuse is the existing domain pack.

The combined gate is the only place the components meet

Amendment (2026-07-01). ADR-0013 (proposed) moves benchmark execution into a pluggable rig backend, which produces a metrics bundle. Scoring stays here: the engine runs the domain's measure_cmd (now over that bundle), so measure_cmd remains the single engine↔domain contract. The backend runs the workload; it never computes the score.

Everything component-specific stays in its component (tools name-prefixed — vllm-profile vs the EPP's profile; skills per-domain; profiler backends per-runtime). The single integration contract is the combined measure_cmd: it is the one artifact that knows both components exist, because it assembles them and runs the cross-cut workload. Keeping the meeting point to one command is what keeps the engine generic — mirrors the single-domain rule that measure_cmd is the only engine↔domain contract.

Merging surfaces without collisions (a real gotcha)

"Union of component surfaces" is not a free cp. Three things need care at assembly time:

• Tools are safe; skills are not. Component tools are already name-prefixed (vllm-profile, vllm-throughput vs the EPP's profile/bench), so PATH merges cleanly. But skills are not prefixed: two components router and engine both ship skills/profile, literally named profile, so merging both into one .claude/skills collides. The composite must namespace skills by component on assembly (e.g. router-profile / engine-profile), rather than each domain pre-prefixing (which would uglify the single-domain case). Namespacing is a merge-time concern, owned by the composite.
• The method prompt is composite-level. A component's method_prompt describes optimizing that component. A composite needs its own prompt that frames the cross-cut goal and both surfaces (when to edit the EPP scorer vs the vLLM signal it consumes). Components contribute their toolbox/skills; the composite owns the standing method.
• Frozen-judge injects stay per-component ([[workspace.inject]], ADR-0006) — each lands in its own component workspace; the combined gate is the cross-cut judge on top.

Where the combined gate runs (locus)

For a GPU composite the gate can't run in the agent's sandbox (no cluster reach, and measuring through a port-forward confounds the result — the EPP lesson, ADR-0006). It runs engine-side, the same locus as the EPP measure/build broker tools (ADR-0005/0006): the loop pod (or a broker-dispatched job) stands up the assembled rig and runs the cross-cut workload in-cluster. The composite therefore requires the broker on, with both runtimes' backends — which is why ADR-0006's multi-backend broker is load-bearing here.

What the two drivers exercise (and why both)

• LoRA-aware routing is a scorer (soft preference) — the gate rewards routing warm adapters; a bad policy costs TTFT but never breaks correctness. The signal is vLLM residency gauges; the agent edits the EPP scorer (and possibly the vLLM eviction policy), profiles both, and the gate scores end-to-end TTFT.
• P/D rollout protection is a filter (hard constraint) — the gate must show zero cross-incompatible KV transfers under a rollout while preserving throughput; too strict strands capacity, too loose corrupts. The signal is a vLLM P/D config hash; the agent edits the EPP filter and the vLLM hash it exposes. The frozen workload includes a rollout schedule (the cross-cut analogue of a request mix).

Carrying both in the ADR keeps the design honest: the engine change (multi-workspace world + tuple identity + combined gate) must serve a soft scorer and a hard filter, edits on the EPP side and the vLLM side, without either feature leaking into engine code.

Provisioning + cost (ADR-0002)

The combined rig is the heaviest yet: real vLLM replicas (GPU, and for P/D a prefill+decode pair with NIXL) behind a live EPP, plus a rollout/adapter-mix harness. The broker (ADR-0002) still mediates the expensive grants; the composite just needs two runtimes' worth of capacity and the multi-backend profiler from ADR-0006. This cost is the reason composites are opt-in, not the default shape.

Consequences

• Positive: the cross-cut feature class becomes optimizable at all — LoRA-aware routing and P/D rollout protection get a real fitness instead of hand-tuning; the win is measured where it actually exists (end-to-end), closing the neither-domain-can-see-it gap; the engine gains one general capability (multi-workspace runs) rather than per-feature code; component packs are reused verbatim, so a composite is mostly a manifest + a combined gate.
• Negative / cost: the heaviest rig to stand up and keep reversible (GPU P/D + EPP + a rollout/mix harness); multi-workspace git memory + tuple run-identity is new engine work (the one non-trivial change here); the combined gate is bespoke per cross-cut feature (authoring, not engine); a re-compose of any component retires the composite's cross-run memory epoch. Composites are opt-in for exactly these costs.
• Forward pointer: validates the component-agnostic discipline the recent work already follows — generic crucible-broker with injected per-domain backends, name-prefixed tools, per-domain skills, measure_cmd as the sole contract. Hold that line and a composite needs no engine special-casing beyond the multi-workspace world.

ADR 0010: Candidate portfolios — explore/exploit search over reviewable candidates

Status: Accepted; implemented (v1, 2026-07-02: parallel propose turns + serialized measurement + the TopK SearchPolicy behind --wide N / [search]; round-robin and successive-halving policies remain future impls) Date: 2026-06-28 Related: ADR-0004 (the core loop is a sequential refinement of one line; this generalizes it to a population), ADR-0009 (a composite candidate is a tuple of per-component edits; the portfolio is a set of such tuples), ADR-0001 (the frozen judge is what makes a portfolio rankable instead of a taste test), ADR-0002 / ADR-0003 (a candidate surfaces as a reviewable draft PR; a composite candidate as a set of linked PRs), ADR-0005 (deploy-to-rank: a candidate that changes the rig can only be scored once the broker builds+deploys it).

Context

The loop today (ADR-0004) is a single line of descent: iteration N refines iteration N−1, --iterations N turns deep. That is pure exploit — it makes one approach better. Running the composite P/D rollout-protection goal one-shot (--iterations 1, openshell backend) showed two things at once:

1. One-shotting the right shape works. Cold, in one turn, claude found that compute_nixl_compatibility_hash() already exists in vLLM, exposed a P/D config-hash, and added an EPP filter pairing only matching prefill↔decode — the correct cross-cut architecture, reasoned from scratch.
2. A single shot is one sample from a wide space. The same goal has architecturally distinct solutions with very different blast radius, and the model "knows" several of them. Betting the whole run on the first one it picks throws away the others — and "which is best" is not an eyeball question.

The competing approaches for this goal, concretely (each a legitimate, different candidate):

These are not refinements of each other — they are different families. And the frozen gate (pd-rollout-bench) can actually rank them, including behaviorally: a proactive filter jumps to ~100% success immediately; the reactive dead-peer ramps as it learns, so its score depends on how long the fixed workload runs. That difference is measured, not argued.

Decision

Generalize the loop from a single line of descent to a candidate portfolio with two explicitly different search modes, ranked by the frozen judge, and surfaced as reviewable PRs.

Design review amendments (2026-07-01, ahead of implementation). The approved v1 design pins: propose turns run in parallel (thread-per-candidate, worktree-per-candidate under the run's state dir) while measurement serializes on the shared rig by staging each candidate's patch through the main workspace (no World/Judge trait changes); the wide→deep hand-off is a pluggable SearchPolicy ([search].policy, v1 ships top-k only; round-robin / successive-halving are future impls — top-k-after-one-measurement picks winners from a single noisy reading, which those policies fix by re-measuring survivors each round); session-log rows gain an additive phase: "wide" field; and [search].approaches is required when wide > 0 — no auto-generated fallback, per §3's engineered-diversity rule.

1. Two investigation strategies, not one knob. Iteration count is not the axis; strategy is:

• Wide / explore. N independent candidates in parallel, each a different angle (method-prompt biased to a distinct approach from the table above), each with minimal context — "implement a minimal version of approach X." Each runs in its own sandbox, so the fan-out costs ~one turn of wall-time. Goal: discover which families clear the gate at all.
• Deep / exploit. Take the gate's top-k and iterate them sequentially with full context — the prior diff, its measured score, and what specifically failed — "make this actually solve it." This is ADR-0004's loop, unchanged, applied to a seeded starting point.

The method prompt differs by mode; the difference is the strategy.

2. Explore → exploit (a tournament with a refinement final). The default run is: a wide round → rank by the gate → a deep round on the winner(s). The wide round seeds the deep round. Today's engine only does the deep half; this ADR adds the wide half and the hand-off. (A pure-wide or pure-deep run remains valid for the degenerate cases.)

3. The judge is the ranker (ADR-0001). A portfolio is only better than guessing because the frozen gate gives an objective, reproducible score per candidate. No candidate is "picked" by review aesthetics; review chooses among gate-ranked options.

4. Candidates are reviewable artifacts (ADR-0002/0003), composite-aware (ADR-0009). Each candidate is published as a draft PR per edited component, cross-linked (the P/D candidate → one PR on the vLLM fork + one on the EPP fork). The PR is based on the exact sha the agent edited against (reachable in the fork's object network), so it is immune to the fork having diverged from upstream. A portfolio is therefore a set of PR-sets a human can diff side-by-side while the gate orders them.

5. Deploy-to-rank is the gate to real scores (ADR-0005). A candidate that changes the rig (vLLM + EPP) is only scored after the multi-backend broker builds+deploys it. Until then a candidate is captured and reviewable (its diff + PRs) but scores at baseline. Ranking a portfolio by true fitness requires the deploy path; reviewing it does not.

Consequences

• The loop gains a population and a (wide, deep) schedule; ADR-0004's Segment/Iteration typestate becomes the deep leg, and a new fan-out drives the wide leg over the same diff-capture + PR-publish plumbing (World::staged_diff, the native publish.rs single- and multi-fork draft-PR path).
• Cost scales with breadth: N wide candidates = N sandboxes (parallel) + N deploys to rank. Breadth is a budget knob, not free.
• Diversity must be engineered (distinct method prompts / forced approaches); N identical turns are not a portfolio. Prompt-diverse fan-out beats one-turn-self-enumerate because the latter shares the model's first framing.
• "Solved" is still the gate's call, now over the best of a population rather than the end of a single line.

Alternatives considered

• Single-line refinement only (status quo). Simplest, but bets the run on the first approach and has no way to compare families — exactly what the one-shot experiment exposed.
• One turn self-enumerates and implements K approaches. Cheaper than fan-out, but the K attempts share one context and one framing, so they cluster; weaker diversity for the same review/rank cost.
• Human picks the approach up front. Throws away the gate's whole reason to exist (objective ranking) and the model's ability to surface approaches a human wouldn't enumerate (e.g. #6 reactive dead-peer).

ADR 0012: Crucible-rendered deployments — generate the loop/broker/rig manifests from the manifest + a deploy profile

Status: Accepted (slice 1 implemented) Date: 2026-06-28 Related: ADR-0002 (the broker whose env the pod hand-duplicates), ADR-0005 (the FORGE_* build/deploy config the pod hand-duplicates), ADR-0008 (immutable composes — the rendered deploy must pin digests, not tags), ADR-0009 (the composite loop pod that drove this), ADR-0011 (the controller runs "deploy" as a step — it should call a renderer, not hand-edit yaml).

Context

A crucible run is a pod (the loop), usually a child broker process, the live rig it measures, and the RBAC

• secrets + volumes that wire those together. Today all of that is hand-written, per-domain YAML. Bringing a composite rig online requires hand-editing, in lockstep:

• loop-pod-openshell.yaml — ~250 lines: the wrapper script, ~20 env vars (FORGE_*, VLLM_*, BROKER_COMPOSITE, RIG_NAMESPACE, KUBECONFIG, Vertex creds), six volumes/mounts (kube token, kubeconfig, push authfile, forge storage, prior-candidate, quay auth), serviceAccountName, a digest-pinned image.
• loop-rbac.yaml — a cross-namespace RoleBinding.
• modelservers.yaml — the rig.
• ad-hoc configmaps (prior-candidate, loggers).

Three structural problems:

1. The pod env duplicates the manifest. FORGE_REGISTRY/FORGE_DOCKERFILE/FORGE_DEPLOY_*, VLLM_BASE_REF, BROKER_COMPOSITE, the broker bind/name — these are facts crucible already knows from crucible.pd-rollout.toml ([agent.broker], the components, the deploy targets). They are copied into the pod spec by hand and kept in sync by hand. The manifest is the source of truth; the pod is a stale shadow of it.

2. Manual digest pinning is a footgun, not a convenience. Every image (loop-openshell, the sandbox) is pinned to a sha256:… with a comment "re-pin after each rebuild — a mutable tag serves the stale cached layer." Re-pinning a milestone tag by hand after a rebuild is error-prone; forget it and the node silently runs the old binary. The digest is mechanically derivable from the tag at deploy time.

3. The pod spec is ~90% identical across domains. The EPP loop pod and the composite loop pod differ only in domain-specific env + which broker binary + which manifest. The shared structure (openshell privileged pod, projected kube token, IRSA, forge storage, broker child) is copy-pasted and diverges.

This is the same theme as ADR-0011: the engine is clean; the deployment of it is manual toil that should be the engine's job.

Decision

Crucible renders its own deployment. A crucible deploy render (emit YAML) / crucible deploy apply command generates the loop pod + broker wiring + RBAC + rig references from two inputs:

• the domain manifest (already the source of truth: components, [agent.broker], [agent.env], build /deploy targets), and
• a thin deploy profile — the only hand-written part: the environment-specific facts crucible can't know (cluster/namespace, secret names, creds source, GPU/resources, the backend = openshell vs local, the results bucket / IRSA role). One small file per cluster, not per run.

What the renderer owns:

• Project manifest config into pod env, instead of hand-duplicating it. The [agent.broker] + build /deploy targets become the broker child's FORGE_*/BROKER_* env; the components become VLLM_*/the apply-hook env. Change the manifest, re-render — no second copy to forget.
• Resolve + pin image digests at render time. The profile names a tag (or :latest of a build); the renderer looks up the digest and emits the @sha256:… pin. The stale-layer footgun disappears; the pin is always correct because it's computed, not typed.
• Share a template per backend pattern. One openshell-loop template (privileged pod, projected token, IRSA, forge storage, broker child) parameterized by manifest + profile; domains stop copy-pasting it.
• Emit the RBAC + secret references the run needs (the SA → edit binding in the rig namespace, the push-authfile mount), as part of the same render — not a separate hand-authored file that drifts.

What it deliberately does not own: the contents of secrets (it references them; a human/External Secrets provisions them), and the rig's domain YAML where the rig is a fixed artifact (ADR-0001 frozen workload) — the renderer references the rig, it doesn't invent model-server topology.

  crucible.pd-rollout.toml ─┐
                            ├─ crucible deploy render ──► loop-pod.yaml + rbac.yaml (+ apply)
  deploy-profile (cluster) ─┘        │
                                     ├─ env projected from the manifest (no hand-duplication)
                                     ├─ image refs resolved to @sha256 (no manual re-pin)
                                     └─ one shared openshell-loop template across domains

Consequences

• One source of truth. The manifest already defines the run; the deploy stops being a second, hand-synced copy of it. "Change the gate / the broker / a component" is a manifest edit + re-render, not a manifest edit + a yaml safari.
• Reproducible + footgun-free deploys. Digests are computed; a re-render after a rebuild is always correct. The "forgot to re-pin, ran the stale binary" class of bug is gone.
• Composes with the controller (ADR-0011). The autoresearch controller's "deploy" step becomes crucible deploy apply — it never hand-writes pod specs. And with ADR-0008, the rendered deploy pins the composed image digests, so the immutable-compose guarantee reaches all the way to the running pod.
• Lowers the bar for a new domain. Onboarding a domain (the vLLM/EPP/composite progression) stops requiring a 250-line pod spec; it's a manifest + a profile entry.
• Open questions.
  • Render engine. A native Rust renderer in crucible (typed, knows the manifest) vs emitting kustomize /helm (gitops-native, but another layer). Lean native: crucible already parses the manifest and resolves images; a typed render keeps the pod spec honest to the config types.
  • How much k8s to model. Just the loop pod + RBAC + broker wiring (the run), or the rig too? Start with the run (the part that's pure duplication); leave the frozen rig as a referenced artifact. Resolved by ADR-0013 (proposed): the rig half is deferred to a pluggable external renderer ([rig].backend / RigBackend); this renderer keeps owning the run.
  • Profile granularity. Per-cluster vs per-domain-per-cluster. Start per-cluster with per-domain overrides only where a domain genuinely needs more (GPU, extra secrets).
  • Apply vs emit. Emit YAML (review/gitops) by default; apply as a convenience that shells kubectl. The renderer stays declarative; applying is a thin wrapper, same split as ADR-0005's build-vs-deploy.
  • Secrets. Reference-only (the renderer never holds a credential); document the expected secret names in the profile so a missing secret is a clear render-time error, not a CreateContainerConfigError later.

Implementation (slice 1)

crucible deploy render|apply (crucible/src/deploy/) renders a composite's loop Pod + cross-namespace RoleBinding from the manifest + a per-cluster profile, as real k8s-openapi objects serialized to YAML. Decisions taken, resolving the open questions above:

• Native Rust renderer, real k8s-openapi types (the spec stays honest to the API). Emit YAML by default; apply pipes it through kubectl apply -f -.
• Build/deploy targets are typed manifest config ([deploy]): the generic forge build contract (buildah → FORGE_*, deploy_name → FORGE_DEPLOY_NAME) is typed; a per-component env map carries the domain hook's own env names verbatim. On a composite, [deploy.<component>] overrides the base.
• Perfectly generic engine. The renderer names only env it itself consumes (BROKER_*, FORGE_*, OPENSHELL_SUPERVISOR_IMAGE, mount-path consts). Every domain/vendor name (EPP_*, VLLM_*, GCLOUD_CREDENTIALS, BENCH_*) lives in the profile's generic env/secret_env maps or [deploy].env. The nested podman was originally logged into a single registry parsed from the sandbox image ref (forge::oci::registry_of); the wrapper now sets REGISTRY_AUTH_FILE at the mounted authfile instead, which podman honors ahead of its own lookup and which covers every registry a run pulls from.
• Digests resolved in-process via oci-client (forge::oci::pin_digest, no crane/skopeo), using the operator's existing registry login. Verified: the milestone tag resolves to the exact @sha256:… the loop pod previously hand-pinned.
• Profile (profile.<cluster>.toml) is the only hand-written part: namespaces, SA, secret names, resources, the loop image tag (not digest), and the generic hook/gate env.

Scope held to the run (loop pod + RBAC). Single-domain (non-composite) render landed 2026-07-02 (a plain manifest renders as a degenerate composite of one; it needs its own [deploy] block). Still deferred: a local-backend template, and modeling the rig (since answered by ADR-0013).

kube-rs migration — the engine no longer depends on kubectl

The goal here is that the engine and the [world] hooks never depend on shelling kubectl — not to strip the binary. kubectl stays in the loop images as an operator/agent debugging affordance (too handy to drop); the engine just doesn't call it. forge::kube (typed kube-rs 4.0 + k8s-openapi 0.28) owns the client, patches, rollout-wait, pod exec/exec-streaming, GPU headroom, and dynamic server-side apply, behind a blocking facade (a block_on that uses block_in_place under the broker's runtime, else a private one). Migrated off kubectl:

• forge::deploy / current_image → typed patch + get (the jsonpath is gone); tag_of → oci-client Reference; rollout timeouts parse via jiff.
• the broker GPU admission (gpu_check) → typed Node/Pod sums + apply_yaml (no more serde_json::Value walking).
• crucible deploy apply → server-side apply_yaml.
• the pd-rollout [world] hooks (apply/snapshot/restore) → a generic forge-kube CLI (set-image/set-pull-secret/set-rolling-update/rollout-restart/rollout-status/current-image/ pod-name/exec/apply); the domain orchestration stays in the hook.
• the remote viewer (crucible view --pod) → a streaming exec (tail -F) + exec-with-stdin for steer/stop.

Both loop images ship forge-kube (the hooks' typed kube path) and keep kubectl (the debugging affordance). The EPP rig's engine-critical [world] hooks (rig-snapshot/restore/down) are migrated to forge-kube; the EPP debug/profiling tools (inspect-rig, epp-metrics, epp-pprof) deliberately stay on kubectl — they're interactive debugging, and epp-pprof is better migrated alongside ADR-0006.

ADR 0014: Scoping as a governed pipeline — crucible scope <issue>

Status: Proposed; S0+S1 implemented (the crucible scope pipeline skeleton and the [judge.selftest] primitive are live; S2, the propose turn, is in progress) Date: 2026-07-02 Related: ADR-0001 (the scoping phase this finally implements — "an agent may propose the harness config"), ADR-0007 (the preflight, here a pipeline stage), ADR-0008 (the freeze the pipeline ends in), ADR-0010 (the wide leg that makes a freshly scoped issue cheap to attack from several angles), the judge-tier issue ranking (onboarding triage), crucible check + crucible init (the BYO on-ramp), crucible::identity (the freeze fingerprint).

Context

The engine side is now rich — composites, mediated builds/deploys, profiling, portfolio search, identity stamping — but every run still starts from a hand-authored harness: a human writes the manifest, the gate, the world hooks, picks the workload and the baseline. That is roughly a day of expert work per issue, and it has become the throughput bottleneck: the loop can explore five approaches in parallel, and it still takes us a day to give it something to explore.

It is also the most dangerous manual step, not just the slowest. The two best incident stories in this repo are both harness failures: #1109 burned a loop on a gate that could not isolate its target (ADR-0007 exists because of it), and #1489's reward hack lived in a gate that under-counted the PreRequest path. ADR-0001 designed the answer — scoping is adaptive, once, then frozen, with the proposal automated and the output validated and human-approved — but only the freeze was ever built. This ADR builds the rest.

Decision

Introduce crucible scope <issue-url>: a five-stage pipeline that turns an upstream issue into a validated, human-approved, frozen domain pack. Every stage exists to kill a bad harness as early and as cheaply as possible.

1. Ingest + triage. goal-from-issue extracts the goal; the judge-tier ranking (T0/T1/T2/N) decides whether the issue is scopeable at all and biases the proposal (a T0 bug wants a test gate; a T1 perf issue wants a measured workload). An N-tier issue exits here with the reason.
2. Propose. One agent turn drafts the harness: the manifest, the gate script, the workload shape, the baseline choice, and the negative controls (a known-bad and a known-good configuration the gate must tell apart). This is ADR-0001's sanctioned adaptivity, in its sandbox: the proposer gets solution-space tools plus the issue, nothing else.
3. Validate mechanically. Three checks, all existing machinery, all unbypassable:
   • crucible check — the contract lint (manifest parses, files resolve, the measure contract emits {valid, score}, the gate is not agent-editable without a frozen inject);
   • the gate self-test (ADR-0001 safeguard 1) — the proposed known-bad must score strictly worse than the known-good, else the gate cannot discriminate and the proposal dies;
   • the isolation preflight (ADR-0007) — the target's own contribution to the gated metric must be material, measured from the target's instrumentation, else the issue is mis-scoped and the pipeline emits that finding instead of a harness. A failure at this stage kills the proposal before any human reads it.
4. Approve. The surviving domain pack opens as a draft PR (the ADR-0002 approval muscle): the manifest, gate, controls, and the validation evidence (self-test scores, preflight attribution) in one reviewable diff. A human signs off before the loop spends budget — ADR-0001's checkpoint, unchanged.
5. Freeze. On approval the manifest is pinned, the pack's RunIdentity digest is computed and recorded as its fingerprint, and the issue is runnable: crucible run (or a wide round, ADR-0010) against a world whose provenance is one content-addressed line.

Why an agent proposing the judge stays honest

This is the reward-hacking surface ADR-0001 fenced, so the trust story must be explicit:

• Proposer ≠ optimizer. The scoping turn and the optimizing loop are different runs with no shared context; the scoping agent never runs against — and never sees the results of — its own gate.
• Validation is mechanical. The self-test and preflight are computed, not judged; a gate that cannot discriminate, or a target that is a sub-percent sliver of the metric, fails arithmetic, not taste.
• A human approves the evaluation. Nothing the proposer emits becomes a judge until a person has read the gate and its evidence — the same rule as every other judge change in the system.
• The freeze is stamped. The approved pack's identity digest makes post-approval drift detectable (ADR-0008); the gate that runs is provably the gate that was approved.

What this is NOT

• Not an adaptive loop. The ADR-0001 wall stands: scoping adapts once, then freezes. The pipeline never runs concurrently with an optimizing loop on the same domain.
• Not auto-run. Stage 4 is mandatory. A wrong harness is wasted or misleading research; it keeps its human checkpoint.
• Not a replacement for author-m2-rig. v1 targets test-gate and local/GitWorld domains (the #1474/#1489 shape, and the BYO on-ramp). Composite GPU rigs need rig topology judgment that stays with a human until the ADR-0013 rig backends make "stand up the rig for the preflight" cheap.

Consequences

• Positive: onboarding drops from a day of expert work to review-of-a-draft-PR; the two known harness failure modes (#1109 mis-scoping, #1489 gate blind spots) are caught by construction — stage 3 runs exactly the checks that would have caught each; every prior wave's machinery becomes load-bearing (check, identity, publish, the wide leg).
• Negative / cost: the gate self-test becomes a first-class engine contract (the manifest needs a place to declare the controls — likely [judge.selftest] with good_cmd/bad_cmd), which is new surface; scope turns cost real tokens on issues that may die in stage 3 (bounded by a per-scope budget cap); the preflight needs per-target attribution metrics, which not every issue has (absent instrumentation, stage 3 degrades to self-test + check and says so in the PR).

Workstreams

Slicing: S0 + S1 first (they harden hand-written harnesses immediately, before any agent proposes one); S2 once the validation net is real; S3/S4 to close the loop.

ADR 0017: Turn result contract — structured state back from turn pods

Status: Proposed (supersedes the 2026-07-06 stub; amends one ADR-0016 ruling, see §"Amendment"; refined 2026-07-07: harness crate + OTLP telemetry restoration + MLflow export workstream) Date: 2026-07-06 Related: ADR-0015 (the WorkPod primitive whose results this carries), ADR-0016 (the ledger the results land in, and the truth model this must not break), the private control plane's workpod dispatcher (today's log-marker scrapers), a run of internal incident PRs (the fragility class that motivated this), opendatahub-io/agentic-ci (the vendored harness whose OTLP collector and library role this restores). External prior art: Tekton results + TEP-0086/TEP-0127, Argo Workflows output offloading, GitHub Actions runner auth, Buildkite job tokens, Temporal task tokens (condensed in §"Prior art"; full cited notes in the appendix).

Context

Turn pods (grounded rank, scope) and run pods return everything to the controller by printing to stdout, scraped from the kube log after the pod goes terminal. Six marker literals, each duplicated across crucible/ and crucible-controller/ with "keep in sync" comments as the only contract:

One night of production use (2026-07-05) surfaced the fragility class four ways:

• the openshell event stream glues messages without newlines, breaking a last-line parse contract (a validated $8 pack died on the adversary verdict parse — fixed by a reverse-scan fallback in an internal PR);
• every payload fights the kubelet's ~10 MiB log-rotation budget, forcing per-payload byte caps and truncation policies — and rotation can still eat a marker line entirely, which no scan direction survives;
• binary artifacts ride logs as giant base64 lines — the least log-shaped data imaginable;
• markers duplicate literals across crates because the crate DAG forbids sharing (crucible and crucible-controller deliberately do not depend on each other).

Two adjacent gaps compound it: startup adoption re-ingests grounded-rank verdicts but not scope reports (reconcile_turn_on_startup only calls parse_verdict_logs, workpod.rs:1506), so a scope turn that finishes while the controller is down is lost; and turn pods are invisible to the completion watch — they are polled in-band by KubePodDispatcher::await_terminal() (workpod.rs:1376), which discards the pod status it already holds.

Markers were the right zero-infrastructure bootstrap. They are the wrong permanent contract for control-flow-critical, sized, structured data.

The harness gap

Crucible began as a consumer of the vendored agentic-ci harness and re-implemented its stream decode + NDJSON event model natively when the vendored package was dropped. Two things fell on the floor in that move:

• The OTLP collector never made the jump. agentic-ci ran a small OTLP http/json receiver next to the agent, injected CLAUDE_CODE_ENABLE_TELEMETRY=1 + the OTEL_* exporter env into the sandbox, logged every export to a jsonl, derived a live token rate, and rolled it all up into the otel_summary NDJSON event at run end. Crucible kept the consumer half — AgentEvent::OtelSummary (event.rs:135), the TUI rendering, the prefer-OTEL-cost logic — but nothing produces the event anymore: cost is estimated from a hardcoded pricing table (agent.rs:283) that silently drifts with every model release, Tokens.rate is permanently None (stream_json.rs:214), and per-model splits, API request latency, and active time are simply gone from the evidence.
• The harness is trapped in a binary. Agent spawn, stream-json decode, the event pump, and session logging live inside crucible's bin target (agent.rs, stream_json.rs, event.rs, session.rs). Nothing outside this repo can drive an agent run the way agentic-ci's library consumers could; there is a standing requirement to expose that capability as a crate again.

The contract work below forces the crate question anyway (the marker literals need a home shared across a crate DAG that forbids crucible ↔ crucible-controller dependencies); this ADR answers it once for both gaps.

Prior art (survey, 2026-07-06)

The industry converged on exactly two mechanisms, and on abandoning everything else:

• Tekton returns task results via the container termination message and lives inside its limits: 4096 bytes per container, 12 KiB per pod divided equally among containers. Hitting that ceiling produced TEP-0086 (options: storage API service, sidecar, ConfigMaps, CRDs, PVCs, logs — logs rejected outright for having "no availability guarantee") and TEP-0127 (sidecar-logs escape hatch: ~3 s pod overhead, controller needs log RBAC, still capped by the 1.5 MiB CRD ceiling). Lesson: termination message for small verdicts only, and don't build the large tier on logs or CRDs.
• Argo Workflows started with output parameters in pod annotations (256 KiB limit), migrated to a WorkflowTaskResult CRD, and still fights the 1 MiB etcd ceiling with compression and optional database offload. Artifacts (the large tier) go to an artifact repository, not through the Kubernetes API. Lesson: the Kubernetes control plane is not a result store; two tiers are load-bearing, not an optimization.
• Kubernetes itself: terminationMessagePath (default /dev/termination-log) is read by the kubelet into containerStatuses[].state.terminated.message, atomically, as part of pod status — no scraping, no rotation exposure. Truncation at 4096 bytes is silent. FallbackToLogsOnError returns at most 2048 bytes / 80 log lines and only on error exits — useful as a debugging aid, not a contract.
• GitHub Actions / Buildkite runners push results and artifacts to the control plane over authenticated HTTP with a per-job token: minted by the control plane at dispatch, scoped to exactly one job, lifetime = job timeout plus a small grace (GitHub: +10 minutes), never persisted, scrubbed from logs. This is the standard shape for our Tier 2 — credential scoped to one unit of work — for which Kubernetes has a native carrier (bound service-account tokens, below).
• Temporal completes activities by RPC with a single-use task token and is explicit that delivery is at-least-once: the completer must be idempotent, keyed on the stable task identity, not the token. Lesson: the controller deduplicates; a retried POST is normal, not an error.
• Prow uploads artifacts to GCS from a sidecar and signals completion with marker objects an external watcher polls — the pattern we'd converge on if we ever outgrow a single controller; noted, not adopted.

Every migration in the survey moved away from logs, annotations, and CRDs and toward "tiny critical payload in pod status, everything else over authenticated push to a store." That is the design below.

Decision

A two-tier result contract, defined in one shared crate. Stdout demotes to a human-facing live/debug channel: progress beats stay, payloads leave.

Tier 1 — the verdict rides the termination message

The turn command (both rank-grounded and scope, behind the existing --marker-style flag, renamed in spirit to "result mode") writes its result JSON to /dev/termination-log as its final act. The controller reads it from pod.status.containerStatuses[].state.terminated.message — atomically, from the same pod status object every read point already holds. No scraping, no rotation exposure, no glue.

• Budget: turn pods are single-container, so the kubelet grants the full 4096 bytes. The engine self-caps the JSON at 3584 bytes (headroom against the silent kubelet truncation) with honest field-level truncation of rationale text, the same discipline the transcript cap uses. A truncated verdict is still valid JSON; a kubelet-truncated one is garbage — the self-cap is what keeps parse failures structural rather than probabilistic.
• Schema: a versioned envelope in the contract crate: {"v":1,"kind":"verdict"|"scope_report","payload":{…},"artifacts":[{"kind","digest","bytes","delivered"}],"usage":{…}}. usage is the optional OTEL rollup (§"The OTLP collector comes back"): {"cost_usd","total","models":[…],"api_requests","api_ms","active_seconds"} — a few hundred bytes that put authoritative cost on the ledger row instead of a pricing-table estimate. Under self-cap pressure the per-model rows drop first; the totals stay. The artifacts manifest is the load-bearing novelty: Tier 1 is the authoritative index of every Tier 2 upload the pod attempted, with content digests. The controller trusts the termination message (authenticated by the kubelet), then checks the drop-box against the manifest — a missing or digest-mismatched artifact is detected without trusting the POST path at all.
• terminationMessagePolicy stays File. FallbackToLogsOnError truncates to 2048 bytes/80 lines and only fires on error exits; it would hand us exactly the parse-a-maybe garbage contract this ADR exists to kill. During migration the marker scrape is the fallback instead (§Migration).
• Read points, all three: await_terminal() grows a return of the terminated container state (it polls api.get(name) already and throws the status away); the completion watch's pod_completion_key() path for run pods; and startup adoption, which reads the same pod status — closing the scope-turn adoption gap as a side effect of the mechanism rather than as a special case.

Tier 2 — artifacts POST to the controller's evidence drop-box

Packs, transcripts, and run sessions are files, and they move as files: an authenticated POST /api/pods/{pod}/artifacts/{kind} (kinds: scope-pack, scope-transcript, run-session, otel-log) with the raw gzipped bytes as the body. Typed, validated, size-checked at the door — no base64, no line discipline, no rotation budget.

• Storage is the state volume, not the database. The controller streams the body to state/evidence/<pod>/<kind> and records only pointers + digests in SQLite, preserving ADR-0016's rows-are-pointers rule. S3 can replace the directory later without touching the contract (the drop-box is an interface, not a place).
• Auth: a pod-bound projected ServiceAccount token, validated via TokenReview. The pod spec gains a serviceAccountToken projected volume with a dedicated audience (crucible-ingest) and a short TTL; render_turn() adds the volume and the CRUCIBLE_INGEST_URL env var (the default automountServiceAccountToken: false stays — this is one explicit, audience-locked file, not the kube-API passport). The ingest extractor sends the bearer to TokenReview and requires three things from the response: the audience matches, the service account is the turns' own, and the bound-pod claim (authentication.kubernetes.io/pod-name) equals the {pod} path segment. A turn is exactly one pod, so pod-binding is turn-scoping. Nothing is minted, stored, or expired by us: the kubelet rotates the token, the API server invalidates it when the pod dies, and a controller restart changes nothing about validation. The cost is a kube API call in the handler (cached briefly per token; the endpoint sees a handful of requests per turn, not a request stream), accepted because the controller is already a kube client everywhere else. Entirely separate from the human-facing oauth2-proxy/role stack.
• Delivery is at-least-once; the controller deduplicates. Uploads are content-addressed: a re-POST of a digest the drop-box already holds returns 200 and changes nothing. The engine retries a failed POST 3 times with backoff and then keeps going — it still writes Tier 1 with that artifact's manifest entry marked delivered:false, so the failure lands loudly on the work_pods row instead of silently truncating like the log path does today. Oversize at the door is a 413 with the limit in the body; the caps move from log-shaped guesses to real HTTP limits (pack 16 MiB, transcript 32 MiB, session 128 MiB, otel-log 32 MiB, all compressed — generous because they no longer fight the kubelet for log budget).
• Ordering: artifacts POST first, termination message last. The Tier 1 manifest can then state the truth about every upload, and the controller's fold has everything it needs the moment the pod goes terminal.

The fold stays pull-shaped (ADR-0016 amendment)

ADR-0016 ruled "a push endpoint is explicitly not v1 … never a correctness dependency." This ADR amends that ruling narrowly rather than reversing it. The ingest endpoint is a data-plane evidence drop — the network equivalent of the S3 bucket and the session log, just closer. Pods still never touch the database; no DB write happens in the request handler beyond recording the artifact pointer. The fold — outcomes into scopes/runs/candidates/ ledger rows — still happens in the controller's own reconcile transaction, triggered by the pod going terminal, reading evidence (now: pod status + drop-box instead of scraped logs). The database remains a rebuildable index; crucible db rebuild reads the evidence directory the same way it reads session logs. What actually changed from ADR-0016's assumption: logs turned out not to be a durable evidence store (the 2026-07-05 incidents, and Tekton's identical finding), so the evidence needed a real drop point. Push is the delivery mechanism for evidence, not the ingestion model.

One contract crate

A new crucible-contract crate (no async, no kube, serde only) that crucible, crucible-controller, and crucible-broker all depend on. It owns: the Tier 1 envelope and payload types, the artifact-kind enum + per-kind size caps, the ingest request/response types, the env-var names, the managed-by label, the AgentEvent NDJSON types (the session-log format both the engine and the controller's SSE relay read), and — until the migration completes — the six marker literals, deleting every "keep in sync" comment in the workspace. The termination message and the ingest bodies are serde round-trips of the same types on both sides; schema drift becomes a compile error instead of a 3 a.m. parse failure.

The harness crate — the agentic-ci replacement

A second crate, crucible-harness (library; async), extracted from the crucible bin target: agent spawn + env assembly, the stream-json decoder, the event pump + NDJSON session logging, and the OTLP collector below. It depends on crucible-contract and nothing controller-shaped. Consumers: the crucible binary (loop + turn commands), and external projects — the library role the vendored agentic-ci package used to play, restored as a Rust crate.

The boundary rule between the two: contract = wire types both sides serde-round-trip; harness = everything that runs next to an agent. The controller depends only on the contract; the crate DAG rule (crucible ↮ crucible-controller) survives intact.

The OTLP collector comes back, in-process

The harness embeds the collector agentic-ci ran as a subprocess, as an in-process task:

• Receiver: binds a configurable address, port 0 auto-assign; accepts /v1/metrics, /v1/logs, /v1/traces OTLP http/json POSTs; appends every export raw to otel.jsonl in the run dir (the otel-log Tier 2 artifact); answers {"partialSuccess":{}}.
• Live: a 60 s sliding-window token rate, fed to the event stream over a channel — Tokens.rate stops being permanently None. The rate/port files of the Python implementation were subprocess IPC; in-process they die.
• Rollup: at agent exit the collector builds the otel_summary event (authoritative cost, per-model input/output/cache tokens, api_requests, api_ms, active_seconds) — the event finally has a producer again — and the same rollup rides Tier 1 as the envelope's usage field.
• Env injection: the harness sets CLAUDE_CODE_ENABLE_TELEMETRY=1, OTEL_{METRICS,LOGS,TRACES}_EXPORTER=otlp, OTEL_EXPORTER_OTLP_PROTOCOL=http/json, the endpoint, OTEL_METRIC_EXPORT_INTERVAL=10000, plus full-fidelity spans exactly like agentic-ci did: CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 (tool nesting + subagent spans) and OTEL_LOG_USER_PROMPTS=1 / OTEL_LOG_TOOL_DETAILS=1 / OTEL_LOG_TOOL_CONTENT=1 — on the spawned agent (local exec) or into the sandbox env (openshell), following agentic-ci's per-harness matrix (docs/otel-configuration.md upstream). Consequence, stated plainly: the otel-log evidence contains prompt text and tool input/output, so it is confidential the same way session transcripts already are; anywhere it is exported to (§MLflow) must be as trusted as the state volume.
• Sandbox reach: local agents export to 127.0.0.1; an openshell sandbox reaches the collector exactly the way it reaches the broker — the collector binds 0.0.0.0 on the loop/turn pod and the sandbox gets a host.containers.internal:<port>:full egress rule (the proven 8849 pattern).
• Degradation: telemetry off or collector unreachable is graceful and identical to today — no otel_summary, no usage, pricing-table estimate as the cost fallback. The collector is evidence enrichment, never a run dependency.

MLflow export — a controller background task over the evidence

The MLflow exporter is a private control-plane component: it lives in the controller deployment and is a per-installation opt-in, no part of the public engine depends on it.

Capturing the raw OTLP jsonl as a first-class artifact is what makes an MLflow exporter a bolt-on instead of a rework. agentic-ci proved the shape (mlflow-push: a decoupled, allow-failure follow-up job that re-encodes the captured /v1/traces records OTLP-JSON → protobuf and POSTs them to MLflow), and MLflow grew the native half since. Crucible runs map onto the experiment-tracking ethos directly, so this is a workstream (R6), not a someday note.

• Locus: the controller, only. The exporter is a controller background task fed by evidence that already exists — the otel-log artifact in the drop-box and the folded ledger rows. It runs post-fold, never inline with a run or a fold, and failure marks the export row and retries later (the agentic-ci allow_failure: true ethos). Pods never see MLflow credentials; local crucible --manifest runs don't export.
• Traces: MLflow ≥ 3.6 ingests OTLP at POST /v1/traces — OTLP/HTTP + protobuf only (no gRPC), experiment selected by the x-mlflow-experiment-id header, bearer auth, gzip accepted from 3.7, SQL backend store required. Association is experiment-level only; there is no documented run-id header, so trace↔run correlation rides span/resource attributes we stamp ourselves (run id, turn, domain) plus matching tags on the MLflow run.
• Runs: plain tracking REST — experiments/get-by-name/create, runs/create, runs/log-batch (params: domain, model, effort, manifest knobs; metrics: gate score / cost / tokens, per-turn as steps), artifact upload via the proxied mlflow-artifacts route, runs/update to close. Mapping: domain (or rig) → experiment, crucible run → MLflow run, turn → metric step.
• Dependencies: no maintained Rust MLflow client exists; the exporter is a thin reqwest REST module plus opentelemetry-proto (serde feature) for the JSON→protobuf re-encode. Both already-idiomatic, well-maintained picks; nothing exotic.
• Confidentiality: with the content flags on (§env injection), exported traces carry prompt text and tool output — the MLflow instance must sit inside the same trust boundary as the state volume, and the exporter stays a per-deployment opt-in (tracking URI + token
  • experiment mapping in controller config, off when absent).

What stays on stdout

Live progress (CRUCIBLE_SCOPE_PROGRESS: beats, human-readable narration) stays on stdout by design: it is loss-tolerant, ordering-tolerant, and consumed live by the SSE relay (turn_live.rs) — logs are exactly the right transport for it. The boundary rule: stdout carries what a human tails; the contract carries what the controller acts on.

Migration

Three phases, each shippable alone, old/new compatible in both directions throughout:

1. Contract crate + Tier 1. Engine writes the termination message and keeps emitting markers; controller prefers the termination message and falls back to the marker scrape (which also covers pods launched by an old engine image). Startup adoption switches to pod status, fixing the scope-turn gap. Ships alone with zero coordination risk.
2. Tier 2. Controller mints tokens and injects CRUCIBLE_INGEST_*; engine POSTs artifacts when the env vars are present and falls back to marker emission when they are absent (old controller + new image keeps working). Run-session delivery moves onto the same endpoint family, retiring the === SESSION === delimiter and the 64 MiB scrape.
3. Demotion. After both sides are rolled and a soak, the engine stops emitting payload markers (pack/transcript base64 lines and the report/verdict lines disappear from logs), and the controller's scrapers + duplicated literals are deleted. Progress beats remain.

The compatibility matrix is the fallback chain itself: new controller + old image → markers; old controller + new image → markers (no ingest env, marker emission still on until phase 3); new + new → the contract. Phase 3 is the only step that burns a bridge, and it is gated on the fleet being past phases 1–2.

Failure modes, named

• Kubelet truncation: prevented by the 3584-byte self-cap; a manifest digest that doesn't parse is a structural bug, not an operational one.
• Pod dies before writing the message (OOM, eviction, wrapper crash): phase=Failed with an empty message — the same information content as today's no-marker log, handled by the same failed-turn path, now distinguishable from "succeeded but unparseable."
• Controller down during the turn: auth state lives in the API server, not the controller, so nothing to recover; the pod's POSTs retry through the outage window, and whatever didn't land is recovered by adoption reading the termination message + manifest and flagging undelivered artifacts. A turn is never lost to a controller restart again.
• Duplicate delivery (pod retry, network retry): content-addressed dedup; idempotent 200.
• Stolen token: audience-locked to the ingest endpoint (useless against the kube API or any other service), pod-bound (only impersonates the one turn pod, and the API server kills it when that pod dies), and the endpoint is write-only — blast radius is "attacker can upload a wrong artifact whose digest won't match the kubelet-authenticated manifest."
• Kube API unavailable at validation time: TokenReview fails closed (503 to the pod, which retries on its backoff budget); the termination-message manifest keeps the miss loud.

What this is NOT

• Not a message bus, not steering. One-shot result delivery at turn end. The ADR-0001 wall stands: nothing flows controller→pod mid-turn through this contract, and progress stays advisory.
• Not a CRD, not annotations, not a sidecar. The survey is unambiguous that all three are migrations waiting to happen (Argo's 256 KiB→1 MiB ladder, Tekton's TEP-0127 RBAC/overhead costs). We have a controller with a disk; we use it.
• Not the inner loop's state. A local crucible --manifest run still never needs a controller; result mode is opt-in exactly like --marker is today, and the session log + git remain a run's complete record (ADR-0016 unbroken).
• Not human-facing API surface. The ingest route lives outside the SPA/OpenAPI-typed surface's auth model, is write-only, and never appears in the UI's generated client.
• Not the controller's observability stack. The OTLP collector receives the agent's telemetry as run evidence; the controller's own prometheus /metrics + ServiceMonitor stack is a different concern and untouched by this ADR.

Consequences

• Positive: the verdict path stops depending on log rotation, newline discipline, and base64-in-logs; scope turns survive controller downtime (adoption gap closed by construction); artifact caps become honest HTTP limits instead of log-budget gymnastics; six keep-in-sync literal pairs collapse into one typed crate; the failure story upgrades from "silently truncated" to "row flagged with exactly which artifact didn't land"; authoritative cost and per-model usage return to the ledger (the pricing-table estimate demotes to a fallback), and the harness becomes a consumable crate again instead of a bin-shaped fossil.
• Negative / cost: the controller gains a pod-facing HTTP surface (new attack surface, mitigated by write-only + audience/pod-bound tokens) and its ingest handler couples to the kube API (TokenReview, cached, failing closed); the state volume now holds evidence payloads, so disk sizing and retention become real (pointer-rows stay small; the volume doesn't); a three-phase migration must actually be driven to phase 3 or we run both contracts forever; crucible-contract and crucible-harness are two more crates whose semver discipline matters from day one, and the harness extraction is a large mechanical refactor of the engine's core.

Workstreams

Slicing: R0+R1 ship together (pure win, no coordination). R2 behind them; R3 rides R2's plumbing; R4 only after a fleet soak proves both directions of the compat matrix were honored. R5 is independent of R2–R4 and can run in parallel once R0 lands: the extraction + collector are engine-local, usage piggybacks on R1's envelope, and the otel-log upload piggybacks on R2's endpoint (until R2, the otel jsonl just stays a run-dir file like every local artifact). R6 is last by construction — it consumes what R2 delivers and R5 produces.

Appendix: prior-art survey (full notes, 2026-07-06)

The condensed lessons live in §"Prior art"; these are the full survey notes with sources, kept so the numbers and rejections above stay auditable.

Tekton Pipelines

Task results ride the container termination message: 4096 bytes per container, 12 KiB per pod divided equally among all containers (init containers included — 12 containers means 1024 bytes each). Hitting the ceiling produced two design cycles:

• TEP-0086 "Changing the way result parameters are stored" (status: Proposed) surveyed the alternatives: a dedicated storage API service (OIDC tokens from pods, backend-agnostic), a result sidecar uploading to external storage, ConfigMaps per TaskRun (rejected: 3+ API requests per TaskRun, still 1.5 MiB-limited), a custom CRD (same ceiling), PVCs (complexity/ perf concerns), and logs — rejected outright for having "no availability guarantee". https://github.com/tektoncd/community/blob/main/teps/0086-changing-the-way-result-parameters-are-stored.md
• TEP-0127 "Larger Results via Sidecar Logs" (status: Implemented) shipped the escape hatch: a Tekton-injected sidecar watches /tekton/run, waits for all steps, prints results to its stdout in a parsable pattern, and the controller reads the sidecar's logs. Costs: ~3 s extra pod startup, the controller needs pod-log RBAC, per-result size still gated by the max-result-size flag (default 4096 bytes), total still capped by the 1.5 MiB CRD limit (TaskRun fails beyond it), and tasks assuming large results break on unconfigured installs. https://github.com/tektoncd/community/blob/main/teps/0127-larger-results-via-sidecar-logs.md

Motivating issues: https://github.com/tektoncd/pipeline/issues/4012, https://github.com/tektoncd/pipeline/issues/4060; docs: https://tekton.dev/docs/pipelines/tasks/.

Argo Workflows

Output parameters are capped at 256 KiB (the pod-annotation limit — parameters were originally reported by the wait sidecar patching pod annotations) (https://argo-workflows.readthedocs.io/en/latest/walk-through/output-parameters/); reporting later moved to the internal WorkflowTaskResult CRD (higher capacity, needs garbage collection) (https://github.com/argoproj/argo-workflows/blob/main/manifests/base/crds/full/argoproj.io_workflowtaskresults.yaml). Aggregate workflow status fights the 1 MiB etcd object limit with, in order: node-status compression (~20:1), opt-in SQL offload (nodeStatusOffLoad: true), and (v3.7+) container-args offload to ConfigMaps past 128 KiB (https://argo-workflows.readthedocs.io/en/latest/offloading-large-workflows/). The documented recommendation for anything larger is artifacts to an artifact repository (S3/GCS), never parameters — emit a small count parameter and index into the artifact (https://github.com/argoproj/argo-workflows/blob/main/examples/handle-large-output-results.yaml). The wait sidecar collects outputs and reports to the controller; it originally mounted docker.sock (bypassing RBAC) and was migrated to the Kubernetes API via service account.

Kubernetes primitives

terminationMessagePath (default /dev/termination-log) is read by the kubelet into containerStatuses[].state.terminated.message, atomically, as part of pod status, immediately on container termination. Limits: 4096 bytes per container, 12 KiB per pod total (divided equally), and truncation is silent. terminationMessagePolicy: FallbackToLogsOnError substitutes the log tail only when the file is empty and the exit was an error, capped at 2048 bytes or 80 lines, whichever is smaller. The path is immutable after pod creation, and the docs scope the mechanism to "brief final status, such as an assertion failure message". https://kubernetes.io/docs/tasks/debug/debug-application/determine-reason-pod-failure/

GitHub Actions runner

Per-job credential pattern: a job-scoped OAuth token is generated when the job is queued, lives for the job duration (default 6 h timeout) plus 10 minutes, is held only in memory, and is delivered inside a job message encrypted to the runner's RSA public key; each action subprocess receives it as an env var registered as a secret (scrubbed from logs). Artifacts upload through the authenticated API with that token. https://github.com/actions/runner/blob/main/docs/design/auth.md, https://docs.github.com/actions/security-guides/automatic-token-authentication

Buildkite agent

Same shape: an internal per-job access token generated at job start, exposed as BUILDKITE_AGENT_ACCESS_TOKEN, scoped to the single job, dead when the job finishes — distinct from the long-lived cluster-wide agent registration token. Used for artifact upload, annotations, metadata. https://buildkite.com/docs/agent/v3/tokens, https://buildkite.com/docs/apis/rest-api/artifacts

Temporal

Activity completion is an RPC carrying an opaque task token, single-use and scoped to one execution attempt — invalidated when the attempt retries, which is why Temporal recommends external services key off Workflow Run ID + Activity ID, never the token (https://docs.temporal.io/activities). Semantics: workflows are exactly-once, activities are at-least-once — the server does not deduplicate completions, so idempotency is the completer's job, enforced at the receiving service (https://temporal.io/blog/idempotency-and-durable-execution, https://community.temporal.io/t/is-a-system-generated-activity-id-suitable-to-use-as-an-idempotency-key/13181/2).

Prow, Airflow, Kueue (brief)

• Prow: pod utilities (initupload, sidecar) push artifacts to GCS under workload identity; finished.json marker objects signal completion and the crier watcher reports onward — sidecar-upload plus polled completion markers, the multi-controller shape we'd converge on past a single controller. https://docs.prow.k8s.io/docs/spyglass/
• Airflow: XCom results default into the metadata database, which bloats under artifact traffic; the documented fix is a custom XCom backend on object storage — the same pointers-not-payloads split as ADR-0016, by convention rather than contract. https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/xcoms.html, https://github.com/apache/airflow/issues/15761
• Kueue: admission-controller model over Job status; no custom result-return path — not relevant to this contract.

Synthesis carried into the decision

ADR 0018: Declarative image builds — build backends + the building state

Status: Proposed Date: 2026-07-08 Related: ADR-0005 (forge, the in-loop candidate builder this extends), ADR-0008 (the composed-world images these builds produce), ADR-0012 (digest pinning; rendered pods consume image@digest), ADR-0013 (the pluggable-backend precedent this copies), ADR-0015/ADR-0016 (the controller + ledger the building state lands in).

Context

Four tiers of image production exist today, and only the first two are automated:

1. CI-built: the engine/loop image (Containerfile.crucible, docker.yml) and the ADR-0008 world-base image, whose CI workflow ends with printed instructions telling a human to repin WORLD_BASE in Containerfile.crucible.
2. Hand-built: every sandbox and loop-openshell image. The Containerfiles embed the procedure ("on the arm64 laptop run the build remotely on the cluster"): mutable :dev/:m1 tags, digests hand-plumbed into manifests, deploy profiles, and gitops values.
3. In-loop candidate builds: forge — buildah build_and_push for real Dockerfile builds, and the native-OCI derive_layer (oci-client: deterministic overlay layer, server-side blob mount, manifest rewrite) for base+edited-files candidates. This tier works.
4. Version-knob choreography: VLLM_VERSION, REPO_SHA, WORLD_BASE are ARGs kept in sync by convention and comments.

Onboarding a new composite therefore means handwriting 2–3 Containerfiles, hand-running remote builds, and hand-plumbing digests — and the outer loop has no answer to "an approved pack implies a rebuilt world image" (the WorkKind::Build gap). Two prior calls shape the fix:

• On-cluster builds borrow buildit's detached rootless-buildah Job pattern: owner-ref'd registry secret, digest-file output, ttl-reaped, idle-node targeting. The controller must not stream build contexts; detached Jobs only.
• GitHub Actions dispatch is the second backend. Other teams already build images in Actions; for the ADR-0013 self-serve story a team without cluster build capacity brings a repo with a build workflow and crucible dispatches it. (The earlier "GH runners are very weak" ruling was about candidate test runs, which stay on-cluster; plain image builds are what those runners are fine at.)

Decision

1. One BuildBackend contract, two implementations

Cluster (detached rootless-buildah Job, folded into forge next to the existing buildah/oci paths — we do not ship a separate buildit binary in the controller image) and GithubActions (workflow_dispatch + poll). A backend's only obligations: run the build, push to the tag it was given, report success/failure. Crucible resolves the digest itself via the existing forge::oci::pin_digest against the pushed tag — the registry is the single source of truth for both backends, retries are idempotent (a re-dispatch of an already-built context is a no-op push), and no output ever has to travel back through Job logs or Actions artifacts.

2. Named builds, declared in the domain manifest

[build.backend-sandbox]
backend = "cluster"
image   = "ghcr.io/example-org/backend-sandbox"
timeout = "30m"
[build.backend-sandbox.cluster]
containerfile = "domains/backend/Containerfile.sandbox"
context       = "."                 # or an OCI-artifact ref (buildit's ctx_ref trick)
platform      = "linux/amd64"
[build.backend-sandbox.watch]
paths = ["domains/backend/Containerfile.sandbox", "domains/backend/tools/**"]

[build.sandbox]
backend = "github-actions"
image   = "ghcr.io/example-org/fullstack-sandbox"
needs   = ["backend-sandbox"]       # ordering + digest availability
[build.sandbox.github]
repo     = "neuralmagic/crucible"   # profile default; must pass the allowed-orgs whitelist
workflow = "crucible-build.yml"     # must exist on the dispatched ref; we never generate it
ref      = "main"
[build.sandbox.github.inputs]       # forwarded after template expansion, nothing else
containerfile = "domains/fullstack/Containerfile.sandbox"
base-image    = "{{ builds.backend-sandbox.digest_ref }}"
image         = "{{ image }}:{{ tag }}"

Load-bearing pieces:

• needs is a plain dependency list (reconcile won't dispatch a build until its deps have digests), and {{ builds.<name>.digest_ref }} hands the pinned output of a dependency to a downstream build. Together they retire the manual world-base repin dance mechanically.
• watch.paths defines "build-needed": hash the watched paths at the pinned ref → the context digest; rebuild when it differs from the last recorded build for this image. The same predicate drives both backends. No [build] block ⇒ today's behavior exactly (statically configured image, never rebuilt) — the schema is purely additive. An empty watch.paths inside a build block is the opposite: nothing to compare ⇒ always rebuild (a force signal that never equals a recorded digest), not a constant "nothing changed".
• The template vocabulary is closed: sha, tag, image, correlation_id, builds.<name>.digest_ref. Nothing else expands. Anything richer turns the config into a programming language and — because the manifest is agent-influenced content post-pack-approval — a string-injection path into someone's CI. For the same reason there is no env/secret pass-through: credentials live on the backend side (the workflow repo's own secrets, the cluster Job's mounted authfile), never in our config.

3. The workflow-file contract (versioned), with declared escape hatches

A conforming workflow (crucible-build.yml, contract v1) honors exactly two behaviors:

1. echo the correlation_id input into the run name — workflow_dispatch returns 204 with no run id, so the run name is how we find our dispatch;
2. push the built image to the image input's tag.

Arbitrary pre-existing workflows that can't be modified declare how we adapt instead:

[build.sandbox.github.outputs]
digest      = "registry-tag"    # default; or { from = "artifact", name = "digest", path = "digest.txt" }
correlation = "run-name"        # default; or "dispatch-window" (serialize dispatches per workflow)

Introspection is validation, not magic: at manifest load / admin time, fetch the workflow YAML and parse on.workflow_dispatch.inputs — fail fast when the declared mapping misses a required input or names one that doesn't exist, warn on type mismatches. Introspection checks the wiring a human declared; it never invents mappings. (Later it can power the admin UI: discovered inputs rendered next to the mapping.)

4. The controller blocks measurement on an explicit building state

Remote builds take single-digit minutes, so this is a reconcile state, not an inline preflight wait: a build-needed row enters building with the dispatched Job/run and expected tag recorded; reconcile polls the backend the way it already watches work pods; success pins the digest onto the row and only then does the run become dispatchable (rendered pods consume image@digest, ADR-0012). Failure parks the row with a build-log pointer as evidence; a timeout cap keeps a wedged run from holding a rig slot; controller restart re-adopts in-flight builds by listing Jobs/runs, never from memory. Budgeting: per-kind concurrency cap, no $ ledger — builds are compute, not LLM spend.

5. crucible build <name> — the same path, human-invoked

The CLI subcommand dispatches a named build (either backend), waits, and prints the digest-pinned ref. It replaces today's hand-run remote cluster build and is the exact code path the controller dispatches later — one implementation, two callers.

6. Self-test on the self-hosting domain

The self-hosting domain (the engine-develops-itself pack, the cheapest full integration test we own) gets a [build.loop] block dispatching this repo's own Actions build of Containerfile.crucible. That proves the GithubActions backend end-to-end — dispatch, correlation, poll, tag push, pin_digest — against CI we fully control, before any external team's workflow is in the loop. The discrimination check mirrors ADR-0014: known-good = dispatch at HEAD and assert the resolved digest matches the pushed tag; known-bad = a workflow ref whose build must fail, asserting the row parks with the log pointer.

Alternatives considered

• Generate workflow files on the fly — rejected: repos own their build environment; we version a contract instead, like the turn-result envelope (ADR-0017).
• Digest reporting via artifacts/job outputs as the default — rejected: the registry already knows; kept only as a declared escape hatch for unmodifiable workflows.
• Inline preflight blocking instead of a building state — rejected: minutes-long remote builds would wedge reconcile; explicit state is also what restart re-adoption needs.
• Ship buildit as a CLI inside the controller image — rejected: forge already has kube-rs, buildah args, and authfile handling; folding the detached-Job mode in avoids a second binary with its own release pipeline.
• Auto-map introspected inputs by name — rejected: silent mis-wiring into someone's CI; introspection validates declared mappings only.

Migration

• M0 — forge grows the detached-Job cluster backend + crucible build (CLI-only).
• M1 — controller building state + build dispatch/adoption in reconcile; approved packs with a [build] block flow through it.
• M2 — GithubActions backend + workflow introspection validation; crucible-build.yml contract v1 lands in this repo.
• M3 — dogfood: the self-hosting domain's [build.loop] self-test; convert the hand-built sandbox and loop-openshell images to declared builds; retire the world-base manual repin.

Consequences

Onboarding a domain's images becomes a manifest edit plus (optionally) a workflow file; digests flow mechanically from build to consumer; the measurement rig can never run on a stale or half-built image; and the hand-run build skill becomes a fallback rather than the procedure. The cost: a new reconcile state, a versioned workflow contract to maintain, and GH-side coupling (App installation needs actions:write) for the remote backend.

ADR 0019: The loop pod stops being a container host — OpenShell's Kubernetes driver

Status: Proposed Date: 2026-07-08 Related: ADR-0002 (the broker the sandbox reaches — note the concrete reach hostname is not in that ADR; it lives in manifest.rs's BrokerCfg docs and the contract §6.1, which is what this ADR amends), ADR-0005 (buildah in the loop image), ADR-0012 (the rendered loop pod, which is what changes), ADR-0018 (moves candidate builds off the loop pod; this ADR depends on it to finish the job).

Context

The openshell backend runs the agent's sandbox as a container nested inside the loop pod. Podman is the compute driver. That single choice is upstream of most of the accidental complexity in the deploy path:

• The loop pod runs privileged: true (crucible/src/deploy/render.rs:376, with no comment explaining why), and so does every one-shot turn pod (render.rs:1211). Note that the test named turn_pod_renders_the_reduced_privileged_shape means reduced machinery, not reduced privilege. De-privileging is therefore two call sites, not one.
• The loop image installs podman, buildah, and fuse-overlayfs (Containerfile.crucible:100).
• crucible/src/openshell/gateway.rs must scrub KUBERNETES_SERVICE_HOST/PORT before launching the gateway, because otherwise it detects the in-cluster signal and demands a Kubernetes driver config that conflicts with the podman driver. This is upstream-patch 1.1, and the module docs call it "load-bearing." We suppress the k8s driver on purpose.
• The nested podman has no imagePullSecret, so a private sandbox image needs its own credential path (pull_authfile → REGISTRY_AUTH_FILE). The kubelet solves this problem for free, for every other pod in the cluster.

Crucible touches podman in exactly one place: gateway.rs:120 starts podman system service to expose a rootless API socket for the gateway. Nothing else in the engine shells out to it.

OpenShell ships crates/openshell-driver-kubernetes, which runs the sandbox as a sibling pod. It is linked into openshell-server in-process (openshell-server/Cargo.toml:21), and it is present in the exact fork commit our gateway binary is built from (wseaton/OpenShell@f25ab2e4) — so this is a configuration and plumbing change, not a fork rebase. Per its README the openshell-sandbox supervisor still owns "agent isolation, credential injection, policy polling," so the deny-by-default egress allowlist that our contamination guard depends on (ADR-0001, openshell/policy.rs) survives the switch unchanged.

Decision

Make the compute driver a configuration knob, and default in-cluster deployments to kubernetes. The sandbox becomes a sibling pod scheduled by Kubernetes. The loop pod stops being a container host.

Podman stays a supported driver. It is what --agent-backend=openshell uses on a laptop, and other agent infrastructure runs on EC2 where there is no Kubernetes API to talk to. Deleting it would trade one lock-in for another. gateway_toml() currently hardcodes compute_drivers = ["podman"]; that becomes a selected value with kubernetes chosen by crucible deploy render.

What the switch buys

The five things that actually have to change

1. The broker reach contract (the load-bearing break). Today the sandbox reaches the broker at host.containers.internal:8849 over the pod-internal podman bridge. The Kubernetes driver injects hostAliases for host.docker.internal and host.openshell.internal only (driver-kubernetes/src/driver.rs:1538) — host.containers.internal does not exist. Every domain manifest hardcodes the old name in [agent.broker].url and in [agent.openshell].endpoints. The alias is only injected when host_gateway_ip is set, and it is a static config string, so crucible must fill it with the loop pod's own IP at render time via the downward API (status.podIP).

The broker's default URL and the default egress endpoint become driver-dependent. The engine resolves the host and templates it into both .mcp.json and the policy allowlist, so a domain manifest never names the transport again. Recon (2026-07-08) settled the shape:

• ComputeDriver::broker_host() returns host.containers.internal (podman) or host.openshell.internal (kubernetes — the alias the driver actually injects). Hostname only; the port stays where it already lives, in bind and the URL builder.
• BrokerCfg.url becomes Option<String> with #[serde(default)]: None means engine-resolved, Some is an explicit override the engine honors. Not the "compare the parsed value against the old default to guess whether the author set it" heuristic — that silently steals a URL from anyone who writes the default out longhand. deny_unknown_fields is no obstacle: it rejects unknown keys, not omitted ones. DeployProfile's Secrets already pairs deny_unknown_fields with omitted Option fields, and the single-domain render test parses a profile with no [secrets] block at all.
• The engine auto-appends the broker endpoint to the resolved allowlist when, and only when, [agent.broker].enabled. Not unconditionally. This keeps inherit_defaults = false honest: the allowlist stays fully determined by the manifest, and a broker-less domain that opts out still resolves to nothing (the opting_out_with_an_empty_table_denies_everything test must keep passing). It also makes the old footgun impossible — the URL and the allowlist entry can no longer disagree, because one derives from the other.
• Verified: no shipped manifest sets [agent.broker].url, and ADR-0002 never names the hostname, so the blast radius is the four domain manifests that hand-list the endpoint, plus doc comments.

P2 must land this doc change too: the contract §6.1 currently states that the broker endpoint "is not a built-in, so a domain that enables [agent.broker] already lists it." True today, false the moment the engine appends it. Update §6.1 in the same PR, not before.

2. The NetworkPolicy stops being a formality. render.rs:823 documents that the sandbox reaches the broker "over the pod-INTERNAL podman bridge — traffic a NetworkPolicy never sees." Once the sandbox is a sibling pod, gateway traffic (:17670) and broker traffic (:8849) become real cluster networking, and the loop pod's deny-all-ingress policy will drop it. The netpol must allow ingress from sandbox pods on exactly those two ports, and nothing else. This is a security-relevant change: the broker, which holds every credential, goes from unreachable-by-construction to reachable-by-policy. It must be reviewed as such, not waved through as plumbing.

3. A new cluster-wide prerequisite. The driver creates agents.x-k8s.io/v1alpha1 Sandbox objects (driver.rs:82-84), not pods. That CRD and its controller come from kubernetes-sigs/agent-sandbox. This is the one step that needs cluster-admin and is not reversible by editing a manifest.

Pin the controller release at install time (do not track the latest URL OpenShell's chart README suggests), so the cluster cannot drift under you on someone else's release. It registers no admission webhook, so a controller outage cannot wedge unrelated workloads on a shared cluster. One finding worth carrying in the ADR:

• A deprecation fuse. The CRD serves v1beta1 (storage) and v1alpha1 (deprecated, served=true). Our pinned driver hardcodes v1alpha1 (driver.rs:83), so it works today and emits a deprecation warning. The first agent-sandbox release that stops serving v1alpha1 breaks the driver. Pin the controller version, and treat "OpenShell moves to v1beta1" as a prerequisite for any upgrade.

4. RBAC. From OpenShell's chart, the gateway's ServiceAccount needs a namespaced Role over agents.x-k8s.io sandboxes + sandboxes/status (create/delete/get/list/patch/update/watch), events (get/list/watch), and pods (get); plus a ClusterRole granting authentication.k8s.io tokenreviews (create — for the IssueSandboxToken bootstrap) and nodes (get/list/watch). The cluster-scoped half is new; today's loop SA is namespaced. crucible deploy render emits the RBAC, so this landed in the rendered RBAC output.

5. Supervisor delivery. supervisor_sideload_method defaults to image-volume, which needs the Kubernetes ImageVolume feature gate (beta ≥1.33, GA ≥1.36). Default to init-container, which the driver documents as working on all versions, and revisit once the target cluster reaches 1.36.

What this does not buy, yet

Removing podman does not by itself remove privileged: true. Rootless buildah in the loop pod is almost certainly the other reason for it, and buildah is there for ADR-0005 in-loop candidate builds. Dropping the privilege escalation needs both this ADR and ADR-0018's Cluster build backend, which moves candidate builds out to detached rootless-buildah Jobs. That is the actual prize, and it is why this ADR sequences after ADR-0018. Claiming de-privileging before then would be a lie.

Migration

Each phase is independently landable and independently revertible. No phase is a big-bang cutover.

• P0 — Prerequisite (cluster-admin, out-of-band). Install the agent-sandbox controller, version-pinned. Nothing in this repo changes.
• P1 — Driver becomes a knob. gateway_toml() takes a ComputeDriver enum instead of hardcoding ["podman"]; [openshell.drivers.kubernetes] is emitted when selected. Default stays podman, so this is a pure no-op refactor with unit tests on the rendered TOML.
• P2 — Transport-agnostic broker reach. The engine resolves the broker host from the active driver and templates it into .mcp.json and the policy allowlist. Domain manifests stop naming host.containers.internal. Ships under the podman driver first, where it must be a no-op — that is the test.
• P3 — Render the Kubernetes shape. crucible deploy render gains: host_gateway_ip from the downward API, the sandbox imagePullSecrets, the two RBAC objects, and the NetworkPolicy ingress rules. Behind a profile flag ([cluster].sandbox_driver = "kubernetes"), default off.
• P4 — First real run. Flip one domain (the one with the cheapest gate) on the target cluster. Verify: the sandbox lands as a sibling pod, the agent reaches the broker, openshell policy update still denies by default, and the gate score matches a podman-driver baseline. The measurement must not move.
• P5 — De-privilege. After ADR-0018's Cluster backend lands, drop podman, buildah, and fuse-overlayfs from the loop image, and privileged: true from the pod. Retire pull_authfile and REGISTRY_AUTH_FILE.

Alternatives considered

• Keep the podman driver. Zero work, and it keeps laptop/EC2 parity in one code path. Rejected as the default because it forces a privileged pod, a bespoke credential path, and an upstream patch that exists only to hide Kubernetes from a program running in Kubernetes. Kept as a supported driver for exactly the laptop/EC2 reasons.
• The docker or vm drivers. docker has the same nesting problem with a worse security story. vm is heavier than the isolation we need, given the supervisor already enforces egress.
• Run the sandbox as a bare Pod we render ourselves, skipping the CRD. Rejected: it forks the supervisor bootstrap (IssueSandboxToken, TLS material, relay) that the driver already implements, and we would own it forever.

Consequences

• Positive: the loop pod stops being a container host; the sandbox gets ordinary Kubernetes scheduling, limits, GPU admission, and image pull; one upstream patch and one credential path retire; the path to privileged: false opens.
• Negative / cost: a cluster-scoped CRD + controller + ClusterRole become prerequisites (a real onboarding tax for a new cluster, and it needs cluster-admin); the broker becomes network-reachable and must be protected by policy rather than by topology; the workspace moves onto a PVC that upstream itself calls "a stopgap persistence model," so the workspace-sync path needs re-verification.
• Risk: the driver's README labels its driver_config schema an "RFC 0005 POC." We are adopting a component that upstream has not frozen. P1–P3 are cheap and revertible precisely because P4 might reveal that it is not ready.

Gateway user auth under this driver (added 2026-07-13, PR #249). The gateway hard-rejects mTLS user authentication with the Kubernetes compute driver (openshell-server/src/cli.rs — podman gets it implicitly, k8s is told to bring OIDC or a fronting proxy), while auto-enabling its authenticator chain from certgen's JWT bundle — so crucible's own bearer-less RPCs die UNAUTHENTICATED at CreateProvider. The rendered k8s-driver gateway config therefore sets [openshell.gateway.auth] allow_unauthenticated_users = true (crucible/src/openshell/gateway.rs); podman rendering is untouched. This is an escape hatch, not an auth downgrade: the socket stays gated by transport mTLS with a per-pod CA (require_client_auth), and the client cert exists only in crucible's turn pod and the supervisor sidecar container — never the agent container — so possession of the cert is the authorization, exactly what mtls_auth grants under podman. The hatch holds until upstream accepts mTLS user auth under the kubernetes driver; the acceptance criterion for that change is that crucible deletes this line. See the fork ledger.

Open questions

Resolved 2026-07-08 by source recon (kept here because the answers are load-bearing):

• Does the tar-upload workspace path survive the PVC-backed /sandbox? Yes, unchanged. openshell/sandbox.rs only builds openshell sandbox upload/download argv. The CLI transfers over the gateway's gRPC CreateSshSession relay (openshell-cli/src/ssh.rs:94, :751, :976), which reaches the sandbox through the supervisor. There is no bind-mount or podman reference in that path, and the podman driver has no upload path of its own — the relay is shared by both drivers.
• Does anything besides gateway.rs:120 assume a local podman? Only the socket-path helper (gateway.rs:72-77), which is reachable solely from the podman boot path. No fuse-overlayfs or podman invocation exists anywhere else in the engine; no domain hook script references either. crucible view is unaffected (local files or kubectl exec).
• Does the "container runtime reaps detached daemons" teardown story break? No, and it was the wrong thing to worry about. That claim (gateway.rs:15-17) is about the gateway and podman daemons dying with the loop pod, not about the sandbox. Sandbox teardown is already explicit and driver-agnostic: run.rs:187-190 calls openshell sandbox delete through gRPC, which under the Kubernetes driver deletes the Sandbox object.

Still open:

1. Does the sandbox pod need enable_user_namespaces / app_armor_profile: Unconfined on the target cluster's nodes? The chart defaults to Unconfined because RuntimeDefault can block the supervisor's netns setup — a claim worth confirming rather than inheriting.

ADR 0020: Candidate build modes — how a proposal becomes a measured artifact

Status: Proposed Date: 2026-07-09 Related: ADR-0001 (the frozen judge, which the build recipe is part of), ADR-0005 (engine-side candidate builds, brokered), ADR-0008 (the pinned base a candidate derives onto), ADR-0009 (a composite mixes modes per component), ADR-0018 (the BuildBackend contract — for infrastructure images; this ADR is the tier it deliberately left alone), ADR-0019 (whose P5 de-privileging depends on this).

Context

Between "the agent edited a file" and "the judge read a score" sits a step nobody has written down: the edit has to become the thing that is actually measured. How that happens differs per component, is currently expressed as bash inside per-domain apply hooks, and is the single biggest driver of per-iteration wall-clock. It is load-bearing and undocumented.

Three modes exist today.

1. No artifact. The gate compiles and runs in place, engine-side, on the loop pod. The bug-fix domains do this: crucible.test.toml → measure-test, crucible.1489.toml → measure-1489, both go test against the agent's workspace. examples/counter doesn't even compile. There is no image, no registry, no deploy. Cost: seconds.

2. No rebuild. The gate measures a live rig whose configuration the agent changed. The EPP perf domain (crucible.toml → bench) is config tuning: scorer weights are applied onto the running EPP. ADR-0005 records this as the reason the perf domain "dodged" the build problem entirely. Cost: a rollout.

3. Build and roll. The candidate must become a running image before measure_cmd executes. This splits by language, not by cleverness, and a composite runs both halves at once (the composite's apply hook, fullstack-lora-apply.sh):

• derive-layer (interpreted). vLLM's changed .py files are appended onto a pinned base as a real OCI layer via forge-derive-layer (forge/src/oci.rs) — no buildah, no compile, no runtime configmap overlay. It takes ~8 seconds only because the base is a same-repo mirror tag, so the registry mounts the base blobs server-side. oci.rs:10 is explicit: "server-side MOUNT when registries match (instant), else stream pull→push". Point the base at the upstream Docker Hub tag and 8 seconds becomes 20+ minutes, silently.
• image (compiled). EPP is Go, so it cannot skip the compile. forge build-candidate runs a real buildah bud against Dockerfile.epp. The compile lives inside the image build, inside apply_cmd, inside the gate's critical path, on every candidate.

Two existing properties of mode 3 are assets and must survive any redesign:

• A compile error is free. build-candidate exits nonzero on a compile error (BuildOutcome::CompileError, forge/src/lib.rs:79); the apply hook maps that to exit 3 (fullstack-lora-apply.sh); the broker returns CompileError { log } and the agent retries without spending a candidate. The compile is not only a tax, it is the fastest feedback signal in the loop. A build mode that cannot distinguish "your code does not compile" from "your candidate scored badly" is a regression, not a refactor.
• An unchanged component does not rebuild. build-candidate --skip-if-exists keys the tag on a diff hash and does a manifest HEAD against the registry; a hit reuses the digest-pinned ref with no build (forge/src/bin/build-candidate.rs:104). ensure-deployed is the apply_cmd backstop that guarantees the measured candidate is the built one.

What ADR-0018 does and does not cover. Its Context enumerates four tiers of image production and says of tier 3, in-loop candidate builds, "This tier works." [build.<name>] targets the infrastructure images — sandboxes, the loop image, the world base. The per-iteration candidate build was left in forge, as shell, per domain. That is the gap.

Decision

Name the modes, declare them per component, and let the engine enforce their preconditions.

1. mode is a typed, per-component manifest field

A composite genuinely mixes modes, so this belongs on the component, not the domain.

[component.vllm.build]
mode    = "derive-layer"
base    = "registry.example.com/team/vllm-candidate:v0.23.0"   # MUST share a registry with `target`
target  = "registry.example.com/team/vllm-candidate"
paths   = ["vllm/"]                             # the subtree whose edits become the layer

[component.epp.build]
mode          = "image"
backend       = "cluster"                       # ADR-0018's BuildBackend
containerfile = "Dockerfile.epp"                # MUST be a frozen inject (see §4)
target        = "registry.example.com/team/epp-candidate"

mode = "none" (the default) covers modes 1 and 2 above: no artifact, no rebuild. A domain that omits [component.<n>.build] behaves exactly as it does today.

2. crucible check enforces the preconditions, before a turn is spent

The failure modes here are silent and expensive, which is precisely what crucible check exists to catch (ADR-0014):

• derive-layer where base and target are on different registries → hard error, naming the 8s vs 20min consequence. This is the single sharpest edge in the current design and today nothing warns.
• derive-layer whose paths include compiled sources → hard error. Appending a .go file to an image changes nothing that runs; the loop would measure the base image forever and report the change as a no-op. (This is the same class of bug as ADR-0007's isolation pre-flight.)
• image whose containerfile is not a frozen = true inject → hard error. See §4.

3. Caching: cache the compiler, not the layers

For image mode the per-iteration cost is dominated by the compile, and layer caching does not help: the source layer changes by definition every turn, so every cache below the COPY is invalidated. What makes a compiled candidate fast is a warm compiler cache (Go build cache, sccache) on a persistent volume attached to the builder. That is a different mechanism from anything ADR-0018's backends provide, and it is the thing worth building.

--skip-if-exists remains the zeroth-order cache and is strictly better than any of this: an unchanged component performs no build at all.

4. The build recipe is part of the judge

Dockerfile.epp currently lives inside the agent's workspace (the EPP checkout under the composite's workspace), the build context is the agent's checkout (build-candidate --context epp), and the lora-routing composite manifest (crucible.lora-routing.toml) declares zero [[workspace.inject]] blocks. The agent can edit the recipe that builds the artifact it is scored on: vendor a prebuilt binary, neuter the compile, COPY around whatever the gate assumes. This is the same shape as the #1489 reward hack, and making build modes declarative widens the surface unless we close it.

Per ADR-0001 the agent may change its solution, never its evaluation. A Containerfile on the measured path is part of the evaluation. It must be a frozen = true inject, re-copied before every scored measure, exactly like a judge harness.

5. The pack-designing agent must be told, and the contract is the only channel

crucible scope writes docs/crucible-contract.md verbatim into the pack-designing agent's context directory (crucible/src/scope.rs:303 embeds it, :1393 writes it out). That agent authors the crucible.toml and the measure script for a new issue. Today crucible/src/prompts/scope-propose.md says nothing about builds at all, so the agent picks a mode by accident — or, more often, copies whichever apply hook it saw last.

An ADR the agent never reads changes nothing. The modes therefore live in the contract (§3.1), not only here. This ADR is the rationale; the contract is the interface. Anyone tempted to "clean up" the duplication should move the rationale, never the rules.

6. Where builds run

Per-iteration candidate builds stay on-cluster (ADR-0018's Cluster backend). GithubActions stays for infrastructure images. Dispatch latency, queue wait, and a cold runner would land on every iteration, and a runner cannot hold the warm compiler cache §3 depends on. ADR-0018 already drew this line for candidate test runs; the same reasoning applies to candidate builds.

This is also what unblocks ADR-0019 P5: moving the candidate build off the loop pod is what removes buildah from the loop image. Together with the Kubernetes compute driver removing podman, that is what finally makes privileged: false possible.

Alternatives considered

• Leave it as bash in apply hooks. It works, and it is invisible. Onboarding a composite means copy-pasting fullstack-lora-apply.sh and hoping the base tag is on the right registry. Rejected: the preconditions are unenforced and the failure modes are silent.
• One mode for everything (always a full rebuild). Uniform and slow. It would take vLLM's 8-second derive to a multi-minute buildah build for zero benefit, since nothing in a Python candidate compiles.
• Always derive-layer. Impossible for compiled components; a Go binary is not the sum of its .go files.
• Full rebuild in GitHub Actions per iteration, relying on layer cache. The tempting one. Rejected in §3/§6: the source layer invalidates the cache every turn, and dispatch latency is per-iteration.

Consequences

• Positive: the mode a domain uses becomes readable, checkable, and reviewable rather than implied by which shell script it copied. The two silent 20-minute/no-op failure modes become crucible check errors. The reward-hacking hole in the build recipe closes. ADR-0019 P5 gains its precondition.
• Negative / cost: a new manifest surface and a BuildBackend extension; the warm-cache volume is real infrastructure; freezing the Containerfile means a domain author can no longer iterate on it inside a run (which is the point, and is the same cost ADR-0001 already accepted for the judge).
• Risk: paths-based derive is a heuristic for "did anything compiled change." A component that is mostly Python with a compiled extension will fool it. crucible check can only catch the declared shape, not a .so built at import time.

Open questions

1. What marks a candidate build-needed at all? Today it is "the diff is non-empty for this component's subtree." That is a per-component paths glob, and it is the same predicate ADR-0018 needs for its [build.<name>.watch] block. One predicate, two consumers — worth unifying.
2. Does the warm compiler cache belong to the builder Job (a PVC per component) or to a long-lived builder Deployment? The Job model is simpler and matches ADR-0018; a PVC that outlives the Job is the minimum.
3. Do we need a mode = "config" distinct from "none" to name mode 2 (config tuning with a rollout) so apply_cmd stops being the only place that knowledge lives?

ADR 0022: Measure task DAGs — the engine walks the ladder

Status: Proposed Date: 2026-07-18 Related: ADR-0001 (the frozen judge this walker becomes part of), ADR-0005 (the brokered codegen tools the walker dispatches), ADR-0016 (session.jsonl as the source of truth the DB re-indexes — the property the per-task cache leans on), ADR-0017 (the event contract per-task results extend), ADR-0020 (how a candidate becomes the digest these tasks measure). Supersedes the walker half of the kernel domain's gate; subsumes #8 (named-job registry); completes #291 (regrade). Tracking issue: #292.

Context

A GPU-measured code domain grades a candidate through a sequence of oracles: a CPU reference self-check, a single-GPU numerical diff, an ncu tensor-pipe capture, a fused multi-GPU equality test, a compute-sanitizer racecheck. Today that sequence lives inside an opaque domain gate script (call it kernel_gate.py, a GPU kernel domain's rung-walker): the engine runs one measure_cmd, the script walks the ladder by calling broker MCP tools itself, and eight hundred lines later the engine receives a single {valid, score, detail} blob. The individual oracles already run as separate Kueue GPU jobs — the walk is the only part that is a black box.

The kernel domain's bring-up paid for that black box three times in two days:

• No per-step resume. Run 13 proved steps 1–5 on a cached digest, then died at step 6. The rerun repaid all five proven steps. Regrade (#291) should mean "re-run the failed step on the cached digest", keyed on (digest, step) — but no such key exists anywhere.
• Silent retries. Step 6 hit its 90-minute job deadline five consecutive times. Kubernetes retried behind everyone's back; the engine saw nothing until a verdict arrived ~8 hours late. There was no engine-visible per-step state to even ask about.
• Policy in per-domain Python. Transport-error-versus-measured-failure classification, advisory-past-terminal semantics, substrate capability filtering, fail-closed hardware truncation — all generic judge policy, all hand-implemented (and hand-patched, twice) in one domain's gate script. The next codegen domain would copy-paste the lot.

And the sequence is not actually a ladder. The ncu capture and the racecheck both depend only on the single-GPU diff; on two GPUs they could run concurrently. A ladder is a small DAG that nobody wrote down as one.

Decision

The manifest declares the DAG as data. The engine walks it. The domain keeps exactly one thing: the frozen per-task command baked into the sandbox image.

Vocabulary: the unit is a task

Declared [[measure.task]], identified by name, connected by depends_on edges. Not "rung" (a ladder word — the whole point is that this is not a ladder), and not stage/step/node/check/job, which all name live things elsewhere in the engine (the scope pipeline's Stage, the journey's JourneyStep, the SPA's journey nodes, crucible check, the broker's Kueue jobs). No new code or prose says "rung"; existing occurrences are cleaned up by whichever change rewrites their file.

The manifest schema

MeasureCfg (crucible/src/manifest/measure.rs) stops being an opaque pass-through table and becomes typed, because the engine now consumes half of it. The fixed benchmark/lm_eval/ profile tool trio is replaced by named job templates — this is the #8 registry, and it kills both the case-select env-toggle overload and the dummy [measure.lm_eval] block the kernel domain carries only because the broker demands a command:

[measure]
gpus = 1
score_task = "calc-diff"     # optional; default = last-passing required task in topo order

[measure.build]              # unchanged — the broker build contract (ADR-0005/0020)
base_image = "..."

[measure.job.oracle]         # named job templates replace the benchmark/lm_eval/profile trio
command = 'python $REFERENCE_DIR/run_task.py --task "$CRUCIBLE_TASK" --out "$OUT"'
kind    = "metrics"          # or "trace" (+ trace_ext); optional per-job gpus override

[[measure.task]]
name       = "calc-diff"     # identity; injected as $CRUCIBLE_TASK, frozen, agent-invisible
job        = "oracle"
depends_on = ["refcheck"]    # default []
needs      = "fp8-tc"        # substrate capability, default "any"
threshold  = 0.001
direction  = "lower"         # an explicit `pass` bool in $OUT wins over threshold grading
required   = true            # default true

Validated at parse time: unique names, edges resolve, acyclic, referenced jobs exist. Exactly one of [judge].measure_cmd or [[measure.task]] — declaring both is an error, declaring neither where a judge is needed stays an error. Domains without tasks (the vLLM brief's single-shot gate) are untouched; the legacy trio keys still project to the broker.

Substrate capabilities are profile-side facts: the deploy profile's [measure] block gains caps = ["any", "fp8-tc"], projected as CRUCIBLE_MEASURE_CAPS into the loop pod. This replaces the capability map hardcoded in the Python gate and the substrate env hack. Fail-closed survives the move: tasks declared but caps unset means the walker refuses to measure.

The broker never sees the DAG. BROKER_CODEGEN_TOOLS_DEFAULTS grows a job map of named templates; the [[measure.task]] array is stripped before projection. One new broker tool, codegen_task {digest, task, job}, dispatches a template with the frozen CRUCIBLE_TASK env; the memo key gains the task name, which also fixes the collision where two profile captures with identical fixed toggles shared a cache entry.

An escape hatch for exotic scoring exists but is deliberately narrow: an optional [measure].grade_cmd receives the assembled per-task results as JSON on stdin and emits the standard {valid, score, ...} line, overriding the engine's default fold. Grading rules (threshold, direction, pass-key) are data and stay in the manifest; there are no per-task grader commands. The kernel domain needs neither; its gate script is deleted, not shrunk.

Validity: required, not "terminal"

"Terminal rung" and "advisory past the terminal" are ladder concepts; with parallel branches "past" has no meaning. The DAG-native contract:

• A task is runnable iff its needs is in the substrate caps and every transitive dependency is runnable.
• valid = every required task is runnable and passed.
• Hardware truncation is computed before dispatching anything: a required task filtered out by the substrate means valid:false with an explicit note and zero GPU spend. A truncated DAG can never produce an honest pass, so it fails fast instead of measuring toward a foregone verdict.
• Advisory (required = false) tasks: unrunnable means skipped; a failure is recorded and never gates validity, but the failed task's own dependents are blocked — nothing runs on top of a failure.
• Short-circuit: a required task's measured failure (or transport-retry exhaustion) fails the reading immediately; undispatched tasks are marked blocked, in-flight ones complete and cache.

The old per-profile terminal difference (one numeric format gates on the fused test, another on the single-GPU diff) becomes plain data: each manifest marks different tasks required.

The walker

A new TaskDagJudge behind the existing Judge trait, selected by build_judge when the manifest declares tasks; CommandJudge is untouched and the loop driver does not change. It calls the broker's codegen tools over the same MCP streamable-HTTP surface the Python gate used — the broker stays a dumb, trusted executor where one tool call is one Kueue job is one memo entry.

The walk is a ready-set loop over the DAG. v1 pins max_inflight = 1: the kernel domain's DAG is nearly a chain, the single-GPU queue serializes anyway, and in-flight-sibling cancellation is machinery no domain needs yet. The structure is parallel-ready; lifting the cap is a constant, not a redesign.

Retry policy splits on the classification the broker already puts on the wire: CodegenReply::JobFailed is a measured failure and is never retried — a kernel that failed calc_diff failed. CodegenReply::Error and HTTP failures are transport and get a bounded auto-retry (default 2), every attempt an engine-visible event. The forge GPU job pins backoffLimit: 0 so Kubernetes never again retries where the engine can't see; the 90-minute deadline itself stays a broker/forge-owned substrate fact.

Cache and regrade

The durable (digest, task) record is the session log, per ADR-0016: every attempt appends a SessionEvent::TaskResult, and on resume or regrade the engine folds the log into a (digest, task) → result map, skipping recorded passes and re-running recorded transport failures. No new persistence machinery; the run-13 scar (die at task 6, repay 1–5) closes by construction. The broker's in-memory memo remains the fast path within a broker lifetime.

crucible regrade --digest <D> [--task <name>] (#291) builds a plan restricted to the target's transitive closure, serves dependencies from cache, force-reruns the target, and re-folds the verdict. So regrade-after-broker-restart works without a rebuild, the broker's is_built provenance check re-verifies the candidate image in the registry instead of only trusting its in-memory set.

Telemetry

SessionEvent::TaskResult { iter, digest, task, job, attempt, status, metric, note, secs } with status ∈ pass | fail | transport | skipped | blocked | truncated — additive, no wire-version bump. The controller folds it into a task_results table (child of candidates, rebuildable like everything else), the SPA gets a per-candidate task grid with live progress from the session tail, and the candidate row keeps an assembled detail.tasks[] so the existing single-row view degrades gracefully. The walker opens one OTLP span per task attempt under the per-turn traceparent; the broker's existing per-tool spans nest beneath it unchanged. "Task 6 running, attempt 2" is visible within seconds instead of eight hours after the fact.

Consequences

• Generic judge policy (classification, retry, truncation, short-circuit, caching) is written once, in Rust, tested in the engine — the next codegen domain declares ~40 lines of TOML and bakes one frozen command, instead of copy-pasting an 800-line walker.
• #8 and #291 stop being separate work: the task→job mapping is the registry, and per-task cache keys make regrade incremental.
• The gate-script tier shrinks: the kernel domain's kernel_gate.py and its tests are deleted, their pure semantics ported into the walker's unit tests; run_rung.py becomes run_task.py switching on $CRUCIBLE_TASK. crucible check prints the resolved runnable set and the truncation verdict, replacing the gate's --selftest.
• The broker gains a tool and a config key but no orchestration; its trust posture is unchanged.
• Single-shot domains keep measure_cmd forever — the DAG path is opt-in per manifest.
• Follow-ups deliberately deferred: max_inflight > 1 (needs sibling-cancellation semantics), richer score composition (the answer is grade_cmd, not a bigger TOML grading language), and baseline-differential grading (count only new racecheck hazards vs the base SHA) as a task attribute.

Alternatives considered

Broker-side orchestration — the engine calls one codegen_walk tool and the broker runs the DAG. Rejected: it structurally recreates the one-long-opaque-call problem (per-task progress again trapped behind a single tool call), and it moves judge policy into the component whose job is to be a dumb executor. The transport/measured split the walker needs is already visible on the reply wire; the broker has nothing to add but opacity.

Keep the gate script as a thin grader — engine walks, then shells out to domain Python for the verdict fold. Rejected: the fold (required-set over task results) is five lines of generic policy; leaving it in per-domain scripts preserves exactly the copy-paste channel this ADR exists to close. grade_cmd remains for domains that genuinely need custom scoring, as an explicit opt-in rather than the default architecture.

A generic engine DAG framework — model tasks on a general workflow engine (or grow WorkKind into one). Rejected as scope creep: this is a judge implementation detail behind an existing trait, not a new orchestration primitive. If a second DAG consumer appears, extract then.