crucible

Deterministic agent orchestration

Agents do the work. The engine selects the next task.

Crucible is a deterministic orchestrator for agent workflows. It runs agent turns in sandboxes. You declare the graph, and the engine schedules the tasks, isolates them, and records their cost.

Write a playbook → Run the control plane GitHub

A workflow is a graph of agent turns, commands, and checks. The executor selects the next task, not the agent. The selection rules are fixed, and you can read them before the run starts.

Two lanes

The same engine runs two kinds of workflow.

A playbook, end to end

A pack is a directory that holds a crucible.toml, a workflow.star, and the scripts that the tasks run. This pack reads a ticket, routes it to one queue, and files it:

read = command(name = "read", run = "./classify.sh", emits = ["bucket"])

gate = route(
    name = "gate",
    depends_on = [read],
    source = read,
    questions = {
        "bucket": choice(
            ask = "Which queue owns this ticket?",
            options = ["outage", "billing", "feature"],
        ),
    },
)

oncall = command(name = "oncall", run = "./file.sh oncall", depends_on = [gate],
                 when = gate.bucket, answers = "outage")
finance = command(name = "finance", run = "./file.sh finance", depends_on = [gate],
                  when = gate.bucket, answers = "billing")
backlog = command(name = "backlog", run = "./file.sh backlog", depends_on = [gate],
                  when = gate.bucket, otherwise = True)

filed = command(name = "filed", run = "./file.sh filed",
                depends_on = [oncall, finance, backlog], join = "passed")

workflow(type = "playbook", tasks = [read, gate, oncall, finance, backlog, filed], result = filed)

$ crucible plan run --manifest crucible.toml --max-cost 1 --max-time 5m
  read                 pass       attempts=1 cost=$0.0000  out={"bucket":"billing"}
  gate                 pass       attempts=1 cost=$0.0000  out={"bucket":{"confidence":1.0,"label":"billing",...}}
  oncall               not_taken  attempts=0 cost=$0.0000  (gate.bucket resolved to billing)
  finance              pass       attempts=1 cost=$0.0000  out={}
  backlog              not_taken  attempts=0 cost=$0.0000  (gate.bucket resolved to billing)
  filed                pass       attempts=1 cost=$0.0000  out={}
plan v1: completed — spent $0.0000 of $1
verdict: valid

The branches that no answer selects settle not_taken. They do not dispatch and do not spend. If you replace source = read with min_confidence = 0.8, a decision model answers the question. The engine records a low-confidence answer as uncertain, and the otherwise task takes it.

Where to go next

The pieces

Your first playbook

A playbook is a graph that runs once. This page builds a playbook from an empty directory. An agent writes a haiku, and a command counts its lines. If the count is wrong, the command sends the draft back for a second round. The first run uses a stand-in for the agent and costs nothing. The second run uses a real agent.

You need a crucible binary on PATH (installation). Every build includes the playbook lane. Only the scored loop needs --features autoresearch.

The pack

A pack is a directory that holds all the files a run needs:

haiku/
├── crucible.toml    the manifest: the agent, the workspace, which graph to run
├── workflow.star    the graph
├── count.sh         the reviewer
└── stand-in.sh      a fake agent, for runs without a model

1. The manifest. crucible.toml sets the backend for agent turns and the initial content of the workspace. Without [repo], the workspace starts empty and holds only the files that inject lists.

   [workspace]
   inject = ["count.sh", "stand-in.sh"]
   
   [agent]
   backend = "command"
   agent_cmd = "./stand-in.sh"
   goal = "Write a haiku about molten metal."
   
   [workflow]
   type = "playbook"
   file = "workflow.star"
   

   backend = "command" runs agent_cmd instead of a model. The rest of the pack sees no difference, so you can test the pack in CI.

2. The graph. workflow.star declares the tasks and the edges between them.

   poem = agent(
       name = "poem",
       prompt = "Write a haiku about molten metal to HAIKU.md. Report its line count as `lines`.",
       emits = ["lines"],
       emits_files = ["HAIKU.md"],
   )
   
   check = command(
       name = "check",
       run = "./count.sh",
       depends_on = [poem],
       revise = poem,
       max_rounds = 2,
   )
   
   workflow(type = "playbook", tasks = [poem, check])
   

   emits declares the output fields. A passing poem must return a lines field. If the field is missing, poem fails, so the error appears at its source and not in a later task. emits_files names the file that the engine stages for check. revise = poem makes check a reviewer. If check fails, poem runs again with the verdict. There are at most two rounds.

3. The scripts. A command task passes if it exits with status zero. Its last stdout line is its JSON output. This is the reviewer:

   #!/bin/sh
   n=$(grep -c . HAIKU.md)
   printf '{"lines": %s}\n' "$n"
   [ "$n" -eq 3 ]
   

   The stand-in writes four lines in the first round. When its prompt contains the reviewer's verdict under revision, it writes three lines:

   #!/bin/sh
   case "$CRUCIBLE_PROMPT" in
   *'"revision"'*) printf 'molten in the dark\nthe crucible holds its breath\npour, and it is steel\n' > HAIKU.md ;;
   *) printf 'molten in the dark\nthe crucible holds its breath\nand then some more\npour, and it is steel\n' > HAIKU.md ;;
   esac
   echo "{\"lines\": $(grep -c . HAIKU.md)}" > PLAN_TASK_RESULT.json
   

   An agent turn reports its result in PLAN_TASK_RESULT.json in the workspace root. The engine adds this instruction to every agent prompt. Name the emits fields in the prompt, as poem does with lines.

Run it

chmod +x count.sh stand-in.sh
crucible check --manifest crucible.toml
crucible plan run --manifest crucible.toml --max-cost 1 --max-time 5m

  poem                 pass       attempts=1 cost=$0.0000  out={"lines":3}
  check                pass       attempts=1 cost=$0.0000  out={"lines":3}
plan v1: completed — spent $0.0000 of $1
verdict: valid

--max-cost and --max-time are required. A pack cannot set its own limits. The person or system that launches the pack sets them. Once spend reaches --max-cost, no further agent task, model-decided route, or fan-out lane starts. At most eight lanes of one fan-out run at once, so the overrun is at most what the lanes already running go on to spend. Commands still run after the cost ceiling, so the run reports the work it paid for. --max-time stops everything.

crucible check validates the manifest, resolves every file that it references, and lists the egress and credentials that the pack would get. It does not run a task.

The verdict is valid because every required task passed. Use the verdict in scripts. plan run exits with status 3 for any other verdict, and with status 1 if it cannot run the plan.

What the run left behind

The run writes two directories next to the manifest. workspace/ is a Git repository, and each passing task makes a commit:

$ git -C workspace log --format=%s
task poem
task poem
autoresearch: baseline workspace

There are two task poem commits. The first draft passed as a task because it wrote its result. The review failed it, and the second round replaced it. plan run prints the final state of each task. state/session.jsonl records every round:

$ jq -r 'select(.task != null and .status != null) | "\(.task) \(.status)"' state/session.jsonl
poem[round-1] pass
check[round-1] fail
poem[round-2] pass
check[round-2] pass
poem pass
check pass

Report a pull request or a pushed branch

If a task makes something outside the run, declare the field that holds its URL as link. Declare a list of URLs as links:

deliver = skill(
    name = "deliver",
    skill = "open-pr",
    depends_on = [check],
    emits = {"pushed": "links", "pr": "link"},
)

{"pushed": ["https://github.com/neuralmagic/crucible/tree/haiku"], "pr": "https://github.com/neuralmagic/crucible/pull/123"}

Only http and https URLs are valid. Any other value fails the task. The engine reads the host and path of each URL to name it (github, a pull request, #123). The run and task views show a provider mark that links to the URL.

If the run is interrupted

plan run --resume continues the run in state/:

crucible plan run --manifest crucible.toml --max-cost 1 --max-time 5m --resume

A task that the session log already settled does not run again. This includes single items of a fan-out and single rounds of a revise loop. The workspace goes back to the commit where the run started. A later task sees the files that earlier tasks declared in emits_files, not other files they left in the tree. Spend and elapsed time from before the interruption count against the limits. Elapsed time includes the time when the run was down. A resume fails if the pack, a parameter, or --max-cost is different from the start of the run. If the run already shut down, a resume only exits with its verdict again.

A pod that the control plane dispatches resumes automatically. If the engine stops, the container restarts and resumes, up to three times. A pod that you render by hand does the same if its profile sets [cluster] state_pvc.

Rerun from a task

If a late task fails, fix it and rerun from that task. You do not have to run the whole graph again. Move the state of the finished run to a different directory, and give that directory to --rerun-of:

mv state parent
crucible plan run --manifest crucible.toml --max-cost 1 --max-time 5m \
  --rerun-of parent --from check

check and all tasks downstream of it run again. Each task that did not pass in the parent run also runs again. The rerun reuses all other tasks. Their output and declared files go to their dependents as before, and the rerun replays their commits onto the commit where the parent started. The session log marks each reused row with reused_from, which names the run that settled it. The spend of reused rows does not count against the new limits.

You can edit the tasks that you rerun. If you edit a task that the rerun would reuse, the rerun fails and names the task. This prevents a stale result from being reused. Add the task to --from to run it again. This check compares only what the graph declares. It does not compare the scripts in the pack. If you edit only check.sh, no task runs again because of that edit.

In the control plane, the page of a finished launch has a Rerun button. It shows which tasks the rerun reuses and the expected cost. From the CLI, launch the draft or playbook again and name the run to rerun:

crux draft-launch haiku --max-cost 1 --max-time 5m --rerun-of <run id> --from check

A draft rerun runs the newest save. A pod rerun gets the parent's evidence from the controller at [cluster].ingest_url. The profile of a pod executor must set this value.

Put a real agent on it

Remove agent_cmd and select a backend:

[agent]
backend = "local"
harness = "claude"
goal = "Write a haiku about molten metal."

The harness is claude, codex, opencode, or pi. The --harness and --model options of plan run override the manifest for one run. A task can set its own values with agent(..., harness = "codex", model = "..."). A real run costs money. --max-cost limits that cost.

With openshell, a task can run in its own sandbox. Declare the sandbox in the manifest and name it on the task:

[agent.sandbox.go]
image = "ghcr.io/acme/sandbox-go@sha256:..."
secrets = []                                  # [[secret]] names with an env projection
relays = []                                   # [[agent.relay]] destinations
broker = false                                # reach the [agent.broker]
mcp = []                                      # [mcp] servers this sandbox reaches
endpoints = ["proxy.golang.org:443:read-only"]

analyze = agent(name = "analyze", prompt = "...", sandbox = "go")

The turn starts from that image. It adds those endpoints to the endpoints of [agent.openshell]. From the secrets, relay files, and [mcp] servers that the pack declares, the turn gets only the ones that the sandbox lists. The turn can reach the broker only if broker = true. Every turn still gets the model credentials of the deployment. A task without sandbox runs in sandbox_image and gets every declared secret and relay. Tasks that share a session must use the same sandbox.

A sandbox limits what the engine provisions for a task. It does not limit what a task reads from upstream tasks. If a task had a secret, its output, files, and workspace changes still go to the tasks after it. The capability disclosure lists each sandbox. At launch, the controller checks each sandbox image against the catalog.

MCP servers

With openshell, a pack can start MCP servers on the loop pod. Use them for tools that need credentials that the sandbox must not have:

[mcp.buildit]
bin = "/usr/local/bin/buildit"
args = ["mcp"]
env = { BUILDIT_NAMESPACE = "builds" }        # set on the server
inherit = ["KUBERNETES_SERVICE_HOST", "KUBERNETES_SERVICE_PORT"]  # copied from the loop pod
tools = ["build", "run", "logs"]              # passed as MCP_TOOLS

[agent]
mcp = ["buildit"]                             # turns without a named sandbox

A turn can reach only the servers that its scope names. The scope is [agent].mcp, or the mcp list of the named sandbox that the turn runs in. The default is no servers. The harness configuration of the turn lists only those servers.

Each server runs as a separate process on a separate port. Ports start at 8850 and follow key order. A server gets only these environment variables: PATH, HOME, its env and inherit names, MCP_NAME, MCP_BIND, MCP_TOKENS_FILE, and MCP_TOOLS. Every turn gets a new token for each server. The engine writes the token to MCP_TOKENS_FILE with the sandbox and workdir of the turn (format). The engine revokes the token when it deletes the sandbox. The server identifies the caller from the token alone.

Parameters

A pack that takes input declares a params block as the first statement of workflow.star:

params = {
    "topic": {"type": "string", "default": "molten metal", "doc": "what the haiku is about"},
}

poem = agent(
    name = "poem",
    prompt = "Write a haiku about " + param("topic") + " to HAIKU.md. Report its line count as `lines`.",
    emits = ["lines"],
    emits_files = ["HAIKU.md"],
)

crucible plan params --file workflow.star           # the JSON Schema, without running anything
crucible plan run --manifest crucible.toml --param topic=slag --max-cost 1 --max-time 5m

A string parameter that sets "multiline": True takes its value in a text area on the control plane's launch, schedule and ask forms. The value is still one string.

Parameters go into prompts and skill arguments. They do not go into a command line. If you put a parameter into run, the compiler rejects the pack:

argument "run" carries a value supplied from outside the pack. A prompt marks such a span so
an agent can tell it from an instruction; nothing else can, so do not build it into "run". A
command or evaluate task reads it as data from the "params" entry of the inputs JSON in $CRUCIBLE_INPUTS_FILE.

A command or evaluate task reads each declared parameter from the params entry of the inputs JSON. CRUCIBLE_INPUTS_FILE names that file. Each parameter has its declared name and type, and the file also holds the outputs of the task's dependencies. If the source has no params block, the entry is an empty object. Agent tasks do not get this entry. They get parameter values only through the prompt.

fetch = command(
    name = "fetch",
    run = "python3 -c 'import json, os; p = json.load(open(os.environ[\"CRUCIBLE_INPUTS_FILE\"]))[\"params\"]; print(json.dumps({\"topic\": p[\"topic\"]}))'",
)

params is a reserved name. A task cannot name a dependency params.

The control plane validates a launch against the same schema.

History

A playbook that runs on a schedule can read the results of its last runs. The pack names one task as its record task. An agent, skill, command, or evaluate task can request up to 30 earlier runs:

triage = agent(
    name = "triage",
    prompt = "Find what is broken. Check the run history first: skip what earlier runs could not fix.",
    history = 5,
    emits = ["broken", "tried"],
)
workflow(type = "playbook", tasks = [triage], history_record = triage)

Each run records only the status of the record task and the fields that it declares in emits. A later run of the same standing launch gets those records under history, oldest first. The list includes failed runs and runs that timed out:

{"records": [{"run": "...", "started_at": "...", "ended_at": "...", "outcome": "finished",
              "verdict": "valid", "revision": "...", "link": "...",
              "entry": {"task": "triage", "status": "pass", "output": {"broken": 2, "tried": ["..."]}}}],
 "dropped": 0}

A command reads the history from the inputs file that CRUCIBLE_INPUTS_FILE names. An agent sees it in its prompt, marked as external input, because an earlier agent wrote part of it. If the records are larger than the size limit that the operator sets, the engine removes the oldest full records and counts them in dropped. crucible check prints the limit.

A manual launch is not part of a series, so it gets an empty list. A local plan run also gets an empty list, unless CRUCIBLE_HISTORY is set. To record a result when the main graph fails, make the record task an epilogue task. The record shape of a pack can change between revisions, so a reader must accept older entries. Each record contains the revision that wrote it.

Launch it from the control plane

The control plane has a registry of playbooks, a draft studio to edit them, and a ledger of all launches. Start a controller and point crux at it. Then:

crux draft-create haiku --description "a haiku, reviewed"
crux draft-push haiku ./haiku --base-version 1
crux draft-launch haiku --max-cost 1 --max-time 5m

draft-push saves the directory as the next version of the draft and compiles it. When the draft is ready, crux draft-graduate opens a PR that exports it to a repository. crux playbook-import owner/repo --path haiku proposes to register it from that repository. After registration, crux launch starts the playbook with parameters, and crux playbook-runs lists its runs.

A registration runs one variant of the pack's manifest (variants). Add --variant offline to playbook-import, or pick the variant in the import wizard, which lists the base manifest and every variant of each pack. Every launch of the registration then runs that variant. To run another variant of the same pack, register it again under another id.

Let your own agent write it

If an agent writes the pack, give it the co-draft skill before it starts. The controller generates the skill at <controller>/api/playbooks/drafts/skill, and the skill contains the controller URL. The skill tells the agent how to use the draft tools, how to use base_version when it saves, and when to stop. The Agent skill button on the drafts page downloads the skill, and crux draft-create prints the link. Install the skill and register the MCP endpoint of the controller:

mkdir -p ~/.claude/skills/crucible-co-draft
curl -fsSL -o ~/.claude/skills/crucible-co-draft/SKILL.md \
  -H "Authorization: Bearer $CONTROLLER_API_TOKEN" \
  "$CONTROLLER_URL/api/playbooks/drafts/skill"
claude mcp add --transport http crucible "$CONTROLLER_URL/mcp" \
  --header "Authorization: Bearer $CONTROLLER_API_TOKEN"

Then ask the agent for a pack. The agent saves into a draft, and you can see it compile in the studio. When the diagnostics are clean, the agent gives you the studio link.

Launching from webhooks

A webhook launches a registered playbook when a sender outside the controller posts a delivery. A sender is any system that POSTs JSON to a URL: quay.io after an image push, GitHub after a release, a CI job, or your own script. The controller verifies the delivery and puts it in a queue. Within a few seconds, the controller takes the delivery from the queue and runs a small CEL transform. The transform decides if the delivery launches a run, which event it is, and which params the launch gets.

flowchart LR
  S[sender] -->|POST /hooks/id| L[delivery listener]
  L -->|genuine: queue it, 202| D[(delivery queue)]
  L -->|anything else: 404| S
  D --> W[controller, every few seconds]
  W -->|filter, dedupe, rate, params| R{outcome}
  R -->|launched| P[playbook run]
  R --> F[filtered / duplicate / throttled / failed]

The listener does not launch runs. It answers 202 when the delivery is in the queue. The controller then processes the queue of each webhook in arrival order. Each delivery gets exactly one outcome, and you can read that outcome later.

Verifiers

A webhook verifies each delivery with one of two verifiers. You select the verifier when you create the webhook, and you cannot change it later. To use a different verifier, delete the webhook and create a new one.

A sender that signs in a different way (a base64 signature, or a signature over a timestamp and the body) cannot use hmac_sha256. Give that sender a path_token URL. The token is the credential.

The controller generates every secret (256 random bits). It shows the secret once, in the response to the create or rotate request. The controller stores path tokens as SHA-256 digests. It seals HMAC secrets with AES-256-GCM under the webhook key. A controller without a webhook key rejects hmac_sha256 webhooks.

If the controller does not record a delivery, the answer is always 404 with {"error":"not found"}. This applies to an unknown or deleted webhook, a paused webhook, a wrong token or signature, and a body larger than 512 KiB. Webhook ids are random UUIDs, so the address alone gives no information.

Creating a webhook

From the UI

Open a playbook and select Add webhook. The default Sender is Custom, for any sender that POSTs JSON. With Custom, the filter launches every delivery, the dedupe key is delivery, and every param is a fixed value until you derive it. The named senders (quay.io repository push, GitHub release, GitHub push) set the verifier, the transform, and the params that they can match by name. If you select Custom again, the transform starts from the beginning. For each param that the playbook declares, select fixed value or derived (CEL). The form checks each expression as you type and marks the location of each error. Run preview evaluates the transform on a sample body and headers. It does not store anything.

When you save, the form shows the delivery URL and the secret one time. Configure the sender with them:

• Any sender: POST JSON to the delivery URL (it includes the token). See A custom sender.
• quay.io: repository settings, Create Notification, event Push to Repository, method Webhook POST, URL = the delivery URL.
• GitHub: repository settings, Webhooks, Add webhook, payload URL = the delivery URL, content type application/json, secret = the HMAC secret, then choose the events.

Webhooks in the navigation lists every webhook that you can read. The page of a webhook shows its delivery URL, its transform, the pause/resume, rotate, and delete controls, and its delivery log. Edit opens the form again. Its preview can load any recorded delivery as the sample.

A custom sender

Any system that can send an HTTP POST with a JSON body can use a path_token webhook:

curl -X POST -H 'content-type: application/json' \
  -d '{"service": "api", "version": "1.4.2", "env": "staging"}' \
  "$DELIVERY_URL"

You choose the body format. Write the transform for that format:

• a filter, for example body.env == "staging";
• a dedupe key, for example body.service + "@" + body.version, so that a retried POST launches only once;
• a derive for each param, for example version: body.version.

To test the transform on a real delivery, load a recorded delivery into the preview of the edit form. If a sender sometimes omits a key, guard it: has(body.env) && body.env == "staging".

An hmac_sha256 sender also signs the raw body. It puts the signature in the header that the webhook names (x-signature in this example):

body='{"service": "api", "version": "1.4.2"}'
sig=$(printf '%s' "$body" | openssl dgst -sha256 -hmac "$SECRET" -hex | sed 's/^.* //')
curl -X POST -H 'content-type: application/json' -H "x-signature: sha256=$sig" \
  -d "$body" "$DELIVERY_URL"

The body must be JSON. The controller records any other body with the outcome failed.

From crux

crux webhook-presets                    # starting points, as JSON
crux webhook-check --file hook.json     # CEL diagnostics, field:line:col; fails on any
crux webhook-create --file hook.json    # prints the webhook, its secret, and delivery URL, once
crux webhooks                           # list
crux webhook <id>                       # one, as JSON
crux webhook-deliveries <id> [--before <delivery-id>] [--limit N]
crux webhook-update <id> --file hook.json
crux webhook-pause <id>                 # its address answers 404; queued deliveries fail
crux webhook-resume <id>
crux webhook-rotate <id>                # new secret, shown once; the old one stops working
crux webhook-delete <id>
crux webhook-preview --file preview.json

A create body:

{
  "playbook": "on-image-push",
  "verifier": "path_token",
  "filter": "\"latest\" in body.updated_tags",
  "dedupe": "delivery",
  "derive": { "image": "body.docker_url", "tags": "body.updated_tags" },
  "params": { "release": "3.2" },
  "max_launches_per_hour": 10,
  "retention_days": 14,
  "max_cost": 5,
  "max_time": "30m"
}

params are fixed values. The controller validates them against the schema of the playbook, as it does for a launch form. derive maps a declared param to the expression that derives it. A param is fixed or derived, not both. An hmac_sha256 body also contains "header": "x-hub-signature-256".

From the API

POST /api/webhooks, GET /api/webhooks, GET|PUT|DELETE /api/webhooks/{id}, POST /api/webhooks/{id}/enabled, POST /api/webhooks/{id}/secret, GET /api/webhooks/{id}/deliveries?before=&limit=, POST /api/webhooks/preview, POST /api/webhooks/check, GET /api/webhooks/cel, GET /api/webhooks/presets. The schemas are in /api/openapi.json. For authorization, a webhook is a standing launch. To read a webhook, you need read on it. To change it, you need update. If a save changes what the webhook launches, the policy also evaluates the save as a launch of the playbook.

The transform

The transform has three kinds of CEL expression. All of them use the same variables:

• filter is a bool. A delivery that it rejects ends filtered. If there is no filter, the value is true.
• dedupe is a string or int, at most 512 bytes. If a key already launched a run, a new delivery with that key ends duplicate. This is true while the webhook exists, also after the controller prunes the delivery that used the key. With delivery, each delivery is a separate event.
• derive.<param> returns a string, number, bool, or list of strings for that param.

Functions: size, int, uint, double, string, bytes, type, timestamp, duration, and on a target contains, startsWith, endsWith, matches. Macros: has, all, exists, exists_one, map, filter. GET /api/webhooks/cel returns the same lists. There is no join.

The controller rejects a save if an expression has one of these problems. The error names the field, and the line and column when the parser knows them:

• it does not parse, it nests brackets more than 32 levels deep, or it is longer than 2 KiB;
• it reads a variable other than the four above, or it calls a function that is not in the list above;
• it nests comprehensions more than two levels deep;
• it has a comprehension whose condition or step reads values other than its own variables. For example, body.l.map(x, body.l) is rejected, and body.items.exists(i, i.tags.exists(t, t == "x")) is allowed. This rule keeps the evaluation time linear in the size of the body.

Examples

quay.io push of a release tag:

filter  body.updated_tags.exists(t, t.matches("^v[0-9]+"))
dedupe  delivery
image   body.docker_url + ":" + body.updated_tags.filter(t, t.matches("^v"))[0]

GitHub release:

filter  headers["x-github-event"] == "release" && body.action == "published"
dedupe  string(body.release.id)
tag     body.release.tag_name

The quay.io push payload names the pushed tags but no digest. For this reason, the quay preset uses delivery as the dedupe key, and every push launches a run. If a playbook pushes the image that it listens to, the rate limit stops it from launching itself in a loop.

Outcomes

A delivery shows queued until the controller processes it, which takes a few seconds at most. The controller processes deliveries in arrival order. Each delivery stops at the first step that gives it an outcome:

Failures that a delivery causes do not count toward disabling the webhook. Failures of the launch do count: the playbook is gone, its stored params do not fit its schema, or the policy denies it. A webhook that fails CONTROLLER_SCHEDULE_AUTO_DISABLE_FAILURES times in sequence pauses itself. Schedules and watches use the same threshold. When you pause or delete a webhook, its queued deliveries fail. The controller keeps finished deliveries for retention_days (1 to 90, default 14). max_launches_per_hour is 1 to 120.

Any user who can read the webhook can read the body and headers of its deliveries. The controller does not record a header that can contain a credential (authorization, cookie, the signature header).

Deploying

The listener uses a different port from the API, so a deployment can publish only the listener. In the Helm chart, webhooks.enabled adds the hooks container, the Service port, and a Route on the external host for the /hooks path only. webhooks.key.existingSecret mounts the webhook key. The webhook key is separate from CONTROLLER_CREDENTIAL_KEY, which also enables offline credentials at sign-in.

Generate a key with head -c 32 /dev/urandom | base64. To rotate the key, put the new key first. Keep the old key after it until you rotate every secret that it sealed.

Trying it locally

just controller-local serves the delivery listener on 127.0.0.1:8871 (set HOOKS_PORT to change it). It generates a webhook key one time in its state directory. A pack with only commands can test webhooks without an agent:

mkdir -p /tmp/image-pushed && cd /tmp/image-pushed
cat > crucible.toml <<'EOF'
[workspace]
inject = ["announce.sh"]

[agent]
backend = "command"
agent_cmd = "true"
goal = "Record which image a registry push named."

[workflow]
type = "playbook"
file = "workflow.star"
EOF
cat > workflow.star <<'EOF'
params = {
    "image": {"type": "string", "required": True, "pattern": "^quay\\.io/"},
    "tags": {"type": "list<string>", "default": ["latest"]},
}

announce = command(name = "announce", run = "./announce.sh")

workflow(type = "playbook", tasks = [announce], result = announce)
EOF
printf '#!/bin/sh\necho "$CRUCIBLE_INPUTS"\n' > announce.sh && chmod +x announce.sh

crux draft-create image-pushed --description "Record which image a registry push named"
crux draft-push image-pushed /tmp/image-pushed --base-version 1
crux draft-publish image-pushed --playbook on-image-push

Create a quay webhook for on-image-push, with the body above or in the UI. Then post a push as quay.io does, to the delivery_url that the create command printed:

curl -X POST --data '{"repository":"org/img","docker_url":"quay.io/org/img","updated_tags":["latest"]}' \
  "$DELIVERY_URL"
crux webhook-deliveries <id>

The controller processes the delivery within a few seconds. POST /api/reconcile processes it immediately. A GitHub delivery to an hmac_sha256 webhook signs its body:

sig="sha256=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$SECRET" | awk '{print $NF}')"
curl -X POST -H "X-GitHub-Event: release" -H "X-Hub-Signature-256: $sig" --data "$BODY" "$DELIVERY_URL"

Troubleshooting

ADR-0055 records the design. The webhook clauses of RFC-0003 give the requirements.

Branching and review

The task set is fixed at compile time. Three constructs make decisions at run time, and each has a limit:

• route() selects branches.
• revise sends work back to an earlier task.
• over fans out one instance for each item of a list.

This page describes these three constructs and the related tools: sessions, gates, joins, advisory tasks, and reports.

The DSL reference lists every argument. Work graphs describes what the executor does with each result.

Branch on a decision

A route() asks typed questions about the output of its dependencies. It records one answer for each question. A downstream task uses when to name the answer that it runs on.

flowchart LR
    read --> gate{gate.bucket}
    gate -- outage --> oncall
    gate -- billing --> finance
    gate -- otherwise --> backlog
    oncall --> filed
    finance --> filed
    backlog --> filed

read = command(name = "read", run = "./read.sh", emits = ["bucket"])

gate = route(
    name = "gate",
    depends_on = [read],
    source = read,
    questions = {
        "bucket": choice(
            ask = "Which queue owns this ticket?",
            options = ["outage", "billing", "feature"],
        ),
    },
)

oncall = command(name = "oncall", run = "./act.sh oncall", depends_on = [gate],
                 when = gate.bucket, answers = "outage")
finance = command(name = "finance", run = "./act.sh finance", depends_on = [gate],
                  when = gate.bucket, answers = "billing")
backlog = command(name = "backlog", run = "./act.sh backlog", depends_on = [gate],
                  when = gate.bucket, otherwise = True)

filed = command(name = "filed", run = "./act.sh filed",
                depends_on = [oncall, finance, backlog], join = "passed")

If the route does not select the answer of a branch, the branch settles not_taken. It does not dispatch, does not spend, and does not change the verdict. A task that depends on it with the default join = "all" is also not taken. For this reason, filed uses join = "passed".

To run a task whether or not one branch was taken, keep join = "all" and mark that dependency optional(...):

summary = command(name = "summary", run = "./summarize.sh",
                  depends_on = [read, optional(oncall)])

summary runs when oncall passed or was not taken, and its inputs have no oncall entry in the second case. If oncall failed, summary is blocked as before. Every other dependency keeps the all rule.

Source of the answer. Set exactly one of these two:

If a source emits a label that the question does not declare, the route fails and the run stops at that route. If a model-backed route has no decision binding, the engine truncates the plan before any task spends.

Question types.

• choice(ask, options) selects one of N labels. options can be a dict, and its values describe each label for the model.
• noul(ask) answers "yes" or "no". when = gate.urgent without answers runs on "yes".
• score(ask, levels) is an ordinal question. levels lists two or more labels, lowest first, optionally as a dict of descriptions. It branches as a choice on its most probable level (the lower level if two are equal). It also records a score in [0, 1], which is the expected level position. For low, medium, high, the positions are 0, 0.5, and 1:

"risk": score(ask = "How risky is this change?",
              levels = {"low": "cosmetic", "medium": "one component", "high": "crosses services"}),

A source answers a score with one level label, not a boolean, and the route records the position of that level. "uncertain" records no score. System One has no ordinal question type, so the engine asks a model-backed score as a choice over the levels and records it with "asked_as": "choice".

Each answer must have a task. The compiler rejects a route if an answer that a when can see does not reach a task. This includes "uncertain". Use otherwise = True to cover the remaining answers, or use drop to declare that an answer goes to no task:

"urgent": noul(ask = "Does the customer need a reply within the hour?", drop = ["no", "uncertain"]),

If you add an option to a question, otherwise takes the new answer, or the pack does not compile until a task takes the new label. examples/route is the full pack. It has a model-backed gate and instructions to serve the decision model.

Let a decision model answer

A decision model reads the inputs of the route and returns a probability for each label of each question. It does not return free text that an agent must format correctly. Crucible supports two decision APIs:

• the OpenAI Decisions API (POST /v1/decisions, protocol decisions);
• any server that implements the System One decision API (POST /v1/systemone, protocol system_one), for example the hosted Jev API or the vLLM DiffusionGemma structured-reads server.

The route declares the questions and the confidence that it needs. The route is the same for both APIs:

read = command(name = "read", run = "./read.sh", emits = ["ticket"])

gate = route(
    name = "gate",
    depends_on = [read],
    min_confidence = 0.8,
    questions = {
        "bucket": choice(
            ask = "Which queue owns this ticket?",
            options = {
                "outage": "the product is down or unusable",
                "billing": "charges, invoices, refunds",
                "feature": "a request for something new",
            },
        ),
        "urgent": noul(
            ask = "Does the customer need a reply within the hour?",
            drop = ["no", "uncertain"],
        ),
    },
)

The workflow does not name the endpoint. The person or system that launches the run binds an endpoint in CRUCIBLE_INFERENCE:

export CRUCIBLE_INFERENCE='{"version":1,"bindings":[{"role":"decision","protocol":"decisions",
  "model":"gpt-6-luna","key_env":"OPENAI_API_KEY"}]}'
crucible plan run --manifest crucible.toml --max-cost 1 --max-time 5m

A decisions binding without a url uses https://api.openai.com/v1/decisions. To use a gateway, set url. A system_one binding must always name its server:

export CRUCIBLE_INFERENCE='{"version":1,"bindings":[{"role":"decision","protocol":"system_one",
  "url":"https://api.example.com/v1/systemone","model":"jev-1","key_env":"JEV_API_KEY"}]}'

key_env names the variable that holds the bearer key. The document does not contain the key. Without a decision binding, the route cannot run, so the engine truncates the plan before any task spends.

Model input. The engine sends one request for each route. The request holds the outputs of the route's dependencies and each question with its instructions and label descriptions. System One takes the outputs as a state object:

{
  "model": "jev-1",
  "state": {"read": {"ticket": "I was charged twice for the March invoice. No rush, but please refund the duplicate."}},
  "questions": {
    "bucket": {"type": "choice", "instructions": "Which queue owns this ticket?",
               "criteria": {"outage": "the product is down or unusable",
                            "billing": "charges, invoices, refunds",
                            "feature": "a request for something new"}},
    "urgent": {"type": "noul", "instructions": "Does the customer need a reply within the hour?"}
  }
}

The OpenAI API takes the same outputs as JSON text in input. A noul becomes a predicate question, and a choice becomes a choice question with the question id as its name:

{
  "model": "gpt-6-luna",
  "input": "{\"read\":{\"ticket\":\"I was charged twice for the March invoice. No rush, but please refund the duplicate.\"}}",
  "questions": [
    {"type": "choice", "name": "bucket", "instructions": "Which queue owns this ticket?",
     "choices": [{"value": "outage", "description": "the product is down or unusable"},
                 {"value": "billing", "description": "charges, invoices, refunds"},
                 {"value": "feature", "description": "a request for something new"}]},
    {"type": "predicate", "name": "urgent", "instructions": "Does the customer need a reply within the hour?"}
  ]
}

The label descriptions are the only definition of each queue that the model gets. Write them as you would write instructions for a person.

Run record. The output of the route is the decision. For each question, it holds the label, its confidence, and the full distribution that the model returned. The label is the most probable declared label. The engine ignores any separate choice or confidence value that the API returns.

{"bucket": {"label": "billing", "confidence": 0.9,
            "probabilities": {"billing": 0.9, "feature": 0.05, "outage": 0.05}},
 "urgent": {"label": "no", "confidence": 0.9, "probabilities": {"no": 0.9, "yes": 0.1}}}

If the probability of a label is less than min_confidence, the engine records "uncertain", and otherwise or drop must cover it. The engine also records "uncertain", with confidence 0 and an empty distribution, if OpenAI does not answer a question.

Control plane. A deployed controller binds the decision model from its provider registry. On the Providers page, add a default with role decision for the playbook class, for the platform or for one domain. The default names an OpenAI provider (which uses gpt-6-luna if the default names no model) or a custom provider with protocol decisions or system_one. Every playbook launch then gets this binding and the provider key, together with the binding of its agent (ADR-0065). A controller that you start with just controller-local does not read secret values. It passes its own CRUCIBLE_* variables to the runs that it launches. Export CRUCIBLE_INFERENCE before you start it, and give the key variable the same prefix ("key_env":"CRUCIBLE_DECISION_KEY").

examples/route is the runnable pack. It has five sample tickets and instructions to serve DiffusionGemma yourself.

Send work back

A task with revise = <task> or revise = [<task>, ...] is a reviewer. If it fails and rounds remain, the tasks that it names run again with the verdict. Then the reviewer runs again.

author = agent(
    name = "author",
    prompt = "Write PROBE.md, a probe that demonstrates the reported bug.",
    session = "author",
    emits_files = ["PROBE.md"],
)

review = command(
    name = "review",
    run = "./check.sh",
    depends_on = [author],
    emits_files = ["evidence/review.json"],
    revise = author,
    max_rounds = 3,
)

From the second round, the inputs of the author contain the last verdict of the reviewer under revision. The engine stages the declared files of the reviewer under inputs/review/:

{"round": 2, "max_rounds": 3, "reviewer": "review",
 "review": {"status": "fail", "note": "exit 1: ", "files": true,
            "output": {"accepted": false, "why": "the probe hit an unrelated 400"}}}

• Rounds have a limit. max_rounds is 2 to 5. It has no default.
• Each round settles. A passing round makes a commit. The engine discards a failing round. The session log records author[round-1], review[round-1], and so on. The verdict uses the rows of the last round.
• Sessions continue. An author with a session continues the same conversation in each round, so it knows what it already tried.
• Chains. revise = [pick, build] sends back a chain. Every listed task runs again in dependency order, and then the reviewer runs. You must list every task between a target and the reviewer, so that no task in a round reads a stale result.
• One loop per task. A loop cannot contain a fan-out. A task cannot be in two loops. Loops cannot be nested or chained.

examples/revise-loop runs a revise pair through a real OpenShell sandbox with a fake model.

Fan out

over = producer.field maps a task over a list that its dependency emits. There is one instance for each item:

scan = skill(
    name = "scan",
    skill = "skills/scan-issues",
    args = {"repo": param("repo")},
    emits = ["issues"],
)

triage = skill(
    name = "triage",
    skill = "skills/triage-issue",
    depends_on = [scan],
    over = scan.issues,
    max_fanout = 12,
    isolated = True,
    required = False,
    emits = ["classification", "severity"],
    emits_files = ["TRIAGE.md"],
)

roundup = command(
    name = "roundup",
    run = "python3 roundup.py",
    depends_on = [scan, triage],
    join = "passed",
    emits_files = ["REPORT.md"],
)

The item is the key of each instance. triage[123] is issue 123 in every retry and in every report row. An instance reads its own key from the reserved item input. max_fanout sets the maximum count, and the engine limit is 256. isolated = True runs the instances at the same time in private worktrees, at most eight at a time, and each one starts only while the run is under --max-cost. required = False lets the sweep finish when some instances fail. join = "passed" gives roundup only the instances that passed. examples/triage is the full pack.

If you know the list at compile time, write the fan-out in Starlark. It unrolls into normal tasks:

AUDITS = [("headings", True), ("bullets", True), ("freshness", False)]

def auditor(topic, blocking):
    return agent(
        name = "audit-" + topic,
        prompt = prompt_file("prompts/audit.md") + "\nAUDIT: " + topic.upper() + "\n",
        depends_on = [polish],
        isolated = True,
        required = blocking,
        emits = ["findings"],
    )

auditors = [auditor(topic, blocking) for topic, blocking in AUDITS]

Decide per item

A route can also map over a list. It asks one time for each item. A task mapped over the same list reads the answer for each item with when, so the turn for item k runs only if the answer for item k selects it:

prepare = command(name = "prepare", run = "./stage.sh",
                  emits = {"members": "list", "facts": "object"})

gate = route(
    name = "gate",
    depends_on = [prepare],
    over = prepare.members,
    max_fanout = 120,
    keyed = [prepare.facts],
    min_confidence = 0.8,
    questions = {"depth": choice(
        ask = "How much work does this ticket need?",
        options = {"settled": "the staged facts already answer it",
                   "quick": "a short look", "deep": "a full investigation"},
        drop = ["settled"],
    )},
)

quick = agent(name = "quick", prompt = prompt_file("analyze.md"), effort = "low",
              depends_on = [prepare, gate], over = prepare.members, max_fanout = 120,
              keyed = [prepare.facts], when = gate.depth, answers = "quick")
deep = agent(name = "deep", prompt = prompt_file("analyze.md"), effort = "high",
             depends_on = [prepare, gate], over = prepare.members, max_fanout = 120,
             keyed = [prepare.facts], when = gate.depth, otherwise = True)

• Items pair by key. Tasks mapped over the same producer.field are aligned. quick[T-1] reads gate[T-1]. An aligned dependency gives an instance only its instance for the same key, not the whole fan-out.
• Items that are not taken stay visible. quick[T-2] settles not_taken, with the answer in its note. It does not fail the fan-out. The fan-out output counts it under not_taken. An aligned task that joins all on it is also not taken.
• One request for each item. max_fanout limits the number of requests. A resumed run keeps the decisions that it has and asks only for the others.
• keyed limits the input. prepare.facts is an object keyed by item. Each instance, route or task, gets only its own entry under facts. A fan-out of one hundred items does not send the facts of every item to every turn.
• Judge a result file. files = ["RESULT.json"] puts the declared JSON file of a dependency in the model state under files. A route mapped after the analysis can ask if the evidence supports the verdict, and use when to send uncertain items to a review lane.
• Deterministic route first. A route mapped over the list can read each answer from an output object keyed by item ({"reachable": {"T-1": "no"}}). Put that route first, and give the model route a when on it. The model then gets questions only about the items that the facts do not decide.

Choose among an item's own options

A choice may take its options from a dependency instead of the pack. options = prepare.packages reads the field at each decision: the list itself on an unmapped route, or, on a route with over, the item's entry in an object keyed by item.

match = route(
    name = "match",
    depends_on = [prepare],
    over = prepare.members,
    max_fanout = 120,
    min_confidence = 0.7,
    questions = {"package": choice(ask = "Which shipped package does the flaw affect?",
                                   options = prepare.packages)},
)
analyze = agent(name = "analyze", prompt = prompt_file("analyze.md"),
                depends_on = [prepare, match], over = prepare.members, max_fanout = 120,
                when = match.package, otherwise = True)
review = agent(name = "review", prompt = prompt_file("review.md"),
               depends_on = [prepare, match], over = prepare.members, max_fanout = 120,
               when = match.package, answers = "uncertain")

prepare emits {"packages": {"T-1": [{"value": "p3", "description": "golang.org/x/net 0.17.0"}, {"value": "none", "description": "No shipped package matches"}]}}.

• Labels are identifiers. Put raw text such as a package name in description and a short key in value. An option every item should have, like none, is the producer's to emit.
• Bounded. At most 64 options; a repeated label keeps its first entry. An item whose list is malformed or holds fewer than two options fails without a request, and the rest decide.
• Branch on uncertain, read the pick. The labels are unknown when the pack compiles, so a when lists only "uncertain", and otherwise = True covers every option. The task it gates reads which option was chosen, and the options it was chosen among, from the route's output.

Keep a conversation

Agent tasks that share a session continue one conversation in dependency order:

scribe = session(name = "scribe")

draft = agent(name = "draft", prompt = prompt_file("prompts/draft.md"), session = scribe, emits = ["entries"])
shape = command(name = "shape", run = "./shape.sh", depends_on = [draft])
polish = agent(name = "polish", prompt = prompt_file("prompts/polish.md"), session = scribe, depends_on = [shape])

polish continues from where draft stopped. Each pair of tasks in a session needs a dependency path between them. A session task cannot be isolated. session() can also set the default harness, model, and effort for its tasks.

shape, between the two turns, shows a second pattern: a command as a gate. It reads the declared entries of draft from CRUCIBLE_INPUTS, counts the files on disk, and exits nonzero if the numbers are different. A bad draft does not get a second, paid turn.

Joins

The join of a task sets what it needs from its dependencies before it runs:

Use passed to join branches and to collect fan-outs. Use settled for a task that must see failures, for example a summary that reports which tasks failed.

Advisory tasks

required = False makes a task advisory. If it fails, its dependents are blocked, but the run can still be valid. Use it for useful checks that must not fail the result: a freshness audit, a slow benchmark, one instance of a sweep.

A required task can be unable to run on the substrate, because the substrate does not declare its needs capability. In that case, the engine truncates the whole plan before it dispatches any task. Without a required task, the graph cannot produce a valid pass, so the engine does not start it.

Report the result

report() is a task that the engine owns. It publishes a rendered template to a destination that the controller configures:

publish = report(
    name = "publish-report",
    destination = {"kind": "slack"},
    template = "reports/slack.md.j2",
    result = roundup,
    required = True,
)

The workflow names a destination key. It does not name a URL, channel, or credential. result puts only the declared fields of that task into the message. An agent cannot skip the call or write the payload.

The template is a Jinja file in the pack. The engine renders it into the card body. The template can read these values:

A field that the selected task did not declare is not defined. Guard optional reads with is defined. The engine escapes every inserted value for Slack, also with |safe. A result that holds <!channel> shows as that literal text. If the body is larger than CRUCIBLE_REPORT_BODY_MAX_BYTES (default and maximum 3000), the report task fails.

{% if result.output is defined %}{{ result.output.summary }}{% else %}roundup {{ result.status }}{% endif %}

severity_field names one declared field of result. Its value sets the accent color of the card:

publish = report(
    name = "publish-report",
    destination = {"kind": "slack"},
    template = "reports/slack.md.j2",
    result = roundup,
    severity_field = "severity",
)

"good", "warning", and "danger" set the accent for any verdict. The card shows the verdict on a separate line. Any other value gives a neutral accent. A selected task that was skipped or not taken also gives a neutral accent. A selected task that failed, had a transport failure, or was blocked gives "danger". The field sets only the accent, not part of the payload. It is a compile error to set severity_field without result, or to name a field that result does not declare in emits.

Where to look next

Workflow DSL reference

The dialect is Starlark with assignments, if, for, comprehensions, def, lambda, load(), at the top level or inside a def. Every one of them runs at compile time: the compiled plan is a static graph, so a loop in the source unrolls into tasks rather than becoming a cycle.

prompt_file(), schema_file(), and load() are the only file access, all confined below the pack directory, and a loaded module cannot re-export what it loaded. The surface has no processes, network access, clock, or randomness.

For what the engine does with the compiled graph, see Work graphs; for the normative rules, see the implementation contract.

Every lane

Available in every workflow type, playbooks included.

agent()

An agent turn driven by a prompt.

skill()

An agent turn whose prompt is a skill's instructions plus its arguments.

command()

A deterministic shell task in the candidate workspace.

evaluate()

A measurement command. Its last non-empty stdout line is a JSON object; pass = false vetoes the result and numeric score feeds grade() and top_k().

report()

Publish a rendered template to a controller-configured destination. The workflow selects a destination key, never an endpoint or a credential.

session()

Declare a durable agent conversation. Tasks that share one run serially under one agent config, across dependency order and loop iterations.

param()

Read a launch parameter. The params block must be the source's first statement, and a source that declares one compiles per run. A supplied value may reach a prompt, with + and inside a region marked as external input, or a skill argument; str(), %, .format(), and string methods on it are refused, as is a value carrying the marker text. A command or evaluate task reads it from params in the inputs JSON at CRUCIBLE_INPUTS_FILE instead of its command line.

Takes one positional argument, name.

prompt_file()

Embed a UTF-8 file below the pack directory. Absolute paths, .., symlinks, non-files, and oversized inputs are refused.

Takes one positional argument, path.

schema_file()

Read a JSON Schema (draft 2020-12) below the pack directory, for a field type in emits or a file in emits_files. Its content, not its path, is compiled into the plan. Paths are confined as for prompt_file; a file that is not a valid 2020-12 schema, declares another $schema, or references a remote $ref is refused.

Takes one positional argument, path.

workflow()

The source's final expression: the lane, the tasks that ship, and the result. A task constructed but not listed is a compile error.

Takes one positional argument, tasks.

Scored lanes only

Available to type = "autoresearch" and type = "custom". A playbook does not have these in scope at all, so naming one is an unknown-name error and a did-you-mean never offers one.

propose()

The loop's candidate-producing agent turn.

apply()

Make the candidate live through the configured world. A failure means unscoreable, not worse.

measure()

Run the manifest's frozen judge as one opaque measurement task.

grade()

Fold evaluation evidence into a measurement. Evidence includes tasks that failed or never ran, which is what the score source alone cannot see.

decide()

Apply the engine's keep-or-discard rule to a measurement. An autoresearch workflow must end here.

top_k()

Engine-owned reducer: the best k dependency outputs by numeric score.

default_autoresearch()

Expand the built-in propose/apply/measure/decide loop into visible nodes, plus the tasks passed to it.

Takes one positional argument, extra_tasks.

Playbook and custom lanes only

Available to type = "playbook" and type = "custom". An autoresearch workflow keeps or discards on a frozen measure, so it has neither these nor when.

route()

Engine-owned decision: answers typed questions about its dependencies' outputs and records one label per question, which other tasks branch on with when. Every label of a question some when refers to, "uncertain" included, must be listed by a when or by the question's drop.

noul()

A yes/no question for route(). It answers "yes" or "no".

choice()

A one-of-N question for route().

pick()

A question for a human = True route whose options are a list a dependency produced. The route's output holds the picked values under the question id, so over = gate.question fans out over them. No when routes on it.

score()

An ordinal question for route(). It answers its most probable level, the lowest on a tie, and records a score in [0, 1]: the expected level position over the normalized distribution, 0 for the lowest level and 1 for the highest. A source answers one level label, scored by its position, and "uncertain" records no score.

optional()

Mark an entry of depends_on that may settle not_taken. Under join = "all" the task dispatches past it as though it passed, and its inputs carry no entry for it; one that failed or was blocked still blocks. Refused under another join, on a dependency that can never settle not_taken, and on a task the reviewer revises.

Takes one positional argument, task.

Reserved fields

Names the engine reads and writes for itself. They are not constructor arguments; they appear in a task's own JSON output and in the inputs it receives.

A task's own JSON output

Read out of the object the task returns.

Inputs the engine writes

Present alongside the dependency entries, never wrapped in one.

Work graphs

A plan is a versioned DAG of tasks that a deterministic executor runs. A task is an agent turn, a command that the plan defines, or a reducer that the engine owns. The executor advances the plan. A task does not select the next task.

The engine supplies a default loop graph and a wide-tournament template. A person or a pack can supply a plan as TOML or JSON through the plan CLI. Workflow admission keeps the graph shape separate from authority. An orchestrator must declare each workflow type and each engine operation that it can execute safely.

Running a plan

# compile and print it, without executing
crucible plan show --file plan.toml
crucible plan show --file plan.toml --mermaid      # flowchart source
crucible plan show --file plan.toml --render       # PNG, inline if the terminal supports it

# execute: command tasks run as subprocesses, agent tasks through the real harness
crucible plan run --file plan.toml --manifest crucible.toml

# execute without a manifest: agent tasks run a stand-in command instead
crucible plan run --file plan.toml --agent-cmd ./role.sh

# replace the manifest's [agent] harness and model for this run; a task that pins its own keeps it
crucible plan run --manifest crucible.toml --harness codex --model gpt-5.6-luna

--cap <name> (repeatable) declares a capability of the substrate. See needs below.

plan run exits with a nonzero status if the plan does not reach a valid verdict.

File format

version = 1                 # the only format version accepted today
# reason = "..."            # reserved for a future replan protocol

[budget]
usd = 5.0                   # required, positive; execution fails closed on overrun

[params]                    # optional; command and evaluate tasks read it as inputs["params"]
topic = "slag"

[[task]]
name = "propose"            # unique within the plan
kind = "agent"
prompt = "..."
model = "claude-opus-4-6"   # optional per-task overrides of the manifest's [agent]
harness = "claude"
effort = "high"
session = "solver"         # optional durable logical conversation

[[task]]
name = "measure"
kind = "command"
command = "./bench.sh"
depends_on = ["propose"]
needs = "gpu"               # default "any"
required = true             # default true
isolation = "worktree"      # optional
join = "all"                # default "all"
emits = ["score"]           # optional declared output fields; absent = undeclared
                            # or typed: emits = { score = "number", tier = ["high", "low"] }
timeout = "20m"             # optional per-attempt limit; agent, command, and evaluate only

[[task]]
name = "pick"
kind = "top_k"
k = 1
direction = "lower"         # or "higher"
depends_on = ["measure"]

Task kinds

A workflow can declare and order engine nodes, but that does not give it authority to run them. Admission needs the matching capabilities, for example workflow.autoresearch, engine.apply, and engine.measure. The generic plan runner rejects engine nodes because it has no World and no frozen Judge.

A command string alone is not a trust boundary. If a command runs a script that must stay trusted after an agent task edits the workspace, declare that script as a frozen [[workspace.inject]] in the manifest. Before each task, the plan runner restores frozen injects in the workspace of the task. This includes isolated worktrees.

A logical session is serial state, so each pair of tasks that share a session must have a dependency path between them. A session task cannot use disposable worktree isolation. The private ledger keeps only an opaque harness cursor and the count of completed turns. The engine does not copy that cursor or the native Claude transcript into the plan or the public session log. Normal streamed harness events keep their usual visibility. The native Claude Code transcript stays in its private local store, or in a mode-0600 engine store that the engine restores into each new OpenShell sandbox. If you omit session, each turn starts a new conversation.

Task output

The output of a task is JSON. It becomes the input of the task's dependents.

• command: the last non-empty stdout line. A nonzero exit status is a measured failure. A spawn failure is a transport failure. Upstream outputs arrive as a JSON object keyed by task name, in the file that CRUCIBLE_INPUTS_FILE names. CRUCIBLE_TASK gives the task name. The same JSON is also in CRUCIBLE_INPUTS if it is 64 KiB or smaller. If it is larger, the variable is not set, because Linux rejects an environment string larger than 128 KiB. Read the file. In an edition-2 pack (edition = 2 at the top of crucible.toml), the command gets besides these only PATH, HOME, LANG, TMPDIR, the run's identity (CRUCIBLE_RUN_NAME, CRUCIBLE_RUN_DISPLAY_NAME, CRUCIBLE_UI_BASE_URL), and the variables [workflow].env names in the manifest (env = ["GH_TOKEN", "GPUS?"]). A named variable the engine does not have refuses the run, unless its name ends in ?: an optional variable is passed when the engine has it and left out when it does not. In an edition-1 pack the command inherits the engine's whole environment, and [workflow].env fails the parse.
• agent with --manifest: the turn writes one JSON object to PLAN_TASK_RESULT.json in the workspace root. If the file is missing after a normal turn, this is a measured failure. An explicit spawn, harness, or stream error is a transport failure and follows the retry policy.
• agent with --agent-cmd: the stand-in gets CRUCIBLE_PROMPT, CRUCIBLE_HARNESS, CRUCIBLE_MODEL, and CRUCIBLE_EFFORT. It returns JSON on its last stdout line.
• evaluate: needs a JSON object. pass = false fails the task, and a malformed pass also fails it. A threshold with a direction compares the numeric score (lower is <=, higher is >=). Without a threshold, a successful command passes unless pass is false.

The controller keeps each attempt's output with the run's task results, and a failing attempt's too, such as an evaluate's verdict with pass = false. crux graph <run> prints what each task's latest attempt emitted, crux graph <run> --json (GET /api/runs/{run_id}/graph) returns every attempt's output, and the run view's task panel shows it as the task's payload.

top_k reads a finite numeric score from each input, so an upstream task that ranks must emit a score. You can declare this: emits = ["score"] on an agent, command, or evaluate task names the fields that its JSON output must contain. Before the run starts, validation rejects these tasks if their declared emits do not include score: a top_k dependency, a grade score source or tiebreak, and a thresholded evaluate. At run time, if a passing attempt does not have a declared field, the engine changes it to a measured failure at the producing task. The engine does not retry it, and it blocks dependents. The error appears at the task that caused it, not downstream. An empty or absent emits declares nothing, and the engine does not check it. top_k and engine tasks cannot declare emits, because the engine defines their outputs.

emits can also declare the type of each field, as a dict from field name to type:

scan = command(
    name = "scan",
    run = "./scan.sh",
    emits = {"targets": "list", "count": "integer", "severity": ["high", "low"]},
)

A type is "string", "integer" (a number with no fractional part), "number", "boolean", "list", "object", or a list of labels. A list of labels declares a string that is equal to one of the labels. Labels are identifiers, as route labels are. The list form of emits is still valid, and it declares only that the field is present.

At run time, if a passing attempt has a field of the wrong type, or a string that is not one of its labels, it is a measured failure at the producing task. This is the same as a missing field. The note names the field, the value type that arrived, and the declared type. A task that settled itself as skipped or failed does not have to supply its fields.

At compile time, before any spend, the compiler checks the types at each place where the graph reads a field:

• over = scan.targets needs a field declared "list". The error points to the over argument.
• A top_k dependency, a grade score source or tiebreak, and a thresholded evaluate need score declared "number" or "integer".
• route(source = scan, ...) needs the field of each question declared as labels that the question answers (its options, or yes/no, plus uncertain), or as "boolean" for a noul. The compiler rejects "string", because it does not limit which label arrives.

The compiler does not check a field declared without a type (list form) or a task without emits. The generated TOML holds the types as a table (emits = { score = "number" }). The plan_admitted event lists each field with its type.

Schema types

schema_file(path) reads a JSON Schema (draft 2020-12) from the pack. You can use it as a type in emits and in the dict form of emits_files:

plan = command(
    name = "plan",
    run = "./plan.sh",
    emits = {"lanes": schema_file("crucible/schemas/lanes.json"), "tickets": "integer"},
    emits_files = {"RESULT.json": schema_file("crucible/schemas/result.json"), "REPORT.md": None},
)

The compiler puts the schema content into the plan, so a change to the schema changes the plan digest. These are compile errors at the call, and the compiler fetches nothing: a file that is not a valid 2020-12 schema, a file that declares a different $schema, and a file that references a remote $ref. At run time, the producing task fails, as for a wrong type, in these cases: the schema rejects a field of a passing output, or a declared file is not JSON or does not match. The note gives the first three errors as instance path and message: output field "lanes" does not match its schema: /1: value is not of type "string". Notes do not repeat the rejected value.

Each consumer checks the schema exactly. It reads type, const, enum, local $ref, allOf, and every branch of anyOf/oneOf. It rejects a schema if it cannot prove the requirement:

• over needs "type": "array" whose items (and any prefixItems) are strings that the schema can prove: "type": "string", or a const/enum of strings. The item names a mapped instance. For this reason, {"type": "array"} without items, or "items": {"type": "integer"}, is a compile error at the over argument.
• A score needs a numeric type, and its const/enum must contain only numbers.
• A route(source = ...) question needs every value that the schema limits (const, enum, or "type": "boolean", which answers yes/no) to be a label that the question accepts. The compiler rejects a schema that limits no value, as it rejects "string".

A route source can limit a question to a finite set of answers, with a label list, "boolean", or a schema. In that case, a when that names a label that the source cannot give is a compile error ("can never be answered"). Such a label needs no task, and otherwise covers only the labels that the source can give. A schema without one top-level type does not satisfy any consumer. On the wire, a schema is its JSON text ({ schema = "..." } in the TOML, { path = "RESULT.json", schema = "..." } for a file), because TOML cannot hold the null values that schemas often contain.

Repair

An agent task can repair its own output before the engine fails it:

plan = agent(
    name = "plan",
    prompt = prompt_file("prompts/plan.md"),
    emits = {"lanes": schema_file("crucible/schemas/lanes.json")},
    emits_files = {"RESULT.json": schema_file("crucible/schemas/result.json")},
    repair = 2,
)

A turn can pass but have output that misses a declared field or file, or does not match a type or schema. In that case, the engine continues the same conversation with the validation notes (masked as above). It tells the agent to repair the output and files in place, and then checks again. It does this up to repair times (at most 3).

Repairs run in the timeout of the attempt, and their cost counts against the run budget. A repair does not start if the attempt has already spent the remaining budget. Repairs are not retries: attempts stays 1. When no repairs remain, the task fails with the same note as without repairs. Each mapped instance repairs separately. A task without a session gets a conversation for the attempt. A task in a session continues that session. command and evaluate do not accept repair.

The task_result event of the task records each repair under repairs, as {label, round, of, cost_usd, notes}, with labels such as audit[RHAI-1] repair 1/2.

Execution semantics

Plan execution states is a generated diagram of how the executor walks a graph, the states of one task, and how the plan ends.

Readiness. A task dispatches when its dependencies are terminal and its join is satisfied. Dispatch follows declaration order, so the event stream is deterministic.

Truncation fails closed. If a required task cannot run on this substrate (its needs is not in the declared caps, or a dependency cannot run), the engine truncates the whole plan and dispatches no task. A truncated DAG cannot produce a valid pass. If an advisory task is in the same position, the engine skips it and its dependents, and validity does not change.

Failure. If a required task fails, the plan stops. All tasks that did not dispatch are blocked. An advisory failure blocks only its dependents.

Early completion. In a playbook, a passing main-graph command or evaluate task can return "complete": true, with an optional "reason" string. This ends the run. No other main-graph task dispatches. Tasks that did not dispatch stay unsettled, not blocked. Epilogue tasks still run. The run is valid unless a required task already settled with a status other than pass. The shutdown outcome is complete, with the reason. Both fields are reserved, so emits cannot list them. A revised command that completes ends its loop without another review. A mapped instance that completes ends the run after its node folds.

An agent task, or any other task whose output a model writes, cannot end the run this way. If it returns "complete": true, the engine refuses it: the task keeps its own outcome, its row's note records the refusal, and the run continues. Put the "nothing to do" decision in a command or evaluate task that reads the agent's output.

Retries apply only to transport failures. Transport failures retry, up to a limit (2 by default). A measured failure does not run again.

Budget. The cost of an attempt is known only after the attempt completes. For this reason, an attempt in progress can report a total larger than budget.usd. Any overrun makes the plan invalid and blocks all further dispatches and retries. A spend equal to the budget is valid only if no other retry or task is necessary.

Time. Elapsed time is known at all times, so it limits each attempt while the attempt runs. Each attempt has a deadline: the timeout of its task, counted from the start of the attempt, or the --max-time limit of the run if that is earlier. A task without timeout runs only under the run limit. At the deadline, the runner stops the whole process group of the attempt (a command and all processes that it started, or the agent turn). The task settles as a measured failure with a note that names the limit: timed out: the task ran past its 20m limit. The engine does not retry it, because a retry would use the same time again. A required task that times out stops the plan, as any other failure does. If the run limit ends an attempt, the run ends on that limit, and all tasks that did not dispatch settle as blocked. A playbook rejects a timeout longer than --max-time before it dispatches a task. In a concurrent batch, each task has its own deadline.

needs

The substrate capability that a task needs. "any" runs on all substrates. You must declare any other value with --cap. If you do not, the task cannot run. plan show reports the truncation verdict before any spend.

isolation

isolation = "worktree" gives the task a private clone of the workspace, with its uncommitted state. This has two effects:

• Isolated tasks that are ready at the same time run concurrently. Without isolation, they would all write the one PLAN_TASK_RESULT.json in the shared workspace.
• The engine discards the edits of the task. Only the declared output leaves the task. Use isolation for review and analysis. Do not use it for a task whose diff must be kept.

A runner that cannot isolate rejects the task. It does not run the task in the shared workspace.

join

join = "all" (default) needs every dependency to pass, except that a dependency marked optional(...) may also have settled not_taken. join = "passed" waits for every dependency, then collects the passing dependencies. There must be at least one. If no dependency can run or pass, the task fails.

revise

A reviewer sends a failing verdict back to the tasks that it names, for a limited number of rounds. Only playbooks support revise. It is the only repetition that the graph declares.

author = agent(
    name = "author",
    prompt = "Write PROBE.md, a probe that demonstrates the reported bug.",
    session = "author",
    emits_files = ["PROBE.md"],
)

review = command(
    name = "review",
    run = "./check.sh",
    depends_on = [author],
    emits_files = ["evidence/review.json"],
    revise = author,
    max_rounds = 3,
)

review runs against the draft as any dependent does. If it fails and rounds remain, author runs again, and then review runs again. This continues until review passes or the third round ends. Any other outcome of the reviewer ends the loop. The loop also ends if a revision fails and the reviewer is blocked. The engine checks the limits before each round.

revise also accepts a list, for a fix that needs more than one task to run again:

confirm = evaluate(
    name = "confirm",
    run = "python3 tools/probe.py --rung confirm",
    depends_on = [rig],
    revise = [pick, build, rig],
    max_rounds = 3,
)

The targets and the reviewer are the body of the loop. Each round runs the body one task at a time in dependency order. Each task uses its own join and when. If build fails, it blocks a reviewer with join = "all", and the loop ends. A reviewer with join = "settled" sees the failure and can send the chain back. The loop starts after every task outside the body that a body task reads has settled. A task outside the body that reads a body task waits until the loop ends.

From the second round, the inputs of each target contain the reserved revision key:

{"round": 2, "max_rounds": 3, "reviewer": "review",
 "review": {"status": "fail", "note": "exit 1: ", "files": true,
            "output": {"accepted": false, "why": "the probe hit an unrelated 400"}}}

files tells if the engine staged the declared files of the reviewer from that failing round. The engine stages them under inputs/<reviewer>/, where a join = "settled" consumer also finds them. The reviewer does not get revision. A body task that declares a session continues it in each round, so it knows what it already tried.

Each round settles separately. A passing round makes a commit, and the engine discards a failing round. Each round reports as task[round-N], the same naming that the instances of a mapped node use. When the loop ends, each task reports one row under its own name. The row holds its last round and the spend of all rounds. Only these rows decide the verdict. A dependent reads the last round.

The limit is 2 to 5, and it has no default. Validation rejects:

• a target that the reviewer does not depend on, directly or through other tasks;
• a body that omits a task on a path between two of its tasks (that task would read a result that a later round replaces);
• a body task that is not an agent, command, or evaluate task, or that fans out;
• a task in two bodies;
• a reviewer that is also a target.

If the number of repetitions depends on what a task finds, put that repetition inside one task.

The loop as a plan

Each loop iteration runs as an autoresearch workflow that capabilities admit. Without an authored workflow, the default workflow expands to normal tasks:

flowchart LR
    propose["propose (engine)"] --> apply["apply"] --> measure["measure"] --> decide["decide"]

The author defines the task names and the tasks between the engine operations. The result of an autoresearch workflow must be a decision that a frozen measurement feeds, with apply and proposal ancestors. A custom workflow has no shape requirement. The outer orchestrator admits it only if it declares workflow.custom.

[workflow]
type = "autoresearch"
result = "keep-if-better"

[[workflow.task]]
name = "invent"
kind = "engine"
op = "propose"
session = "solver"

[[workflow.task]]
name = "review"
kind = "command"
command = "./review.sh"
depends_on = ["invent"]

[[workflow.task]]
name = "deploy-preview"
kind = "engine"
op = "apply"
depends_on = ["review"]

[[workflow.task]]
name = "benchmark"
kind = "engine"
op = "measure"
depends_on = ["deploy-preview"]

[[workflow.task]]
name = "keep-if-better"
kind = "engine"
op = "decide"
source = "benchmark"
depends_on = ["benchmark"]

To admit this workflow, the orchestrator needs workflow.autoresearch, engine.propose, engine.apply, engine.measure, and engine.decide. Because the workflow binds solver, it also needs agent.session.persist. A custom orchestrator can admit type = "custom" and any subset of the operations that it implements. Task-level needs still controls where an admitted task can run. Workflow capabilities control which operations the orchestrator is authorized to run.

Each iteration writes plan_admitted and task_result lines to the session log, in addition to the decision and session events. The loop driver keeps cross-round state, keep/discard, and all controls between rounds (parking, steering, re-scoping, budget). The wide round runs as a template that the engine compiles from [search].

The templates have no budget of their own. The driver owns the run budget and checks it between rounds, so the engine still measures and decides a turn that goes over the limit.

Authored measurement subgraphs

You can still use measure(). To make the measurement steps visible in the graph, use evaluate() and grade(). Dependencies define the rungs, isolated peers can run concurrently, and grade() selects the score source for decide().

candidate = propose(name = "invent")
live = apply(name = "deploy", depends_on = [candidate])

refcheck = evaluate(
    name = "cpu-reference",
    run = "./refcheck.sh",
    depends_on = [live],
)
diff = evaluate(
    name = "single-gpu-diff",
    run = "./diff.sh",
    depends_on = [refcheck],
    threshold = 0.001,
    direction = "lower",
    needs = "gpu",
    isolated = True,
)
latency = evaluate(
    name = "latency",
    run = "./latency.sh",
    depends_on = [diff],
    direction = "lower",
    threshold = 12.5,
    isolated = True,
)
racecheck = evaluate(
    name = "racecheck",
    run = "./racecheck.sh",
    depends_on = [diff],
    required = False,
    isolated = True,
)
measurement = grade(
    name = "final-grade",
    evidence = [diff, latency, racecheck],
    score = latency,
)
decision = decide(name = "choose", measurement = measurement)
workflow(
    type = "autoresearch",
    tasks = [candidate, live, refcheck, diff, latency, racecheck, measurement, decision],
    result = decision,
)

The renderer groups evaluate and grade as Measurement and keeps the edges. During crucible scope, validation writes the admitted graph to WORKFLOW.png. Agents cannot replace this file.

Run-scoped epilogue tasks

stage = "epilogue" on a [[workflow.task]] removes the task from the graph of each iteration. In Starlark, set stage = "epilogue" on agent(), command(), or evaluate(). The epilogue subgraph runs one time, after the loop ends cleanly (finished, budget, or solved). It runs against the final kept candidate, and only if the run kept a candidate. Use it for slow checks, for example a 90-minute compute-sanitizer racecheck or a slow performance benchmark.

The CRUCIBLE_INPUTS of each task holds the kept candidate under the reserved kept key: {"iter", "score", "tiebreak", "sha", "snapshot", "note"}. Dependencies cannot cross stages. Engine operations cannot be in the epilogue. The workflow result must be in the iteration stage.

In a playbook, the epilogue runs after the main graph completes or fails. It reads the reserved outcome input: {"exit", "tasks": {name: {"status", "note", "output", "files"}}}. Each entry is the same as the value that a join = "settled" consumer gets, with per_instance. The engine stages the declared files of each main-graph task under inputs/<name>/ (for a mapped node, under inputs/<node>[<key>]/, where %, /, \, NUL, and a leading . in the key are written %25, %2F, %5C, %00, and %2E, so llm-d/router stages under inputs/<node>[llm-d%2Frouter]/). It stages files from failed tasks and from passing tasks. files tells if the files of that task are present. A skipped, blocked, or transport-failed task stages no files.

Epilogue results are advisory. They cannot undo the keep of the candidate. The engine writes their rows to the session log and RESULTS.md (epilogue / epilogue-skip / epilogue-fail). The PR body gets an "Epilogue checks (advisory)" section, with failures marked FAILED.

report(name = "publish-report", destination = {"kind": "slack"}, template = "reports/slack.md.j2", result = roundup, required = True) is the publication epilogue that the engine owns. A destination is a key that the engine knows, not a URL or a secret name. slack is the only destination. The optional result selector puts only the declared JSON fields of that main-graph task into a Block Kit card that the engine builds. The card does not show prompts, stdout, workspaces, undeclared fields, raw Slack blocks, channels, or credentials. The default limit for the selected output is 16 KiB encoded. An operator can set a different limit, up to 64 KiB, with CRUCIBLE_REPORT_RESULT_MAX_BYTES. CRUCIBLE_REPORT_BODY_MAX_BYTES limits the rendered template body (default and maximum 3000, the Slack section limit). Data that is too large fails. The engine does not truncate it. For a required report, a render or delivery failure fails the workflow. An agent does not have to call a tool to send the report.

Worked example

examples/revise-loop is the smallest revise loop. An agent writes a draft probe. A command rejects the first draft, which the agent wrote without a verdict. The revision, which the agent wrote with the verdict, passes. just revise-loop-e2e runs it in real OpenShell sandboxes on local podman with a claude image that has no model. It checks the rounds, the continued session, and the commits.

examples/adversarial-review puts a review task between a code node and the gate below it. It has a single-reviewer shape and a two-reviewer panel shape. The panel runs isolated reviewers concurrently and joins them on a policy gate. The correctness reviewer blocks, and the copy-edit reviewer is advisory. It runs at no cost against a stand-in manifest, or against real models with the live manifest.

Plan execution states

Generated from crucible/src/plan/machine.rs by crucible plan states; scripts/state-docs.sh --check keeps it current. The executor walks both tables at every decision, so an edge missing here is a path it cannot take. The graph itself, what the tasks are and how they depend on each other, is described in Work graphs; this page is how the executor walks one.

One task

A task is pending until its dependencies settle, runs (retrying transport-class failures up to the configured count), and settles on one of the six statuses the session log reports. Everything that reaches a settled state without an attempt is a blocked task, and the edge names why.

The plan

The plan dispatches ready tasks in topological order until every task has settled or a required task fails or a ceiling is reached. After a halt the remaining tasks settle as blocked; epilogue tasks still run after a required task fails so the failure is reported, and tasks that cannot spend still run after the cost ceiling so the run reports the work it paid for.

The sources are docs/img/plan-task-states.dot and docs/img/plan-states.dot (crucible plan states --format dot).

Why a task is blocked

The note on a blocked task's result is one of these.

How a plan ends

Example: build, run, measure, iterate

examples/optimize-loop is a playbook in which a sandboxed agent optimizes a program against real measurements on a cluster, and a verifier checks the result against evidence rather than the agent's word. It uses the two MCP servers a run can be granted with agent tokens (see Agent and people MCP architecture):

• buildit builds the agent's workspace into an image in a builder pod and pushes it to the run's push repository by digest;
• crucible-kueue submits each benchmark as a Kueue Job with fixed CPU and memory, waits on it, and reads its logs.

The agent measures a baseline, changes the slow function, rebuilds and reruns for up to rounds rounds, and runs two or three variants as parallel Jobs in at least one round. It reports each round's Job name, image digest and numbers.

How the result is checked

The loop pod, where the verifier runs, holds no kueue token and no rights in the workload namespace. A second agent task, collect, reads crucible-kueue for every Job this run created and writes the replies to a file; its prompt carries nothing the optimizer wrote. The verify evaluate task then checks the report against those replies, against the registry (read with the loop pod's push authfile), and against its own reference answer, computes the speedup itself, and passes only above min_speedup. The pack's README lists every check.

Where the numbers show up

verify prints its verdict as its result: pass, speedup (also its score), baseline and best (each a Job, its image and the ops/s the evidence gave), rounds (each round's Jobs with their measured ops/s and p50, or why one was rejected, and the round's best), and any failures. A failing verdict keeps everything it measured. The controller stores it as the task's output on the run's task results, so it outlives the loop pod:

• crux graph <run> prints it under the task table, and crux graph <run> --json returns each attempt's whole output;
• the run view's verify task panel shows it as the task's payload;
• just optimize-loop and just optimize-loop-e2e print it when the run ends, pass or fail.

Launch it

just optimize-loop-test
just optimize-loop <kube context> <controller namespace>            # CPU, 3 rounds, 5x
just optimize-loop <kube context> <controller namespace> true 2 3   # GPU, 2 rounds, 3x

The controller's profile needs [buildit] with a push_repo, [secrets] push_authfile, and [kueue] with that repository among its pull secrets. The GPU variant also needs a LocalQueue with NVIDIA GPU flavors.

How the loop works

This diagram shows the autoresearch lane. A goal goes in, and an agent proposes a change. The engine applies the change to a reversible world. A frozen judge measures it one time. The loop keeps the change only if it is strictly better than the best result so far. Git stores the accepted states, and the operator can steer the run. Privileged operations run on the host in a broker. The agent can only send requests to the broker.

flowchart TD
    issue["GitHub issue / Jira ticket"] -->|scope --issue| goal["Run goal<br/>(frozen objective)"]
    goal --> wide["wide round (optional)<br/>N parallel propose → rank → winner seeds"]
    wide --> propose

    subgraph control["Control plane (operator, human-in-the-loop)"]
        direction LR
        budget["budget"]
        steer["steer"]
        stoppark["stop / park"]
        resume["resume"]
        escalate["escalate"]
    end

    subgraph loop["The keep / discard loop"]
        direction TB
        propose["propose<br/>agent backend: local · openshell · command"]
        apply["apply<br/>World: edit tree, or build + set image"]
        measure["measure<br/>Judge: frozen, scores once"]
        accept{"accept?<br/>strictly better?"}
        remember["remember<br/>git commit + session NDJSON"]
        restore["restore<br/>World rollback to last good"]

        propose --> apply --> measure --> accept
        accept -->|keep| remember
        accept -->|discard| restore
        remember -->|next iteration| propose
        restore --> propose
    end

    control -. "steer / stop / resume / budget" .-> loop

    subgraph broker["MCP broker: the host keeps the credentials, the agent sends requests"]
        direction TB
        prov["provisioning<br/>GPU / Kueue, capture RBAC"]
        jira["issue-tracker grounding"]
        prof["profiler<br/>pprof / GPU traces"]
        forge["forge build + deploy<br/>buildah / native-OCI derive-layer, digest-pinned"]
        draft["draft PR per fork"]
    end

    propose -. "MCP ask" .-> broker
    forge -. "image + set image" .-> apply

    remember -->|publish-on-keep| s3[("S3 run records<br/>session.jsonl · summary.json · diffs")]
    remember --> draft
    draft -->|review comments re-steer| steer
    remember -->|evidence upload, pod token| ctl["control plane<br/>(crucible-controller)"]

Loop control states is a generated diagram of the controls around the loop: the gates between iterations and the ways a run ends.

The stages

Glossary

Some words have a specific meaning in the loop. World and Judge are the most important, because they define the trust boundary.

The loop in one read describes the same model in text.

The autoresearch loop, in one read

Autoresearch is the scored lane of Crucible. It is a keep/discard loop with a goal and a gate, in which an agent improves a codebase against a frozen objective. The loop does not depend on a domain. A domain is a problem packaged for the loop. For a graph that runs one time and ships, see Your first playbook.

This page describes the system in general concepts. Then it shows how the examples/counter domain maps to those concepts. ADR-0001 gives the reason that the judge is frozen.

Scope. The core is the engine, the World/Judge boundary, CommandWorld and CommandJudge, and the crucible.toml manifest. A composite manifest (ADR-0009) runs the engine across more than one component. Around the loop there are: provisioning and issue-tracker grounding through an MCP broker (ADR-0002), a profiler over MCP (ADR-0006), build and deploy in the engine (ADR-0005) with rendered deployment manifests (ADR-0011), and publish-on-keep with a draft PR for each fork. Authorized review comments on the PR steer the run again.

Summary

The loop is an optimization loop with these properties:

• The proposal step is an LLM agent.
• The objective is a frozen, expensive black-box evaluation.
• The world is state that the engine can change and restore.
• The loop has durable memory and human-in-the-loop control.

The loop

flowchart LR
    controls["budget / steer / stop / escalate"] -.-> propose["propose<br/>(agent)"]
    propose --> apply["apply<br/>(World)"]
    apply --> measure["measure<br/>(Judge)"]
    measure --> accept{"accept?<br/>keep or discard"}
    accept -->|"keep"| remember["remember<br/>(Git + session log)"]
    accept -->|"discard"| restore["restore<br/>(World)"]
    remember --> next["next iteration"]
    restore --> next
    next --> propose

• propose: the agent edits the world toward the goal. A backend supplies the proposal policy: local (the harness on the host), openshell (sandboxed), or command. [agent].backend selects it.
• apply: makes the candidate live. For a code repository, the edits are the apply step.
• measure: the frozen Judge scores the candidate.
• accept?: keeps the candidate if it is strictly better. Otherwise, restores the last good state.
• remember: kept states are Git commits. Each step is an NDJSON event.

By default, the loop has one candidate in each iteration. A wide round (ADR-0010) can first run N independent propose turns in parallel, each biased to a different [search].approaches entry. The same Judge ranks them, and the winner is the start point of the loop. With --wide 0 (the default), there is no wide round. The wide round is a step before the loop and does not change the engine.

The contract

Crucible defines a small contract. A repository implements it in any language. A domain is a manifest and some executables. It is not Rust code. Each command runs in the workspace:

What of the engine's environment measure, apply, snapshot, and restore get depends on the pack's edition, a top-level key of crucible.toml:

edition = 2

A manifest without edition is edition 1. Its commands inherit the engine's whole environment, credentials included, and crucible check and every run warn that the pack should move to edition 2. The capability disclosure says so per table: host-env [judge] commands inherit the engine's whole environment.

Under edition 2 the commands get a minimal environment, not the engine's: PATH, HOME, LANG, TMPDIR, the run's identity (CRUCIBLE_RUN_NAME, CRUCIBLE_RUN_DISPLAY_NAME, and CRUCIBLE_UI_BASE_URL, when the platform sets them), the variables the engine sets for that command (CRUCIBLE_BASELINE_*, CRUCIBLE_BEST_SCORE, CRUCIBLE_TOKEN), and the variables its table declares. The pod token and the controller URL it authenticates against are not part of the base. The agent can edit files a command runs, so nothing else of the engine's environment reaches it: no kubeconfig, registry authfile, broker token, or provider key unless you name it.

edition = 2

[judge]
measure_cmd = "./measure.sh"
direction = "lower"
env = ["KUBECONFIG", "EPP_ENDPOINT?"]        # measure_cmd only

[world]
apply_cmd = "./deploy.sh"
env = ["KUBECONFIG", "REGISTRY_AUTH_FILE"]   # apply, snapshot, and restore

[workflow].env does the same for a workflow's command and evaluate tasks. An env list in an edition-1 manifest fails the parse, and so does an edition this engine does not know. A name is a letter or underscore followed by letters, digits, and underscores. A required variable that the engine's environment does not set refuses the run before any of these commands runs, naming every missing one; crucible check reports it as a finding. A name with a trailing ? is optional: the command gets it when the engine's environment sets it and runs without it otherwise, so a script can keep a shell default (${EPP_ENDPOINT:-127.0.0.1:8000}) that a deployment overrides. Only the trailing ? is allowed (? alone, A??, and A?B fail the parse), and a name is either required or optional, not both. crucible check lists each table's declared variables in the capability disclosure as host-env, the optional ones marked (optional). setup runs with the engine's environment in every edition.

The accept policy is the same for all domains. The engine keeps a candidate if it is valid and its score is better than the best score in the direction. The run is solved only if measure reports it. A domain does not supply code for the keep logic.

Carrying pipeline artifacts across a discard

A discard resets the workspace and removes untracked files. Any files that an iteration generated for its candidate (code traces, a generated port tree) are gone in the next turn, and the agent must generate them again. [workspace] carry_forward names paths, relative to the workspace, that the discard keeps:

[workspace]
carry_forward = ["codegen-out/"]

The engine writes each entry to .git/info/exclude, and the clean step of the discard keeps it. Carried content does not go into a candidate diff, a snapshot commit, or a tree hash. If a turn only generates the carried content again, the engine sees no candidate change.

Limits:

• A path that the repository tracks is not protected. Git excludes apply only to untracked files, and git reset --hard in the discard reverts tracked content. Carry only pipeline output that the repository does not track.
• A path can be absent when the run starts. The exclude applies when the path appears.
• Nested paths (target/codegen-out) work. The clean step goes into the parent directory and does not delete it. Entries must be plain relative paths, without ., .., or a leading /.
• Wide-tournament rounds run in new worktrees without the carried directories, so the setting does not apply to them.
• Composite manifests reject the key. Carry-forward for each component is not supported.

If you omit the key, each iteration starts from a clean workspace. Use this default for a campaign whose method needs a clean start.

Declared artifacts: carried AND published

[[workspace.artifact]] is carry_forward with publishing. Use it for generated output that a reviewer must see, but that must not be in the candidate diff:

[[workspace.artifact]]
path   = "codegen-out/runtime"   # carried, same rules as carry_forward
embed  = "summary.txt"           # newest match → PR body (collapsed, 16 KiB cap) + record
upload = "summary.pptx"          # newest match → record only (binaries a PR can't inline)
title  = "Runtime investigation" # PR-body section heading (default: the path)

The newest match is used. If versioned pipeline directories (V1 … V7) each have a summary with the same name, the summary of the final iteration is published. The record destination accepts s3://bucket[/prefix] or file:///abs/path. The file form writes the same layout to a mounted PVC, for clusters that cannot reach S3. If two or more iterations have a measurement, the PR body also shows the score of each iteration as a mermaid xychart-beta chart.

Languages

The contract is JSON and exit codes, so you can write each command in any language:

• Small glue commands (apply, operations, steer, stop): we recommend nushell. These commands run kubectl or Prometheus, parse the output, and compute a value. Nushell works with structured data (... | from json | ... in, a record | to json out). Nushell is not a POSIX shell, so pin nu in the sandbox image.
• A large measure with real math (percentiles, replay): use a compiled binary (Rust or Go).
• Bash, Python, and other languages are also valid. Only the JSON and exit status format is fixed.

measure_cmd is any executable that prints one JSON object on stdout ({"valid": bool, "score": number, "note": string}). A plain POSIX shell script satisfies the full contract:

#!/bin/sh
p99=$(make bench | jq .p99_ms)
printf '{"valid": true, "score": %s, "note": "p99 %sms"}\n' "$p99" "$p99"

The engine reads only the JSON verdict, so all languages work the same way.

What the engine supplies

The control plane (crucible-controller) is optional. It finds candidate issues, scopes them into packs, launches runs, and shows run records as leaderboards. The controller links the engine. The engine binary does not link the controller, and everything on this page works without a controller.

Trust boundary (ADR-0001): the engine gives the agent a World but not a Judge. The objective is frozen, and the agent cannot reach it, so the agent cannot change the test that grades it. The same boundary applies to provisioning. The agent can only send requests to the broker over MCP. The privileges (GPU and Kueue, capture RBAC, the forge token, the issue-tracker credential) stay on the host and do not go into the sandbox of the agent.

[preflight]: prove the environment before spending an iteration

A rung that grades a candidate can fail for reasons that are not related to the candidate: a missing dev header, a pip install that hides the built wheel, a cache directory that is not writable. If the run finds these problems in iteration 1, it loses a full agent turn. [preflight] runs the same ladder against the unmodified tree first, with no agent cost:

[preflight]
commands = [
  "python3 tools/gate.py --only-rung 1 --mode {mode}",
  "python3 tools/gate.py --only-rung 2 --digest {digest} --limit 1",
]
baseline = "python3 tools/gate.py --only-rung 3 --digest {digest}"

• Each command runs with sh -c in the workspace. Its last non-empty stdout line must be a JSON object. The rung passes if the command exits with status 0 and pass is absent or true. The recognized keys are digest, score, tiebreak, note, and logs.
• {mode} runs the command once for each build mode in [measure.build].mutable_kwargs.mode, in declared order. A {mode} command without that list is a manifest load error. Without this rule, a preflight could pass in derive mode while full mode is broken.
• {digest} inserts the most recent digest that an earlier rung emitted, so the build rung gives its artifact to the correctness rung. If no rung produced a digest yet, the command fails.
• Any failure is an environment verdict. The run does not start. It logs the failed command, its note, and the end of its stderr as a preflight-failed row. The run uses no iteration.
• baseline is optional. Its score and tiebreak become segment.baseline_score. A skip_baseline (codegen) domain then decides iteration 1 against a real number, not against the worst-score sentinel of the direction. Without baseline, the sentinel is used.

The three extension points

• New domain: usually a manifest that points CommandWorld and CommandJudge at a repository, and a measure command. No Rust. Write a custom World or Judge only for in-process measurement or a world that is not files.
• Composite domain: combine N existing component domains into one run. It has a top-level [composite] table that references the component domain directories (used without change), a CompositeWorld (a tuple of Git overlays, one for each component, and the live deployment), and one combined gate over the full stack. The engine does not change. It runs a list of workspaces (ADR-0009).
• New frontend: implement Reporter.
• New agent transport: add an AgentSource.

The canonical example: the counter, mapped to the contract

examples/counter/ is the full system, with each large part replaced by the smallest part that satisfies the contract. Each of its five small files has a contract role:

There is no [world] block, so GitWorld supplies the restore: kept iterations are commits, and discards use git reset --hard. The backend is command, so the whole loop runs in milliseconds and costs nothing. It is still a real end-to-end run (manifest → setup → propose → measure → keep/discard → Git history).

A production domain fills the same roles with larger parts. A serving-stack domain has two reversible parts: the code tree and a live deployment. It adds snapshot_cmd and restore_cmd that capture the image and configuration together with the Git state. Its measure is a compiled benchmark binary. Its propose backend is local or openshell. Its glue commands (apply pipelines, deployment inspection, steer and stop) are on PATH as bare names, so the contract names are the same for every implementation. A normal repository needs none of this. It needs one measure command and the Git default.

Add a new domain

1. crucible init creates a starter crucible.toml and a measure script in the repository. You can also write the manifest by hand:

   [repo]      url|path, ref
   [workspace] setup_cmd          # optional; default: git clone + checkout
               carry_forward      # optional; untracked derived paths a discard keeps
               [[workspace.artifact]] path, embed?, title?   # carried + published to PR/S3
   [agent]     model, method_prompt, goal_file|goal, toolbox_dir, env
   [judge]     measure_cmd, direction = "lower"|"higher"
   [world]     apply_cmd?, snapshot_cmd?, restore_cmd?   # omitted → git
   

2. Write a measure command (any language) that prints {valid, score, solved?}.
3. crucible check --manifest crucible.toml validates the manifest without an agent turn. It checks that every referenced file resolves, and it runs measure_cmd one time to check the output format.
4. Run crucible --manifest crucible.toml. The engine supplies the loop, the budget, all frontends, steer, stop, and resume, the session log, and escalation.

Any other repository can run this way with no new Rust.

The task lane: no [judge]

Some work has no objective to score, for example "consolidate the open dependabot PRs" or "fix flaky tests every night". If you omit [judge], the run is a task. The loop runs as usual (sandbox, broker, session log, publish, resume), with the built-in TaskJudge, which keeps everything. There is no baseline measure and no score. The engine keeps and snapshots every completed turn. The run exits with status 0 when no iterations remain (ADR-0021).

[repo]
url = "https://github.com/you/repo"
[agent]
backend = "openshell"
goal = "Consolidate the open dependabot PRs into one branch; run the tests."
[publish]
pr_repo = "you/repo-fork"   # the deliverable: kept commits as one draft PR

On the wire, the run has the normal format, with gate: "task" as the discriminator: an iteration-0 baseline-skipped row, then keep rows with score: null, and Summary.best_score null. crucible check does not probe the measure. It requires a goal and prints a "task mode" notice. Notes:

• Exit status 0 means that the run completed. It does not mean that the task succeeded. Examine the rows or the PR.
• solved does not occur. Only the iteration limit ends the run (--no-early-stop has no effect).
• The trust boundary does not change. The agent has no credentials. The output is a draft PR. Privileged write actions (merge, close issues) stay in broker tools.
• Composites still need [judge]. [search], [workflow], and [preflight] also need it, and a task manifest rejects them at load. The scope pipeline rejects proposed packs without a judge, and a scored judge cannot use objective = "task".

examples/task/ is the runnable reference (deterministic command backend, no LLM). Tasks: the no-judge lane is the full guide.

Distress: the agent can page you

The broker has a distress tool. The agent uses it to report that a run cannot succeed. The required severity sets the result:

Use error for a run that cannot succeed without an operator: the same environment failure in many iterations, a missing capability, or a goal that is not possible as written. Do not use it for a task that is difficult but possible. The tool replies {"status":"suspending"}, and the agent finishes its turn. The suspend occurs at the start of the next iteration, after the engine records that turn, so no data is lost. More error calls during the suspend have no effect. If the engine cannot write the marker (a full or read-only volume), the tool returns an error, not suspending, and posts nothing. The run is not suspended, so the agent and the page must not report that it is.

The suspend is a pending approval. The loop opens the same approval_wait bracket that provisioning uses, so suspended wall-clock time goes to parked_total and does not count toward --max-time. The distressed iteration still counts, because its turn ran. The suspended time counts against no limit. --max-park limits the wait.

The signal is one file on the forge-storage volume of the loop pod: /var/lib/forge/distress. To resume the run, delete the file:

oc exec <pod> -n <ns> -- rm /var/lib/forge/distress   # resume in place

If the fix must be in place at pod start (image pins, the pack, resources), replace the pod. The marker is on an emptyDir, so the new pod starts without it. A resume that finds the open distress bracket does not park again. Info and warn notes go to /var/lib/forge/distress-notes.jsonl, and the engine adds them to the next row.

The render sets these variables on the loop pod. The broker inherits them, so the page can name the run: CRUCIBLE_RUN_NAME, CRUCIBLE_ITERATIONS, CRUCIBLE_LOOP_IMAGE, and the downward-API CRUCIBLE_POD_NAME and CRUCIBLE_POD_NAMESPACE. A pod that the controller dispatches also has CRUCIBLE_RUN_DISPLAY_NAME, the two-word name of the run (benevolent-monkey). The page and report cards show this name instead of the run id. Links still use the id. Slack supports only webhooks. Set SLACK_WEBHOOK_URL to an incoming webhook through the [[secret_env]] of the profile. Set DATADOG_BASE_URL if your Datadog site is not app.datadoghq.com. Without a webhook, the suspend still occurs. The engine sends the page one time and does not retry.

crucible flow: explain a finished run

crucible flow --session state/session.jsonl --out flow.html converts a finished run into a small flow-model IR and renders it. The --out extension selects the format: .json (the IR), .dot, .mmd, or .html. The HTML format is a self-contained page with a header, a swimlane of iterations (agent turn and check ladder in each column, scores against the baseline), detail for each iteration, and the draft-PR footer. The optional --spans <dd-export.json> (Datadog APM) adds real wall-clock time: columns in proportion to time, check durations, and the tool-call timeline of the agent.

Concept to code

• loop, budget, keep/discard: crucible/src/runloop/driver.rs, crucible/src/runloop/graph.rs
• the contract traits: crucible/src/crucible.rs
• World and Judge implementations: crucible/src/command_world.rs (GitWorld, CommandWorld, CompositeWorld), crucible/src/command_judge.rs (CommandJudge)
• composite domains: crucible/src/manifest/mod.rs (CompositeManifest), crucible/src/cli/scored.rs (run_composite)
• broker (provisioning, issue tracker, profiler, draft PR): crucible-broker/
• distress (suspend from the agent, and the Slack page): crucible-broker/src/distress.rs, crucible/src/control/distress.rs
• build and rendered deploy in the engine: forge/ (build, native-OCI derive, kube-rs), crucible/src/deploy/
• agent transport: crucible/src/agent/
• frontends: crucible/src/report/{console,stream}.rs
• session log wire format: crucible/src/report/session.rs
• the example domain: examples/counter/ (manifest, measure.nu, bump.nu)

Getting started: zero to a running loop

This page shows how to start an autoresearch loop from nothing. For a graph that runs one time, start with Your first playbook. This page has two parts:

1. Run a loop locally. The example domain needs no cluster and no LLM, and it runs in milliseconds.
2. Onboard your own domain. You write two things: the manifest and the measure command.

How the loop works (the diagram and glossary) and The autoresearch loop, in one read explain the design. The manifest and command formats on this page expand on the contract, which is the normative reference. If this page and the contract are different, the contract is correct.

Before you start

Tools. To build, you need Rust and Cargo. Part 1 also needs nu (nushell, the glue language). Development and deployment use more tools: Go, kubectl or oc, helm and helmfile, just, podman and buildah, gitleaks, and the claude CLI.

Credentials. Part 2 with a real agent backend needs Claude access for the claude CLI. The command backend and crucible init, check, and scope need no credentials.

Build the binary. Clone the repository (part 1 uses the examples/counter files) and build from source:

git clone https://github.com/neuralmagic/crucible.git
cd crucible
cargo build --release -p crucible --features autoresearch
install -m 755 target/release/crucible ~/.local/bin/   # or anywhere on PATH

You can also download a prebuilt binary from the releases page, if there is one for your platform. The binary alone is sufficient to onboard your own repository with the crucible init → crucible check → crucible scope steps of part 2.

Before part 2, note this convention: a manifest refers to the tools of a pack by bare name. The counter example does not use this convention, because bump.nu and measure.nu are in the pack directory and the manifest calls them by relative path. A pack that ships tool binaries must put them on $PATH before the loop runs.

Part 1: run a loop locally

examples/counter is the fastest way to see the full engine. The repository under test is a Git tree with one integer in value.txt. The score is that integer. A deterministic command backend (bump.nu) replaces the LLM, so the run costs nothing. It uses the real generic path (manifest → setup → propose → measure → keep/discard → Git history) without a cluster or a model.

crucible --manifest examples/counter/crucible.toml --iterations 6
# or, from a source checkout: cargo run -p crucible --features autoresearch -- --manifest examples/counter/crucible.toml --iterations 6

The manifest (examples/counter/crucible.toml) defines the full run:

[repo]
path = "."

[workspace]
dir = "workspace"
setup_cmd = "mkdir -p workspace && cp value.txt measure.nu bump.nu workspace/ && git -C workspace init -q && git -C workspace add -A && git -C workspace -c user.email=crucible@local -c user.name=crucible -c commit.gpgsign=false commit -qm baseline"

[agent]
backend   = "command"
agent_cmd = "./bump.nu"   # deterministic proposer: value.txt += 1
goal      = "Raise the integer in value.txt as high as you can (target: 5)."

[judge]
measure_cmd = "./measure.nu"
direction   = "higher"
objective   = "value"

# No [world] block -> GitWorld: kept iterations are commits, discards reset --hard + clean.

Each iteration: the engine runs agent_cmd to change the workspace and runs measure_cmd to score it. It keeps the change as a Git commit only if the score is strictly better in the direction. If not, it rolls the workspace back. The run writes:

• examples/counter/state/session.jsonl: the append-only NDJSON event log. It is the source of truth for the run.
• The Git history of the workspace. Each kept iteration is a commit.

To use the engine on real work, replace the deterministic proposer with a real agent and point measure at a real objective.

Part 2: onboard your own domain

A domain is a manifest and some executables. It is not Rust. A new domain runs without engine changes. You write two things: a crucible.toml and a measure command.

crucible init creates both in the current directory: a minimal manifest and a measure script that always reports the same score. You then edit the two generated files. After you point [judge].measure_cmd at the real gate, run crucible check --manifest crucible.toml. It validates the configuration before you spend an agent turn:

• every referenced file resolves;
• measure_cmd runs one time and prints the contract format;
• it warns if the gate is unprotected in the workspace that the agent edits.

crucible scope --pack <dir> runs one pipeline (ADR-0012): it resolves the goal, runs check, renders WORKFLOW.png, and freezes SCOPE.md with a RunIdentity digest. Add --issue owner/repo#N to take the goal from a GitHub issue. It fetches the issue through the GitHub API and uses GITHUB_TOKEN or GH_TOKEN.

The manifest

The contract gives the full schema and the meaning of each field. The parser rejects unknown keys, so a typo causes an error.

The most important rule: if you omit [world], you get GitWorld, and Git is the only state that the engine rolls back. If you set any of the three *_cmd keys, you get CommandWorld: Git and your snapshot and restore commands, for example for a live cluster.

The measure contract

measure_cmd is the judge. The engine parses the last stdout line that starts with { as JSON:

{ "valid": true, "score": 234.1, "solved": false, "note": "p99 234.1ms" }

• valid (required bool), score (number), solved (bool, default false), note and detail (optional).
• A nonzero exit status sets valid=false, whatever the command printed.
• The engine sets CRUCIBLE_BASELINE_SCORE, CRUCIBLE_BASELINE_TOTAL, and CRUCIBLE_BEST_SCORE, so your gate can compare.
• Keep rule: the engine keeps a candidate only if it is valid and its score is strictly better than the best in the direction. If solved is true, the engine keeps the candidate and ends the run. Use solved for a bug-fix gate.

measure_cmd is any executable that prints one JSON object. A plain sh script is a complete judge:

#!/bin/sh
p99=$(make bench | jq .p99_ms)
printf '{"valid": true, "score": %s, "note": "p99 %sms"}\n' "$p99" "$p99"

You can use any language. The engine reads only stdout.

Backends

[agent].backend selects the proposer:

• local (default): runs the selected harness (default claude) on this machine in the workspace. Needs the harness CLI and its credentials.
• openshell: runs the turn in a sandboxed pod (Landlock and deny-by-default egress). Needs sandbox_image and an egress allowlist in [agent.openshell].endpoints. Use this backend in a cluster.
• command: runs a fixed sh -c command as the proposal. It is deterministic, uses no LLM, and costs nothing. The counter example in part 1 uses it.

Composite domains

A manifest with a top-level [composite] table combines two or more existing component domains, without changes to them, into one run with one combined gate. For example, a server domain and a router domain each keep their own workspace and repository. One measure_cmd scores the combined system, so the gate judges a change to either component by the behavior of the pair. The engine does not change. It runs a list of workspaces. In addition to the manifest, a composite needs a combined gate that tests both components, and deploy targets for each component if the domains rebuild images.

Where to go next

• How the loop works: the full diagram and glossary.
• The loop in one read: the concepts and the contract table.
• Implementation contract: the normative manifest and command specification.
• JIRA tools (mediated): take a goal from an issue.

Tasks: the no-judge lane

The autoresearch loop needs an objective, but some jobs have no objective. The task lane runs the same loop (sandbox, broker, session log, publish, resume) for work without a score: consolidate the open dependabot PRs, fix flaky tests every night, regenerate docs, triage an issue backlog. You need one manifest. You do not need Rust or a gate script.

To use the task lane, omit [judge]. No other setting is necessary (ADR-0021).

The minimal manifest

[repo]
url = "https://github.com/you/repo"

[agent]
backend = "local"        # or "openshell" for the sandboxed path
model   = "claude-opus-4-6"
goal    = """
Consolidate the open dependabot PRs into a single branch.
Resolve conflicts, run the test suite, and write a summary to RESULTS.md.
"""

[publish]
pr_repo = "you/repo-fork"   # kept commits ship as one draft PR

Run it:

crucible check --manifest task.toml     # validates; prints the task-mode notice
crucible --manifest task.toml --iterations 3

Each iteration is one agent turn. The engine keeps every completed turn and commits it to Git. There is no baseline, no score, and no discard. The run exits with status 0 when no iterations remain.

What the engine supplies

• History: each turn is a commit. An interrupted run continues from the session log with --resume. The engine rolls back an incomplete turn to the last kept state.
• The deliverable: the engine pushes kept commits as a draft PR ([publish].pr_repo). If a results bucket is configured, it also records the run log and the diff of each turn to S3.
• The trust model: with backend = "openshell", the agent runs under Landlock with a deny-by-default egress allowlist, and it has no credentials. All privileged operations go through the broker tools, not the agent environment. You merge the PR.
• Forensics: state/session.jsonl is the source of truth. crucible flow --session state/session.jsonl --out flow.html renders a self-contained page that shows what the agent did in each turn.
• Steering: STEER.md, the control bridge, distress paging, and the limits (--max-cost, --max-time) work as in scored runs.

Semantics to know

The runnable reference

examples/task/ is the reference domain. The deterministic command backend replaces the LLM, so the full path (manifest → turns → keeps → session log → resume) runs in milliseconds:

crucible --manifest examples/task/crucible.toml --iterations 3

Use its manifest as the template for tools that generate task manifests.

Loop control states

Generated from crucible/src/runloop/machine.rs by crucible loop-states; scripts/state-docs.sh --check keeps it current. The driver advances through this table at every gate, so an edge missing here is a transition the loop cannot take.

Each Turn is one iteration's work graph (propose → apply → measure → decide), rendered in Work graphs. Everything else is the control shell around it: the gates at the Head, the parks, and how a run ends. Dashed states are idle; the colored edges are the ways out.

The source is docs/img/loop-states.dot (crucible loop-states --format dot).

How a run ends

The edge label after the arrow is the shutdown token on the session log.

An error inside the loop reports error and takes none of these edges.

Images for a new domain

A Crucible run uses two pods. A domain that needs a toolchain must install it in both:

The runtime image does not depend on a domain. It contains gcc, openssl-devel, python3, git, and jq, and no other measurement tools. The sandbox contains the agent CLI and Python. The domain image must add all other tools that measure_cmd or the agent turn needs. examples/selfhost (Rust) is the example: docker/Containerfile.runtime-rust and docker/Containerfile.sandbox-rust.

What the two images must satisfy

Loop pod (Containerfile.runtime-<domain>)

• Use FROM ghcr.io/neuralmagic/crucible:<tag>. Keep WORKDIR /opt/crucible. Do not replace /usr/local/bin/{crucible,openshell,openshell-gateway}.
• Install the toolchain in a world-readable location (/usr/local/...), not under /root. On OpenShift, the pod runs as an arbitrary uid.
• Pre-fetch the dependencies that measure_cmd builds against. The loop pod has egress (crates.io answered 200 from crucible-system on waldorf on 2026-08-25), but a cold dependency fetch in the first measure is slow and often fails first.
• dnf is available (full UBI10).

Agent sandbox (Containerfile.sandbox-<domain>)

• Use FROM quay.io/aipcc/agentic-ci/claude-sandbox:<tag>. It is a minimal UBI10: it has rpm but no dnf and no compiler. The uid is sandbox (998), HOME=/sandbox, and claude is at /usr/local/bin/claude.

• To add RPMs, install them into a rootfs in a full UBI10 stage of the same release, and copy that rootfs over the image:

  FROM registry.access.redhat.com/ubi10/ubi:10.2 AS pkgs
  RUN dnf install -y --nodocs --installroot /mnt/rootfs --releasever 10 \
          --setopt install_weak_deps=false gcc glibc-devel binutils ... \
      && dnf clean all --installroot /mnt/rootfs
  FROM quay.io/aipcc/agentic-ci/claude-sandbox:0.3.11
  COPY --from=pkgs /mnt/rootfs/ /
  

  perf is not in the UBI repositories. Sampling profilers with their own sampler (samply) work. cargo flamegraph does not work.

• Install the toolchain under /usr/local, run chmod -R a+rwX on its caches, and put USER sandbox at the end. If you set ENV PATH, include the bin directory of the toolchain. The agent inherits it.

• Pre-fetch more dependencies here than in the loop pod. The sandbox has deny-by-default egress. A cargo add or pip install by the agent during a turn works only if the registry is on the [agent.openshell].endpoints allowlist of the manifest and the binary that fetches is in [agent.openshell].binaries (for example, /usr/local/cargo/bin/cargo). Files that are already in the image cache need neither.

Build

Both images are amd64 and large (compiler and dependency fetch). Build them on the cluster, not under QEMU:

buildit build quay.io/<you>/crucible-sandbox-<domain>:v1 -n weaton-dev \
    --kubecontext coreweave-waldorf --mode job -f Containerfile.sandbox-<domain> \
    --request cpu=8 --request memory=16Gi
buildit wait <job> -n weaton-dev --kubecontext coreweave-waldorf   # last line: digest-pinned ref

.dockerignore keeps target/, .claude/, and the example workspaces out of the build context. Pin CRUCIBLE_TAG for the runtime image (--build-arg CRUCIBLE_TAG=<tag>). If you do not, the engine version in the loop pod can be different from the controller version.

A new quay repository is private. The namespace needs an imagePullSecrets entry (a kubernetes.io/dockerconfigjson secret from ~/.docker/config.json) before the pod can pull the image. Without it, the pod stays in ImagePullBackOff with a 401.

Verify the images

Before a manifest uses an image, run the image as an unexpected uid. Check the toolchain, the cache, and an offline build:

kubectl run smoke --restart=Never --image=<ref> \
    --overrides='{"spec":{"imagePullSecrets":[{"name":"quay-pull"}],"securityContext":{"runAsUser":12345,"runAsGroup":0}}}' \
    --command -- sh -c 'id; cargo --version; cd /tmp && cargo new -q t && cd t && cargo build -q --offline && ./target/debug/t'

Then run crucible check --manifest <pack>/crucible.toml locally. It runs measure_cmd and the gate self-test. Set backend = "openshell" and sandbox_image, and deploy.

Running the controller locally

The controller can run on a laptop without an identity provider, Vault, or a cluster. Playbook launches run as crucible plan run subprocesses on the same machine. If the agent turns of a pack need an OpenShell sandbox, the engine starts one on the podman of the host.

just controller-local

Then open http://127.0.0.1:8870. The recipe needs bun and a Rust toolchain. It needs podman only for OpenShell sandboxes. The recipe:

• builds the UI and the crucible, crucible-controller (with embedded-db), and crux binaries;
• runs crucible-controller autopilot against its own Postgres, with the settings below.

just controller-local 9000 wren uses a different port and login. If no other address is configured, crux uses http://127.0.0.1:8870, so on the default port it needs no setup:

crux whoami
crux draft-create notes --description "release notes"
crux draft-push notes examples/playbook --base-version 1
crux draft-launch notes --max-cost 1 --max-time 5m

What the recipe sets

The recipe unsets CONTROLLER_API_TOKEN, CONTROLLER_OIDC_ISSUER, and VAULT_ADDR for the process, also if your shell exports them.

The embedded database

With DATABASE_URL=embedded, crucible-controller autopilot runs its own Postgres 16 and does not connect to an external one. It needs a build with the embedded-db feature (cargo build -p crucible-controller --features embedded-db). A build without the feature rejects the value at boot.

• On the first start, the controller downloads the Postgres binaries one time into ~/.theseus/postgresql. All embedded controllers on the machine share them.
• The data directory is $CONTROLLER_STATE_DIR/postgres/data, and its password is in postgres/pgpass next to it. The data stays after a restart. To start with an empty database, delete postgres/.
• At each start, the server listens on a free loopback port. Only the controller uses it. crux and the UI use the controller API.
• The server stops when the controller exits. If the controller is killed, the next start stops the old server before it starts a new one.

The dev identity

Without CONTROLLER_API_TOKEN and CONTROLLER_OIDC_ISSUER, the controller starts only with CONTROLLER_DEV_IDENTITY. The guard is then open: it admits every request as that login, and binds loopback, whatever CONTROLLER_API_ADDR is. The controller removes any X-Auth-Request-* header from the client, so a browser and curl get the same identity. The controller does not boot if this variable is set on a guarded deployment, because it would give every caller that identity, and it does not boot with none of the three, because an open controller without a login is never what a deployment meant.

Agent turns

A local run starts with an empty environment. It keeps only PATH, HOME, USER, OPENSHELL_PODMAN_SOCKET, and the CRUCIBLE_* variables from the controller.

• backend = "local" (the default) runs the agent CLI on this machine, under your own login. The harness finds its configuration through HOME and USER. This costs real money.
• backend = "command" runs the command of the pack, without a model.
• backend = "openshell" runs each turn in a sandbox from sandbox_image, which the engine starts on podman. It needs openshell and openshell-gateway on PATH. Podman must be able to pull the image, or the image must already be in podman. just revise-loop-e2e builds an image without a model.

Put any other variable that a run needs in CONTROLLER_LOCAL_SECRET_ALLOWLIST. The pack must disclose the variable as an agent credential before a run gets it.

Webhooks

The recipe serves the webhook delivery listener on 127.0.0.1:8871. Launching from webhooks shows a pack with only commands, from a curl delivery to a finished run.

What is disabled

• By default, the build does not include the autoresearch lane (see Deploying the controller), so the UI shows only playbooks.
• Without Vault, the write routes of the secrets registry answer 503. A pack that binds no secrets launches normally. A launch of a pack that binds secrets fails.
• Runtime overrides (the live caps on the admin page) need the overrides ConfigMap, so the controller uses the configuration that it parsed at start.
• Discovery, triage, and the work-pod paths have no cluster and no watched repositories, so they do nothing.

Deploying the controller

The controller is the control plane around the engine. It has a Postgres ledger of issues, scopes, runs, and launches, a daemon that reconciles them into loop pods, and an HTTP API. The API serves the embedded UI, /api, and the hosted MCP server at /mcp. Run one replica per cluster. The binary is crucible-controller. It ships as ghcr.io/neuralmagic/crucible-controller, built on the runtime image, so the image also contains the engine binaries that the controller launches.

This page describes what the controller needs and how its parts connect. Authentication describes identity, and Vault describes the secrets registry. To try the controller on a laptop first, see Running the controller locally.

Requirements

All durable state is in Postgres. You can replace the controller pod at any time.

Running it

export DATABASE_URL=postgres://crucible:...@postgres:5432/crucible
export CONTROLLER_API_ADDR=0.0.0.0:8080
export CONTROLLER_API_TOKEN=$(openssl rand -hex 32)
export CONTROLLER_PUBLIC_URL=https://crucible.example.com
export CONTROLLER_WATCHED_REPOS=your-org/your-repo
export CONTROLLER_POD_NAMESPACE=crucible-loops
export GITHUB_TOKEN=ghp_...
crucible-controller autopilot

autopilot is the resident daemon. It does discovery, ranking, scoping, launching, and recording, and it serves the API. autopilot --once processes the queue and exits, for a one-time reconcile. Each flag has a CONTROLLER_* environment variable. crucible-controller autopilot --help lists all of them with their defaults.

The admission caps limit what the daemon can spend without a person:

You can disable autopilot at run time on the admin page. This pauses only the spend that the daemon starts.

The autoresearch lane

Playbooks are the core of the controller: the registry, drafts, launches, schedules, watches, and their runs. The autoresearch lane is separate and disabled by default. It contains GitHub discovery and triage, ranking, scope proposal and its approval gate, image builds, and scored loop runs. Two settings enable it:

• the autoresearch cargo feature, which compiles the lane (cargo build -p crucible-controller --features autoresearch). The published image and release binaries include it.
• CONTROLLER_AUTORESEARCH=true, which enables the lane in a build that has it. If the variable is set on a build without the feature, the controller does not start.

When the lane is disabled, the controller does not serve its routes or run its discovery polls, and it does not change rows that are not playbook launches. GET /api/version reports autoresearch, and the UI hides the pages of the lane. The engine has a matching autoresearch feature for the scored loop and crucible scope. A deployment that enables the lane needs both.

On Kubernetes

The controller runs as a Deployment. In a cluster, replicas compete for a Lease (CONTROLLER_LEASE_NAME), and only the holder reconciles. During a rolling update, two daemons do not apply the check-then-act caps at the same time. CONTROLLER_LEADER_ELECTION=off disables leader election for a single-replica deployment that does not use a Lease.

A minimal set of objects:

• a ServiceAccount for the controller, with a Role in the loop namespace bound to it: pods get/list/watch/create/delete and pods/log get (dispatch, the watch, and the log relay); pods/finalizers update (the pod's ConfigMap and Secret block its deletion, which OpenShift's owner-reference admission checks); configmaps and secrets create/delete (the pack ConfigMap and the run Secret, which holds the pod token and the delivered secrets, both owner-referenced to the pod); and leases get/create/update/patch for leader election;
• a Secret for CONTROLLER_API_TOKEN and one for GITHUB_TOKEN, referenced by env;
• a Service on the API port;
• the Deployment, env from the table above, emptyDir at CONTROLLER_SCRATCH_DIR.

Before you expose the Service outside the cluster, put an OIDC login in front of it. See Authentication. You can expose the machine paths (/api, /mcp) on a separate hostname, because they authenticate with a bearer token.

The webhook delivery listener is a third endpoint, on a separate port. It is the only part that you publish to the internet. Launching from webhooks gives its variables and the webhooks values of the chart.

The image labels contain the contract version of the engine (io.crucible.contract-version). Before the controller launches a pod, it reads the same label from the dispatch image. If the versions are different, it does not launch the pod.

Where loops run

By default, loop pods run in the cluster of the controller, in CONTROLLER_POD_NAMESPACE. A deploy profile (CONTROLLER_DEPLOY_PROFILE, a TOML file) sets the cluster values that a launch needs: the loop image, pull secrets, the service account, the results bucket, and the environment for the engine. The engine renders the loop pod from the profile in the controller process, so the controller validates the profile when it starts, not when a launch fails.

If the controller runs loops in other clusters (spokes):

• CONTROLLER_CLUSTERS_DIR holds one kubeconfig for each spoke. CONTROLLER_DISPATCH_CLUSTER names the default target. CONTROLLER_DISPATCH_CLUSTER_BY_CONTRACT selects the cluster by the broker contract of the pack.
• CONTROLLER_CLUSTER_POLICY limits which users can target which cluster (name=group:/path;user:login, comma-separated).
• If the hub cannot reach a spoke, the spoke sends reports to the hub. crucible-controller cluster-snapshot runs on the spoke as a CronJob and posts to the push endpoint of the hub with a bearer token that the hub lists in CONTROLLER_PUSH_TOKENS.

Agent tokens

The controller can act as the OAuth authorization server for sandboxed agents. An OpenShell sandbox supervisor posts its SPIRE JWT-SVID to POST /oauth/token (client credentials with a jwt-bearer client assertion, no client_id); the controller verifies it against SPIRE's JWKS, maps the sandbox to the run whose pod bound it, and returns an ES256 access token living 15 to 30 minutes. MCP servers verify it against /.well-known/jwks.json; /.well-known/oauth-authorization-server describes the endpoints. All three are served outside the human auth stack.

The token's sub is the person the run was dispatched for: whoever unparked, scoped, redispatched, or approved the issue for that dispatch, else the launch's creator (the owner, for a schedule, watch, or webhook). A GitHub review approval counts only when an operator paired the reviewer's GitHub login with a crucible login in CONTROLLER_GITHUB_LOGINS (<github login>=<crucible login>, comma-separated); equal names are never assumed to be the same person. The request is consumed by the run it gets, so later runs act for the owner again. An override by an actor that is not a login records nobody. act.sub is the sandbox's SPIFFE ID. It also carries run_id (the run pod), sandbox (the sandbox's name), workdir (its workspace path), kube_user (below) and, for runs on the [kueue] cluster, namespace (crucible-kueue's default namespace for the run's workloads). Claims name who is calling and for which run; what that identity may do is the resource server's to decide (crucible-kueue runs SubjectAccessReview as kube_user). Refusals return only the OAuth error code and a fixed description; the reason is logged.

Sandbox placement. A sandbox's SPIFFE ID is only as trustworthy as whoever can create pods where its supervisor runs, so with agent clusters the controller refuses to start unless the deploy profile runs each run's gateway in its own pod: [cluster] sandbox_namespace (which means the kubernetes driver), which profile load refuses to share with the loop, rig, or crucible-kueue namespaces. It also refuses to start the issuer when its own namespace, where the signing key Secret is mounted from, is any of those namespaces or [buildit] namespace (the loop account can exec into any pod there), and when a SPIRE's namespace in CONTROLLER_AGENT_SPIRE is one pack code reaches (loop, rig, sandbox, buildit or crucible-kueue workload namespace), since a pod there could exec into the SPIRE server and mint any SVID. Its namespace is POD_NAMESPACE, else the service account's namespace file; a controller rendered by crucible deploy render runs in the loop namespace and so cannot hold the key.

Gateway identity. Each run's gateway has its own OpenShell gateway id, the run pod's name. The gateway stamps it as openshell.ai/gateway-id on every supervisor pod and Sandbox it creates, from its own config, and driver_config cannot set labels or annotations. Scope the sandbox ClusterSPIFFEID to the sandbox namespace and build the ID from that label and the supervisor's openshell.ai/sandbox-id annotation:

spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/openshell/gateway/{{ index .PodMeta.Labels "openshell.ai/gateway-id" }}/sandbox/{{ index .PodMeta.Annotations "openshell.ai/sandbox-id" }}'

Each cluster's gateway prefix is everything before the gateway id (spiffe://<trust domain>/openshell/gateway/, from its discovered trust domain, or its CONTROLLER_AGENT_SPIRE entry). The controller reads that SPIRE's keys from its OIDC discovery provider through the cluster's API server Service proxy, with the same client it dispatches with, so its identity on each cluster needs get on services/proxy in SPIRE's namespace. The iss of an SVID picks the cluster whose keys verify it, and a token is issued only when the sandbox's run is on that cluster.

The engine binds each sandbox it creates by posting its id and workdir to /api/sandbox-bindings/runs/{run} with the run's pod token (CRUCIBLE_POD_TOKEN, the same token that uploads its evidence), and only when the loop pod carries CRUCIBLE_AGENT_TOKEN_ENDPOINT. A spoke that reaches the controller through an external route needs /api/sandbox-bindings and /oauth/token there, beside /api/evidence and the issuer's /.well-known/jwks.json. The controller binds only a Sandbox resource (agents.x-k8s.io) in the sandbox namespace carrying that id and the run's gateway id, reads the sandbox's name from it, and owner-references it to the gateway pod. At token issue it refuses an SVID whose gateway id is not the bound run's. Name the controller's identity on each cluster in [cluster] sandbox_readers (namespace/name) and the loop render grants what it needs (below). A sandbox that was never bound, whose run pod is no longer running, or whose SVID names another run's gateway gets invalid_grant.

Isolated gateways. A run's OpenShell gateway does not run in the loop pod. The loop render creates the crucible-openshell-gateway ServiceAccount in the sandbox namespace and binds the sandbox rights (Sandboxes, pods, secrets) to it alone; the loop service account gets no verbs there. For each run the controller first creates the gateway Service in the sandbox namespace and mints a CA, a server certificate for that Service's ClusterIP, and client certificates for the loop pod and the supervisors, valid for the run's deadline plus a day (30 days without one); it keeps no copy of the CA key. The loop pod's client material and the gateway URL arrive through the run Secret. Before the loop pod it creates a loop Service (the sandboxes' host.openshell.internal), a NetworkPolicy admitting only that run's loop pod and its own supervisors to the gateway port, the gateway pod, and the server and supervisor TLS Secrets owned by that pod. Collection deletes the gateway's Sandboxes and supervisors, the loop Service, then the gateway's own objects; a sweep on the discovery tick does the same for any gateway whose run pod is no longer running. Dispatch to a cluster the hub only receives pushes from is refused, since the hub cannot create the gateway there.

For the controller's identity, sandbox_readers is granted: sandboxes get/list/patch/delete; pods get/list/create/delete and pods/finalizers update (the gateway's Secrets block its pod's deletion); services, secrets and networkpolicies list/create/delete in the sandbox namespace; and services list/create/delete in the loop namespace.

Grounded-rank and scope turns get a gateway of their own the same way, with certificates valid for the turn's deadline; a failed launch tears it down, and the gateway sweep removes it once the turn's row closes. Turn sandboxes are never bound, so they get no agent tokens.

Supervisors of runs that get agent servers mount the SPIFFE Workload API (the csi.spiffe.io driver) at /spiffe-workload-api/spire-agent.sock; [cluster] sandbox_spiffe_socket names another path. Runs without agent servers mount none, so a cluster without SPIRE still starts sandboxes.

With [cluster] aws_sandbox_role_arn, the gateway pod assumes that role with an STS-audienced token for its own service account, as the in-pod gateway does with the loop pod's, so the role's trust policy must allow system:serviceaccount:<sandbox namespace>:crucible-openshell-gateway.

Pack code holds its run's gateway client certificate, so it can still create sandboxes for its own run with the driver_config pod fields the gateway accepts (node_selector, tolerations, host_aliases, runtime_class_name, priority_class_name, PVC volumes); resource admission keeps PVCs and priority classes to those labelled openshell.ai/sandbox-attachable=true, and those sandboxes carry its own gateway id. The pinned OpenShell (3441f91) predates v0.1.2: supervisors are separate pods, and nothing here depends on the sidecar supervisor topology or other v0.1.2 features.

kube_user. Tokens carry the dispatcher's Kubernetes username on the run's cluster when the controller knows it, by the same rule people's tokens use there (Settings > Cluster identities, Cluster identities); a server that sees it runs SubjectAccessReview as that user. A link cluster's username is the one the person linked there. On the hub a platform administrator can override it with PUT /api/users/{login}/kube-user ({"kube_user": "<name>"}, or null to clear it); the override is audited on user:<login>, and no administrator sets their own. system: identities are refused. crucible-kueue acts only as a token's kube_user, so a token carrying crucible-kueue scopes is refused when the username is unknown; other tokens are issued without the claim, logged at warn. The linking and override routes are served whenever the controller mints agent tokens, people tokens, or has a [cluster_link] table.

The issuer needs CONTROLLER_AGENT_ISSUER, a signing key and the deploy profile. Audiences, people resources, each cluster's SPIRE and platform, the servers' trusted issuer and the cluster link callback are derived from those; the rest of the table is overrides. See Minimal configuration and the derived values beside it.

A run dispatched onto one of the [kueue] clusters (clusters = ["spoke-a", "spoke-b"], or cluster = "spoke-a"), or of any server the profile describes, gets each server's name, URL, scopes and transport (CRUCIBLE_AGENT_SERVERS), the token endpoint (CRUCIBLE_AGENT_TOKEN_ENDPOINT) and the issuer as SVID audience (CRUCIBLE_AGENT_SVID_AUDIENCE) as plain env. The engine registers a token-grant provider for each, so each sandbox's supervisor obtains its own token and injects it on traffic to the server alone; no token is minted per run. A server is granted only on a cluster whose SPIRE the issuer trusts. The run records what it got, with the server's scopes and, for a server that acts as kube_user such as crucible-kueue, the dispatcher's username there, and the run page shows it. A third server needs an [[agent_servers]] entry with its url and scopes, nothing else.

The deploy profile describes each server that takes agent tokens (DeployProfile::agent_servers): [kueue] needs a kube_user in every token and names workload_namespaces, the first as the namespace claim, for runs on its clusters; [buildit] needs neither. Each server's URL on each of its clusters is an agent audience, with its scopes: kueue.read kueue.submit for crucible-kueue and buildit.read buildit.build for buildit. The rendered server trusts runs' tokens from the controller's CONTROLLER_AGENT_ISSUER (crucible deploy render --kueue --issuer <it>), fetching keys from <issuer>/.well-known/jwks.json unless run_jwks_uri names another URI (a route, for a server that cannot reach the issuer's address). kube_authz (openshift or token) is where the server takes a caller's groups for its SubjectAccessReviews. With people = true the rendered server also accepts people's tokens from the same issuer (--person-issuer), and each cluster's server becomes a resource people's MCP clients get tokens for, at its Service URL or the cluster's people_urls entry.

buildit. A run dispatched onto one of the [buildit] clusters starts buildit mcp in its loop pod on port 8848, and its sandboxes reach it at http://host.openshell.internal:8848/mcp with tokens their supervisors obtain like crucible-kueue's. buildit takes only tokens for that audience whose run_id is its own run, reads each build's context from the token's sandbox and workdir through the run's gateway, and runs builder pods in [buildit] namespace, which profile load refuses to share with the loop, rig, sandbox or crucible-kueue namespaces. The loop render grants the loop service account pods create/get/list/watch/delete, pods/exec and pods/log there and nowhere else, since those rights in the loop namespace would reach every run's loop pod and Secret. Create the namespace before the first run; builder pods run as its default service account, so on OpenShift bind that account to an SCC that admits them (uid 1000). The rights still let runs see each other's builder pods in that namespace, including a push's authfile while it is copied in; buildit under a service account of its own is the clean end state. A build's RUN steps run the agent's code unsandboxed in the builder pod, so the render also creates the buildit-builders NetworkPolicy there: no ingress, egress only to DNS and [buildit] egress (default TCP 443 anywhere). Policies match networks, not host names: list your registries' networks, or put the cluster's pod and service networks in except, to keep builds off in-cluster services. push publishes a succeeded build to push_repo only, tagged <sandbox>-<build_id>, with the loop pod's [secrets] push_authfile; without either, buildit pushes nothing. queue submits builder pods to that Kueue LocalQueue in namespace; set it where Kueue manages the namespace and holds pods that name no queue (manageJobsWithoutQueueName), or builds wait 10 minutes and fail. [buildit.builders] sizes builder pods on every cluster: node_arch (their kubernetes.io/arch), max_builds live per sandbox, pod_deadline_secs, requests/limits, and storage_driver. [buildit.cluster_builders.<cluster>] replaces any of those on one cluster, so a mixed-architecture cluster can pin its builds. Without node_arch, builder pods run on the loop pod's architecture; unset limits are buildit's defaults (4 builds, 7200 s, no requests or limits). storage_driver is vfs (the default), a full copy of the filesystem per image layer, or overlay, each layer stored once with kernel overlayfs in buildah's user namespace; overlay needs a kernel that allows that (5.13 and later, such as RHEL 9's).

[buildit]
clusters     = ["spoke-a", "spoke-b"]
namespace    = "crucible-builds"                    # builder pods, apart from loop pods
egress       = [{ cidr = "0.0.0.0/0", except = ["10.128.0.0/14", "172.30.0.0/16"] }]  # optional
run_jwks_uri = "https://crucible-ext.example/.well-known/jwks.json"  # optional
push_repo    = "registry.example.com/team/crucible-smoke"  # optional
queue        = "crucible-builds"                    # optional, a Kueue LocalQueue in namespace
image        = "ghcr.io/neuralmagic/crucible-buildit@sha256:..."  # optional, default the loop image's

[buildit.builders]                                  # optional
max_builds        = 2
pod_deadline_secs = 3600
requests          = { cpu = "8" }
limits            = { cpu = "32" }
storage_driver    = "overlay"

[buildit.cluster_builders.spoke-b]                  # optional, per cluster
node_arch = "amd64"

[buildit.cache]                                     # optional, a cache builds share
clusters     = ["spoke-a"]                          # default the hub; deploy it on each
token_secret = "buildit-cache-token"
size_gib     = 500
write        = true

image pins buildit apart from the loop image: loop pods copy its /usr/local/bin/buildit in with an init container and run it ahead of their own, and the cache runs it unless [buildit.cache] image says otherwise. Every main build publishes ghcr.io/neuralmagic/crucible-buildit, so a buildit-only change rolls by moving this pin; the controller reads the profile at each dispatch, and the cache needs crucible deploy --buildit-cache applied again.

[buildit.cache] adds a buildit cache in the loop namespace that every build may use: ccache objects and stored build outputs shared across runs. crucible deploy --buildit-cache --profile <profile> renders its PersistentVolumeClaim, Deployment, Service and a NetworkPolicy that admits builder pods only; apply it on each cluster in clusters. The loop render opens builder egress to it, and runs on those clusters, and no others, get its URL. Create the token Secret first, in the loop namespace, with a URL-safe value:

oc create secret generic buildit-cache-token --from-literal=token="$(openssl rand -hex 32)"

With write (the default) loop pods mount the token and builds on the cache's clusters may write to it. A build's RUN steps can read it, so one run's agent could store objects other runs' builds use: keep write for clusters whose runs trust each other, and set write = false otherwise so builds read only.

openssl ecparam -genkey -noout -name prime256v1 | openssl pkcs8 -topk8 -nocrypt -out agent-signing.pem
oc create secret generic crucible-agent-signing-key --from-file=agent-signing.pem

Observability

• /metrics serves Prometheus text on the API port.
• Tracing goes to stderr as leveled tracing events. Set OTEL_EXPORTER_OTLP_ENDPOINT to also export OTLP spans over HTTP.
• /healthz answers when the database is reachable.
• The GET /api/system payload names the commit the binary was built from.

The API and the CLI

GET /api/openapi.json is the API document. It needs the same bearer token as the other routes. crux is the CLI for that API and the tool library that the controller hosts at /mcp:

cargo install --locked --git https://github.com/neuralmagic/crucible.git crux
export CONTROLLER_URL=https://crucible.example.com
export CONTROLLER_API_TOKEN=crk_...
crux whoami

You get the crk_ key on the Settings page of the UI. It has the identity of its owner. The static CONTROLLER_API_TOKEN also works on /api, as anonymous or as the identity that CONTROLLER_API_TOKEN_IDENTITY sets. It does not work on /mcp. The crux README is the only reference for its flags, environment variables, and configuration file.

Authentication

Three kinds of callers use the controller. Each kind has its own credential:

Authorization is separate from the credential. Teams, ownership, and the policy set (RFC-0003) decide what each caller can do, from the login and groups in the credential. On a new deployment, four lists set the first members:

• CONTROLLER_ADMINS: logins that administer the platform;
• CONTROLLER_OPERATORS: logins that can operate;
• CONTROLLER_OPERATOR_GROUPS: IdP groups whose members can operate;
• CONTROLLER_PUBLISHERS: logins that can publish their own drafts without review.

These lists seed the platform-administrators, platform-operators, and playbook-publishers teams while those teams have no members. A login on the admin or operator list has that team membership on every request, also if the team does not list it. Membership that you add in the UI has the same effect. An owner of platform-administrators is an admin. Any other member of platform-administrators or platform-operators is an operator. This applies to the routes that need admin or operator, and to the role that /api/whoami reports.

Each team has a workspace at /teams/<slug>. Each list has a team path (for example /teams/<slug>/secrets) that shows only the resources of that team.

Opening a deployment to an organization

A signed-in user who is in no team can read and launch the playbooks and providers that team:platform-administrators owns, and dispatch to its targets. The user owns all resources that they create. To give one organization more access, without giving it to every login that the issuer accepts, create a team whose members come from a rule. Then give the shared resources to that team. The Teams page makes the first call:

POST /api/teams {"slug": "my-org", "display_name": "My org",
  "members": [{"kind": "rule", "member": "group-suffix:my-org-group", "role": "member"},
              {"kind": "user", "member": "<you>", "role": "owner"}]}
PUT /api/playbooks/<id>/owner {"owner": "team:my-org"}
PUT /api/providers/<id>/owner {"owner": "team:my-org"}

Members can read, launch, and dispatch what the team owns. Maintainers and owners can also edit it. A provider uses its key on every dispatch, so each user who can select the provider spends with that key. Move a provider to a team only if the whole team must use it. email-domain:<domain> rules match the email that the controller records at each login.

The autoresearch lane

On a deployment that runs autoresearch (CONTROLLER_AUTORESEARCH=true), only entitled callers see the pages and routes of the lane (issues, inbox, turns, builds, repos, explore, the autopilot switch). Other callers get a 404, and their navigation does not show these pages. The entitlement is read on the lane, which the autoresearch team owns. Its members have the entitlement, platform administrators have all entitlements, and a policy set can give it to other users. The controller seeds the team with platform-operators as a nested team. To give a person access to the lane, add a group rule or the person to the team. /api/whoami reports the entitlements of the caller under entitlements.

Viewing as another user

A platform administrator (platform:impersonate) can view the controller as any user who has signed in, from the row of the user on a team page. The browser session then uses the identity of that user, with the groups from their last sign-in, for one hour or until the administrator stops it. The view is read-only. The controller rejects every write except DELETE /api/impersonation. The event log records the start and the stop under the login of the administrator. API keys and bearer tokens cannot use a view.

The controller as the OIDC relying party

With CONTROLLER_OIDC_ISSUER set, the controller serves /auth/login, /auth/callback, and /auth/logout. It runs the authorization-code flow with PKCE against the issuer, validates the ID token against the JWKS of the issuer, and creates its own server-side session (a cookie id and rows in Postgres). There is no proxy in front, and the controller reads no identity header on any path: it removes every X-Auth-Request-* header from each request before it authenticates the request.

The session holds the login, the email, and the groups claim from the issuer. Team membership rules match on groups. If the issuer emits no groups claim, no user gets access through groups, and there is no error. Test the claim before you point a deployment at the issuer:

CONTROLLER_OIDC_ISSUER=https://sso.example.com/realms/crucible \
CONTROLLER_OIDC_CLIENT_ID=crucible-controller \
CONTROLLER_OIDC_CLIENT_SECRET=... \
crucible-controller oidc-claims

oidc-claims runs one login and prints every claim that the issuer gives this client. The test passes if the output has a groups array with the expected paths. After the change, GET /api/whoami in a browser reports the login and a list of groups that is not empty.

Offline credentials for scheduled work

A scheduled launch has no session, and a launch that uses a team secret needs the current group membership of the owner. Set CONTROLLER_CREDENTIAL_KEY (or CONTROLLER_CREDENTIAL_KEY_FILE) to a base64 AES-256 key. The callback then seals the offline refresh token of each user with that key, and a schedule uses that credential to refresh the groups of its owner when it fires. Without a key, the controller stores nothing. A schedule then uses the group snapshot from its last save, and it parks when the snapshot is older than CONTROLLER_SCHEDULE_OWNER_TTL_SECS.

Three conditions are necessary. Two of them are on the issuer:

1. offline_access is in CONTROLLER_OIDC_SCOPES.
2. The issuer grants offline_access to the client and to the user.
3. The client allows refresh-token reuse. Two schedules of one owner can fire in one sweep. The controller runs their refreshes one at a time for each subject, but an issuer that revokes on any reuse can revoke the credential when two refreshes overlap.

To rotate the key, put the new key on the first line and keep the old key below it. The first line seals, and every line decrypts. CONTROLLER_SESSION_GROUP_REFRESH_MINUTES sets the maximum age of the groups of a live session before the next request reads them again. If the issuer refuses, the session becomes a viewer until the user signs in again. Users revoke their own credential on the Settings page.

Keycloak

Use a confidential client with the authorization-code flow and a group-membership mapper. The test suite of the controller uses the realm in crucible-controller/tests/keycloak/crucible-realm.json. It imports without changes with start-dev --import-realm. It is the reference for the settings below.

1. Client crucible-controller: confidential, Standard flow on, valid redirect URI https://<host>/auth/callback (plus the exact CONTROLLER_OIDC_REDIRECT_URL if it differs).
2. Group mapper on that client: mapper type Group Membership, token claim name groups, full group path on, added to the ID token and the userinfo endpoint. The controller reads groups as paths (/platform-devs). Team rules and CONTROLLER_OPERATOR_GROUPS match against these paths.
3. Offline access, if schedules run with team secrets: assign the offline_access role of the realm to the users (or to the default roles), and allow refresh-token reuse on the client (Revoke Refresh Token off, or Refresh Token Max Reuse at least 1).

With the reference realm running locally:

docker run -d --name keycloak -p 8180:8080 \
  -e KC_BOOTSTRAP_ADMIN_USERNAME=admin -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin \
  -v "$PWD/crucible-controller/tests/keycloak:/opt/keycloak/data/import:ro" \
  quay.io/keycloak/keycloak:26.4 start-dev --import-realm
export CONTROLLER_OIDC_ISSUER=http://127.0.0.1:8180/realms/crucible
export CONTROLLER_OIDC_CLIENT_ID=crucible-controller
export CONTROLLER_OIDC_CLIENT_SECRET=$(jq -r '.clients[] | select(.clientId=="crucible-controller") | .secret' crucible-controller/tests/keycloak/crucible-realm.json)
export CONTROLLER_OIDC_REDIRECT_URL=http://127.0.0.1:8080/auth/callback

alice (password alice-password) is in /platform-devs; mallory (mallory-password) is in /other-team.

Dex

Dex emits groups as a top-level claim when the upstream connector supplies groups, so you do not need a mapper. The controller matches the exact strings that Dex emits. For most connectors, these are bare names (platform-devs), not paths. Use the same names in CONTROLLER_OPERATOR_GROUPS and in team rules.

A static client:

issuer: https://dex.example.com
staticClients:
  - id: crucible-controller
    secret: ...
    name: crucible
    redirectURIs:
      - https://crucible.example.com/auth/callback
oauth2:
  skipApprovalScreen: true
connectors:
  - type: github
    id: github
    name: GitHub
    config:
      clientID: ...
      clientSecret: ...
      redirectURI: https://dex.example.com/callback
      orgs:
        - name: your-org
          teams:
            - platform-devs

Dex issues refresh tokens if the client requests offline_access and the connector supports it (GitHub, LDAP, and OIDC upstreams do). Before you point the controller at Dex, use oidc-claims to check that groups has the expected format.

Without an issuer

A controller without CONTROLLER_OIDC_ISSUER runs no login. It starts in one of two shapes:

• With CONTROLLER_DEV_IDENTITY, it is an open controller for local development. Every request uses that login, and the controller binds its HTTP surface to loopback. The controller refuses to start if CONTROLLER_API_TOKEN is also set.
• With CONTROLLER_API_TOKEN, only machine callers get in: the static token, with the login in CONTROLLER_API_TOKEN_IDENTITY if it is set, and API keys.

With neither, the controller refuses to start. A deployment that forgets its issuer does not become open.

Retired settings

The controller refuses to start when one of these is set, and the error names the setting:

API keys

The Settings page creates crk_<id>_<secret> keys. The controller stores only the SHA-256 of the secret, so you cannot recover a lost key. Create a new one. By default, keys expire after 90 days. You can revoke a key on the same page. A key has the login of its owner and the groups from their last browser sign-in. A key never has more access than its owner, and its groups are never newer than the last login. A caller that authenticated with a key cannot create or revoke keys. A key cannot create a replacement key or stay valid after revocation.

Each rejection on the bearer path tells the cause: no header, not a key, a key of a different controller, expired, or revoked. If crux whoami answers credential: bearer (anonymous), the string in the header does not start with crk_.

Cluster identities

MCP servers that check Kubernetes permissions (SubjectAccessReview) need to know who a person is on the cluster they run on. People's access tokens carry it as kube_user, and Settings → Cluster identities lists every cluster the deploy profile knows, the rule that cluster uses, and the caller's username there. The deploy profile declares each cluster's rule:

[cluster]                                 # the hub
identity = { claim = "login" }            # usernames are SSO logins

[clusters.spoke-b]
kubeconfig_secret = "spoke-b-kubeconfig"
identity = { claim = "email" }            # the cluster's OIDC login uses the same email

[clusters.spoke-a]
kubeconfig_secret = "spoke-a-kubeconfig"
identity = "link"                         # the default: each person proves it to the cluster

prefix puts the API server's OIDC username prefix in front of a claim ({ claim = "login", prefix = "sso:" }). A claim cluster shows the resolved name read-only. A link cluster shows Link, Relink and Unlink, and records only what the cluster itself said: the link's username, how it was proven, and when. The token used to prove it is never stored or logged. On the hub, an administrator's override (PUT /api/users/{login}/kube-user, audited, never self-applied) wins over the rule and shows as such; it is also what the person's runs act as there.

Linking through an OpenShift cluster's OAuth server

The controller registers as a confidential OAuthClient on each cluster listed in the profile's [cluster_link] table. Link sends the browser to that cluster's authorize endpoint (read from /.well-known/oauth-authorization-server on its API server) with PKCE and a state bound to the person's session. The callback redeems the code for a user:info token, the cluster's TokenReview names its owner, the username is recorded, and the token is dropped.

[cluster_link]
openshift = ["spoke-a"]
# external_url = "https://crucible.apps.hub.example.com"  # default: CONTROLLER_AGENT_ISSUER
# client_id = "crucible-controller"
# access_token_max_age_secs = 300

The callback sits under the agent issuer, CONTROLLER_AGENT_ISSUER, unless external_url names another URL browsers reach the controller at. Render the client for one cluster, then create it there with a fresh secret, and give the same secret to the controller under the cluster's name:

crucible deploy render --profile profile.toml --cluster-link spoke-a \
  --issuer https://crucible.apps.hub.example.com > oauthclient-spoke-a.yaml
secret=$(openssl rand -hex 32)
oc --context spoke-a apply -f oauthclient-spoke-a.yaml
oc --context spoke-a patch oauthclient crucible-controller --type merge -p "{\"secret\":\"$secret\"}"
oc --context hub -n crucible-system create secret generic crucible-cluster-link \
  --from-literal=spoke-a="$secret" --dry-run=client -o yaml | oc --context hub apply -f -

OAuthClient is cluster-scoped, so creating it needs cluster-admin on that cluster. Mount the crucible-cluster-link Secret into the controller and point CONTROLLER_CLUSTER_LINK_SECRETS at the directory: one file per cluster, named for it. A listed cluster without a file offers the paste fallback only. The controller's own credential on each cluster needs tokenreviews create (system:auth-delegator) to review the token, and the controller reaches the cluster's token endpoint (its oauth-openshift route) directly, not through the cluster's proxy_url.

Pasting a token

A link cluster without OAuth takes a pasted oc whoami -t or an OIDC ID token the cluster's API server accepts. The cluster's TokenReview names the owner; the token is dropped the same way.

API keys can read but never link or unlink.

MCP clients

People sign in with public OAuth clients (no secret, PKCE S256 required). Each client has its own client_id, and every code, approval and refresh token is bound to the client it was issued to: redeeming a code as another client is invalid_grant, and presenting a refresh token as another client is invalid_grant and revokes that approval. Access tokens carry the client in azp. Without configuration, three clients are registered:

Redirect matching follows RFC 8252 §7.3. A registered http URI on a loopback host matches any port on that same host with the same path, and nothing else: no query, no fragment, no credentials, and the request's URI must already be in canonical form. localhost, 127.0.0.1 and [::1] are three different hosts. A client registered on localhost cannot use 127.0.0.1, and the reverse. Any other registered URI must be https and matches only character for character. http on a host that is not a loopback host is never accepted. Codex adds a 12 character id of the MCP server to a pre-registered client's callback path. A registered path that ends in /* allows that one extra segment (1 to 64 of A-Z a-z 0-9 - _), and the built-in codex client uses it.

To register other clients, or to change the built-in ones, point CONTROLLER_PEOPLE_CLIENTS_FILE at a TOML file. The file replaces the built-in list:

[[clients]]
client_id = "claude-code"
name = "Claude Code"
redirect_uris = ["http://localhost/callback"]

[[clients]]
client_id = "codex"
name = "Codex"
redirect_uris = ["http://127.0.0.1/callback", "http://127.0.0.1/callback/*"]

[[clients]]
client_id = "lab-cli"
name = "Lab CLI"
redirect_uris = ["http://127.0.0.1/oauth/done"]
resources = ["https://crucible-kueue.apps.spoke-a.example.com/mcp"]   # default: every audience

A registered client_id is 1 to 64 of A-Z a-z 0-9 . _ -, so it can never look like a URL. If you remove a client, or narrow its resources, its refreshes are refused from the next refresh.

Self-registered clients

pi 1.x, Codex (registration auto) and Claude Code without --client-id register a client of their own before they sign in (RFC 7591). With people OAuth configured, the controller accepts such registrations at POST /oauth/register and advertises registration_endpoint. CONTROLLER_PEOPLE_DCR=false turns registration off and removes registration_endpoint from the metadata. Registration accepts only native public clients:

• token_endpoint_auth_method is absent or none. No secret, registration access token or management endpoint is issued;
• grant_types stay within authorization_code and refresh_token, and response_types within code;
• every redirect_uris entry is http on localhost, 127.0.0.1 or [::1], with no query, fragment or credentials, and at most 8 entries. At authorize, any port on the registered host with the registered path matches.

The client gets a random dyn-… client_id. Its codes and refresh tokens are bound to that id like any other client's. The name it chose is cleaned and cut to 100 characters, and the consent page and Settings always show it as " (self-registered)". The consent page also shows the host:port the code goes to. A registration that has made no successful token exchange (a code or a refresh) for 30 days stops authorizing and refreshing, and the hourly sweep deletes it with its approvals. Naming it to /oauth/authorize, signed in or not, does not keep it alive. Each source address may register 20 times an hour, and about 10000 registrations live at once (429 and 503 past those limits; concurrent registrations on several replicas can go slightly over the cap). The source is the TCP peer, or the last X-Forwarded-For hop when the peer is in CONTROLLER_PEOPLE_TRUSTED_PROXIES (the networks the cluster router connects from). Without that setting every registration behind a router counts against the router's address.

Client ID Metadata Documents

When CONTROLLER_PEOPLE_CIMD_HOSTS lists hosts, a client_id that is an https:// URL on one of those hosts is resolved as an OAuth Client ID Metadata Document (draft-ietf-oauth-client-id-metadata-document, the MCP 2025-11-25 client registration). The metadata then advertises client_id_metadata_document_supported: true. This is off by default, and every URL client_id is refused when it is off. The controller fetches the document when it is asked to authorize and refuses the request, before any consent, unless all of these are true:

• the URL is canonical https with a host name (not an IP address) and a path, and has no credentials, query or fragment;
• the host is on the allowlist, and resolves to a public address, never a loopback, private, link-local or other RFC 6890 special-use address (with HTTPS_PROXY set, the proxy resolves names, so this check moves to the proxy);
• the answer is 200 (redirects are not followed), JSON, at most 5 KiB, within 5 seconds;
• its client_id equals the URL exactly, it has a client_name and at least one redirect_uris entry, token_endpoint_auth_method is none, it carries no client_secret, and any grant_types or response_types include authorization_code and code.

Its redirect URIs follow the same rules as registered ones. A loopback URI in a document matches any port, but a document cannot use /*. A valid document is cached for its Cache-Control max-age (minus Age), 5 minutes if it sends none, and 1 hour at most. A document with no-store or no-cache, and any refused document, is never cached. The token endpoint does not fetch anything. It checks that the URL's host is still allowed, and the grant it redeems is bound to that URL. The consent page and Settings show the document's client_name with the URL's host (for example "Codex (chatgpt.com)"), so a document cannot pass for a registered client.

Codex publishes https://chatgpt.com/oauth/codex/client.json and pi publishes https://pi.dev/oauth/client.json. Allowlist chatgpt.com and pi.dev to accept them.

Signing in from each tool

Use the MCP server's URL, a people resource of the deploy profile ([kueue] people, at its Service URL or its people_urls entry). Each tool finds this controller through the server's protected resource metadata.

With self-registration on (the default), no client options are needed:

# Claude Code: then run /mcp in a session and choose Authenticate.
claude mcp add --transport http crucible-kueue https://crucible-kueue.apps.spoke-a.example.com/mcp

# pi
pi mcp add crucible-kueue --url https://crucible-kueue.apps.spoke-a.example.com/mcp
pi mcp login crucible-kueue

# Codex (mcp add starts the sign-in itself; mcp login repeats it)
codex mcp add crucible-kueue --url https://crucible-kueue.apps.spoke-a.example.com/mcp
codex mcp login crucible-kueue

Each tool then signs in as a self-registered client. To sign in under the registered client instead, so that consent and Settings show the verified name, or when CONTROLLER_PEOPLE_DCR=false, pass the registered id: claude mcp add --client-id claude-code …, pi mcp add … --oauth-client-id pi, codex mcp add … --oauth-client-id codex. Without self-registration and without a client id, Codex ends with "Dynamic client registration not supported", and pi ends with "Authorization server does not support dynamic client registration".

If chatgpt.com is on CONTROLLER_PEOPLE_CIMD_HOSTS, Codex in auto mode prefers its metadata document over registering, and signs in as https://chatgpt.com/oauth/codex/client.json. pi signs in under its document when its mcp.json entry has "oauth": {"clientRegistration": "cimd", "callbackUrl": "http://127.0.0.1/callback"} and pi.dev is allowlisted.

pi used to borrow claude-code. Under claude-code, pi's 127.0.0.1 callback is now refused. Re-add the server without OAuth options (or with --oauth-client-id pi) and sign in again. An existing pi approval keeps refreshing as claude-code until it expires or someone signs it out.

The consent page

GET /oauth/authorize from an MCP client parks the request on the browser session and sends the browser to /consent?id=… in the controller UI. The page names the client (its registered name, a metadata document's client_name with its host, or a self-registered client's chosen name marked "(self-registered)") and shows its client_id, the host:port the code goes to, the server, scopes and login, and the person's username on the server's cluster. A link cluster nobody has linked offers Link (and the paste box where the cluster has no OAuth) right there, and the sign-in lands back on the same page. Allow and Deny are a JSON call that returns the client's redirect URI with a code or an error, and the page navigates to it. A request answers once, and only from the browser and person that started it.

People's tokens and refresh

Access tokens last 15 minutes. Codes and refresh tokens live in Postgres, keyed by the SHA-256 of the secret, so every replica honours them and a restart signs nobody out. A refresh token is spent on use and the next one returned; presenting a spent one again revokes the whole approval. An approval dies after 14 days without a refresh or 90 days after it was granted. Every refresh reads the person again: a deleted account, one whose stored credential the identity provider refused, or a cluster username that was unlinked ends the approval, and a relink shows up in the next token. Settings → MCP clients lists the live approvals by client. Each client has its own Sign out (DELETE /api/me/mcp-sign-ins?client_id=<id>), and one button signs every client out (DELETE /api/me/mcp-sign-ins). Expired codes and approvals are swept hourly.

Vault and the secrets registry

Runs need credentials that belong to the people who launch them: a registry push token, a Jira token, a GitHub App key. The secrets registry delivers these credentials to a run, so nobody has to put them in a pod spec. A user or a team registers a secret in the UI, and the controller keeps the value in Vault. When a launch binds the secret, the controller delivers it to the pod of the run as a Kubernetes Secret that the pod owns (ADR-0028, ADR-0036).

The controller is the only Vault client in the fleet. Spokes do not have a Vault token. They get Kubernetes Secrets that the hub writes for one run and deletes with the run.

What the controller needs from Vault

• A KV v2 mount, VAULT_MOUNT (default crucible). The registry uses only the crucible/ prefix in the mount. A registered secret is at <mount>/crucible/registry/<owner>/<name>, and every write makes a new version. Other applications can share the mount, because the registry does not use its top level.
• A login method with a policy over that mount, one of:

Also set VAULT_ADDR, VAULT_CACERT for a private CA, VAULT_NAMESPACE on Vault Enterprise, and VAULT_RENEW_THRESHOLD_SECS (default 300), which sets how early the controller renews the login token.

The controller manages the token. It gets the token on the first call, renews it before its TTL ends, and gets a new token when a renewal fails or Vault denies a call. The controller retries a denied call exactly one time after a new login. Secret values cannot go into logs: tokens, secret ids, and payloads are redacted in Debug and have no Display.

Without VAULT_ADDR, the controller is not a Vault client. The write routes of the registry answer 503, and all other functions work.

Bootstrapping a mount and an AppRole

export VAULT_ADDR=https://vault.example.com VAULT_TOKEN=...
vault secrets enable -path=crucible -version=2 kv
vault auth enable approle
vault policy write crucible-hub - <<'EOF'
path "crucible/data/crucible/*"     { capabilities = ["create", "read", "update", "delete"] }
path "crucible/metadata/crucible/*" { capabilities = ["read", "list", "delete"] }
path "crucible/delete/crucible/*"   { capabilities = ["update"] }
EOF
vault write auth/approle/role/crucible-hub token_policies=crucible-hub \
  token_ttl=20m token_max_ttl=60m secret_id_num_uses=0 secret_id_ttl=0
vault read -field=role_id auth/approle/role/crucible-hub/role-id
vault write -f -field=secret_id auth/approle/role/crucible-hub/secret-id

Mount the secret id as a file and point the controller at it:

export VAULT_AUTH=approle
export VAULT_ROLE_ID=<role_id>
export VAULT_SECRET_ID_FILE=/var/run/secrets/vault-approle/secret-id
export VAULT_MOUNT=crucible

For the JWT method, enable jwt auth with the OIDC discovery URL of the cluster as the issuer. Create a role bound to the ServiceAccount of the controller and to the audience of the projected token. Set VAULT_AUTH=jwt, VAULT_JWT_ROLE=crucible-hub, and VAULT_JWT_TOKEN_FILE to the mount path of the projected token.

A local dev server

vault server -dev is sufficient for development. The end-to-end suite of the controller uses it (crucible-controller/tests/vault_e2e.rs, which creates its own mount, roles, and policies). Run the bootstrap above against docker run -d -p 8200:8200 -e VAULT_DEV_ROOT_TOKEN_ID=root hashicorp/vault:2.0 to get a working registry. Set VAULT_SECRET_ID directly, not as a file.

How a secret moves

1. Register. A user or a team registers a secret on the Secrets page: a name, a kind (opaque, file, registry_authfile, kubeconfig, inference_api_key), and the value. The value goes directly to Vault as a new version at <mount>/crucible/registry/<owner>/<name>. The ledger keeps the pointer, the kind, the owner, and the audit trail, but not the value. You can also register a secret by reference: a path outside the mount. The controller verifies it one time with the Vault token of the person who registers it. It uses that token for one read and then discards it.
2. Bind. A binding attaches a secret to a scope (a playbook, a schedule) with a projection: an environment variable or a file. To bind, you need bind on the secret and launch on the scope. The policy set decides both. A draft started from a team's blueprint gets that team's bindings for members of the team, and a placeholder for every other secret the pack declares.
3. Redeem. When the controller dispatches a bound launch, it reads the current version and writes a Kubernetes Secret in the namespace of the run, owner-referenced to the pod of the run. The pod spec projects it as the binding specifies. When the pod is deleted, Kubernetes deletes the Secret. The redactor of the engine gets every delivered value, so the value never appears in a session log.

The visibility of a secret tells if the agent can see it (agent_visible) or if only the broker process on the loop pod can see it (broker_only, the default). Members of a team can use secrets that the team owns, under the policy of the team. Sharing gives the same access to named users or groups.

Deployment secrets

The credentials of the controller (the GitHub token, the API token, the OIDC client secret) are deployment configuration, not registry entries. Keep them in the same Vault, and sync them into the namespace of the controller as Kubernetes Secrets. A CronJob or an external-secrets operator that reads <mount>/crucible/deploy/<release>/... on a schedule can do this. The controller is not involved.

Agent and people MCP architecture

This document describes the MCP stack that lets agents and people act on Kubernetes clusters through crucible. It covers these parts:

• the two MCP servers, crucible-kueue and buildit;
• the shared resource server library, crucible-mcp-auth;
• the controller as the OAuth issuer;
• the per-run gateway isolation that makes agent identities trustworthy.

The audience is engineers who maintain or extend this code, and reviewers of changes to it. The text follows ASD-STE100 Simplified Technical English. Code names appear exactly as in the source, in backticks. Code references use path:line from the repository root.

Note: Every part of this stack is off unless you configure it. Section 9 lists what stays inert when a value is absent.

Glossary

Each term has one sense in this document.

1. Overview

1.1 The problem

Agents in sandboxes must start cluster work: Jobs and RayJobs that train models, Deployments that serve models, and image builds. A Kubernetes credential must not enter the sandbox. Pack code in the loop pod must not choose who the agent is, or which namespace and queue it uses. People who use MCP clients on their laptops need the same tools, with their own Kubernetes permissions.

1.2 The shape

The controller is the one OAuth issuer for agents and for people. A sandbox supervisor proves its identity with a JWT-SVID from the SPIRE of its cluster. The controller finds the run that the sandbox belongs to and signs a short token. The token names the launcher (sub), the sandbox (act.sub), the run and the launcher's Kubernetes username (kube_user). A person gets a token through the authorization code flow with PKCE, after upstream SSO and consent. The MCP client is a public client: a registered client, a metadata document client, or a self-registered client.

The MCP servers verify each token against the issuer JWKS. Before each action, they ask the API server a SubjectAccessReview as kube_user. So the person's own RBAC is the ceiling. Each run's gateway runs in a pod of its own in the sandbox namespace. So pack code cannot create a supervisor with the SPIFFE ID of another run.

examples/optimize-loop is a worked playbook on these servers: the agent builds with buildit, benchmarks as Kueue Jobs through crucible-kueue and iterates, and a deterministic verifier checks its report against crucible-kueue's replies and the push registry (see the example).

1.3 Components and trust boundaries

flowchart LR
    subgraph Laptop["Person's machine"]
        MC["MCP client<br/>registered, metadata document<br/>or self-registered"]
        BR["Browser"]
    end
    SSO["Upstream SSO<br/>OIDC provider"]
    CIMD["Client metadata host<br/>(allowlisted only)"]
    subgraph HUB["hub cluster"]
        subgraph CTLNS["controller namespace"]
            CTL["controller<br/>issuer, /oauth/token,<br/>/.well-known/jwks.json"]
            PG[("Postgres")]
            KEY["issuer key Secret"]
        end
    end
    subgraph SPOKE["spoke-a cluster"]
        API["API server<br/>Service proxy,<br/>TokenReview, SAR"]
        subgraph SPIRENS["SPIRE namespace"]
            SPIRE["SPIRE server and agents"]
            OIDC["SPIRE OIDC discovery<br/>provider /keys"]
        end
        subgraph LOOPNS["loop namespace (pack code)"]
            LOOP["loop pod<br/>engine, broker,<br/>buildit :8848"]
        end
        subgraph BUILDNS["buildit namespace"]
            BUILDER["builder pods"]
        end
        subgraph SBXNS["sandbox namespace (gateway SA only)"]
            GW["gateway pod<br/>crucible openshell-gateway"]
            SUP["supervisor pod<br/>sandbox"]
        end
        subgraph KNS["crucible-kueue namespace"]
            KQ["crucible-kueue :8850"]
        end
        subgraph WNS["workload namespace"]
            WL["Jobs, RayJobs,<br/>Deployments, ..."]
        end
    end
    MC -- "people token" --> KQ
    MC -- "register, authorize, token" --> CTL
    BR -- "login" --> SSO
    BR -- "consent" --> CTL
    CTL --- PG
    CTL --- KEY
    CTL -- "GET metadata document" --> CIMD
    CTL -- "kube client: Sandbox reads,<br/>gateway objects, /keys proxy" --> API
    API --> OIDC
    LOOP -- "mTLS" --> GW
    GW -- "creates" --> SUP
    SUP -- "JWT-SVID" --- SPIRE
    SUP -- "assertion" --> CTL
    SUP -- "agent token" --> KQ
    SUP -- "agent token" --> LOOP
    LOOP -- "bind sandbox,<br/>pod token" --> CTL
    LOOP --> BUILDER
    KQ -- "SAR as kube_user,<br/>then acts as own SA" --> API
    KQ --> WL

The diagram shows one spoke. The hub can also host runs; it then has the same namespaces. The trust boundaries are the namespaces:

• The controller namespace holds the issuer key. The controller refuses to start when this namespace is also the loop, rig, sandbox, crucible-kueue or workload namespace (crucible-controller/src/agent_auth/placement.rs:74).
• The loop namespace runs pack code. The loop service account has no verbs in the sandbox namespace under an isolated profile.
• The sandbox namespace allows pod creation only to crucible-openshell-gateway. A SPIFFE ID there is as trustworthy as that rule.
• The crucible-kueue namespace holds the server service account. Workload verbs of that account apply only in the workload namespace.
• The person's machine holds only a token and a refresh token, never a cluster credential.
• A client metadata host is outside every boundary. The controller fetches a JSON document from it only when its host is in CONTROLLER_PEOPLE_CIMD_HOSTS. The document can name a client and its redirect URIs, and nothing more.

2. Trust model and identities

2.1 Principals

A run acts for its launcher. The launcher is runs.dispatched_by, which must be user:<login>. The controller stamps it when it creates the run row (crucible-controller/src/runs/store.rs:334). The value is the person who last asked for this dispatch: an unpark, a redispatch, a scope, or a paired GitHub approval. Without such a request, the value is the owner of the run. A run that has no person behind it gets no token (GrantError::NoLauncher).

2.2 Token types and claims

All tokens are compact JWS, algorithm ES256, with kid in the header. The issuer signs both kinds with the same key and the same iss.

The agent token is built in crucible-controller/src/agent_auth/issuer.rs:820. The people token is built in crucible-controller/src/agent_auth/people.rs:299.

Note: Claims name identity only. There is no role and no queue list in a token. Each resource server decides what the caller may do. crucible-kueue ignores role and queues claims if a token carries them.

2.3 Issuer keys and JWKS

The issuer key is a PKCS#8 P-256 key. The kid is the RFC 7638 thumbprint of its public half, so a new key gets a new kid with no configuration (issuer.rs:152). The JWKS at /.well-known/jwks.json publishes the current key and, when configured, the previous key (issuer.rs:820). The previous key verifies only. It signs nothing.

To rotate the issuer key, do these steps:

1. Move the current key file to CONTROLLER_AGENT_PREVIOUS_SIGNING_KEY_FILE.
2. Mount a new key at CONTROLLER_AGENT_SIGNING_KEY_FILE.
3. Restart every controller replica.
4. Wait for one token lifetime. The default is 900 seconds; people tokens can live up to 3600.
5. Remove the previous key variable and restart again.

Warning: All replicas must mount the same keys. A replica with another key signs tokens that the other replicas do not publish.

Resource servers cache the JWKS for 600 seconds. A token with an unknown kid causes a refetch, but not more than one fetch each 30 seconds (crucible-mcp-auth/src/jwks.rs:152). A failed fetch keeps the cached keys. The client refuses a key set larger than 256 KiB and follows no redirects.

2.4 Lifetimes

2.5 What each component verifies

The controller token endpoint verifies the assertion. Section 3 gives the full list.

Every resource server verifies a token through ResourceServer::validate (crucible-mcp-auth/src/server.rs:201) and Verifier::verify (crucible-mcp-auth/src/verify.rs:65):

1. The header algorithm is asymmetric (RS, PS, ES or EdDSA). HS* and none never verify.
2. The header has a kid, and the JWKS of the issuer has that key. Under an issuer trusted for runs (the controller), the header typ is at+jwt (RFC 9068 §2.1), for a run's token and a person's alike. The controller types every access token it mints, so an ID token, an assertion or any other JWT under its iss and key is refused. An issuer trusted only for people, such as an OIDC realm, is not held to it.
3. The algorithm of the key, when the JWK declares one, is the header algorithm.
4. The signature is correct.
5. iss is exactly a trusted issuer string. There is no URL normalization.
6. aud names at least one configured resource URL.
7. exp and nbf hold, with 30 seconds of leeway.
8. exp, iss, aud and sub are present.
9. The token has every scope in required_scopes (kueue.read or buildit.read).

Note: the controller started typing its access tokens and the servers started requiring it in the same change. A live hub and its spokes need the controller and crucible-kueue/buildit rolled together: servers built with the check refuse an older controller's untyped tokens. An older server ignores typ, so rolling the controller first is safe.

The configuration of each trusted issuer says whom its tokens stand for: runs or people (Callers). When one issuer is trusted for both, a token with act is a run token and a token without act is a people token. The token cannot choose its kind in any other way.

Then each server applies its own rules:

2.6 Why kube_user RBAC is the ceiling

crucible-kueue acts with its own service account. That account can create workloads in the workload namespace. Before each call, the guard turns the call into access checks. For each check it asks the API server a SubjectAccessReview as kube_user, with the groups of that user (crucible-mcp-auth/src/kube.rs:209). The server does the call only when every review allows it. So:

• A token can do at most what its kube_user can do with kubectl. The token scopes, the tool policy, the allowlist profile and the server account reach can only reduce that.
• An agent token acts as its launcher. A run cannot do more than the person who launched it.
• The issuer and the reviewer both refuse a system: username (crucible-controller/src/identity/kube_identity.rs:27, crucible-mcp-auth/src/kube.rs:67). So no token can act as a node, a service account or the cluster. Groups from a token claim also lose every system: entry.
• The server creates pods with its own identity. So the caller must be able to get each Secret and PersistentVolumeClaim that a manifest names (crucible-kueue/src/access.rs:149).
• A new RoleBinding applies on the next call, because denials are not cached. A removed permission can still allow calls for up to 30 seconds, because allowed decisions are cached.
• workload_status and tasks/get read the workload's Kueue Workloads, its pods and their events with the server's account, and show each only when kube_user may get and list it in the namespace. A part the caller may not read is left out, and the reply's omitted names it and why (crucible-kueue/src/access.rs, StatusPart).
• Any caller who may use a queue can discover its hardware. This is a guarantee, and a deliberate exception to the ceiling: hardware_describe and queue_describe check only get or the submit verb (use) on the LocalQueue, and hardware_describe reads the nodes behind the queue's flavors with the server's account, so an agent can pick a flavor and a GPU product without RBAC on nodes. It returns each flavor's node labels and taints, and for the nodes those select, how many there are and how many are schedulable per GPU product, and their accelerator capacity. Node names are not returned. Tests hold the guarantee (crucible-kueue/tests/http_auth.rs, and cluster_e2e.rs against a cluster).

The issuer never decides what a person may do. It decides only who the person is on each cluster (section 5).

3. Agent token flow

3.1 From dispatch to the first MCP call

sequenceDiagram
    autonumber
    participant C as controller (hub)
    participant A as API server (spoke-a)
    participant L as loop pod (engine)
    participant G as gateway pod
    participant S as supervisor
    participant P as SPIRE agent
    participant K as crucible-kueue
    C->>A: create gateway Service in sandbox namespace
    A-->>C: ClusterIP
    Note over C: mint run CA, server cert for the ClusterIP,<br/>loop and supervisor client certs, drop CA key
    Note over C: run Secret gets loop client TLS, gateway URL,<br/>CRUCIBLE_AGENT_TOKEN_ENDPOINT, CRUCIBLE_AGENT_SVID_AUDIENCE,<br/>CRUCIBLE_AGENT_SERVERS, run_agent_tools row written
    C->>A: loop Service, NetworkPolicy, gateway pod, TLS Secrets owned by the gateway pod
    C->>A: create loop pod
    L->>G: health check over mTLS, register the CLI
    L->>G: import token-grant provider profiles (crucible-kueue, buildit)
    L->>G: CreateSandbox with those providers
    G->>A: create Sandbox and supervisor pod,<br/>label openshell.ai/gateway-id, annotation openshell.ai/sandbox-id
    G-->>L: sandbox id
    L->>C: POST /api/sandbox-bindings/runs/{run}<br/>Bearer pod token, {sandbox_id, workdir}
    C->>A: list Sandbox by sandbox-id and gateway-id labels
    A-->>C: the Sandbox and its name
    Note over C: write bind record, owner-reference the Sandbox to the gateway pod
    C-->>L: 204
    S->>P: fetch JWT-SVID, audience = issuer
    P-->>S: JWT-SVID
    S->>C: POST /oauth/token client_credentials,<br/>client_assertion = JWT-SVID, audience = kueue URL, scope
    C->>A: GET SPIRE /keys through the Service proxy (cached)
    A-->>C: SPIRE JWKS
    Note over C: verify the assertion, read gateway id and sandbox id,<br/>find the bind record, check cluster and gateway,<br/>resolve the launcher and kube_user
    C-->>S: access_token, expires_in, scope
    Note over S: inject Authorization on egress<br/>to the kueue host and port only
    S->>K: MCP request with Bearer token
    K->>A: SubjectAccessReview as kube_user
    K->>A: act with the server service account

The steps are these:

1. Dispatch reserves the gateway. The Service comes first because its ClusterIP goes into the server certificate (crucible-controller/src/runs/workpod/gateway.rs:184).
2. The controller folds the loop pod client material and the agent server access into the run Secret before it renders the loop pod (crucible-controller/src/runs/workpod/run.rs:813, access in crucible-controller/src/runs/workpod/agent_servers.rs). A server is granted only where the issuer can mint its tokens: it trusts the run's cluster's SPIRE, discovered or named in CONTROLLER_AGENT_SPIRE (crucible-controller/src/runs/workpod/agent_servers.rs, granted). The loop pod gets them as one CRUCIBLE_AGENT_SERVERS list. The run records the servers that it got, with the scopes of the audience.
3. The gateway starts before the loop pod, so it boots while the loop pod schedules (gateway.rs:256).
4. The engine reads CRUCIBLE_OPENSHELL_GATEWAY_URL and talks to the remote gateway over mTLS (crucible/src/openshell/gateway.rs:422-437). It creates one token-grant provider, one egress entry and one .mcp.json entry for each server in CRUCIBLE_AGENT_SERVERS the same way, after starting a server whose transport is the loop pod (crucible/src/openshell/agent_servers.rs). A provider holds no secret and gives the sandbox no placeholder (crucible/src/openshell/provider.rs:368).
5. After CreateSandbox, the engine binds the sandbox (crucible/src/openshell/run.rs:469). It binds only when the loop pod carries CRUCIBLE_AGENT_TOKEN_ENDPOINT. A failed bind does not fail the turn. The sandbox then gets no token.
6. The bind route trusts only the pod token of that run (crucible-controller/src/agent_auth/bindings.rs:298). It reads the sandbox name from the cluster, never from the request (crucible-controller/src/agent_auth/directory.rs:60).
7. The supervisor asks SPIRE for a JWT-SVID with the audience in token_grant.jwt_svid_audience. That value is the issuer identifier.
8. The token request has exactly the fields that OpenShell sends: grant_type, client_assertion_type, client_assertion, audience, scope. Any other field, for example client_id or resource, is refused (crucible-controller/src/agent_auth/mod.rs:149).
9. The supervisor injects the token as a bearer header only on egress to the host and port of the provider endpoint, from a binary the provider's profile lists. The profile lists the same binaries as the sandbox policy: OpenShell injects a grant only when the profile's own rule admits the request, and with binary identity required, a rule admits only the binaries it names. The agent never sees the token.

3.2 What the token endpoint checks

issue (crucible-controller/src/agent_auth/mod.rs:314) does these checks in this order:

1. The issuer mints agent tokens: the profile isolates gateways and describes a server, or CONTROLLER_AGENT_SPIRE names a cluster.
2. The deploy profile is isolated and names a sandbox namespace (crucible-controller/src/agent_auth/placement.rs:48).
3. grant_type is client_credentials.
4. client_assertion_type is urn:ietf:params:oauth:client-assertion-type:jwt-bearer.
5. The assertion iss is the SVID issuer of exactly one configured cluster. That cluster's SPIRE keys verify the assertion (crucible-controller/src/agent_auth/issuer.rs:701).
6. The assertion aud is exactly one value, the issuer identifier.
7. The assertion has a numeric iat, not more than 30 seconds in the future.
8. exp minus iat is not more than 900 seconds.
9. sub is <gateway prefix><gateway id>/sandbox/<sandbox id> for the prefix of that cluster.
10. A bind record exists for the sandbox id, and its loop pod is in the running state.
11. The cluster of the run is the cluster whose SPIRE signed the assertion.
12. The requested audience is the URL of a server the deploy profile describes on the run's cluster (Audience::of).
13. The gateway id in the SPIFFE ID is the gateway id of the bound loop pod.
14. The run has a launcher (dispatched_by is user:<login>).
15. Each requested scope is in the audience entry. A request with no scope gets all the scopes of the entry.
16. When the audience requires a kube_user (the profile describes its server as one that authorizes as kube_user, as [kueue] does), it is known for the launcher on the run's cluster. For other audiences, the token omits an unknown kube_user, and the controller logs a warning. Every audience is described: at startup the controller refuses one for which the profile describes no server at that URL on that cluster, comparing URLs once parsed (Audience::described_by, crucible-controller/src/agent_auth/issuer.rs).

Note: Pack code holds the run CA's client certificate, so it can make its gateway create sandboxes and request a token for any audience on its run's cluster, with that audience's scopes. The audience allowlist, per cluster, is the boundary: list only the servers a run on that cluster may use, with the scopes it may hold.

The controller reads SPIRE keys through the API server Service proxy of each cluster, with the same client it dispatches with (crucible-controller/src/agent_auth/spire.rs:21). The path is /api/v1/namespaces/<ns>/services/<scheme>:<service>:<port>/proxy/keys (issuer.rs:189). The scheme is https unless the entry starts with http://.

3.3 Refusals

Every refusal returns only the OAuth error code and a fixed description. The detail goes to the controller log as agent token refused with reason and the user_agent.

The bind route has its own refusals (bindings.rs:320):

Note: Pack code holds the gateway client certificate of its own run. It can create sandboxes for its own run with the driver_config fields that the gateway accepts. Those sandboxes carry the gateway id of its own run, so the issuer refuses them for any other run.

4. People token flow

4.1 MCP clients

People sign in with public OAuth clients. A public client has no secret, so PKCE S256 is required. Each code, approval and refresh token is bound to the client ID that it was issued to. The people token names that client in azp.

4.1.1 Client ID forms

ClientId::parse (crucible-controller/src/agent_auth/people/clients.rs:133) puts each client_id in one of three forms. The form decides who vouches for the client.

Clients (crucible-controller/src/agent_auth/people/registry.rs:27) holds the registry, the metadata document resolver and the registrations. It has two entry points:

• resolve (registry.rs:66) serves /oauth/authorize. It returns the client with its name and redirect rules. For a metadata document client, it fetches the document.
• admits (registry.rs:87) serves /oauth/token. It fetches nothing. It refuses a registered client that the registry no longer lists, a metadata document client whose host is no longer allowlisted, and a registration that is no longer live. The grant that the request redeems is already bound to its client ID.

flowchart TD
    REQ["client_id from the request"] --> P{"ClientId::parse"}
    P -- "fails" --> BAD["refuse: error page<br/>or invalid_client"]
    P -- "https:// URL" --> MD{"CONTROLLER_PEOPLE_CIMD_HOSTS<br/>set and lists the host?"}
    MD -- no --> BAD
    MD -- yes --> FETCH["authorize: fetch or cache hit<br/>token: host check only"]
    FETCH --> DOC{"document valid?"}
    DOC -- no --> BAD
    DOC -- yes --> OK["Client: id, name, redirect rules"]
    P -- "dyn- id" --> DYN{"people_dynamic_clients row,<br/>used in the last 30 days?"}
    DYN -- no --> BAD
    DYN -- yes --> STAMP["stamp last_used_at"] --> OK
    P -- "registered name" --> REG{"in the client registry?"}
    REG -- no --> BAD
    REG -- yes --> OK

4.1.2 The client registry

Without configuration, the registry has three clients (clients.rs:448):

CONTROLLER_PEOPLE_CLIENTS_FILE names a TOML file of [[clients]] tables. Each table has client_id, name, redirect_uris and an optional resources list. The file replaces the built-in list, so list the built-in clients too if you want to keep them (clients.rs:479). The controller does not start when the file is malformed, when it lists no client or one client twice, or when a resources entry is not a people resource (crucible-controller/src/agent_auth/people.rs:172). Without resources, a client can get tokens for every audience.

[[clients]]
client_id = "lab-cli"
name = "Lab CLI"
redirect_uris = ["http://127.0.0.1/oauth/done"]
resources = ["https://crucible-kueue.apps.spoke-a.example.com/mcp"]

When you remove a client, its next refresh gets invalid_client. When you remove a resource from its resources, its next refresh for that resource revokes the approval.

Redirect matching follows RFC 8252 §7.3 (RedirectRule, clients.rs:237):

• A registered http URI on localhost, 127.0.0.1 or [::1] matches any explicit port on the same host, with the same path. It matches no query, no fragment and no credentials.
• localhost, 127.0.0.1 and [::1] are three different hosts. A client registered on localhost cannot use 127.0.0.1.
• A registered path that ends in /* also matches one more segment of 1 to 64 base64url characters. Codex adds such a server id to the callback path. Only the registry can use /*.
• Any other registered URI must be https. It matches only character for character.
• The requested URI must already be in canonical form (Client::redirect, clients.rs:348).

4.1.3 Client ID Metadata Documents

CONTROLLER_PEOPLE_CIMD_HOSTS is a comma-separated list of host names. When it is set, a client_id that is an https:// URL on one of those hosts is resolved as an OAuth Client ID Metadata Document (draft-ietf-oauth-client-id-metadata-document). When it is unset, every URL client_id is refused (ClientError::MetadataOff).

At /oauth/authorize, Cimd::client (crucible-controller/src/agent_auth/people/cimd.rs:370) returns a cached document or fetches it, and only for a signed-in browser: a request naming a metadata document without a session goes to /auth/login first, so nobody can make the controller fetch without signing in. The fetch (cimd.rs:390) has these rules:

• The host must be on the allowlist (cimd.rs:362).
• The resolver gives only public addresses. It drops loopback, private, link-local and other RFC 6890 special-use addresses (cimd.rs:127, cimd.rs:170). With HTTPS_PROXY set, the proxy resolves the names, so this check moves to the proxy.
• The client uses only https and follows no redirects (cimd.rs:338).
• The answer must be 200, JSON, at most 5 KiB, and arrive within 5 seconds.

The document must describe a public client of its own URL (client_of, cimd.rs:200):

• client_id is the URL exactly.
• client_name is 1 to 100 printable characters.
• redirect_uris has at least one entry. The entries follow the registry rules, but /* is not accepted.
• token_endpoint_auth_method is none. There is no client_secret and no client_secret_expires_at.
• grant_types and response_types, when present, include authorization_code and code.

A valid document is cached for its Cache-Control max-age minus Age. Without max-age it is cached for 300 seconds. The limit is 3600 seconds (freshness, cimd.rs:266). A document with no-store or no-cache is not cached. A refused document is not cached.

The consent page and Settings show the client_name of the document with the host of the URL, so a document cannot look like a registered client. Codex publishes https://chatgpt.com/oauth/codex/client.json. pi publishes https://pi.dev/oauth/client.json.

4.1.4 Self-registered clients

pi 1.x, Codex and Claude Code without a client id register a client before they sign in (RFC 7591). The controller accepts these registrations at POST /oauth/register (crucible-controller/src/agent_auth/people/dynamic.rs:422). Registration is on by default when the deploy profile serves people. CONTROLLER_PEOPLE_DCR=false turns it off.

sequenceDiagram
    autonumber
    participant M as MCP client
    participant C as controller (any replica)
    participant D as Postgres
    M->>C: GET /.well-known/oauth-authorization-server
    C-->>M: registration_endpoint = issuer/oauth/register
    M->>C: POST /oauth/register {client_name, redirect_uris, ...}
    Note over C: check: public client, loopback http redirect URIs,<br/>grant and response types, at most 8 URIs
    Note over C: rate limit: 20 each hour for each source address
    C->>D: INSERT people_dynamic_clients<br/>only while fewer than about 10000 are live
    C-->>M: 201 {client_id: dyn-..., token_endpoint_auth_method: none}
    M->>C: GET /oauth/authorize?client_id=dyn-...
    C->>D: UPDATE last_used_at WHERE client_id AND not idle, RETURNING the client
    Note over C,D: the hourly sweep deletes registrations idle for 30 days<br/>and the approvals of deleted registrations

check (dynamic.rs:194) accepts only native public clients:

• token_endpoint_auth_method is absent or none. The controller issues no secret, no registration access token and no management endpoint.
• grant_types includes authorization_code and stays within authorization_code and refresh_token. response_types is code.
• redirect_uris has 1 to 8 entries, each at most 512 characters. Each entry is http on localhost, 127.0.0.1 or [::1], with no query, fragment or credentials.
• The controller removes control characters and repeated white space from client_name and keeps at most 100 characters. An empty name becomes Unnamed client.

The client ID is dyn- and 24 random bytes in base64url. The consent page and Settings show the name as " (self-registered)".

Dcr::register (dynamic.rs:287) applies two limits:

• Each source address can register 20 times in each hour. The limit is in memory, for each replica. Past it, the answer is 429 with temporarily_unavailable.
• About 10000 registrations used in the last 30 days can exist. Past it, the answer is 503 with temporarily_unavailable. The count and the insert are one statement but not serialized across replicas, so concurrent registrations can go slightly over: the cap is soft.

The source address is the TCP peer. When the peer is in CONTROLLER_PEOPLE_TRUSTED_PROXIES, as the cluster router is, the source is the last X-Forwarded-For hop (source, dynamic.rs:428). Any other peer, an in-cluster caller included, is its own source and cannot pick a bucket.

Dcr::client (dynamic.rs:345) reads a registration and stamps last_used_at in one statement. A registration that is not used for 30 days stops working. The hourly sweep deletes it and the approvals issued to it (Dcr::sweep, dynamic.rs:375).

4.1.5 Clients on consent and in Settings

The consent API (crucible-controller/src/identity/api/consent.rs:63) returns these client fields:

GET /api/me/mcp-sign-ins returns the same client_id, client_name, client_host and client_kind for each live approval. The approval stores the name that the person saw on the consent page.

4.2 Discovery, authorization and code exchange

sequenceDiagram
    autonumber
    participant M as MCP client
    participant K as crucible-kueue
    participant C as controller (any replica)
    participant B as browser
    participant U as upstream SSO
    participant O as cluster OAuth server
    participant H as client metadata host
    participant D as Postgres
    M->>K: POST /mcp, no token
    K-->>M: 401, WWW-Authenticate names the resource metadata URL
    M->>K: GET /.well-known/oauth-protected-resource/mcp
    K-->>M: resource, authorization_servers = [issuer], scopes_supported
    M->>C: GET /.well-known/oauth-authorization-server
    C-->>M: authorization_endpoint, token_endpoint, S256, iss parameter,<br/>registration_endpoint, client_id_metadata_document_supported
    opt no client id configured and registration on
        M->>C: POST /oauth/register
        C-->>M: client_id = dyn-...
    end
    M->>B: open /oauth/authorize?client_id, code_challenge, state, resource, redirect_uri=http://127.0.0.1:port/callback
    B->>C: GET /oauth/authorize
    alt metadata document client
        C->>H: GET client_id URL (unless cached)
        H-->>C: client metadata JSON
    else self-registered client
        C->>D: read and stamp people_dynamic_clients row
    end
    Note over C: resolve the client, match redirect_uri
    alt no native browser session
        C-->>B: 303 /auth/login?rd=...
        B->>U: OIDC login with PKCE
        U-->>B: code
        B->>C: /auth/callback, session row in Postgres
        C-->>B: 303 back to /oauth/authorize
    end
    Note over C: validate, park the request on the session
    C-->>B: 303 /consent?id=...
    B->>C: GET /api/oauth/consent?id=...
    C-->>B: client name, kind, host, redirect origin,<br/>resource, scopes, login, identity on the cluster
    opt link cluster with no link
        B->>C: POST /api/me/cluster-identities/{cluster}/link
        C-->>B: authorize_url of the cluster OAuth server
        B->>O: authorize with PKCE, scope user:info
        O-->>B: code
        B->>C: GET /oauth/cluster-link/{cluster}/callback
        C->>O: redeem code
        C->>C: TokenReview on that cluster, write link, drop the cluster token
    end
    B->>C: POST /api/oauth/consent {id, decision: approve}
    C->>D: insert people_codes row (SHA-256 of the code), bound to the client ID
    C-->>B: {redirect_url: redirect URI?code, state, iss}
    B->>M: navigate to the redirect URI
    M->>C: POST /oauth/token grant_type=authorization_code, client_id, code, code_verifier, redirect_uri, resource
    Note over C: admit the client ID (no fetch)
    C->>D: delete people_codes row, returning the grant
    Note over C: check client ID, redirect_uri, PKCE S256,<br/>resource, resolve kube_user again
    C->>D: insert people_grants with client_id and client_name,<br/>and the first people_refresh_tokens row
    C-->>M: access_token (azp = client ID), refresh_token, expires_in, scope
    M->>K: POST /mcp, Bearer token

The steps are these:

1. crucible-kueue answers a request without a token with 401 and a challenge that names its RFC 9728 metadata. The metadata lists the trusted issuers as authorization_servers (crucible-mcp-auth/src/metadata.rs:42).
2. The MCP client reads the RFC 8414 metadata of the controller (crucible-controller/src/agent_auth/issuer.rs:831, crucible-controller/src/agent_auth/mod.rs:445). People support adds authorization_endpoint, the authorization_code and refresh_token grants, the code response type, S256, the none client authentication, and authorization_response_iss_parameter_supported. Two fields depend on the client configuration:
   • registration_endpoint is <issuer>/oauth/register while registration is on. It is absent with CONTROLLER_PEOPLE_DCR=false.
   • client_id_metadata_document_supported is true while CONTROLLER_PEOPLE_CIMD_HOSTS is set. It is absent otherwise.
3. /oauth/authorize (crucible-controller/src/agent_auth/people.rs:554) resolves the client (section 4.1) and matches redirect_uri against its redirect rules. state is required, at most 1024 characters. The method must be S256. resource must be a people resource of the deploy profile (DeployProfile::people_resources), and a registered client with resources must list it (validate, people.rs:373).
4. An unknown client, a refused metadata document, an expired registration or a bad redirect gives an error page. The controller never redirects to a URI that the client did not register. When the client and the redirect are correct, the controller sends every other error to the redirect URI with error, state and iss.
5. Without a native session, the controller sends the browser to /auth/login. After the login, the browser returns to /oauth/authorize. A session whose group refresh failed counts as no session.
6. The request waits on the session under a random id. The consent page shows it only to the same browser and the same person. The page names the client and shows the host:port that the code goes to (section 4.1.5). No page may frame it: every SPA response and the authorize page send X-Frame-Options: DENY and Content-Security-Policy: frame-ancestors 'none'.
7. The approval resolves kube_user for the cluster of the audience. An unlinked link cluster answers 409. The controller parks the request on the session again, so the page can link and try again.
8. The code is 32 random bytes, base64url. Postgres stores only its SHA-256. The stored grant names the client ID and the client name.
9. The token endpoint admits the client ID without a fetch (registry.rs:87). It marks the code spent under a row lock, so a code works once on any replica (redeem_code in crucible-controller/src/agent_auth/people/grants.rs). A code issued to another client ID is invalid_grant (redeem_code, people.rs). The code is spent all the same. The approval the redemption starts is recorded against the code, and a spent code stays a day past its expiry. Presented again by the client it was issued to, it is invalid_grant and revokes that approval, so the refresh token the first redemption issued stops working (OAuth 2.1 §4.1.3, RFC 9700 §4.5.3); a replay that lands before the approval starts stops it from starting. Presented by another client, it is only unknown, so a code someone saw cannot revoke a person's approval.

Note: Claude Code does not step up after a 403. So a person always gets every scope that the audience lists. Requested scopes outside that list are refused; the OIDC scopes openid, profile, email and offline_access are ignored.

4.3 Refresh with rotation and reuse detection

sequenceDiagram
    autonumber
    participant M as MCP client
    participant C as controller (any replica)
    participant D as Postgres
    M->>C: POST /oauth/token grant_type=refresh_token, refresh_token, client_id
    Note over C: admit the client ID (no fetch),<br/>else invalid_client and nothing is spent
    C->>D: UPDATE people_refresh_tokens SET used_at WHERE key = sha256 AND used_at IS NULL
    alt row updated
        C->>D: read people_grants FOR UPDATE
        Note over C: refuse when revoked
        alt client_id is not the client of the approval
            C->>D: revoke the approval and delete its live token
            C-->>M: invalid_grant
        else same client
            Note over C: refuse when idle over 14 days,<br/>or older than 90 days
            alt another resource or a scope beyond the approval
                C->>D: roll back: the token stays unspent
                C-->>M: invalid_target or invalid_grant
            end
            C->>D: issue the next refresh token
            Note over C: client still allowed for the resource,<br/>read the person again: users row, stored<br/>credential not refused, kube_user on the cluster
            alt person still allowed and kube_user known
                C-->>M: new access_token and new refresh_token
            else refused
                C->>D: revoke the approval
                C-->>M: invalid_grant
            end
        end
    else token already spent
        C->>D: revoke the approval and delete its live token
        C-->>M: invalid_grant
    end

The rotation is one conditional UPDATE, so two replicas cannot both spend one refresh token (grants.rs:223). A spent token stays in the table until its approval goes. When a client presents it again, the controller revokes the whole approval (RFC 9700 §4.14.2). A live token that another client ID presents also revokes the approval (grants.rs:279). A refresh that asks for another resource or a wider scope is refused inside the same transaction, before anything commits (refresh_fits, people.rs:765; grants.rs:309), so the token it presented stays live and the client can refresh with it again.

A refresh also checks the client again. A registered client that the registry no longer lists gets invalid_client. A registered client whose resources no longer lists the resource of the approval loses the approval.

Every refresh reads the person again (crucible-controller/src/identity/people_directory.rs:35):

• A deleted users row revokes the approval.
• A stored offline credential that the upstream SSO refused (user_credentials.last_error) revokes the approval.
• An unlinked cluster username revokes the approval. A new link appears in the next token.

Warning: A refresh does not ask the upstream identity provider. A person deactivated there keeps refreshing until something else refuses them: a stored offline credential the SSO then refuses, the 14-day idle limit, or the 90-day cap on an approval (REFRESH_IDLE_SECS, REFRESH_ABSOLUTE_SECS in crucible-controller/src/agent_auth/people/grants.rs). Without a stored credential (CONTROLLER_CREDENTIAL_KEY unset, or a person who never stored one), nothing else does. To cut a person off sooner, delete their users row, which also drops their stored credential and API keys.

Warning: A client that retries a refresh after a lost response presents a spent token. The controller treats that as reuse and revokes the approval. The person must approve the client again.

A background task deletes expired codes and dead approvals each hour (grants.rs:425). With registration on, the same task deletes registrations idle for 30 days and the approvals of deleted registrations. It starts only when people tokens are configured.

4.4 Several replicas

Any replica can do any step of the flow:

• Browser sessions, the parked authorization request and a started cluster link live in the Postgres session store.
• Codes, approvals and refresh tokens live in people_codes, people_grants and people_refresh_tokens. Each method is one statement or one transaction.
• Every replica signs with the same issuer key and publishes the same JWKS.

Leader election (CONTROLLER_LEASE_NAME) applies only to the daemon. The API, the token endpoint and the bind route answer on every replica.

4.5 Settings and revocation

GET /api/me/mcp-sign-ins lists the live approvals of the caller, with the client of each (crucible-controller/src/identity/api/mcp_sign_ins.rs:57). Settings groups them by client.

• DELETE /api/me/mcp-sign-ins?client_id=<client ID> revokes the approvals of one client and deletes its unused codes (grants.rs:330). A client_id that does not parse gets 400.
• DELETE /api/me/mcp-sign-ins revokes all approvals of the caller and deletes their unused codes.

An API key cannot revoke approvals (mcp_sign_ins.rs:113).

5. Per-cluster identity

5.1 Identity rules

The deploy profile gives each cluster one identity rule. The hub rule is [cluster] identity. A spoke rule is [clusters.<name>] identity. A cluster without a rule is link (forge/src/fleet.rs:61).

Use a claim rule only when the cluster API server trusts the same SSO for the same people. Use link in all other cases.

5.2 Links

A person proves a link in one of two ways:

• Paste a cluster token: PUT /api/me/cluster-identities/{cluster} with {"cluster_token": "..."}. The controller sends a TokenReview to that cluster (crucible-controller/src/identity/kube_links.rs:108).
• Use the OAuth server of an OpenShift cluster: POST /api/me/cluster-identities/{cluster}/link. The controller is a confidential OAuthClient there. It redeems the code with PKCE for a user:info token and sends that token to a TokenReview (crucible-controller/src/identity/cluster_link.rs:204, cluster_link.rs:237).

Either way the controller stores only the username, the source (token-review or openshift-oauth) and the time. It never stores or logs the cluster token. A token that another cluster minted does not authenticate, so it cannot link this cluster. A link to a system: username is refused.

The OAuth link is offered for a cluster only when all of these are true:

• [cluster_link] openshift lists the cluster.
• The controller has a client for the cluster.
• A file with the cluster's name exists in CONTROLLER_CLUSTER_LINK_SECRETS.

The callback is GET /oauth/cluster-link/{cluster}/callback. It checks the state in constant time, the cluster, the person and the 10-minute limit. It then lands on /settings?linked=<cluster> or ?link_error=<code>. The code is a fixed word, so a crafted URL cannot put text on the page.

5.3 The hub override

A platform administrator can set the hub username of a person: PUT /api/users/{login}/kube-user with {"kube_user": "<name>"}, or null to clear it. An administrator cannot set their own. The event log records it on user:<login>. The override applies on the hub only, and it wins over the hub rule and any hub link (crucible-controller/src/identity/cluster_identity.rs:66).

5.4 Resolution

flowchart TD
    START["resolve(person, cluster)"] --> HUB{"cluster is hub and<br/>users.kube_user_override set?"}
    HUB -- yes --> OV["Known: override"]
    HUB -- no --> RULE{"identity rule of the cluster"}
    RULE -- "claim" --> CL{"claim value present<br/>and a valid username?"}
    CL -- yes --> KR["Known: prefix + claim"]
    CL -- no --> UN["Unavailable: reason"]
    RULE -- "link" --> LK{"kube_identity_links row<br/>for (sub, cluster)?"}
    LK -- yes --> KL["Known: link"]
    LK -- no --> NL["Unlinked"]

The same function decides kube_user for every consumer:

Issuance and the tools record name who acts as an authz::model::Principal, and for_principal (crucible-controller/src/identity/cluster_identity.rs) resolves it: a user: principal through their sign-in, a run: principal through the person it was dispatched for. A group: or team: principal, or a run dispatched for one, has no username of its own and is refused with a typed error (PrincipalError), which the token endpoint answers with invalid_grant. A team's own identity per cluster would be one more arm of for_principal; the tables would not change.

A launcher that never logged in has no users row, so its kube_user is unavailable. Because the agent token resolves at each request, a new link applies to the next token of a run that is already live.

Note: The links, the override and the settings panels are served only when the controller mints agent tokens, people tokens are configured, or the profile has [cluster_link] (crucible-controller/src/api/mod.rs:672, crucible-controller/src/api/state.rs:306). So an agent-only deployment on a link cluster serves the paste link route. A person can link a username there, and the runs of that person then get it.

6. Per-run gateway isolation

6.1 Why each run has its own gateway

A SPIFFE ID of a sandbox is only as trustworthy as the pods that can carry it. SPIRE derives the ID from pod metadata. If pack code can create a pod in the sandbox namespace, it can claim the SPIFFE ID of any sandbox. When the gateway runs inside the loop pod, the loop service account needs pod rights in the sandbox namespace. Pack code runs as that account.

Profile load refuses a loop_namespace that is the sandbox namespace, but loop pods run where the controller's CONTROLLER_POD_NAMESPACE (hub) or a spoke kubeconfig's context namespace puts them. So dispatch checks the namespace each run's loop pod actually lands in, and refuses a run with an isolated gateway whose loop pod would run in the sandbox namespace, in that cluster's SPIRE namespace, or on the hub in the controller's own (check_loop_namespace).

So under an isolated profile, the controller starts each run's gateway in a pod of its own, as crucible-openshell-gateway. That account is the only identity with sandbox rights in the sandbox namespace. The gateway stamps its gateway id on every supervisor and Sandbox that it creates, from its own configuration. driver_config cannot set labels or annotations. The ClusterSPIFFEID puts the gateway id in the SPIFFE ID:

spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/openshell/gateway/{{ index .PodMeta.Labels "openshell.ai/gateway-id" }}/sandbox/{{ index .PodMeta.Annotations "openshell.ai/sandbox-id" }}'

The token endpoint then refuses an SVID whose gateway id is not the bound run's gateway id.

A gateway prefix is spiffe://<trust domain>/openshell/gateway/ for a discovered SPIRE. One named in CONTROLLER_AGENT_SPIRE may hold one {tenant} path segment, such as spiffe://<trust domain>/openshell/{tenant}/gateway/. It then matches IDs of the form spiffe://<trust domain>/openshell/<tenant>/gateway/<gateway id>/sandbox/<sandbox id>, where <tenant> is a DNS-1123 label, for a ClusterSPIFFEID that puts a tenant label of the gateway pod before gateway. The tenant is logged with each token (GatewayPrefix::split, crucible-controller/src/agent_auth/issuer.rs). Nothing else reads it yet. The gateway id check is the same with or without a tenant. The default prefix and the rendered template have no tenant segment, and a prefix without one refuses an ID that has one, and the other way round.

6.2 Lifecycle

flowchart TD
    D["dispatch_run"] --> PO{"isolated profile?"}
    PO -- no --> LOCAL["gateway runs inside the loop pod<br/>(GatewayAccess::Local)"]
    PO -- yes --> PUSH{"push-only cluster?"}
    PUSH -- yes --> REF["refuse: PushOnlyCluster"]
    PUSH -- no --> RES["reserve: gateway Service,<br/>ClusterIP, mint run CA and certs"]
    RES --> SEC["loop client TLS and gateway URL<br/>into the run Secret"]
    SEC --> ST["start: loop Service, NetworkPolicy,<br/>gateway pod, TLS Secrets owned by it"]
    ST --> LP["create loop pod"]
    LP -- "create fails" --> TD1["teardown"]
    ST -- "start fails" --> TD1
    LP --> RUN["run executes"]
    RUN --> COL["collection: delete loop pod, then teardown"]
    COL --> TD2["teardown"]
    TICK["discovery tick: sweep_orphan_gateways"] --> CHK{"is the work_pods row of<br/>its loop pod running?"}
    CHK -- no --> TD3["teardown"]
    CHK -- yes --> KEEP["keep"]

The phases are these:

• reserve creates the gateway Service and mints the TLS material for its ClusterIP (crucible-controller/src/runs/workpod/gateway.rs:184).

• start creates the loop Service, the NetworkPolicy and the gateway pod (gateway.rs:256). It then creates the two TLS Secrets with an owner reference to the gateway pod.

• teardown deletes objects in this order (gateway.rs:337):

  1. the Sandbox resources and supervisor pods with the gateway id label;
  2. the loop Service in the loop namespace;
  3. the gateway's own objects.

  The gateway's own objects go last. A teardown that fails halfway leaves the label that the sweep finds.

• sweep_orphan_gateways runs on the daemon discovery tick (gateway.rs:398). It finds the gateway objects in each sandbox namespace by crucible.dev/role=openshell-gateway. It removes each gateway whose loop pod row is absent or not running. The daemon registers it only under an isolated profile.

A failed launch removes its gateway at once (crucible-controller/src/runs/workpod/run.rs:1015). Collection removes it after it deletes the loop pod (run.rs:1116). Each failure logs a warning and leaves the rest to the sweep.

Turn pods (grounded rank, scope) get a gateway the same way, with certificates valid for the turn deadline (crucible-controller/src/runs/workpod/spec.rs:409). Their loop Service opens only the OTLP collector port. Turn sandboxes are never bound, so they get no agent tokens.

6.3 Objects

The gateway objects carry crucible.dev/gateway-for=<run pod> and crucible.dev/role=openshell-gateway. What the gateway creates carries openshell.ai/gateway-id=<gateway id> (crucible/src/deploy/render/gateway.rs:78).

The run Secret holds the loop pod client material under openshell-gateway-ca.crt, openshell-gateway-tls.crt and openshell-gateway-tls.key. The engine reads them as files in /etc/crucible/gateway-tls. The loop pod env gets CRUCIBLE_OPENSHELL_GATEWAY_URL (https://<gateway ClusterIP>:17670) and CRUCIBLE_OPENSHELL_GATEWAY_TLS_DIR.

The gateway pod runs the loop image with the command crucible openshell-gateway (crucible/src/deploy/render/gateway.rs:224, crucible/src/openshell/gateway.rs:886). It gets the driver env of the loop pod (CRUCIBLE_SANDBOX_*, the supervisor and runtime images, the OTLP endpoints), and these per-run values:

The gateway keeps its state in memory (--db-url sqlite::memory:). Under the kubernetes driver, the gateway accepts any client with a certificate from the run CA as an authorized user (crucible/src/openshell/gateway.rs:507). The run CA is the boundary.

6.4 The per-run CA

GatewayTls::mint (crucible-controller/src/runs/workpod/gateway.rs:90) uses rcgen to make:

• a CA, constrained to path length 0, with keyCertSign and cRLSign;
• a server certificate with the gateway Service ClusterIP as its only IP SAN, serverAuth;
• a loop client certificate, clientAuth;
• a supervisor client certificate, clientAuth.

All four start one day before the mint date. They end after the run deadline in whole days plus one day, or after 30 days for a run with no deadline. The controller drops the CA key after the mint. No other run trusts this CA, so a certificate of one run cannot call the gateway of another.

6.5 Network policy

The gateway NetworkPolicy (crucible/src/deploy/render/gateway.rs:362) selects the gateway pod. It admits ingress on port 17670 only from:

• the loop pod of this run (crucible/run=<run> in the loop namespace), and
• supervisor pods with openshell.ai/component=supervisor and this gateway id.

The loop Service exposes the ports that the rendered loop NetworkPolicy lists for sandbox traffic. These are the broker, the OTLP collector, buildit on 8848 with [buildit], and the [mcp] servers of the pack. Under an isolated profile, the loop render omits port 17670 (crucible/src/deploy/render/kube.rs:1342).

6.6 RBAC for each identity

Note: pods/finalizers update is necessary on OpenShift. The TLS Secrets have an owner reference with blockOwnerDeletion to the gateway pod, and the OpenShift owner-reference admission checks that right.

Note: The loop render grants the builder pod rights only when the profile has [buildit], and only in [buildit] namespace. The render does not depend on sandbox_driver.

Warning: The loop service account is shared by every run. Rights to create pods or exec into pods in the loop namespace would let one run's pack code reach another run's loop pod and run Secret: its gateway client certificate, pod token, push authfile and kubeconfig. It could also label a pod crucible/run=<other run> to pass that run's gateway NetworkPolicy. That is why builder pods live in a namespace of their own, which profile load refuses to share with the loop, rig, sandbox or crucible-kueue namespaces. The rights in the buildit namespace still let runs see each other's builder pods there, including a push's authfile while buildit streams it into the builder pod. The clean end state is buildit under a service account of its own.

With [cluster] aws_sandbox_role_arn, the gateway pod gets its own projected token for the sts.amazonaws.com audience. The trust policy of that role must allow system:serviceaccount:<sandbox namespace>:crucible-openshell-gateway.

Pod admission in these namespaces depends on the cluster's platform (section 11.2). The gateway and loop pods set no UID: on OpenShift the namespace's SCC assigns one, and on plain Kubernetes they run as their image's user, root, so their namespaces enforce Pod Security baseline there. OpenShell's sandbox and supervisor pods set their own non-root UID, drop every capability and use RuntimeDefault seccomp, so they would pass restricted on either platform.

6.7 Push-only clusters and directly reachable clusters

The controller creates gateway objects, reads Sandbox resources and reads SPIRE keys through its own API client for the cluster. A push-only cluster sends snapshots to the hub, and the hub has no client for it. So under an isolated profile, dispatch to a push-only cluster fails with PushOnlyCluster (crucible-controller/src/runs/workpod/run.rs:557, run.rs:707).

A cluster is push-only when the hub has a push token for it and no client for it. A cluster has a client when it is the hub, a personal target, or has a kubeconfig in CONTROLLER_CLUSTERS_DIR. A cluster with both a push token and a kubeconfig dispatches normally.

Note: The token path depends on the per-cluster client. When the hub cannot reach a spoke, the sandboxes of that spoke get no tokens until the spoke is reachable again.

6.8 When sandbox_namespace is unset