Agent and people MCP architecture

This document describes the MCP stack that lets agents and people act on Kubernetes clusters through crucible. It covers these parts:

• the two MCP servers, crucible-kueue and buildit;
• the shared resource server library, crucible-mcp-auth;
• the controller as the OAuth issuer;
• the per-run gateway isolation that makes agent identities trustworthy.

The audience is engineers who maintain or extend this code, and reviewers of changes to it. The text follows ASD-STE100 Simplified Technical English. Code names appear exactly as in the source, in backticks. Code references use path:line from the repository root.

Note: Every part of this stack is off unless you configure it. Section 9 lists what stays inert when a value is absent.

Glossary

Each term has one sense in this document.

1. Overview

1.1 The problem

Agents in sandboxes must start cluster work: Jobs and RayJobs that train models, Deployments that serve models, and image builds. A Kubernetes credential must not enter the sandbox. Pack code in the loop pod must not choose who the agent is, or which namespace and queue it uses. People who use MCP clients on their laptops need the same tools, with their own Kubernetes permissions.

1.2 The shape

The controller is the one OAuth issuer for agents and for people. A sandbox supervisor proves its identity with a JWT-SVID from the SPIRE of its cluster. The controller finds the run that the sandbox belongs to and signs a short token. The token names the launcher (sub), the sandbox (act.sub), the run and the launcher's Kubernetes username (kube_user). A person gets a token through the authorization code flow with PKCE, after upstream SSO and consent. The MCP client is a public client: a registered client, a metadata document client, or a self-registered client.

The MCP servers verify each token against the issuer JWKS. Before each action, they ask the API server a SubjectAccessReview as kube_user. So the person's own RBAC is the ceiling. Each run's gateway runs in a pod of its own in the sandbox namespace. So pack code cannot create a supervisor with the SPIFFE ID of another run.

examples/optimize-loop is a worked playbook on these servers: the agent builds with buildit, benchmarks as Kueue Jobs through crucible-kueue and iterates, and a deterministic verifier checks its report against crucible-kueue's replies and the push registry (see the example).

1.3 Components and trust boundaries

flowchart LR
    subgraph Laptop["Person's machine"]
        MC["MCP client<br/>registered, metadata document<br/>or self-registered"]
        BR["Browser"]
    end
    SSO["Upstream SSO<br/>OIDC provider"]
    CIMD["Client metadata host<br/>(allowlisted only)"]
    subgraph HUB["hub cluster"]
        subgraph CTLNS["controller namespace"]
            CTL["controller<br/>issuer, /oauth/token,<br/>/.well-known/jwks.json"]
            PG[("Postgres")]
            KEY["issuer key Secret"]
        end
    end
    subgraph SPOKE["spoke-a cluster"]
        API["API server<br/>Service proxy,<br/>TokenReview, SAR"]
        subgraph SPIRENS["SPIRE namespace"]
            SPIRE["SPIRE server and agents"]
            OIDC["SPIRE OIDC discovery<br/>provider /keys"]
        end
        subgraph LOOPNS["loop namespace (pack code)"]
            LOOP["loop pod<br/>engine, broker,<br/>buildit :8848"]
        end
        subgraph BUILDNS["buildit namespace"]
            BUILDER["builder pods"]
        end
        subgraph SBXNS["sandbox namespace (gateway SA only)"]
            GW["gateway pod<br/>crucible openshell-gateway"]
            SUP["supervisor pod<br/>sandbox"]
        end
        subgraph KNS["crucible-kueue namespace"]
            KQ["crucible-kueue :8850"]
        end
        subgraph WNS["workload namespace"]
            WL["Jobs, RayJobs,<br/>Deployments, ..."]
        end
    end
    MC -- "people token" --> KQ
    MC -- "register, authorize, token" --> CTL
    BR -- "login" --> SSO
    BR -- "consent" --> CTL
    CTL --- PG
    CTL --- KEY
    CTL -- "GET metadata document" --> CIMD
    CTL -- "kube client: Sandbox reads,<br/>gateway objects, /keys proxy" --> API
    API --> OIDC
    LOOP -- "mTLS" --> GW
    GW -- "creates" --> SUP
    SUP -- "JWT-SVID" --- SPIRE
    SUP -- "assertion" --> CTL
    SUP -- "agent token" --> KQ
    SUP -- "agent token" --> LOOP
    LOOP -- "bind sandbox,<br/>pod token" --> CTL
    LOOP --> BUILDER
    KQ -- "SAR as kube_user,<br/>then acts as own SA" --> API
    KQ --> WL

The diagram shows one spoke. The hub can also host runs; it then has the same namespaces. The trust boundaries are the namespaces:

• The controller namespace holds the issuer key. The controller refuses to start when this namespace is also the loop, rig, sandbox, crucible-kueue or workload namespace (crucible-controller/src/agent_auth/placement.rs:74).
• The loop namespace runs pack code. The loop service account has no verbs in the sandbox namespace under an isolated profile.
• The sandbox namespace allows pod creation only to crucible-openshell-gateway. A SPIFFE ID there is as trustworthy as that rule.
• The crucible-kueue namespace holds the server service account. Workload verbs of that account apply only in the workload namespace.
• The person's machine holds only a token and a refresh token, never a cluster credential.
• A client metadata host is outside every boundary. The controller fetches a JSON document from it only when its host is in CONTROLLER_PEOPLE_CIMD_HOSTS. The document can name a client and its redirect URIs, and nothing more.

2. Trust model and identities

2.1 Principals

A run acts for its launcher. The launcher is runs.dispatched_by, which must be user:<login>. The controller stamps it when it creates the run row (crucible-controller/src/runs/store.rs:334). The value is the person who last asked for this dispatch: an unpark, a redispatch, a scope, or a paired GitHub approval. Without such a request, the value is the owner of the run. A run that has no person behind it gets no token (GrantError::NoLauncher).

2.2 Token types and claims

All tokens are compact JWS, algorithm ES256, with kid in the header. The issuer signs both kinds with the same key and the same iss.

The agent token is built in crucible-controller/src/agent_auth/issuer.rs:820. The people token is built in crucible-controller/src/agent_auth/people.rs:299.

Note: Claims name identity only. There is no role and no queue list in a token. Each resource server decides what the caller may do. crucible-kueue ignores role and queues claims if a token carries them.

2.3 Issuer keys and JWKS

The issuer key is a PKCS#8 P-256 key. The kid is the RFC 7638 thumbprint of its public half, so a new key gets a new kid with no configuration (issuer.rs:152). The JWKS at /.well-known/jwks.json publishes the current key and, when configured, the previous key (issuer.rs:820). The previous key verifies only. It signs nothing.

To rotate the issuer key, do these steps:

1. Move the current key file to CONTROLLER_AGENT_PREVIOUS_SIGNING_KEY_FILE.
2. Mount a new key at CONTROLLER_AGENT_SIGNING_KEY_FILE.
3. Restart every controller replica.
4. Wait for one token lifetime. The default is 900 seconds; people tokens can live up to 3600.
5. Remove the previous key variable and restart again.

Warning: All replicas must mount the same keys. A replica with another key signs tokens that the other replicas do not publish.

Resource servers cache the JWKS for 600 seconds. A token with an unknown kid causes a refetch, but not more than one fetch each 30 seconds (crucible-mcp-auth/src/jwks.rs:152). A failed fetch keeps the cached keys. The client refuses a key set larger than 256 KiB and follows no redirects.

2.4 Lifetimes

2.5 What each component verifies

The controller token endpoint verifies the assertion. Section 3 gives the full list.

Every resource server verifies a token through ResourceServer::validate (crucible-mcp-auth/src/server.rs:201) and Verifier::verify (crucible-mcp-auth/src/verify.rs:65):

1. The header algorithm is asymmetric (RS, PS, ES or EdDSA). HS* and none never verify.
2. The header has a kid, and the JWKS of the issuer has that key. Under an issuer trusted for runs (the controller), the header typ is at+jwt (RFC 9068 §2.1), for a run's token and a person's alike. The controller types every access token it mints, so an ID token, an assertion or any other JWT under its iss and key is refused. An issuer trusted only for people, such as an OIDC realm, is not held to it.
3. The algorithm of the key, when the JWK declares one, is the header algorithm.
4. The signature is correct.
5. iss is exactly a trusted issuer string. There is no URL normalization.
6. aud names at least one configured resource URL.
7. exp and nbf hold, with 30 seconds of leeway.
8. exp, iss, aud and sub are present.
9. The token has every scope in required_scopes (kueue.read or buildit.read).

Note: the controller started typing its access tokens and the servers started requiring it in the same change. A live hub and its spokes need the controller and crucible-kueue/buildit rolled together: servers built with the check refuse an older controller's untyped tokens. An older server ignores typ, so rolling the controller first is safe.

The configuration of each trusted issuer says whom its tokens stand for: runs or people (Callers). When one issuer is trusted for both, a token with act is a run token and a token without act is a people token. The token cannot choose its kind in any other way.

Then each server applies its own rules:

2.6 Why kube_user RBAC is the ceiling

crucible-kueue acts with its own service account. That account can create workloads in the workload namespace. Before each call, the guard turns the call into access checks. For each check it asks the API server a SubjectAccessReview as kube_user, with the groups of that user (crucible-mcp-auth/src/kube.rs:209). The server does the call only when every review allows it. So:

• A token can do at most what its kube_user can do with kubectl. The token scopes, the tool policy, the allowlist profile and the server account reach can only reduce that.
• An agent token acts as its launcher. A run cannot do more than the person who launched it.
• The issuer and the reviewer both refuse a system: username (crucible-controller/src/identity/kube_identity.rs:27, crucible-mcp-auth/src/kube.rs:67). So no token can act as a node, a service account or the cluster. Groups from a token claim also lose every system: entry.
• The server creates pods with its own identity. So the caller must be able to get each Secret and PersistentVolumeClaim that a manifest names (crucible-kueue/src/access.rs:149).
• A new RoleBinding applies on the next call, because denials are not cached. A removed permission can still allow calls for up to 30 seconds, because allowed decisions are cached.
• workload_status and tasks/get read the workload's Kueue Workloads, its pods and their events with the server's account, and show each only when kube_user may get and list it in the namespace. A part the caller may not read is left out, and the reply's omitted names it and why (crucible-kueue/src/access.rs, StatusPart).
• Any caller who may use a queue can discover its hardware. This is a guarantee, and a deliberate exception to the ceiling: hardware_describe and queue_describe check only get or the submit verb (use) on the LocalQueue, and hardware_describe reads the nodes behind the queue's flavors with the server's account, so an agent can pick a flavor and a GPU product without RBAC on nodes. It returns each flavor's node labels and taints, and for the nodes those select, how many there are and how many are schedulable per GPU product, and their accelerator capacity. Node names are not returned. Tests hold the guarantee (crucible-kueue/tests/http_auth.rs, and cluster_e2e.rs against a cluster).

The issuer never decides what a person may do. It decides only who the person is on each cluster (section 5).

3. Agent token flow

3.1 From dispatch to the first MCP call

sequenceDiagram
    autonumber
    participant C as controller (hub)
    participant A as API server (spoke-a)
    participant L as loop pod (engine)
    participant G as gateway pod
    participant S as supervisor
    participant P as SPIRE agent
    participant K as crucible-kueue
    C->>A: create gateway Service in sandbox namespace
    A-->>C: ClusterIP
    Note over C: mint run CA, server cert for the ClusterIP,<br/>loop and supervisor client certs, drop CA key
    Note over C: run Secret gets loop client TLS, gateway URL,<br/>CRUCIBLE_AGENT_TOKEN_ENDPOINT, CRUCIBLE_AGENT_SVID_AUDIENCE,<br/>CRUCIBLE_AGENT_SERVERS, run_agent_tools row written
    C->>A: loop Service, NetworkPolicy, gateway pod, TLS Secrets owned by the gateway pod
    C->>A: create loop pod
    L->>G: health check over mTLS, register the CLI
    L->>G: import token-grant provider profiles (crucible-kueue, buildit)
    L->>G: CreateSandbox with those providers
    G->>A: create Sandbox and supervisor pod,<br/>label openshell.ai/gateway-id, annotation openshell.ai/sandbox-id
    G-->>L: sandbox id
    L->>C: POST /api/sandbox-bindings/runs/{run}<br/>Bearer pod token, {sandbox_id, workdir}
    C->>A: list Sandbox by sandbox-id and gateway-id labels
    A-->>C: the Sandbox and its name
    Note over C: write bind record, owner-reference the Sandbox to the gateway pod
    C-->>L: 204
    S->>P: fetch JWT-SVID, audience = issuer
    P-->>S: JWT-SVID
    S->>C: POST /oauth/token client_credentials,<br/>client_assertion = JWT-SVID, audience = kueue URL, scope
    C->>A: GET SPIRE /keys through the Service proxy (cached)
    A-->>C: SPIRE JWKS
    Note over C: verify the assertion, read gateway id and sandbox id,<br/>find the bind record, check cluster and gateway,<br/>resolve the launcher and kube_user
    C-->>S: access_token, expires_in, scope
    Note over S: inject Authorization on egress<br/>to the kueue host and port only
    S->>K: MCP request with Bearer token
    K->>A: SubjectAccessReview as kube_user
    K->>A: act with the server service account

The steps are these:

1. Dispatch reserves the gateway. The Service comes first because its ClusterIP goes into the server certificate (crucible-controller/src/runs/workpod/gateway.rs:184).
2. The controller folds the loop pod client material and the agent server access into the run Secret before it renders the loop pod (crucible-controller/src/runs/workpod/run.rs:813, access in crucible-controller/src/runs/workpod/agent_servers.rs). A server is granted only where the issuer can mint its tokens: it trusts the run's cluster's SPIRE, discovered or named in CONTROLLER_AGENT_SPIRE (crucible-controller/src/runs/workpod/agent_servers.rs, granted). The loop pod gets them as one CRUCIBLE_AGENT_SERVERS list. The run records the servers that it got, with the scopes of the audience.
3. The gateway starts before the loop pod, so it boots while the loop pod schedules (gateway.rs:256).
4. The engine reads CRUCIBLE_OPENSHELL_GATEWAY_URL and talks to the remote gateway over mTLS (crucible/src/openshell/gateway.rs:422-437). It creates one token-grant provider, one egress entry and one .mcp.json entry for each server in CRUCIBLE_AGENT_SERVERS the same way, after starting a server whose transport is the loop pod (crucible/src/openshell/agent_servers.rs). A provider holds no secret and gives the sandbox no placeholder (crucible/src/openshell/provider.rs:368).
5. After CreateSandbox, the engine binds the sandbox (crucible/src/openshell/run.rs:469). It binds only when the loop pod carries CRUCIBLE_AGENT_TOKEN_ENDPOINT. A failed bind does not fail the turn. The sandbox then gets no token.
6. The bind route trusts only the pod token of that run (crucible-controller/src/agent_auth/bindings.rs:298). It reads the sandbox name from the cluster, never from the request (crucible-controller/src/agent_auth/directory.rs:60).
7. The supervisor asks SPIRE for a JWT-SVID with the audience in token_grant.jwt_svid_audience. That value is the issuer identifier.
8. The token request has exactly the fields that OpenShell sends: grant_type, client_assertion_type, client_assertion, audience, scope. Any other field, for example client_id or resource, is refused (crucible-controller/src/agent_auth/mod.rs:149).
9. The supervisor injects the token as a bearer header only on egress to the host and port of the provider endpoint, from a binary the provider's profile lists. The profile lists the same binaries as the sandbox policy: OpenShell injects a grant only when the profile's own rule admits the request, and with binary identity required, a rule admits only the binaries it names. The agent never sees the token.

3.2 What the token endpoint checks

issue (crucible-controller/src/agent_auth/mod.rs:314) does these checks in this order:

1. The issuer mints agent tokens: the profile isolates gateways and describes a server, or CONTROLLER_AGENT_SPIRE names a cluster.
2. The deploy profile is isolated and names a sandbox namespace (crucible-controller/src/agent_auth/placement.rs:48).
3. grant_type is client_credentials.
4. client_assertion_type is urn:ietf:params:oauth:client-assertion-type:jwt-bearer.
5. The assertion iss is the SVID issuer of exactly one configured cluster. That cluster's SPIRE keys verify the assertion (crucible-controller/src/agent_auth/issuer.rs:701).
6. The assertion aud is exactly one value, the issuer identifier.
7. The assertion has a numeric iat, not more than 30 seconds in the future.
8. exp minus iat is not more than 900 seconds.
9. sub is <gateway prefix><gateway id>/sandbox/<sandbox id> for the prefix of that cluster.
10. A bind record exists for the sandbox id, and its loop pod is in the running state.
11. The cluster of the run is the cluster whose SPIRE signed the assertion.
12. The requested audience is the URL of a server the deploy profile describes on the run's cluster (Audience::of).
13. The gateway id in the SPIFFE ID is the gateway id of the bound loop pod.
14. The run has a launcher (dispatched_by is user:<login>).
15. Each requested scope is in the audience entry. A request with no scope gets all the scopes of the entry.
16. When the audience requires a kube_user (the profile describes its server as one that authorizes as kube_user, as [kueue] does), it is known for the launcher on the run's cluster. For other audiences, the token omits an unknown kube_user, and the controller logs a warning. Every audience is described: at startup the controller refuses one for which the profile describes no server at that URL on that cluster, comparing URLs once parsed (Audience::described_by, crucible-controller/src/agent_auth/issuer.rs).

Note: Pack code holds the run CA's client certificate, so it can make its gateway create sandboxes and request a token for any audience on its run's cluster, with that audience's scopes. The audience allowlist, per cluster, is the boundary: list only the servers a run on that cluster may use, with the scopes it may hold.

The controller reads SPIRE keys through the API server Service proxy of each cluster, with the same client it dispatches with (crucible-controller/src/agent_auth/spire.rs:21). The path is /api/v1/namespaces/<ns>/services/<scheme>:<service>:<port>/proxy/keys (issuer.rs:189). The scheme is https unless the entry starts with http://.

3.3 Refusals

Every refusal returns only the OAuth error code and a fixed description. The detail goes to the controller log as agent token refused with reason and the user_agent.

The bind route has its own refusals (bindings.rs:320):

Note: Pack code holds the gateway client certificate of its own run. It can create sandboxes for its own run with the driver_config fields that the gateway accepts. Those sandboxes carry the gateway id of its own run, so the issuer refuses them for any other run.

4. People token flow

4.1 MCP clients

People sign in with public OAuth clients. A public client has no secret, so PKCE S256 is required. Each code, approval and refresh token is bound to the client ID that it was issued to. The people token names that client in azp.

4.1.1 Client ID forms

ClientId::parse (crucible-controller/src/agent_auth/people/clients.rs:133) puts each client_id in one of three forms. The form decides who vouches for the client.

Clients (crucible-controller/src/agent_auth/people/registry.rs:27) holds the registry, the metadata document resolver and the registrations. It has two entry points:

• resolve (registry.rs:66) serves /oauth/authorize. It returns the client with its name and redirect rules. For a metadata document client, it fetches the document.
• admits (registry.rs:87) serves /oauth/token. It fetches nothing. It refuses a registered client that the registry no longer lists, a metadata document client whose host is no longer allowlisted, and a registration that is no longer live. The grant that the request redeems is already bound to its client ID.

flowchart TD
    REQ["client_id from the request"] --> P{"ClientId::parse"}
    P -- "fails" --> BAD["refuse: error page<br/>or invalid_client"]
    P -- "https:// URL" --> MD{"CONTROLLER_PEOPLE_CIMD_HOSTS<br/>set and lists the host?"}
    MD -- no --> BAD
    MD -- yes --> FETCH["authorize: fetch or cache hit<br/>token: host check only"]
    FETCH --> DOC{"document valid?"}
    DOC -- no --> BAD
    DOC -- yes --> OK["Client: id, name, redirect rules"]
    P -- "dyn- id" --> DYN{"people_dynamic_clients row,<br/>used in the last 30 days?"}
    DYN -- no --> BAD
    DYN -- yes --> STAMP["stamp last_used_at"] --> OK
    P -- "registered name" --> REG{"in the client registry?"}
    REG -- no --> BAD
    REG -- yes --> OK

4.1.2 The client registry

Without configuration, the registry has three clients (clients.rs:448):

CONTROLLER_PEOPLE_CLIENTS_FILE names a TOML file of [[clients]] tables. Each table has client_id, name, redirect_uris and an optional resources list. The file replaces the built-in list, so list the built-in clients too if you want to keep them (clients.rs:479). The controller does not start when the file is malformed, when it lists no client or one client twice, or when a resources entry is not a people resource (crucible-controller/src/agent_auth/people.rs:172). Without resources, a client can get tokens for every audience.

[[clients]]
client_id = "lab-cli"
name = "Lab CLI"
redirect_uris = ["http://127.0.0.1/oauth/done"]
resources = ["https://crucible-kueue.apps.spoke-a.example.com/mcp"]

When you remove a client, its next refresh gets invalid_client. When you remove a resource from its resources, its next refresh for that resource revokes the approval.

Redirect matching follows RFC 8252 §7.3 (RedirectRule, clients.rs:237):

• A registered http URI on localhost, 127.0.0.1 or [::1] matches any explicit port on the same host, with the same path. It matches no query, no fragment and no credentials.
• localhost, 127.0.0.1 and [::1] are three different hosts. A client registered on localhost cannot use 127.0.0.1.
• A registered path that ends in /* also matches one more segment of 1 to 64 base64url characters. Codex adds such a server id to the callback path. Only the registry can use /*.
• Any other registered URI must be https. It matches only character for character.
• The requested URI must already be in canonical form (Client::redirect, clients.rs:348).

4.1.3 Client ID Metadata Documents

CONTROLLER_PEOPLE_CIMD_HOSTS is a comma-separated list of host names. When it is set, a client_id that is an https:// URL on one of those hosts is resolved as an OAuth Client ID Metadata Document (draft-ietf-oauth-client-id-metadata-document). When it is unset, every URL client_id is refused (ClientError::MetadataOff).

At /oauth/authorize, Cimd::client (crucible-controller/src/agent_auth/people/cimd.rs:370) returns a cached document or fetches it, and only for a signed-in browser: a request naming a metadata document without a session goes to /auth/login first, so nobody can make the controller fetch without signing in. The fetch (cimd.rs:390) has these rules:

• The host must be on the allowlist (cimd.rs:362).
• The resolver gives only public addresses. It drops loopback, private, link-local and other RFC 6890 special-use addresses (cimd.rs:127, cimd.rs:170). With HTTPS_PROXY set, the proxy resolves the names, so this check moves to the proxy.
• The client uses only https and follows no redirects (cimd.rs:338).
• The answer must be 200, JSON, at most 5 KiB, and arrive within 5 seconds.

The document must describe a public client of its own URL (client_of, cimd.rs:200):

• client_id is the URL exactly.
• client_name is 1 to 100 printable characters.
• redirect_uris has at least one entry. The entries follow the registry rules, but /* is not accepted.
• token_endpoint_auth_method is none. There is no client_secret and no client_secret_expires_at.
• grant_types and response_types, when present, include authorization_code and code.

A valid document is cached for its Cache-Control max-age minus Age. Without max-age it is cached for 300 seconds. The limit is 3600 seconds (freshness, cimd.rs:266). A document with no-store or no-cache is not cached. A refused document is not cached.

The consent page and Settings show the client_name of the document with the host of the URL, so a document cannot look like a registered client. Codex publishes https://chatgpt.com/oauth/codex/client.json. pi publishes https://pi.dev/oauth/client.json.

4.1.4 Self-registered clients

pi 1.x, Codex and Claude Code without a client id register a client before they sign in (RFC 7591). The controller accepts these registrations at POST /oauth/register (crucible-controller/src/agent_auth/people/dynamic.rs:422). Registration is on by default when the deploy profile serves people. CONTROLLER_PEOPLE_DCR=false turns it off.

sequenceDiagram
    autonumber
    participant M as MCP client
    participant C as controller (any replica)
    participant D as Postgres
    M->>C: GET /.well-known/oauth-authorization-server
    C-->>M: registration_endpoint = issuer/oauth/register
    M->>C: POST /oauth/register {client_name, redirect_uris, ...}
    Note over C: check: public client, loopback http redirect URIs,<br/>grant and response types, at most 8 URIs
    Note over C: rate limit: 20 each hour for each source address
    C->>D: INSERT people_dynamic_clients<br/>only while fewer than about 10000 are live
    C-->>M: 201 {client_id: dyn-..., token_endpoint_auth_method: none}
    M->>C: GET /oauth/authorize?client_id=dyn-...
    C->>D: UPDATE last_used_at WHERE client_id AND not idle, RETURNING the client
    Note over C,D: the hourly sweep deletes registrations idle for 30 days<br/>and the approvals of deleted registrations

check (dynamic.rs:194) accepts only native public clients:

• token_endpoint_auth_method is absent or none. The controller issues no secret, no registration access token and no management endpoint.
• grant_types includes authorization_code and stays within authorization_code and refresh_token. response_types is code.
• redirect_uris has 1 to 8 entries, each at most 512 characters. Each entry is http on localhost, 127.0.0.1 or [::1], with no query, fragment or credentials.
• The controller removes control characters and repeated white space from client_name and keeps at most 100 characters. An empty name becomes Unnamed client.

The client ID is dyn- and 24 random bytes in base64url. The consent page and Settings show the name as " (self-registered)".

Dcr::register (dynamic.rs:287) applies two limits:

• Each source address can register 20 times in each hour. The limit is in memory, for each replica. Past it, the answer is 429 with temporarily_unavailable.
• About 10000 registrations used in the last 30 days can exist. Past it, the answer is 503 with temporarily_unavailable. The count and the insert are one statement but not serialized across replicas, so concurrent registrations can go slightly over: the cap is soft.

The source address is the TCP peer. When the peer is in CONTROLLER_PEOPLE_TRUSTED_PROXIES, as the cluster router is, the source is the last X-Forwarded-For hop (source, dynamic.rs:428). Any other peer, an in-cluster caller included, is its own source and cannot pick a bucket.

Dcr::client (dynamic.rs:345) reads a registration and stamps last_used_at in one statement. A registration that is not used for 30 days stops working. The hourly sweep deletes it and the approvals issued to it (Dcr::sweep, dynamic.rs:375).

4.1.5 Clients on consent and in Settings

The consent API (crucible-controller/src/identity/api/consent.rs:63) returns these client fields:

GET /api/me/mcp-sign-ins returns the same client_id, client_name, client_host and client_kind for each live approval. The approval stores the name that the person saw on the consent page.

4.2 Discovery, authorization and code exchange

sequenceDiagram
    autonumber
    participant M as MCP client
    participant K as crucible-kueue
    participant C as controller (any replica)
    participant B as browser
    participant U as upstream SSO
    participant O as cluster OAuth server
    participant H as client metadata host
    participant D as Postgres
    M->>K: POST /mcp, no token
    K-->>M: 401, WWW-Authenticate names the resource metadata URL
    M->>K: GET /.well-known/oauth-protected-resource/mcp
    K-->>M: resource, authorization_servers = [issuer], scopes_supported
    M->>C: GET /.well-known/oauth-authorization-server
    C-->>M: authorization_endpoint, token_endpoint, S256, iss parameter,<br/>registration_endpoint, client_id_metadata_document_supported
    opt no client id configured and registration on
        M->>C: POST /oauth/register
        C-->>M: client_id = dyn-...
    end
    M->>B: open /oauth/authorize?client_id, code_challenge, state, resource, redirect_uri=http://127.0.0.1:port/callback
    B->>C: GET /oauth/authorize
    alt metadata document client
        C->>H: GET client_id URL (unless cached)
        H-->>C: client metadata JSON
    else self-registered client
        C->>D: read and stamp people_dynamic_clients row
    end
    Note over C: resolve the client, match redirect_uri
    alt no native browser session
        C-->>B: 303 /auth/login?rd=...
        B->>U: OIDC login with PKCE
        U-->>B: code
        B->>C: /auth/callback, session row in Postgres
        C-->>B: 303 back to /oauth/authorize
    end
    Note over C: validate, park the request on the session
    C-->>B: 303 /consent?id=...
    B->>C: GET /api/oauth/consent?id=...
    C-->>B: client name, kind, host, redirect origin,<br/>resource, scopes, login, identity on the cluster
    opt link cluster with no link
        B->>C: POST /api/me/cluster-identities/{cluster}/link
        C-->>B: authorize_url of the cluster OAuth server
        B->>O: authorize with PKCE, scope user:info
        O-->>B: code
        B->>C: GET /oauth/cluster-link/{cluster}/callback
        C->>O: redeem code
        C->>C: TokenReview on that cluster, write link, drop the cluster token
    end
    B->>C: POST /api/oauth/consent {id, decision: approve}
    C->>D: insert people_codes row (SHA-256 of the code), bound to the client ID
    C-->>B: {redirect_url: redirect URI?code, state, iss}
    B->>M: navigate to the redirect URI
    M->>C: POST /oauth/token grant_type=authorization_code, client_id, code, code_verifier, redirect_uri, resource
    Note over C: admit the client ID (no fetch)
    C->>D: delete people_codes row, returning the grant
    Note over C: check client ID, redirect_uri, PKCE S256,<br/>resource, resolve kube_user again
    C->>D: insert people_grants with client_id and client_name,<br/>and the first people_refresh_tokens row
    C-->>M: access_token (azp = client ID), refresh_token, expires_in, scope
    M->>K: POST /mcp, Bearer token

The steps are these:

1. crucible-kueue answers a request without a token with 401 and a challenge that names its RFC 9728 metadata. The metadata lists the trusted issuers as authorization_servers (crucible-mcp-auth/src/metadata.rs:42).
2. The MCP client reads the RFC 8414 metadata of the controller (crucible-controller/src/agent_auth/issuer.rs:831, crucible-controller/src/agent_auth/mod.rs:445). People support adds authorization_endpoint, the authorization_code and refresh_token grants, the code response type, S256, the none client authentication, and authorization_response_iss_parameter_supported. Two fields depend on the client configuration:
   • registration_endpoint is <issuer>/oauth/register while registration is on. It is absent with CONTROLLER_PEOPLE_DCR=false.
   • client_id_metadata_document_supported is true while CONTROLLER_PEOPLE_CIMD_HOSTS is set. It is absent otherwise.
3. /oauth/authorize (crucible-controller/src/agent_auth/people.rs:554) resolves the client (section 4.1) and matches redirect_uri against its redirect rules. state is required, at most 1024 characters. The method must be S256. resource must be a people resource of the deploy profile (DeployProfile::people_resources), and a registered client with resources must list it (validate, people.rs:373).
4. An unknown client, a refused metadata document, an expired registration or a bad redirect gives an error page. The controller never redirects to a URI that the client did not register. When the client and the redirect are correct, the controller sends every other error to the redirect URI with error, state and iss.
5. Without a native session, the controller sends the browser to /auth/login. After the login, the browser returns to /oauth/authorize. A session whose group refresh failed counts as no session.
6. The request waits on the session under a random id. The consent page shows it only to the same browser and the same person. The page names the client and shows the host:port that the code goes to (section 4.1.5). No page may frame it: every SPA response and the authorize page send X-Frame-Options: DENY and Content-Security-Policy: frame-ancestors 'none'.
7. The approval resolves kube_user for the cluster of the audience. An unlinked link cluster answers 409. The controller parks the request on the session again, so the page can link and try again.
8. The code is 32 random bytes, base64url. Postgres stores only its SHA-256. The stored grant names the client ID and the client name.
9. The token endpoint admits the client ID without a fetch (registry.rs:87). It marks the code spent under a row lock, so a code works once on any replica (redeem_code in crucible-controller/src/agent_auth/people/grants.rs). A code issued to another client ID is invalid_grant (redeem_code, people.rs). The code is spent all the same. The approval the redemption starts is recorded against the code, and a spent code stays a day past its expiry. Presented again by the client it was issued to, it is invalid_grant and revokes that approval, so the refresh token the first redemption issued stops working (OAuth 2.1 §4.1.3, RFC 9700 §4.5.3); a replay that lands before the approval starts stops it from starting. Presented by another client, it is only unknown, so a code someone saw cannot revoke a person's approval.

Note: Claude Code does not step up after a 403. So a person always gets every scope that the audience lists. Requested scopes outside that list are refused; the OIDC scopes openid, profile, email and offline_access are ignored.

4.3 Refresh with rotation and reuse detection

sequenceDiagram
    autonumber
    participant M as MCP client
    participant C as controller (any replica)
    participant D as Postgres
    M->>C: POST /oauth/token grant_type=refresh_token, refresh_token, client_id
    Note over C: admit the client ID (no fetch),<br/>else invalid_client and nothing is spent
    C->>D: UPDATE people_refresh_tokens SET used_at WHERE key = sha256 AND used_at IS NULL
    alt row updated
        C->>D: read people_grants FOR UPDATE
        Note over C: refuse when revoked
        alt client_id is not the client of the approval
            C->>D: revoke the approval and delete its live token
            C-->>M: invalid_grant
        else same client
            Note over C: refuse when idle over 14 days,<br/>or older than 90 days
            alt another resource or a scope beyond the approval
                C->>D: roll back: the token stays unspent
                C-->>M: invalid_target or invalid_grant
            end
            C->>D: issue the next refresh token
            Note over C: client still allowed for the resource,<br/>read the person again: users row, stored<br/>credential not refused, kube_user on the cluster
            alt person still allowed and kube_user known
                C-->>M: new access_token and new refresh_token
            else refused
                C->>D: revoke the approval
                C-->>M: invalid_grant
            end
        end
    else token already spent
        C->>D: revoke the approval and delete its live token
        C-->>M: invalid_grant
    end

The rotation is one conditional UPDATE, so two replicas cannot both spend one refresh token (grants.rs:223). A spent token stays in the table until its approval goes. When a client presents it again, the controller revokes the whole approval (RFC 9700 §4.14.2). A live token that another client ID presents also revokes the approval (grants.rs:279). A refresh that asks for another resource or a wider scope is refused inside the same transaction, before anything commits (refresh_fits, people.rs:765; grants.rs:309), so the token it presented stays live and the client can refresh with it again.

A refresh also checks the client again. A registered client that the registry no longer lists gets invalid_client. A registered client whose resources no longer lists the resource of the approval loses the approval.

Every refresh reads the person again (crucible-controller/src/identity/people_directory.rs:35):

• A deleted users row revokes the approval.
• A stored offline credential that the upstream SSO refused (user_credentials.last_error) revokes the approval.
• An unlinked cluster username revokes the approval. A new link appears in the next token.

Warning: A refresh does not ask the upstream identity provider. A person deactivated there keeps refreshing until something else refuses them: a stored offline credential the SSO then refuses, the 14-day idle limit, or the 90-day cap on an approval (REFRESH_IDLE_SECS, REFRESH_ABSOLUTE_SECS in crucible-controller/src/agent_auth/people/grants.rs). Without a stored credential (CONTROLLER_CREDENTIAL_KEY unset, or a person who never stored one), nothing else does. To cut a person off sooner, delete their users row, which also drops their stored credential and API keys.

Warning: A client that retries a refresh after a lost response presents a spent token. The controller treats that as reuse and revokes the approval. The person must approve the client again.

A background task deletes expired codes and dead approvals each hour (grants.rs:425). With registration on, the same task deletes registrations idle for 30 days and the approvals of deleted registrations. It starts only when people tokens are configured.

4.4 Several replicas

Any replica can do any step of the flow:

• Browser sessions, the parked authorization request and a started cluster link live in the Postgres session store.
• Codes, approvals and refresh tokens live in people_codes, people_grants and people_refresh_tokens. Each method is one statement or one transaction.
• Every replica signs with the same issuer key and publishes the same JWKS.

Leader election (CONTROLLER_LEASE_NAME) applies only to the daemon. The API, the token endpoint and the bind route answer on every replica.

4.5 Settings and revocation

GET /api/me/mcp-sign-ins lists the live approvals of the caller, with the client of each (crucible-controller/src/identity/api/mcp_sign_ins.rs:57). Settings groups them by client.

• DELETE /api/me/mcp-sign-ins?client_id=<client ID> revokes the approvals of one client and deletes its unused codes (grants.rs:330). A client_id that does not parse gets 400.
• DELETE /api/me/mcp-sign-ins revokes all approvals of the caller and deletes their unused codes.

An API key cannot revoke approvals (mcp_sign_ins.rs:113).

5. Per-cluster identity

5.1 Identity rules

The deploy profile gives each cluster one identity rule. The hub rule is [cluster] identity. A spoke rule is [clusters.<name>] identity. A cluster without a rule is link (forge/src/fleet.rs:61).

Use a claim rule only when the cluster API server trusts the same SSO for the same people. Use link in all other cases.

5.2 Links

A person proves a link in one of two ways:

• Paste a cluster token: PUT /api/me/cluster-identities/{cluster} with {"cluster_token": "..."}. The controller sends a TokenReview to that cluster (crucible-controller/src/identity/kube_links.rs:108).
• Use the OAuth server of an OpenShift cluster: POST /api/me/cluster-identities/{cluster}/link. The controller is a confidential OAuthClient there. It redeems the code with PKCE for a user:info token and sends that token to a TokenReview (crucible-controller/src/identity/cluster_link.rs:204, cluster_link.rs:237).

Either way the controller stores only the username, the source (token-review or openshift-oauth) and the time. It never stores or logs the cluster token. A token that another cluster minted does not authenticate, so it cannot link this cluster. A link to a system: username is refused.

The OAuth link is offered for a cluster only when all of these are true:

• [cluster_link] openshift lists the cluster.
• The controller has a client for the cluster.
• A file with the cluster's name exists in CONTROLLER_CLUSTER_LINK_SECRETS.

The callback is GET /oauth/cluster-link/{cluster}/callback. It checks the state in constant time, the cluster, the person and the 10-minute limit. It then lands on /settings?linked=<cluster> or ?link_error=<code>. The code is a fixed word, so a crafted URL cannot put text on the page.

5.3 The hub override

A platform administrator can set the hub username of a person: PUT /api/users/{login}/kube-user with {"kube_user": "<name>"}, or null to clear it. An administrator cannot set their own. The event log records it on user:<login>. The override applies on the hub only, and it wins over the hub rule and any hub link (crucible-controller/src/identity/cluster_identity.rs:66).

5.4 Resolution

flowchart TD
    START["resolve(person, cluster)"] --> HUB{"cluster is hub and<br/>users.kube_user_override set?"}
    HUB -- yes --> OV["Known: override"]
    HUB -- no --> RULE{"identity rule of the cluster"}
    RULE -- "claim" --> CL{"claim value present<br/>and a valid username?"}
    CL -- yes --> KR["Known: prefix + claim"]
    CL -- no --> UN["Unavailable: reason"]
    RULE -- "link" --> LK{"kube_identity_links row<br/>for (sub, cluster)?"}
    LK -- yes --> KL["Known: link"]
    LK -- no --> NL["Unlinked"]

The same function decides kube_user for every consumer:

Issuance and the tools record name who acts as an authz::model::Principal, and for_principal (crucible-controller/src/identity/cluster_identity.rs) resolves it: a user: principal through their sign-in, a run: principal through the person it was dispatched for. A group: or team: principal, or a run dispatched for one, has no username of its own and is refused with a typed error (PrincipalError), which the token endpoint answers with invalid_grant. A team's own identity per cluster would be one more arm of for_principal; the tables would not change.

A launcher that never logged in has no users row, so its kube_user is unavailable. Because the agent token resolves at each request, a new link applies to the next token of a run that is already live.

Note: The links, the override and the settings panels are served only when the controller mints agent tokens, people tokens are configured, or the profile has [cluster_link] (crucible-controller/src/api/mod.rs:672, crucible-controller/src/api/state.rs:306). So an agent-only deployment on a link cluster serves the paste link route. A person can link a username there, and the runs of that person then get it.

6. Per-run gateway isolation

6.1 Why each run has its own gateway

A SPIFFE ID of a sandbox is only as trustworthy as the pods that can carry it. SPIRE derives the ID from pod metadata. If pack code can create a pod in the sandbox namespace, it can claim the SPIFFE ID of any sandbox. When the gateway runs inside the loop pod, the loop service account needs pod rights in the sandbox namespace. Pack code runs as that account.

Profile load refuses a loop_namespace that is the sandbox namespace, but loop pods run where the controller's CONTROLLER_POD_NAMESPACE (hub) or a spoke kubeconfig's context namespace puts them. So dispatch checks the namespace each run's loop pod actually lands in, and refuses a run with an isolated gateway whose loop pod would run in the sandbox namespace, in that cluster's SPIRE namespace, or on the hub in the controller's own (check_loop_namespace).

So under an isolated profile, the controller starts each run's gateway in a pod of its own, as crucible-openshell-gateway. That account is the only identity with sandbox rights in the sandbox namespace. The gateway stamps its gateway id on every supervisor and Sandbox that it creates, from its own configuration. driver_config cannot set labels or annotations. The ClusterSPIFFEID puts the gateway id in the SPIFFE ID:

spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/openshell/gateway/{{ index .PodMeta.Labels "openshell.ai/gateway-id" }}/sandbox/{{ index .PodMeta.Annotations "openshell.ai/sandbox-id" }}'

The token endpoint then refuses an SVID whose gateway id is not the bound run's gateway id.

A gateway prefix is spiffe://<trust domain>/openshell/gateway/ for a discovered SPIRE. One named in CONTROLLER_AGENT_SPIRE may hold one {tenant} path segment, such as spiffe://<trust domain>/openshell/{tenant}/gateway/. It then matches IDs of the form spiffe://<trust domain>/openshell/<tenant>/gateway/<gateway id>/sandbox/<sandbox id>, where <tenant> is a DNS-1123 label, for a ClusterSPIFFEID that puts a tenant label of the gateway pod before gateway. The tenant is logged with each token (GatewayPrefix::split, crucible-controller/src/agent_auth/issuer.rs). Nothing else reads it yet. The gateway id check is the same with or without a tenant. The default prefix and the rendered template have no tenant segment, and a prefix without one refuses an ID that has one, and the other way round.

6.2 Lifecycle

flowchart TD
    D["dispatch_run"] --> PO{"isolated profile?"}
    PO -- no --> LOCAL["gateway runs inside the loop pod<br/>(GatewayAccess::Local)"]
    PO -- yes --> PUSH{"push-only cluster?"}
    PUSH -- yes --> REF["refuse: PushOnlyCluster"]
    PUSH -- no --> RES["reserve: gateway Service,<br/>ClusterIP, mint run CA and certs"]
    RES --> SEC["loop client TLS and gateway URL<br/>into the run Secret"]
    SEC --> ST["start: loop Service, NetworkPolicy,<br/>gateway pod, TLS Secrets owned by it"]
    ST --> LP["create loop pod"]
    LP -- "create fails" --> TD1["teardown"]
    ST -- "start fails" --> TD1
    LP --> RUN["run executes"]
    RUN --> COL["collection: delete loop pod, then teardown"]
    COL --> TD2["teardown"]
    TICK["discovery tick: sweep_orphan_gateways"] --> CHK{"is the work_pods row of<br/>its loop pod running?"}
    CHK -- no --> TD3["teardown"]
    CHK -- yes --> KEEP["keep"]

The phases are these:

• reserve creates the gateway Service and mints the TLS material for its ClusterIP (crucible-controller/src/runs/workpod/gateway.rs:184).

• start creates the loop Service, the NetworkPolicy and the gateway pod (gateway.rs:256). It then creates the two TLS Secrets with an owner reference to the gateway pod.

• teardown deletes objects in this order (gateway.rs:337):

  1. the Sandbox resources and supervisor pods with the gateway id label;
  2. the loop Service in the loop namespace;
  3. the gateway's own objects.

  The gateway's own objects go last. A teardown that fails halfway leaves the label that the sweep finds.

• sweep_orphan_gateways runs on the daemon discovery tick (gateway.rs:398). It finds the gateway objects in each sandbox namespace by crucible.dev/role=openshell-gateway. It removes each gateway whose loop pod row is absent or not running. The daemon registers it only under an isolated profile.

A failed launch removes its gateway at once (crucible-controller/src/runs/workpod/run.rs:1015). Collection removes it after it deletes the loop pod (run.rs:1116). Each failure logs a warning and leaves the rest to the sweep.

Turn pods (grounded rank, scope) get a gateway the same way, with certificates valid for the turn deadline (crucible-controller/src/runs/workpod/spec.rs:409). Their loop Service opens only the OTLP collector port. Turn sandboxes are never bound, so they get no agent tokens.

6.3 Objects

The gateway objects carry crucible.dev/gateway-for=<run pod> and crucible.dev/role=openshell-gateway. What the gateway creates carries openshell.ai/gateway-id=<gateway id> (crucible/src/deploy/render/gateway.rs:78).

The run Secret holds the loop pod client material under openshell-gateway-ca.crt, openshell-gateway-tls.crt and openshell-gateway-tls.key. The engine reads them as files in /etc/crucible/gateway-tls. The loop pod env gets CRUCIBLE_OPENSHELL_GATEWAY_URL (https://<gateway ClusterIP>:17670) and CRUCIBLE_OPENSHELL_GATEWAY_TLS_DIR.

The gateway pod runs the loop image with the command crucible openshell-gateway (crucible/src/deploy/render/gateway.rs:224, crucible/src/openshell/gateway.rs:886). It gets the driver env of the loop pod (CRUCIBLE_SANDBOX_*, the supervisor and runtime images, the OTLP endpoints), and these per-run values:

The gateway keeps its state in memory (--db-url sqlite::memory:). Under the kubernetes driver, the gateway accepts any client with a certificate from the run CA as an authorized user (crucible/src/openshell/gateway.rs:507). The run CA is the boundary.

6.4 The per-run CA

GatewayTls::mint (crucible-controller/src/runs/workpod/gateway.rs:90) uses rcgen to make:

• a CA, constrained to path length 0, with keyCertSign and cRLSign;
• a server certificate with the gateway Service ClusterIP as its only IP SAN, serverAuth;
• a loop client certificate, clientAuth;
• a supervisor client certificate, clientAuth.

All four start one day before the mint date. They end after the run deadline in whole days plus one day, or after 30 days for a run with no deadline. The controller drops the CA key after the mint. No other run trusts this CA, so a certificate of one run cannot call the gateway of another.

6.5 Network policy

The gateway NetworkPolicy (crucible/src/deploy/render/gateway.rs:362) selects the gateway pod. It admits ingress on port 17670 only from:

• the loop pod of this run (crucible/run=<run> in the loop namespace), and
• supervisor pods with openshell.ai/component=supervisor and this gateway id.

The loop Service exposes the ports that the rendered loop NetworkPolicy lists for sandbox traffic. These are the broker, the OTLP collector, buildit on 8848 with [buildit], and the [mcp] servers of the pack. Under an isolated profile, the loop render omits port 17670 (crucible/src/deploy/render/kube.rs:1342).

6.6 RBAC for each identity

Note: pods/finalizers update is necessary on OpenShift. The TLS Secrets have an owner reference with blockOwnerDeletion to the gateway pod, and the OpenShift owner-reference admission checks that right.

Note: The loop render grants the builder pod rights only when the profile has [buildit], and only in [buildit] namespace. The render does not depend on sandbox_driver.

Warning: The loop service account is shared by every run. Rights to create pods or exec into pods in the loop namespace would let one run's pack code reach another run's loop pod and run Secret: its gateway client certificate, pod token, push authfile and kubeconfig. It could also label a pod crucible/run=<other run> to pass that run's gateway NetworkPolicy. That is why builder pods live in a namespace of their own, which profile load refuses to share with the loop, rig, sandbox or crucible-kueue namespaces. The rights in the buildit namespace still let runs see each other's builder pods there, including a push's authfile while buildit streams it into the builder pod. The clean end state is buildit under a service account of its own.

With [cluster] aws_sandbox_role_arn, the gateway pod gets its own projected token for the sts.amazonaws.com audience. The trust policy of that role must allow system:serviceaccount:<sandbox namespace>:crucible-openshell-gateway.

Pod admission in these namespaces depends on the cluster's platform (section 11.2). The gateway and loop pods set no UID: on OpenShift the namespace's SCC assigns one, and on plain Kubernetes they run as their image's user, root, so their namespaces enforce Pod Security baseline there. OpenShell's sandbox and supervisor pods set their own non-root UID, drop every capability and use RuntimeDefault seccomp, so they would pass restricted on either platform.

6.7 Push-only clusters and directly reachable clusters

The controller creates gateway objects, reads Sandbox resources and reads SPIRE keys through its own API client for the cluster. A push-only cluster sends snapshots to the hub, and the hub has no client for it. So under an isolated profile, dispatch to a push-only cluster fails with PushOnlyCluster (crucible-controller/src/runs/workpod/run.rs:557, run.rs:707).

A cluster is push-only when the hub has a push token for it and no client for it. A cluster has a client when it is the hub, a personal target, or has a kubeconfig in CONTROLLER_CLUSTERS_DIR. A cluster with both a push token and a kubeconfig dispatches normally.

Note: The token path depends on the per-cluster client. When the hub cannot reach a spoke, the sandboxes of that spoke get no tokens until the spoke is reachable again.

6.8 When sandbox_namespace is unset

Profile load also refuses a sandbox namespace that is the loop namespace, the rig namespace, the crucible-kueue namespace or the workload namespace (crucible/src/deploy/profile.rs:859).

6.9 SPIRE discovery

The issuer verifies a sandbox's JWT-SVID against the SPIRE of the cluster that issued it, so it needs each agent cluster's JWT issuer, trust domain and the Service its keys are read through. It finds them itself (crucible-controller/src/agent_auth/discovery.rs), through the same cluster client it dispatches with, for every cluster an agent server serves:

The gateway prefix is spiffe://<trust domain>/openshell/gateway/. Discovery runs at startup and each minute for every cluster not yet read, and every ten minutes for all of them. A cluster it cannot read, or whose SPIRE runs in a namespace pack code reaches, has agents off: its runs get no agent servers, the issuer refuses its SVIDs, the controller logs why, and GET /api/agent-clusters reports each cluster as ready, pending or unavailable with the reason. CONTROLLER_AGENT_SPIRE names a SPIRE installed anywhere else and replaces discovery for that cluster.

The run render grants the identities in [cluster] sandbox_readers what discovery reads, as crucible-spire-reader: get on services/proxy for the discovery Service in SPIRE's namespace, and the spire-server ConfigMap on Kubernetes, or a ClusterRole for the two operator resources named cluster on OpenShift.

7. crucible-kueue internals

7.1 Request pipeline

flowchart LR
    REQ["POST /mcp"] --> AUTH["auth::authenticate<br/>verify token, required scope kueue.read"]
    AUTH --> CAN["ClaimMap::canonicalize<br/>map configured claims to kube_user,<br/>kube_groups, run_id, namespace"]
    CAN --> DNS["default_namespace<br/>the only usable namespace, if one"]
    DNS --> ID["Identity::from_principal<br/>run or person, kube_user required"]
    ID --> GUARD["guard::enforce<br/>method allowlist, per-tool scope,<br/>tool policy, access checks"]
    GUARD --> RMCP["rmcp Streamable HTTP<br/>stateless, JSON responses"]
    RMCP --> TOOL["KueueMcp tool<br/>acts as the server service account"]
    META["GET /.well-known/oauth-protected-resource/mcp"] --> MD["RFC 9728 metadata, no auth"]
    HZ["GET /healthz"] --> OK["ok, no auth"]

The app is built in crucible-kueue/src/lib.rs:62:

1. auth::authenticate verifies the bearer token and checks kueue.read (crucible-kueue/src/auth.rs:117). A token that fails verification gets a 401 with the RFC 9728 challenge. A token without kueue.read gets a 403 step-up challenge.
2. ClaimMap::canonicalize copies the configured claims into the canonical names, and removes a canonical claim that the configured claim does not supply (crucible-kueue/src/config.rs:289). A token cannot set kube_user itself unless kube_user is the configured user claim.
3. A token without namespace gets the one namespace the caller can use, when there is exactly one (auth.rs:36).
4. Identity::from_principal refuses a people token with run_id, a people token on a server with no reviewer, and any token without kube_user (auth.rs:68). The refusal is a 403. A run's identity is its run_id; a person's is its sub. A value that is not a label value gets a hash suffix.
5. The guard decides the request (decide, crucible-mcp-auth/src/guard.rs:148). initialize, ping, server/discover, notifications/* and responses to the server pass. tools/list passes and has injected parameters removed from each inputSchema. Every other method must be listed in KueueRules. JSON-RPC batches are refused.
6. For a listed call, the guard checks the scope. For a tool, it then applies the tool policy. It then sends each access check to the reviewer (review, guard.rs:106). The request goes on as the guard parsed and rewrote it.
7. A refused tool call returns a tool result with isError and the text {"status": "refused", "error", "retryable"}, so the model can read the reason. Only an access review that did not complete (the API server or the group lookup failed) is retryable; a caller the reviewer cannot check (an empty or system: kube_user, malformed kube_groups) is refused for good (ReviewError, crucible-mcp-auth/src/access.rs). An absent scope returns a 403 step-up challenge. A refused method returns JSON-RPC error -32001.

7.2 Tools

The namespace of each check is the call's namespace argument, else the token's namespace claim. Every tool takes an optional namespace. tasks/get and tasks/cancel take no namespace: they act in the namespace the task id names (7.9). Each check covers the object named; what a read returns beyond it is read with the server's account (2.6).

Note: A unit test fails if a tool has no scope in KueueRules. A new tool cannot ship unchecked.

7.3 Plan, then submit

sequenceDiagram
    autonumber
    participant A as agent
    participant K as crucible-kueue
    participant API as API server
    A->>K: workload_plan {queue, manifest}
    K->>API: SubjectAccessReviews (queue use, create kind, Secrets, PVCs)
    K->>API: get LocalQueue and its ClusterQueue
    Note over K: normalize the manifest, compute plan_hash
    K->>API: create with dryRun, plan-hash and actor annotations
    K-->>A: plan_hash, fit, pod_sets, server_changes, manifest
    A->>K: workload_submit {queue, manifest, plan_hash}
    K->>API: the same reviews
    Note over K: normalize again, compare plan_hash
    alt hash differs
        K-->>A: error that names both hashes
    else object absent
        K->>API: create
        K-->>A: submitted, uid, task_id
    else object exists with the same plan hash
        K-->>A: already submitted with this plan
    else object exists with another plan hash
        K-->>A: conflict
    end

workload_plan normalizes the manifest, runs a server-side dry run, and estimates the fit against the ClusterQueue quota as fits_now, waits_for_quota or never_fits (crucible-kueue/src/server.rs:1604). It returns plan_hash, finished_retention_seconds, server_changes, the admission mode, the pod sets, the total requests and the normalized manifest.

workload_submit normalizes the same input again and compares the hash (server.rs:1658). The hash is the SHA-256 of the normalized object as key-sorted JSON (crucible-kueue/src/manifest.rs:504). It depends only on content, so the agent must send the same queue and the same manifest. An identical resubmit is a no-op. A name that holds another plan is a conflict.

After the hash, the server stamps two annotations: <domain>/plan-hash, and for a run, <domain>/actor with the SPIFFE ID of the presenter (server.rs:288).

7.4 Manifest allowlist and profiles

The allowlist is data (crucible-kueue/src/profile.rs). Anything not on the list is refused.

A TOML file starts from base = "strict" or base = "tenant" and has optional [add] and [remove] tables with the same list names. The tool policy can make any rule stricter.

The pod field list has no serviceAccountName, hostNetwork, hostPID, hostIPC, nodeName, schedulerName or runtimeClassName. So a manifest cannot pick another identity, the host, or a way past Kueue. The object-level spec of each kind is also on a list, so fields such as managedBy never reach the cluster.

7.5 Normalization

normalize (crucible-kueue/src/manifest.rs:303) does these steps:

1. Find the kind from apiVersion and kind. Seven kinds are known: Job, RayJob, LeaderWorkerSet, Deployment, StatefulSet, JobSet, PyTorchJob.
2. Remove status, uid, resourceVersion, creationTimestamp and managedFields. Refuse generateName, ownerReferences and finalizers.
3. Require metadata.name as a DNS-1123 label.
4. Set metadata.namespace to the caller namespace. Refuse another value.
5. Refuse reserved labels and annotations that the profile does not allow (manifest.rs:822).
6. Set the object labels: kueue.x-k8s.io/queue-name, <domain>/managed-by=crucible-kueue, <domain>/run-id, <domain>/owner-kind (run or person), <domain>/launched-by. A manifest value that conflicts is refused.
7. With --suspend (default true), create Jobs and JobSets with spec.suspend: true, and PyTorchJobs with spec.runPolicy.suspend: true, for Kueue to start on admission.
8. For batch kinds, add activeDeadlineSeconds (default 86400) and ttlSecondsAfterFinished (default 3600) where the manifest sets none (manifest.rs:228). A JobSet gets the deadline on each replicated Job; a PyTorchJob gets both on runPolicy; a RayJob gets the TTL only with shutdownAfterJobFinishes: true.
9. Check the object-level fields of the kind. A RayJob may not enable the in-tree autoscaler, and its head service must be ClusterIP. Volume claim templates may set only access modes, resources and volume mode.
10. In each pod template: check reserved metadata, set <domain>/workload and <domain>/run-id, check the pod spec (manifest.rs:1142), and set automountServiceAccountToken: false. A manifest with true is refused.
11. Check each container. Refuse privilege, privilege escalation, root and added capabilities. Accept procMount only as Default, and seccomp and AppArmor only as RuntimeDefault. Refuse a probe or hook with a host.
12. Add pull secrets (section 7.6).
13. For a Deployment or StatefulSet, pin spec.selector.matchLabels to the workload and run-id labels, so its controller adopts only pods of this object (manifest.rs:1122).

server_changes in the plan lists what step 12 added.

7.6 Pull secrets

--pull-secret name or --pull-secret name=registry declares a pull secret of the served namespaces. A pod template may name only declared pull secrets. When a container image is on a declared registry (a host, or a host/path prefix), the server adds that secret to the pod itself (crucible-kueue/src/manifest.rs:433). An image on another registry must be public.

Warning: The server does not create the pull secret. It must exist in the workload namespace with that name, or the pods cannot pull. crucible deploy render --kueue does not copy it.

7.7 Access checks and namespaces

The reviewer (crucible-mcp-auth/src/kube.rs) builds a SubjectAccessReview with:

• user: the canonical kube_user, never a system: user;
• groups: OpenShift Group objects that list the user plus system:authenticated and system:authenticated:oauth (--kube-authz openshift), or the kube_groups claim without system: entries plus system:authenticated (--kube-authz token, where the token is not an OpenShift OAuth sign-in);
• the resource attributes of the access check.

The server serves the namespaces in --namespace, else every namespace that holds a LocalQueue. A caller can use a served namespace when its RBAC lets it list LocalQueues there (crucible-kueue/src/namespaces.rs:97-115). The answer is cached 30 seconds per identity (subject, kube_user and kube_groups), so a RoleBinding granted or removed shows in the usable list within 30 seconds; the guard's per-call access decisions are cached as section 2.6 says. Every tool call and both task methods resolve the namespace they act in (the call's namespace argument, the token's namespace claim, or the task id's namespace) and refuse one the server does not serve, or one the token's namespaces claim leaves out, with not_found and the list of usable namespaces. The guard has already run the call's SubjectAccessReviews by then, so those reach the API server first; the check comes before the server reads or writes anything with its own account. Identity::act_in (crucible-kueue/src/caller.rs) makes this check and is the only way to get a Caller, the type every tool acts with, so a tool cannot skip it. A test calls every tool the router lists in an unserved namespace and expects not_found (crucible-kueue/tests/http_auth.rs). The usable list is also cut to the claim. A server with no --namespace that cannot list LocalQueues cluster-wide serves nothing. It never passes on the API server message about the server's own account.

7.8 Status rollup and the wait

Kueue can track one object as several Workloads. A Deployment gets one Workload for each pod. rollup (crucible-kueue/src/status.rs:138) gives each Workload a phase: NoWorkload, Pending, QuotaReserved, Admitted, Evicted, Stopped, Finished. The headline is the phase furthest from the run state, with counts for each phase. So pending with admitted > 0 means partly admitted.

workload_status reports the phase and the reason. While the workload waits, it also reports the queue position and a new fit estimate. It then reports pod counts, up to five problem pods with exit messages, recent events and a next_action. detail: "full" adds all conditions, the object status and every pod. For a caller with a kube_user, the Workload fields (phase, counts, queue position, fit, readiness, next_action), the pod fields and the events each need get and list on Kueue workloads, pods and events in the namespace; a part without them is left out and listed under omitted with the reason.

With until (admitted, running, ready or finished) the server waits for up to timeout_seconds, at most 50, and polls each 2 seconds (crucible-kueue/src/server.rs:1228). The wait ends with one of these:

7.9 MCP tasks

The server enables the MCP tasks extension for batch kinds (Job, RayJob, JobSet, PyTorchJob). A client that declares the extension gets a task from workload_submit. The task id is <namespace>/<Kind>/<name>/<uid> (crucible-kueue/src/task.rs:16), so a new object with the same name is a different task. Clients without the extension get a synchronous reply that still carries task_id.

The server keeps no task state. tasks/get reads the cluster each time. A task completes when Kueue marks the Workload finished. It failed unless the reason is Succeeded. tasks/cancel sets the <domain>/task-cancelled annotation and deactivates the Workload (server.rs:2175).

Serving kinds are never tasks. workload_status reports their readiness instead.

7.10 Ownership modes

workload_list takes filter: mine (default, this run), managed (all objects of this server), or all (every Kueue-queued workload of a served kind). Stop and resume deactivate the Kueue Workload for Job, RayJob, LeaderWorkerSet, JobSet and PyTorchJob. For a Deployment or StatefulSet they scale to zero and keep the replica count in <domain>/stopped-replicas.

7.11 Kueue version and kinds

At startup the server reads /apis/kueue.x-k8s.io and uses v1beta2, else v1beta1. It does not start when the cluster serves neither (crucible-kueue/src/kueue.rs:49). It also lists which of the seven kinds the cluster installs (crucible-kueue/src/kinds.rs:11). whoami and the server instructions report them.

7.12 Generic configuration

The server does not need crucible. These flags make it a general Kueue MCP server for a cluster with a stock OIDC provider:

8. buildit per run

8.1 Where it runs

buildit runs as a child process of the engine in each loop pod of a [buildit] cluster. It is an entry of CRUCIBLE_AGENT_SERVERS like any other, named buildit, whose transport is the loop pod (local.buildit, carrying its run issuer, builder namespace and push repository). The engine starts it before it creates the providers (Granted::start_local, crucible/src/openshell/agent_servers.rs), and gives it a provider, an egress entry and a .mcp.json entry the way it does every server. It listens on the port of its URL, [buildit] url, by default http://host.openshell.internal:8848/mcp, which is the loop Service. The supervisor injects a token for that audience on egress to that host and port only.

The engine runs the buildit first on its PATH. That is the loop image's own unless [buildit] image names another: the buildit-bin init container then copies that image's /usr/local/bin/buildit into an emptyDir the loop container mounts read-only at /opt/crucible/buildit/bin, and the wrapper puts that directory first on PATH, so the engine and the run's own tasks run it. The shared cache runs from the same image unless [buildit.cache] image names one. docker.yml publishes ghcr.io/neuralmagic/crucible-buildit from every main build (docker/Containerfile.buildit), and the controller reads the deploy profile at each dispatch, so moving the pin rolls buildit for the next runs with no new loop image.

The engine starts it with these arguments (Buildit::args, crucible/src/openshell/buildit.rs):

8.2 Who may call it

realm_caller (buildit/src/mcp.rs:832) accepts a token only when all of these are true:

• The trusted issuer is a run issuer. People tokens are refused.
• run_id is this server's --run-id. A token of another run is refused.
• sandbox is a label value. It names the sandbox to read from.
• workdir, when present, is an absolute normal path.
• sub is a label value. It becomes crucible.dev/launched-by on builder pods.

The scopes are buildit.read for status, logs and tasks/get, and buildit.build for build, run, push, clean and tasks/cancel. buildit does no access checks: it builds in its own namespace for the run's own sandboxes.

8.3 How it reads the sandbox

For build, buildit downloads the build context with openshell sandbox download <sandbox> <workdir>/<context> <dest> through the run's gateway (buildit/src/sandbox.rs:101). The context path must be relative, without ... It may not contain .git, .mcp.json, .claude, .buildit, .kube or .jira. sanitize then deletes those names but .git, special files, and symlinks that resolve outside the context (sandbox.rs:207). Of each .git directory it keeps HEAD, index, objects, packed-refs, refs and shallow, so a build can read the checkout's history (vLLM's setup.py takes its version from it), and deletes the rest: the config, where remote URLs and http headers can carry credentials, the hooks and the logs. run writes requested output paths back to <workdir>/.buildit/<build id>/ in the sandbox with openshell sandbox upload.

8.4 Builder pods

Each build gets one builder pod in the buildit namespace (buildit/src/mcp.rs:971). The pod has the labels buildit.dev/managed-by=mcp, buildit.dev/sandbox, buildit.dev/build-id and crucible.dev/launched-by. It runs buildah with automountServiceAccountToken: false, enableServiceLinks: false and an activeDeadlineSeconds of --pod-deadline (default 7200). --max-builds (default 4) limits live builder pods per sandbox. The shipped policy caps a build at 3600 seconds and a run at 1800 seconds; without a value they are 1800 and 600.

Warning: A build's RUN steps and buildit run execute the agent's code in the builder pod, with no OpenShell sandbox around it. The loop render's NetworkPolicy buildit-builders in the buildit namespace admits no ingress to builder pods, and egress only to DNS (UDP and TCP 53 and 5353) and to [buildit] egress, by default TCP 443 anywhere, for the registries a build pulls from and pushes to. A NetworkPolicy matches IP networks, not host names: to keep RUN steps away from in-cluster services, list the registries' networks, or add as except the cluster's pod network, its service network (which holds the API server's ClusterIP, the kubernetes Service in default, such as 172.30.0.1 on OpenShift or 10.96.0.1 on kubeadm), and the API server's own addresses when it is reached outside the service network. The pod and service networks are in oc get network.config cluster -o yaml on OpenShift, or the kube-controller-manager's --cluster-cidr and --service-cluster-ip-range. The loop render logs a warning when an egress entry reaches 0.0.0.0/0 or ::/0 with no except, as the default does (BuilditCfg::unfenced_egress).

The loop account may create any pod in the buildit namespace, and builder pods need AppArmor unconfined, so that namespace cannot enforce Pod Security baseline. Without more, pack code could create a privileged pod, mount the node's / or join its network, and from the node reach every other run. The loop render therefore also emits a ValidatingAdmissionPolicy and its binding to the buildit namespace, both named <namespace>-buildit-builders, from buildit/deploy/builder-admission.yaml (builder_admission, crucible/src/deploy/render/kube.rs). It admits a pod there only when:

• it carries buildit.dev/managed-by=mcp;
• it runs one container, builder, from quay.io/buildah/stable:latest, with no init or ephemeral containers;
• the container is not privileged, adds no capabilities, runs as a non-root runAsUser, does not ask for seccomp Unconfined, keeps the default /proc mount, and sets no SELinux user, role or type;
• it sets no nodeName (on create), runtime class, system priority class, sysctls, or toleration beyond the not-ready and unreachable ones admission gives every pod;
• it shares no host network, PID or IPC namespace, and mounts only emptyDir volumes;
• it runs as the default account with automountServiceAccountToken: false;
• it reads no Secret or ConfigMap into its environment (valueFrom, envFrom) and names no image pull Secret;
• AppArmor is set only as buildit sets it: the annotation for builder, unconfined (or the matching appArmorProfile field on that container).

The policy checks pod creates and updates, so an update cannot swap the image or the labels. It admits what admission defaults every pod with: those two tolerations, a cluster's global default PriorityClass, and an SELinux level an OpenShift SCC assigns. The policy needs Kubernetes 1.30 or later.

The loop render also emits a ResourceQuota in the buildit namespace from [buildit] quota, by default 16 pods and no NVIDIA or AMD GPUs. Builder pods request no CPU or memory, so a quota on requests.cpu or requests.memory would refuse every one. buildit's tests evaluate its own builder pod against the same file and expect it to pass, and expect each escape above to fail (the_builder_admission_policy_admits_buildits_pod_and_no_escape, buildit/src/mcp.rs), so the policy and the builder cannot drift apart.

On OpenShift, SCC admission does the same job when the builder pods' account (default in the buildit namespace) can use only an SCC no broader than this: allowPrivilegedContainer: false, allowedCapabilities: [], allowHostNetwork, allowHostPID, allowHostIPC and allowHostDirVolumePlugin false, volumes: [emptyDir], runAsUser: MustRunAsNonRoot, and the unconfined AppArmor annotation allowed. Do not grant that account, or the loop account, anyuid or privileged; the loop account's pods in that namespace run as whatever account the pod names, so every account there needs the same bound. The ValidatingAdmissionPolicy applies on OpenShift too, so both hold.

8.5 The push path

push publishes a build that succeeded to one repository only: --push-repo, for example registry.example.com/team/smoke. The tag is <sandbox>-<build id>. The repository must be <registry>/<path> with no tag, digest or scheme (buildit/src/mcp.rs:457).

sequenceDiagram
    autonumber
    participant A as agent
    participant B as buildit (loop pod)
    participant P as builder pod
    participant R as registry.example.com
    A->>B: push {build_id}
    Note over B: build must have exited 0
    B->>B: read the push authfile from the loop pod mount
    B->>P: exec a shell that writes stdin to a temp file and runs buildah push<br/>authfile on stdin
    P->>R: push REPO:SANDBOX-BUILD_ID
    P-->>B: sha256 digest on stdout
    Note over P: the temp files are removed on exit
    B-->>A: image REPO@sha256 digest, tag

The authfile is the loop pod's [secrets] push_authfile Secret, mounted in the loop pod only (FORGE_AUTHFILE). buildit reads it on each push and streams it to the builder pod over exec stdin (buildit/src/mcp.rs:479, mcp.rs:1089). The builder pod never mounts it, and the sandbox never sees it. Without a push repository or without the authfile, push is refused.

Warning: the push runs in the same builder pod as the agent's build steps and run commands, as the same user, and nothing stops a process a step left behind when push starts. Such a process could read the authfile while the push holds it, then push to any tag, or any repository the credential reaches. Until the push moves to a container of its own, give [secrets] push_authfile a credential scoped to the one push repository.

8.6 The run's own socket

The same buildit mcp also serves its tools on a unix socket, <state dir>/buildit.sock (mode 0600), for the run's deterministic tasks. The socket takes no token. Every call on it is the run itself, under the sandbox name loop-<first 16 hex of sha256(run id)>, which no openshell sandbox (ci-<hex>) and no other run takes. That name scopes --max-builds, clean and every lookup the way a sandbox name does, and buildit refuses any token that names it. The shipped tool policy, the builder namespace, the queue and the push repository are the same as over HTTP.

The engine exports the socket as CRUCIBLE_BUILDIT_SOCKET to command and evaluate tasks only (ShellRunner, crucible/src/plan/runner.rs). Agent turns and sandboxes never get it, and route and report tasks spawn nothing. Before such a task runs, the engine starts the run's buildit if the controller granted one (run_buildit, crucible/src/openshell/agent_servers.rs), and the task fails as a transport error when buildit does not come up. With the variable set, buildit build, status, run, push, logs and clean call the socket. Each prints one JSON object on stdout and exits nonzero on failure:

buildit build -c image/            # waits for the build to end; --no-wait returns at once
buildit context-digest image/      # the context_digest a build of image/ reports

A build on the socket reads its context from the run's workspace. The CLI sends -c as an absolute path, and buildit refuses one outside the workspace, so a task in an isolated worktree cannot build its own copy. run writes fetch_paths to <workspace>/.buildit/<build id>/.

Every build reply carries context_digest: a sha256 over the context archive buildit uploaded, one record per entry sorted by path (path, file mode bits, content hash or symlink target; owners, times and directory modes do not count). buildit keeps it on the builder pod, so status, a completed build task and push report it after a server restart too. `buildit context-digest

CONTROLLER_AGENT_ISSUER=https://crucible.example.com
CONTROLLER_AGENT_SIGNING_KEY_FILE=/etc/crucible/agent-signing/key.pem
CONTROLLER_DEPLOY_PROFILE=/etc/crucible/profile/profile.toml

[cluster]
# ...the keys every profile has (loop_namespace, service_account, supervisor_image, ...), plus:
sandbox_namespace = "crucible-sandboxes"
sandbox_readers   = ["crucible-system/crucible-controller"]

[kueue]
clusters            = ["pirate", "waldorf"]
workload_namespaces = ["crucible-kueue-e2e"]
people              = true
people_urls         = { pirate = "https://kueue.pirate.example.com/mcp" }

[buildit]
clusters  = ["pirate", "waldorf"]
push_repo = "quay.io/example/crucible-smoke"

erDiagram
    runs ||--o{ run_sandboxes : "run_id"
    work_pods ||--o{ run_sandboxes : "pod_name"
    people_grants ||--o{ people_refresh_tokens : "grant_id"
    runs {
        TEXT run_id PK
        TEXT dispatched_by
    }
    issues {
        TEXT key PK
        TEXT dispatch_requested_by
    }
    users {
        TEXT sub
        TEXT login
        TEXT kube_user
        TEXT kube_user_override
    }
    run_sandboxes {
        TEXT sandbox_id PK
        TEXT run_id FK
        TEXT pod_name FK
        TEXT bound_at
        TEXT sandbox_name
        TEXT workdir
    }
    kube_identity_links {
        TEXT sub PK
        TEXT cluster PK
        TEXT kube_user
        TEXT source
        TEXT linked_at
    }
    people_codes {
        BYTEA key PK
        TEXT sub
        JSONB grant_json
        BIGINT expires_at
    }
    people_grants {
        TEXT id PK
        TEXT sub
        TEXT client_id
        TEXT client_name
        TEXT resource
        TEXT cluster
        JSONB scopes
        BIGINT created_at
        BIGINT last_used_at
        BIGINT revoked_at
    }
    people_refresh_tokens {
        BYTEA key PK
        TEXT grant_id FK
        BIGINT issued_at
        BIGINT used_at
    }
    people_dynamic_clients {
        TEXT client_id PK
        TEXT client_name
        JSONB redirect_uris
        BIGINT created_at
        BIGINT last_used_at
    }
    run_agent_tools {
        TEXT run_id PK
        TEXT server PK
        TEXT cluster
        TEXT url
        TEXT kube_user
        JSONB scopes
        JSONB limits
        TEXT granted_at
    }

openssl ecparam -genkey -noout -name prime256v1 | openssl pkcs8 -topk8 -nocrypt -out agent-signing.pem

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: team-queue-user
  namespace: team-workloads
rules:
  - apiGroups: [kueue.x-k8s.io]
    resources: [localqueues]
    verbs: [list]
  - apiGroups: [kueue.x-k8s.io]
    resources: [localqueues]
    resourceNames: [team-queue]
    verbs: [get, use]
  - apiGroups: [kueue.x-k8s.io]
    resources: [workloads]
    verbs: [get, list]
  - apiGroups: [batch]
    resources: [jobs]
    verbs: [create, get, update, delete]
  - apiGroups: [""]
    resources: [pods, pods/log]
    verbs: [get, list]

BEGIN;
DROP TABLE people_dynamic_clients;
DROP TABLE run_agent_tools;
DROP TABLE people_refresh_tokens;
DROP TABLE people_grants;
DROP TABLE people_codes;
DROP TABLE kube_identity_links;
ALTER TABLE runs DROP COLUMN dispatched_by;
ALTER TABLE issues DROP COLUMN dispatch_requested_by;
DROP TABLE run_sandboxes;
ALTER TABLE users DROP COLUMN kube_user_override;
ALTER TABLE users DROP COLUMN kube_user;
DELETE FROM _sqlx_migrations WHERE version BETWEEN 59 AND 68;
COMMIT;

BEGIN;
DROP TABLE people_dynamic_clients;
DELETE FROM people_grants WHERE client_id <> 'claude-code';
DROP INDEX people_grants_sub_client;
ALTER TABLE people_grants DROP COLUMN client_name;
ALTER TABLE people_grants DROP COLUMN client_id;
DELETE FROM people_codes;
DELETE FROM _sqlx_migrations WHERE version IN (67, 68);
COMMIT;