Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-0068: Agents get identity-only tokens from the controller, bound to their run's own gateway

Status: accepted | Date: 2026-10-08

References: ADR-0067, ADR-0036

Context

ADR-0067 gave sandboxed agents access to crucible-kueue with per-run bearer tokens: the controller minted a random token at dispatch, wrote its digest, run id, namespace, role and queue allowlist as a tenant record ConfigMap in a controller-only namespace, and the server watched those records. Building the next consumers (buildit, the agent MCP servers behind the OpenShell provider token grant) and reviewing the design surfaced its limits:

  • Every resource server would need its own record watch and namespace, or share one, and every new server multiplies the deployment surface the records already cost.
  • The record carried authorization (role, queues) decided by the controller from the deploy profile, which the server then replayed. The person behind the run, and what Kubernetes lets that person do, never entered the decision.
  • The token was a static credential in the loop pod's environment, readable by pack command tasks, valid for the whole run.
  • OpenShell already ships a per-sandbox credential flow: the supervisor fetches a SPIFFE JWT-SVID and exchanges it with an OAuth token endpoint (RFC 7523 jwt-bearer client assertion) for an access token it injects on egress, so the agent never holds it.

A SPIFFE ID is only as trustworthy as the pods it can be issued to. A sandbox ID derived from a pod annotation in a namespace pack code can create pods in is a credential pack code can forge, and with a gateway in the loop pod, running as the loop service account, pack code could do exactly that.

Decision

The controller is the OAuth authorization server for sandboxed agents, and its tokens say who is calling, never what they may do.

  • Each run's gateway runs in its own pod. Under the kubernetes driver with [cluster] sandbox_namespace, the controller starts every run's OpenShell gateway (and every turn's) as crucible-openshell-gateway in the sandbox namespace, the only identity with rights there; the loop service account has none. The controller mints a CA per run, a server certificate for the gateway Service's address and client certificates for the loop pod and the supervisors, and keeps no CA key. A NetworkPolicy admits only that run's loop pod and the supervisors its gateway created.
  • The SPIFFE ID names the gateway. Each run's gateway id is its run pod's name; the gateway stamps it as openshell.ai/gateway-id on every supervisor and Sandbox it creates, and driver_config cannot set labels or annotations. The ClusterSPIFFEID, scoped to the sandbox namespace, is spiffe://<td>/openshell/gateway/<gateway id>/sandbox/<sandbox id>.
  • Binding and issue check the gateway. The engine binds each sandbox it creates to its run with the run's pod token; the controller binds only a Sandbox carrying that id and the run's gateway id, and owner-references it to the gateway pod. POST /oauth/token verifies the SVID, resolves the binding, and refuses an SVID whose gateway is not the bound run's.
  • Claims are identity only. sub (the person the run was dispatched for, consumed per dispatch), act.sub (the SVID's SPIFFE ID), run_id, sandbox, workdir, kube_user (the person's Kubernetes username, from an explicit link or an administrator's override), and namespace, the default namespace for the run's workloads. No role, no queues: a resource server decides what the caller may do, crucible-kueue by SubjectAccessReview as kube_user.
  • Tenant records go away. Neither the controller nor crucible-kueue keeps per-run records; the token is the grant.
  • Keys rotate without a flag day. Tokens are ES256 with an RFC 7638 thumbprint kid. The JWKS publishes the current key and, while configured, the previous one, which no longer signs. Tokens live 15 to 30 minutes (CONTROLLER_AGENT_TOKEN_TTL_SECS).
  • SVID trust comes from each cluster's own SPIRE. The controller verifies an SVID against the JWT authorities of the SPIRE that issued it, read through the controller's existing per-cluster API client rather than a separate network route. SPIRE's bundle ConfigMap carries only the X.509 roots, so the keys come from the SPIRE OIDC discovery provider's JWKS reached through the API server's Service proxy.

Consequences

  • A sandboxed agent's calls carry a token it never sees, scoped to one run and one sandbox, minutes long, naming the person and the sandbox; a stolen token is worth one run for at most half an hour.
  • Pack code can no longer mint a sandbox identity: it cannot create pods in the sandbox namespace, and pods its own gateway creates carry its own gateway id, which the issuer will not accept for another run. It can still create sandboxes for its own run with the placement fields the gateway accepts.
  • Authorization moves to the resource servers and to Kubernetes RBAC. A run whose dispatcher has no linked Kubernetes username gets no crucible-kueue scopes.
  • The controller now owns per-run gateway lifecycle in the sandbox namespace (create, adopt, tear down, sweep) and needs RBAC there, rendered for [cluster] sandbox_readers. A cluster the hub only receives pushes from cannot host isolated gateways.
  • The controller's per-cluster client becomes part of the token path: an unreachable spoke means its sandboxes get no tokens until it is reachable again.
  • The OpenShell pin (3441f91) predates v0.1.2; nothing here depends on v0.1.2's sidecar supervisor topology.

Alternatives Considered

Controller-minted identity-only tokens bound to SVIDs from per-run isolated gateways, authorization at the resource server (accepted)

  • Pros: One issuer for every resource server; no records or watches, The token names the person and the sandbox; Kubernetes RBAC decides what they may do, OpenShell's token grant keeps the token out of the sandbox, Pack code cannot forge a sandbox identity
  • Cons: The controller runs a gateway pod per run and turn, Resource servers must verify JWTs and run SubjectAccessReview

Keep tenant records (ADR-0067) and add a record watch per resource server (rejected)

  • Pros: Already built for crucible-kueue
  • Cons: A namespace, RBAC and watch per server, Authorization decided by the profile, not by the person's rights, A static run-long token readable by pack command tasks
  • Rejected because: It scales deployment surface with every server and keeps authorization away from the person's own rights.

Keycloak federated-jwt with a client per run (rejected)

  • Pros: Proven against SPIRE JWT-SVIDs in the spike
  • Cons: Exact-subject matching needs a Keycloak client per sandbox, created, bound and garbage-collected by the controller, A second identity system in the token path
  • Rejected because: Per-run client lifecycle is more moving state than issuing the token directly.

Role and queues in the token, as the tenant records carried (rejected)

  • Pros: The server needs no Kubernetes access to authorize
  • Cons: The controller decides what the person may do from a profile, not from their rights, Two places to change when access changes
  • Rejected because: Authorization belongs to the resource server and Kubernetes RBAC.

One shared gateway per cluster in the sandbox namespace (rejected)

  • Pros: One gateway pod per cluster instead of per run
  • Cons: A gateway admin certificate shared by every run's pack code, so any run can create supervisors with any sandbox id, No per-run gateway id to bind SVIDs to
  • Rejected because: It shares the credential the isolation exists to separate.

Verify SVIDs against a JWKS URL per cluster over routes or services (rejected)

  • Pros: No dependency on the API server
  • Cons: A network path and TLS trust to configure per spoke, Spokes the hub reaches only through their API server need another route
  • Rejected because: The controller already holds an authenticated client for every cluster it dispatches to.

crucible-kueue acts with its own service account and checks each call with a SubjectAccessReview as kube_user (accepted)

  • Pros: The server's own account bounds what any token can reach, whatever the caller's RBAC, The person's RBAC is the ceiling for every call, decided by the API server, The server adds its pull secret, ownership labels and field manager to what it creates
  • Cons: Every call's access checks are modelled by hand in crucible-kueue's access.rs, and a call it models wrong is allowed or refused wrongly, Allowed decisions are cached for 30 seconds, so a removed permission can still allow calls that long, hardware_describe reads nodes with the server's account, an exception to the ceiling kept so agents can pick a flavor and GPU product

Pass the agent's token through to the Kubernetes API server (rejected)

  • Pros: Kubernetes authorizes every request itself; no checks to model
  • Cons: The MCP authorization spec forbids passing a token through to another resource, since its audience names this server, The API server would trust the issuer only with an OIDC authenticator configured on every spoke, A leaked token would be a Kubernetes credential, not a crucible-kueue one
  • Rejected because: It breaks audience binding and turns every agent token into a cluster credential.

Impersonate kube_user for every Kubernetes call (rejected)

  • Pros: Kubernetes authorizes each request as the person, with no checks to model
  • Cons: The server needs impersonate on users and groups, which RBAC cannot scope to a dynamic set of people, so a compromised server acts as anyone, Objects would be created as the person, losing the pull secret, ownership labels and field manager the server adds
  • Rejected because: Unscoped impersonation makes the server's account worth every user's rights.