Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-0028: Secrets registry: user- and team-owned secrets, Vault custody through the hub, pull-with-grant projection

Status: accepted | Date: 2026-08-23

References: ADR-0046, ADR-0048, ADR-0049, ADR-0027, ADR-0044, ADR-0029, ADR-0030

Context

Packs need credentials: a write token for the dependency-sweep merge gate, clone tokens for private repos, API keys for skills, Vertex ADC for the agent, registry authfiles for candidate pushes. Today every one of them is a Kubernetes Secret an admin creates by hand in the loop namespace of every cluster, named from the deploy profile ([[secret_env]], [secrets].pull_authfile, pull_secret). Nothing records who owns a secret, who may use it, or when it was used; adding a cluster means copying every secret; and a person who wants to run a net-new workflow on the shared control plane has to file a ticket for an admin to mint and place credentials. The goal is self-service: a user registers a secret in the controller UI, owned by them or by one of their teams, binds it to a repo, playbook, or domain, and launches.

Current state. The SSO edge asserts the caller's idp (LDAP) groups as an OIDC claim, forwarded as a per-request header; the controller parses it (Groups) and matches /groups/team-x path-shaped groups for the operator role. Groups are not stored on the session. On the static-API-token path no identity header is stripped, and on the cluster-token path only user and email are stripped, so groups are caller-asserted on both non-proxy paths. Sessions are server-side in Postgres. ADR-0048 made the hub-to-spoke identity credential-free (projected SA token federation) and put the ingest drop-box on the external router shard behind the WAF, TokenReview, and a bound-pod claim; the claim checks the pod name only, pod names are deterministic per run, and the loop service account holds pods create in the loop namespace. Run pods are excluded from the pod timeout sweep. ADR-0046 deferred token rotation to Vault. The broker-token hardening work established the agent-visibility pattern for header-shaped bearers: the sandbox holds a placeholder and the egress proxy injects the real value. ADR-0027 fixed the rule that launch rows, params, and argv are echoed by the API and prefilled into relaunch forms, which is why a secret value must never travel any of those paths. Local mode spawns the engine with the controller's full environment.

Constraints. Corporate Vault is reachable from the hub and from the corp-internal spokes, but not from the GPU spokes (kelp, wharf, and the rest), so nothing on the spoke side may depend on Vault. The hub cannot reach the corp-internal spokes inbound; they receive no pods. the IdP group membership arrives only as a session claim; there is no directory lookup the controller can make offline, which matters for scheduled launches (ADR-0027 schedules fire with no session). Long-lived classic PATs are forbidden by org policy; the pack-PR credential is already a GitHub App installation token, and secrets of that kind stay chart-mounted controller identity. Kubelet image pull secrets are consumed before a pod runs, so they cannot be fetched by the pod. The manifest declared-secrets block, the loop-image redemption step, and the redactor are core-repo changes that ship with a loop pin bump before the controller release that enables this.

Decision

Three parts: an authorization model in Postgres, byte custody in Vault with the hub as the only client, and a pull-with-grant projection.

1. Model

A principal is user:<login> (the SSO login) or group:<group path>, stored normalized (trimmed, lowercased) and compared as exact full paths; the operator role's tail-segment match is not used for ownership. Both come from a controller-validated identity, never from a directory. A secret row carries a name, an owner principal, a kind (opaque, file, registry_authfile, kubeconfig), a visibility (broker_only or agent_visible), a Vault path, and a consumer class: run (projected into pods) or hub (read by the hub for its own clients, introduced by ADR-0030 for the kubeconfig kind). A binding attaches a secret to a scope (repo, playbook, domain) with a projection (an env name or a file path) and records the pack revision and schema digest it was made against. A grant is created per dispatch: the run, the secret ids it resolved, a redeem-by deadline, and the pod UID it is bound to; every redemption attempt is an audit row with outcome and error. An audit row records every register, rotate, bind, unbind, grant, hub read, and redemption with the acting principal (none for hub reads) and, for redemptions, the cluster.

Packs declare the secret names they need in the manifest, mirroring how params are declared; the manifest is the only declaration site, and a binding is the only mapping from a declared name to a value. The preview gate and import review show the declared names to the registering admin. A launch whose scope lacks a binding for a declared name is refused with the name in the error. The deploy profile's [[secret_env]] stays authoritative for a pack until its manifest declares secrets; when both exist for one name the preview gate warns and the binding wins. Kubelet pull secrets (pull_secret, and the measure-job pull secret) remain pre-created cluster Secrets named on the target; registry_authfile covers only in-pod pulls and pushes. Draft and import test-fires never resolve bindings. A pin bump that changes the pack revision parks launches on that scope until an owner member re-binds.

2. Authorization

Server-side on every route; the SPA is convenience. Any group in the caller's validated claims is ownable.

  • Register: any operator, as owner user:self or any group: in their claims.
  • Bind / unbind: a member of the secret's owner principal who also holds the global operator role.
  • Use at launch: the launcher's principals must cover the owner of every secret bound to the scope. One miss refuses the launch with the secret named.
  • Scheduled launch: the owner is the principal who last saved the schedule; a save by anyone else re-owns the row and re-runs the launch check against the saver's own claims. The save snapshots the owner's groups on the schedule row with a TTL. A schedule on a scope with bindings cannot be created or saved without an authenticated principal. At fire time, with ADR-0029 the hub refreshes the owner's groups from their offline credential; if the refresh fails the snapshot is used while inside its TTL, else the launch parks. Without ADR-0029 the snapshot alone applies. Existing rows have no snapshot and park until re-saved. A re-save expires that schedule's parked launches with a reason.
  • Reference: packs and MCP agents may name a secret. No human or agent may read one; there is no read-back endpoint. Admins see metadata and audit, not values.
  • Delete: owner member; refused while bindings exist.
  • Rotate: owner member; writes a new version. A grant resolves the current version at each redemption, so rotation reaches the next pod start; already-redeemed pods keep what they hold.

The global roles remain the outer check. There is no general per-resource ACL engine; ownership plus bindings covers every rule above, and a later rule such as "who may launch playbook X" reuses the principal and binding tables.

Prerequisite. Every rule above rests on Identity and Groups. They must come from a controller-validated source, which is ADR-0029. Until it ships, x-auth-request-groups and the identity headers are stripped on every non-proxy path, and group: owners are refused on the static-token path.

3. Custody

Vault KV v2, path crucible/registry/<principal>/<name>, versions from Vault. Amended 2026-08-29: a dedicated mount was not obtainable — the mount is an IT-managed standard this team does not control — so the registry shares the apps mount with the deploy-sync secrets and is namespaced under the same crucible/ prefix they use, one subtree the hub's policy can be scoped to. Registering at the mount's top level, as an unprefixed <principal>/<name> would, is refused by Vault. The hub's Vault login is a configured auth method behind one client: AppRole (role id chart-mounted, secret id in a pre-created Secret, wrapped response unwrapped at boot) is the method available at launch; JWT auth with the projected SA token replaces it once workload identity is confirmed on the Vault side, with jwt_validation_pubkeys (or jwks_ca_pem) plus bound_subject and bound_audiences pinned; the hub's discovery mirror is not an acceptable trust root for Vault because the mirror and the subject share a hostname. Switching methods is a chart value; the registry code sees one client. Postgres holds everything except the bytes.

Registration is synchronous: the UI POST writes the KV version and returns the id; plaintext lives in the hub for the request only. If Vault is unavailable the registration fails; nothing is queued. Two ways to bring a secret:

  • Value in the UI: the hub is the writer; the user never touches Vault. The self-service default.
  • Reference to an existing Vault path: the user registers vault://<mount>/<path>#<key> and the path's owner grants the hub's Vault identity read under their own policy; that grant is the path owner's consent to registry-governed use. Because the hub's policy may already cover paths other teams granted, registration and every re-point require a Vault token belonging to the registrant, with which the hub performs a one-shot verification read before storing the pointer; each bind requires the binder's own Vault token for the same read. The token is held in memory for the request only and is excluded from tracing, audit, and error bodies. Paths inside the crucible mount are refused as references. After registration, access is governed by the registry owner principal; a registrant who later loses Vault access keeps registry access until the path owner revokes the hub's read or an owner member unbinds.

The controller's own identity secrets (GitHub App key, fleet spoke kubeconfigs, database URL, push tokens, Jira token) are not user data and do not enter the registry. They are sourced from the same Vault mount under a deploy/<release>/ prefix and materialized into the chart's existingSecret objects by a controller vault-sync subcommand run as a chart hook Job and on a schedule, because the hub cluster has no secrets operator; the only chart-mounted secret left is the Vault login itself. Consumers (the controller env, kubelet pull, the spoke kubeconfig mounts) keep reading Kubernetes Secrets.

4. Projection

Dispatch resolves the scope's bindings, checks the launcher, creates the pod, records the pod UID from the create response on the pod row, and writes the grant bound to that UID with a redeem-by deadline; the pod spec carries the grant id and nothing else. A create that returns AlreadyExists voids the grant. On start, the loop image redeems the grant against the hub: it presents its own bound SA token; the hub TokenReviews it against the pod's cluster and requires the token's pod-uid claim to equal the recorded UID. Other identity modes are defined by the ADR that introduces the target kind. The hub reads Vault and returns the bundle. The same pod may redeem again after a restart; nothing else can. A re-drive that creates a new pod issues a new grant. The loop side writes env and files into a tmpfs and hands the loop process its environment, retrying redemption with capped backoff until the deadline.

Reading at redemption means plaintext never rests on the hub; a rotated secret is picked up by the next pod start; Vault availability is a dependency at pod start, not a cache to invalidate. The redemption endpoint sits on the external router shard with its own Route entry, its own WAF rate limit, and reencrypt termination, so a crash-looping redeemer cannot exhaust the ingest limit and the bundle does not cross the router-to-pod hop in plaintext. The chart change ships before the image. Redemption failures are recorded on the grant; reconcile parks the run and deletes the pod after N failed redemptions or when the deadline passes unredeemed, so a pod that cannot redeem does not hold a GPU in CrashLoopBackOff. Spokes never contact Vault and never hold a Vault token. No secret write RBAC is needed on any spoke. A dedicated turn service account with no RBAC in the loop namespace is a prerequisite, so that sandbox pods cannot create a same-named pod and redeem.

Values never appear in the pod spec, argv, ConfigMaps, launch rows, CRUCIBLE_INPUTS, transcripts, or events. The engine gains a redactor; the loop-side fetcher registers every redeemed value, every leaf value of a parsed file kind, and their base64 and percent-encoded forms with it before the loop starts.

5. Visibility

broker_only is the default and is defined per kind. Header-shaped opaque values get placeholder substitution at the egress proxy, following the broker-token precedent. file, registry_authfile, and kubeconfig kinds are broker-side only and refuse agent_visible. Vertex ADC for the agent goes through the gateway's google-cloud provider, not a placeholder. agent_visible is an explicit opt-in for opaque kinds that the UI warns about, because an agent-visible value can end up in a PR.

6. Local mode

No registry. Local runs spawn with a cleared environment and receive only PATH, the run's own CRUCIBLE_* variables, and the declared secret names resolved from an operator-controlled allowlist; the registration API refuses in local mode.

7. Sequencing

ADR-0029 guard changes first (or the interim header stripping above). Then the core release carrying the manifest block, the redemption step, and the redactor, with a loop and turn-profile pin bump. Then the controller release with the tables, routes, and UI.

Consequences

  • Register in the UI, bind on the repo or playbook page, launch; no ticket, no per-cluster copying. A binding on a scope resolves on any cluster the scope's runs land on.
  • The hub is a custodian, not an end-to-end encryptor. It decrypts (reads Vault) to project. A hub compromise reaches every secret the hub's Vault policy can read; the policy is scoped to the crucible mount plus explicitly granted reference paths, Vault audits every read, and path owners can revoke a referenced path themselves.
  • Vault and the hub become dependencies of pod start for any run that uses a secret. Profiles that already ingest through the hub gain nothing new; profiles that relied only on the end-of-run session dump now have a hard dependency at start, and a pod that cannot redeem is parked by reconcile with the recorded error.
  • Scheduled launches under team secrets are only as fresh as the owner's groups; the schedule-row snapshot plus TTL is a new obligation with a staleness window equal to the TTL; ADR-0029 narrows it to the refresh-token lifetime.
  • Reference-mode secrets add an operator-level risk: a registrant who can read a path can expose it to a run. The verification read with the registrant's own Vault token, the per-bind re-read, and the refusal of in-mount paths limit that to paths the registrant and binder could already read; loss of the registrant's Vault access is not detected until the path owner revokes.
  • Grants bound to a recorded pod UID and a no-RBAC turn service account close the same-name pod substitution that the name-only bound-pod claim allowed; the ingest drop-box gains the same UID check.
  • The engine gains a redemption step in the loop image, a declared-secrets block in the manifest, and a redactor; the preview gate gains a secrets section; the profile [[secret_env]] path stays for undeclared packs.
  • Redemption adds a fourth externally exposed path with its own Route and rate limit.
  • Two Vault-side asks precede implementation: a crucible KV mount with an AppRole bound to the hub, and, when workload identity is available, a JWT auth role bound to the hub's service account subject and audience with pinned validation keys for the hub's issuer. Amended 2026-08-29: the dedicated mount was never obtainable, and the policies as granted are inverted against this decision: the human OIDC identity (app-crud-crucible) holds full CRUD on apps/data/crucible/*, while the crucible AppRole the hub logs in with holds only list, read. "The hub is the writer" cannot hold while the writer is the human, which is why the config data was seeded by hand and why the first registration failed. The standing ask is to give the AppRole the registry subtree only, keeping crucible/deploy/* read-only for it, and noting that KV v2 splits the verbs across three API paths so a grant on data/ alone leaves delete broken:
path "apps/data/crucible/registry/*"     { capabilities = ["create", "read", "update"] }
path "apps/delete/crucible/registry/*"   { capabilities = ["update"] }
path "apps/metadata/crucible/registry/*" { capabilities = ["delete", "list", "read"] }

Neither identity can grant this: sys/policies/acl/* and auth/approle/role/crucible are both denied, so it is an IT request.

  • Deployment secrets gain a rotation path (edit in Vault, next sync applies) and a new failure mode: a sync that cannot reach Vault leaves the last materialized Secret in place, so boot never depends on Vault; only registry reads and redemption do.
  • ADR-0030 builds on the kubeconfig kind, the hub consumer class, and the grant path for user-registered cluster targets.

Alternatives Considered

Ownership plus bindings in Postgres, bytes in Vault with the hub as the only Vault client, per-dispatch grants bound to the pod UID and redeemed by the pod against the hub (accepted)

Envelope-encrypted values in Postgres with a chart-mounted KEK, the controller as sole custodian (rejected)

  • Rejected because: Vault is reachable from the hub, so this trades native audit, versioning, and user-side revocation for a KEK whose loss kills every registered secret and a blast radius of KEK plus database equals everything. Reconsider only for a deployment without Vault.

Keep pre-created Kubernetes Secrets per cluster, add ownership metadata on top (rejected)

  • Rejected because: Still copied per cluster and minted by admins; the metadata would describe secrets the controller cannot create, rotate, or place, so registration is not self-service.

Controller writes a per-run Kubernetes Secret onto the target cluster at dispatch (rejected)

  • Rejected because: Needs secrets create/delete for the federated identity on every spoke, widening the read-mostly grant of ADR-0048, and puts plaintext in spoke etcd; the pull model only needs the pod to reach the hub.

Vault Agent injector or CSI driver on each spoke (rejected)

  • Rejected because: The GPU spokes cannot reach Vault, so this would split projection behavior by cluster and break the property that a loop is dispatchable anywhere the hub can dispatch.

End-to-end encryption to the target cluster's key at registration (sealed-secrets style) (rejected)

  • Rejected because: Pins a secret to one cluster at registration; a loop that may land on several clusters would need the value re-registered per target, which is the per-cluster copying this decision removes.

A general per-resource RBAC engine over repos, playbooks, and loops (rejected)

  • Rejected because: A secret has one owner and a set of bindings; those two relations answer every rule in section 2. A generic ACL table would exist for rules nobody has.

A Keycloak service account with group-query rights for offline membership checks (rejected)

  • Rejected because: Groups are already a session claim, and native OIDC (ADR-0029) gives the hub a refresh token to re-fetch them for scheduled launches; a second credential and a directory integration would duplicate that.

Grants bound to the pod name, as the ingest drop-box binds today (rejected)

  • Rejected because: Pod names are deterministic per run and the loop service account can create pods in the loop namespace, so a same-named pod created before or after the hub's pod could redeem another run's grant. The UID from the create response is the only identity the hub itself observed.