Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-0036: Deliver run secrets as owner-referenced Secrets on trusted spokes

Status: accepted | Date: 2026-08-29

References: ADR-0028, ADR-0048, ADR-0030

Context

ADR-0028 settled how a run's secrets are owned, stored, and authorized, and chose how they reach the pod: the controller stamps a grant id into the pod spec, and the pod trades that id plus its bound ServiceAccount token for the values over HTTP. The registry half of that decision is built and in production. The delivery half is not: it needs an endpoint on its own Route, a redeem-by deadline, a failure cap that deletes a pod after repeated bad redemptions, a TokenReview per redemption against the pod's home cluster, an in-pod client with capped backoff, and a per-cluster redemption URL that nothing currently supplies to a pod.

That machinery re-implements what a kubelet already does with a mounted Secret: project values into a container, hold them at owner-only modes, keep them off the workspace volume, and update the mount when the object changes. The pull model was chosen to keep values out of spoke etcd and to avoid widening the hub's credential on spoke clusters.

Two facts have since changed the comparison. The operator has declared the spokes trusted, which is what the at-rest argument turned on. And ADR-0048's recorded spoke grant is pods get/list/watch/create/delete, pods/log get, configmaps create/delete namespaced, and tokenreviews create — already write-capable on a namespaced companion object the controller writes per run, so adding secrets create/delete is symmetric rather than a widening. The privilege is close to notional in any case: anything that can create a pod in a namespace can mount that namespace's Secrets.

Ownership is the remaining question the pull model answered for free, since a grant dies with its deadline. Kubernetes answers it directly with owner references, an idiom this codebase already uses to garbage-collect a turn's published objects with the pod that produced them.

Decision

The controller writes a per-run Kubernetes Secret into the target namespace at dispatch and owner-references it to the pod, and the pod mounts it. This supersedes the delivery half of ADR-0028 and nothing else in it: ownership and bindings stay in Postgres, the bytes stay in Vault, the hub stays the only Vault client, and a launch whose scope cannot bind a declared name is still refused before a pod exists.

Delivery won on the same ground the pull model lost. A trusted spoke makes the at-rest objection a preference rather than a threat, and ADR-0048's grant is already write-capable on the same namespace, so the push costs a symmetric verb rather than a new class of authority. What remains is that the pull model reimplements kubelet: the projection, the file modes, the lifetime, and the update-on-change all exist in Kubernetes, and rebuilding them bought an at-rest property the deployment no longer needs.

The decision is scoped to trusted targets. A bring-your-own cluster under ADR-0030 is not one, and placing a value at rest in a cluster the operator does not administer is a different decision than this one.

Consequences

A run's secret set becomes a mount rather than a protocol. The redemption endpoint and its Route, the redeem-by deadline, the failure cap that deletes a pod after repeated bad redemptions, the TokenReview per redemption, the in-pod client, and the per-cluster redemption URL all stop being needed, and a run no longer races its own credential at start.

The value is at rest in the spoke namespace for the pod's lifetime, readable by that cluster's administrators and by anything holding secret read there. Owner references bound the lifetime, not the exposure, and this is the cost the trust declaration buys.

Reads leave no trail. The registry still audits register, rotate, bind, unbind, and delete, so who authorized a value's use is recorded; that a kubelet mounted it is not. An investigation can establish which runs were entitled to a secret, not which ones read it.

Rotation changes hands. A redemption read the current KV version at every pod start; a mount holds what the controller wrote. Propagating a rotation to a running pod now depends on the controller rewriting the Secret, and a pod that starts before that write holds the previous value.

The hub needs secrets create and delete in each target's loop namespace, which is a change to the RBAC that ADR-0048 records and the chart renders.

The decision holds only while targets are trusted. A bring-your-own cluster reopens it, and the pull model this rejects remains the answer for that case if it arrives.

Alternatives Considered

Controller writes a per-run Kubernetes Secret into the target namespace at dispatch, owner-referenced to the pod (accepted)

  • Pros: Kubelet does the projection: owner-only modes, tmpfs, mount updates on change, no in-pod code, Owner references tie the value's lifetime to the pod, so it is collected with the turn, The Secret exists before the pod, removing the bind-then-redeem race the pull model needs backoff for
  • Cons: The value sits in the spoke namespace at rest for the pod's lifetime, A kubelet mount leaves no per-read audit record

Per-dispatch grant redeemed by the pod against the hub over HTTP (the ADR-0028 delivery half) (rejected)

  • Pros: No value at rest on the spoke: it lives in Vault and reaches a tmpfs that dies with the process, Every redemption is an audited event bound to one pod UID, and reads the current KV version at pod start
  • Cons: Rebuilds kubelet projection in application code: a Route, a WAF rule, a deadline, a failure cap, a TokenReview per redemption, and a retrying in-pod client, The grant is unredeemable until the create response binds the pod UID, so a run's start races its own credential
  • Rejected because: Its two advantages are bought at the cost of reimplementing secret projection: the audit trail records reads nobody has asked to see, and keeping the value out of etcd was the point only while the spokes were untrusted. The operator has declared them trusted, which removes the reason the extra machinery existed.

Keep the grant redemption path and give it a per-cluster redemption URL (rejected)

  • Pros: Smallest change from what is already merged: the registry, grants, and the redemption endpoint all ship today
  • Cons: Keeps every cost of the pull model and adds per-cluster URL configuration that must be right on each target before a run can hold a secret
  • Rejected because: Completing a mechanism is not a reason to keep it. The missing URL is the cheapest of its costs, and paying it would settle the design by sunk effort.