ADR-0031: Deployment secret projection by the Vault Secrets Operator

Status: accepted | Date: 2026-08-25

References: ADR-0028

Context

ADR-0028 put the controller's own identity secrets (database URL, GitHub App key, push tokens, Jira and Datadog tokens, spoke kubeconfigs, registry pull secrets) in the same Vault mount as the registry under a deploy/<release>/ prefix, and projected them into the chart's existingSecret objects with a crucible-controller vault-sync subcommand run as a pre-install hook Job and an hourly CronJob. That projection was chosen on the stated premise that the hub cluster runs no secrets operator.

The premise was wrong. the hub has the Vault Secrets Operator installed: secrets.hashicorp.com/v1beta1 is registered with VaultConnection, VaultAuth, VaultStaticSecret, VaultDynamicSecret, VaultPKISecret, and SecretTransformation, and crucible--runtime-ext may create all three of the kinds this needs. Nothing checked for it, so the chart carried a sync of its own: a mapping ConfigMap, a hook Job, a CronJob, a ServiceAccount with namespace Secret RBAC, a subcommand, and roughly 1300 lines of module behind it.

Two costs of the hand-rolled path had already surfaced. The AppRole secret id has to be mounted into every pod that syncs, so the credential spreads with the schedule. And ADR-0028 left rolling the Deployment after a rotated env secret out of scope, so a rotated value sat in a Secret that the running controller never re-read; the hourly tick made the secret current and the process stale. Separately, every CronJob run since the chart landed had failed before reading a path, which is what prompted looking at the mechanism at all.

The projection is the only part of ADR-0028 at issue. Vault stays the custodian, the mount and the deploy/<release>/ prefix are unchanged, consumers keep reading Kubernetes Secrets, and the registry, the grant path, and the hub-as-only-Vault-client property are untouched.

Decision

Deployment secrets are projected by the Vault Secrets Operator. The chart renders one VaultStaticSecret per mapping entry over a shared VaultConnection and VaultAuth, and the mapping ConfigMap, hook Job, CronJob, ServiceAccount, RBAC, and the vault-sync subcommand are deleted.

It wins because the projection is a solved problem the cluster already solves: the operator owns the login, so the AppRole secret id stops being mounted into sync pods; it reconciles continuously rather than on a schedule; and rolloutRestartTargets restarts the controller Deployment when a synced Secret changes, which closes the rotation gap ADR-0028 deferred rather than adding another mechanism for it. It also removes a component from a chart whose failure mode was invisible until a run was inspected by hand.

The mapping's shape survives the move. destination.transformation.templates rewrites KV fields into the Secret keys the chart's existingSecret references expect, and the transformation excludes every source field by name so that a field nobody mapped stays out of the projected Secret, which is what the subcommand's explicit key mapping did. excludeRaw is set beside it but covers only the whole-payload blob.

The Vault login Secret and the registry pull secrets stay hand-created for their existing reasons: the login is what the operator authenticates with, and the kubelet pulls the controller image before any reconcile has run.

Consequences

The chart loses a component it maintained and a credential mount: roughly 2300 lines deleted against 240 added, and no pod of ours holds the AppRole secret id to sync with. A rotated deployment secret now reaches the running controller without a manual roll.

The hub cluster gains a hard dependency on the Vault Secrets Operator being installed. A cluster without it cannot deploy this chart with controller.vault.enabled, where before the chart carried its own sync; that trade is deliberate for a hub, and is the reason the decision is scoped to deployment secrets rather than to registry projection, which still reaches spokes the operator does not run on.

The operator reads fixed key names, the CA bundle under ca.crt and the AppRole secret id under id, while the controller's own Vault client reads whatever caKey and secretIdKey name. Both Secrets carry both spellings, which is a duplicated value in two keys and a step in the onboarding script that will be wrong if only one is rotated.

Sync failures move from a CronJob exit code and pod logs to the operator's own status on each VaultStaticSecret, so the runbook for "why has this path not landed" changes; and reconcile behavior, including what happens when Vault is unreachable, becomes the operator's to define rather than ours to guarantee.

Alternatives Considered

One VaultStaticSecret per mapping entry, projected by the cluster's Vault Secrets Operator (accepted)

Keep the chart's own vault-sync: fix the startup panic and add the pod-template hash annotation the rotation gap needs (rejected)

• Rejected because: Rebuilds, on our maintenance budget, what the cluster already runs, and keeps the AppRole secret id mounted into every sync pod. The rotation follow-up it still owes is a feature the operator ships.

Vault Agent injector sidecar or the Secrets Store CSI driver on the controller pod (rejected)

• Rejected because: Projects into the pod's filesystem, not into Kubernetes Secrets, so the kubelet still has no pull secret and the spoke kubeconfig mounts and env references would all have to change. ADR-0028 already rejected per-spoke injectors for the registry.

External Secrets Operator with a Vault SecretStore (rejected)

• Rejected because: Equivalent in shape, but it is not what this cluster runs; installing a second secrets operator to avoid using the installed one inverts the reason for the change.

Keep both paths, with the operator objects behind a chart flag (rejected)

• Rejected because: Two projections for one set of Secrets, each able to overwrite the other, and the hand-rolled one keeps its maintenance cost while being the path nobody exercises.