ADR-0055: Webhook triggers: verified deliveries land in an inbox and CEL maps them to a launch

Status: accepted | Date: 2026-09-29

References: ADR-0033, ADR-0037, ADR-0043, ADR-0049, ADR-0054, RFC-0002:C-PLAYBOOK-PARAMS, RFC-0003:C-DECISION, RFC-0003:C-ENFORCEMENT, RFC-0003:C-WEBHOOK-DELIVERY, RFC-0003:C-WEBHOOK-SETTLEMENT, RFC-0003:C-WEBHOOK-LIFECYCLE

Context

Standing launches fire on a time (ADR-0033) or a tracker query hit (ADR-0037). Neither covers an event another system pushes: a new image lands in a quay.io repository, a GitHub release is published, a GitLab pipeline finishes. Polling each source is possible for some (the image catalog already reads OCI registries) but it is per-source code, adds latency, and does not exist for most senders. Every one of these senders can POST JSON to a URL.

ADR-0043 assumed no public ingress. That no longer holds: the prod namespace is an ext namespace and can publish a Route on the public router shard. This ADR does not amend ADR-0043; its pollers stay as they are, and the one public route here is the delivery route.

Senders disagree on authentication. GitHub signs the body with HMAC-SHA256 in X-Hub-Signature-256; the Standard Webhooks spec signs id.timestamp.body in webhook-signature with a timestamp to bound replay; GitLab sends a shared token in X-Gitlab-Token; quay.io signs nothing, and receivers such as Kargo and Flux authenticate it by an unguessable token in the URL path.

Senders also disagree on payload shape, and a playbook's params are typed and validated against its schema. Something owned by the trigger has to decide whether a delivery is relevant and turn it into params, and it runs inside the controller on input from the public internet.

Constraints carried over from ADR-0037: one fire path, the trigger never touches the authorization columns, and immunity to the self-retrigger loop (a playbook that pushes the image it listens to).

Decision

Add a webhook trigger as a fourth ADR-0037 sidecar, one per standing launch. playbook_webhooks holds the verifier kind, the secret, the transform, the launch rate ceiling, and the delivery retention. The contract it implements is RFC-0003:C-WEBHOOK-DELIVERY, RFC-0003:C-WEBHOOK-SETTLEMENT, and RFC-0003:C-WEBHOOK-LIFECYCLE; this records how. Launches record origin webhook; the consumed-key row pairs each launch with the delivery that produced it and outlives delivery retention.

Receiving. The controller serves POST /hooks/{id} (and /hooks/{id}/{token} for the path-token verifier) on a second listener with a concurrency limit. Only that listener is published on the ext Route; the API listener stays behind the auth proxy. The handler applies the body cap, looks the webhook up, and verifies. Every comparison is constant-time and every unrecorded delivery gets the same 404. Response timing is not equalized: an id carries 74 random bits, so an existence probe has nothing to aim at. The concurrency limit sheds load before the lookup, the same way for every id. A verified delivery is written to a webhook_deliveries row with its headers, minus the one that carries the credential, and the handler returns 202. It never launches. The path token is redacted from controller logs and traces.

Verifiers. hmac_sha256 (GitHub's X-Hub-Signature-256) and path_token (quay.io), the two with a sender to serve. A verifier and its header are fixed at creation; changing either is a delete and a new webhook, so no header ever carried a secret the current verifier does not know about. The controller generates every secret, so a sender is configured by pasting what the create response returns once. HMAC secrets are sealed with AES-256-GCM under the controller's credential key and bound to the webhook id, because verifying needs the plaintext; tokens are stored as SHA-256 digests, like API keys.

Firing. The webhook's LaunchTrigger reads unsettled delivery rows as its due set, oldest first. The transform runs at claim time, not in the handler: a transform edited after a delivery arrived applies to it, and the sender never sees a transform result. The claim locks the webhook's sidecar and core rows and evaluates under that lock, so a concurrent disable waits for it and a delivery that settled while the launch was being minted rolls the launch back; the save-time rules keep the hold short. Two changes to the ADR-0037 core follow, and schedules and watches move onto both:

The per-firing overlay becomes a map of param name to JSON value, converted to the stored param form before standing::fire validates it. Today it is one (param, string) pair.
LaunchTrigger::claim gains a third result, settled without launching. It commits the sidecar's settlement and skips fire. Today a false claim rolls back and an Err aborts the sweep, so a delivery could neither settle filtered nor survive a transform error.

A transform error, an underivable dedupe key, an unparseable body, or a schema refusal of a derived value settles failed and does not count toward auto-disable, because a sender chooses all of them. Only failures of the firing itself count: the pack is gone, the stored params drifted from the schema, the decision denied, or an insert failed. Such a firing rolls back its claim, and the webhook's fail settles the delivery failed with the firing's reason in its own transaction, so a failed delivery never blocks the ones behind it. Auto-disable disables the webhook with its standing launch and is audited with the controller as actor; a delivery to a webhook whose standing launch is disabled is treated as a delivery to a disabled webhook. Disabling or deleting a webhook, by a caller or by auto-disable, settles its pending deliveries failed.

Transform, in CEL. A webhook stores a filter expression (bool), a dedupe expression (string), and a derive map of param name to expression. There is no title expression: the launch title is the playbook's, so nothing the sender wrote reaches a run except as a param. The environment is body (the parsed JSON), headers (names lowercased), delivery (the delivery id, unique per delivery), and received_at. Expressions compile at save time with a parser recursion limit, and the save is refused if any does not compile, reads a variable other than the four, calls a function the interpreter does not define (the parser accepts any name, so join() would otherwise fail on every delivery), or derive names a param the target does not declare. The interpreter has no evaluation budget, so the save also walks the AST and refuses an expression that is over a length cap, nests comprehensions more than two deep, or has a comprehension whose loop condition or step references anything but its own iteration and accumulator variables. That last rule is what bounds cost: without it body.l.map(x, body.l) is quadratic in the body and took 113 s on an 11 KB body under 0.14.5. It also refuses the legitimate items.exists(i, i.tags.exists(t, t == i.name)), which is the price. The nesting cap stops [a, a].exists(b, [b, b].exists(...)) from doubling at every level.

body is built by our own JSON conversion, not the crate's serde path, which maps a non-negative integer to uint so that body.count + 1 fails as uint + int. The conversion maps an integral number that fits to int and anything else to double. The ANTLR parser recurses once per bracket level before its own depth limit applies, and a debug build overflowed a 2 MB stack at 20 nested parentheses; a stack overflow aborts the process and no catch_unwind catches it. So a save refuses bracket nesting past 32 before parsing, and parsing and evaluation run on a scoped thread with a 16 MB stack, where a panic is a join error. A parse error is reported as its first diagnostic only, since the crate repeats the source once per recovery attempt. A save may carry a sample payload; the endpoint returns the filter result, the overlay, and the dedupe key without storing anything. A check endpoint answers every refusal a save would, with the parser's line and column when it gave one, so an editor marks expressions as they are typed.

Dedupe. (webhook, dedupe key) is a seen-set written in the launch's transaction, the guard playbook_watch_hits already uses. Only a launch consumes a key, and the seen-set outlives delivery retention. A GitHub push keys on the pushed commit and a release on its id. quay.io's repo_push payload names the pushed tags and no digest, so its preset keys on delivery and every push launches; for quay the loop guard against a playbook that pushes the image it listens to is the rate bound, until the controller resolves the pushed digest from the registry itself. Sender retries settle duplicate. Neither verifier bounds replay by time, so replay protection is exactly the dedupe key.

Rate. A webhook carries a maximum number of launches per hour, set by the owner and capped by a platform maximum. The count is taken under the standing launch's row lock so two replicas cannot both admit the last slot. A delivery past it settles throttled and is never retried.

Presets (quay.io repository push, GitHub release, GitHub push) prefill the verifier, filter, dedupe key, and params map in the form. They are data, not code paths.

Consequences

One public route family on its own listener. Nothing else in the API is reachable from the internet, and a webhook firing is decided as its standing launch's firing under RFC-0003:C-DECISION, with the owner's memberships resolved at firing time.
The delivery route sits beside the ingest drop-box outside RFC-0003:C-ENFORCEMENT; webhook management and delivery reads stay bound routes.
The ADR-0037 claim contract grows a third result and the overlay becomes a map. That touches schedules and watches; their existing tests are the regression net.
Every verified delivery has a recorded outcome, which answers "why didn't it fire" from the SPA instead of from logs. A sender never sees a filter or schema failure; the delivery log is the only place it shows.
A holder of a leaked secret can launch the playbook as its owner, with any param values the filter admits, up to the ceiling, until the secret is rotated. The param schema is the only bound on those values, so a pack meant for webhooks should constrain its params tightly.
A throttled delivery is lost. A burst, or a holder of a leaked token, can exhaust the ceiling and block legitimate deliveries until the token is rotated. Re-running a lost event is an ordinary launch from the form.
A token holder can also claim a predictable dedupe key (a release id, a commit that is already public) before the real event, by getting a launch through the filter first. HMAC senders are safe from this because the key comes from a signed body; a path-token sender is not.
Rotating a secret is an update. This design has no overlap window: deliveries signed with the old secret 404 until the sender is updated.
The path token appears in router access logs, which the controller does not control.
CEL is a new dependency: the cel crate from cel-rust (MIT; antlr4rust BSD-3-Clause, everything else MIT or Apache-2.0). Release 0.14.5 builds map and filter results by copying the accumulator every step, so a harmless map over a 50k-element list took 34 s. Upstream fixed this in #341 after that release; on master the same call takes 6 ms. Until a release carries it, the controller pins cel-rust by git rev under deny.toml's allow-git, as it already does for tower-sessions-stores. Master also changes Context to two lifetimes.
Evaluation is synchronous on the sweep and cannot be preempted by a timeout; the bound is the save-time rules plus the body cap. The sweep serves every standing launch, so a slow expression would delay schedules and watches too, not just its own webhook.
With those rules, parsing the quay filter takes about 13 µs and evaluating it about 15 µs. Nothing in a panic probe set (overflow, division by zero, bad regex, bad timestamp, out-of-range index, deep nesting) panicked; the one reachable panic! in the interpreter is on an unspecified AST node, which the save-time walk refuses.
One URL per webhook. Three playbooks on one quay repository means three notifications on the quay side. Shared receivers with fan-out are a follow-up that would add a subscribe action under RFC-0003.
The image catalog's registry poll stays the no-ingress path for images. It could become a standing trigger later; it is not part of this decision.

Alternatives Considered

Webhook sidecar on the standing-launch core; the handler verifies and stores to an inbox, the sweep transforms with CEL and fires (accepted)

Pros: Reuses the one fire path, the firing decision, and the HA claim., A delivery survives a controller restart between receipt and launch., Every delivery has a recorded outcome.
Cons: A new public listener and Route., A new dependency., Changes the ADR-0037 claim contract and overlay shape., A delivery launches once the leader hears of it over Postgres LISTEN; one missed during a reconnect waits for the next discovery tick.

Launch synchronously in the request handler (rejected)

Pros: Immediate launch., No inbox table.
Cons: A second launch-minting path beside the sweep's., A sender that times out and retries races the first request's launch., A burst of deliveries is a burst of launches before any ceiling is read.
Rejected because: Duplicates the fire path ADR-0037 made single, and gives sender retries a race against dedupe.

A webhook gateway in front (Argo Events, Tekton Triggers, Knative Eventing) that calls the launch API (rejected)

Pros: Mature sender adapters and filters.
Cons: Needs an operator and CRDs in a namespace-scoped deployment., The gateway would hold an API credential that launches as one principal, so per-trigger ownership is lost., Trigger config lives in Kubernetes objects instead of owned controller resources.
Rejected because: Launches would carry the gateway's identity rather than the trigger owner's, and ADR-0043 rules out running as an operator (no CRDs, no reconciling Kubernetes objects it did not create).

Poll the source instead (a registry watch trigger for images) (rejected)

Pros: No public ingress., The image catalog already reads OCI registries.
Cons: Per-source code for every sender., Latency of the poll interval., Most senders have nothing to poll.
Rejected because: Ingress is available, and polling does not generalize past images. It remains the path for a deployment without ingress.

Transform as JSON Pointer extraction with equality filters and string templates (rejected)

Pros: No new dependency., Trivial to evaluate safely.
Cons: Cannot express membership or list logic, such as a tag being among the pushed tags., Every missing case becomes a new filter operator.
Rejected because: The quay case already needs "latest" in body.updated_tags, and the operator set would grow into an ad hoc language.

Transform in Starlark, the language packs are written in (rejected)

Pros: Already a dependency., Familiar to pack authors.
Cons: Owner-authored general-purpose code running in the controller process on public input., Needs step and memory limits the controller does not have today.
Rejected because: More power than a filter and a field map need, and the wrong trust boundary for code evaluated against internet input.

Transform in jq (rejected)

Pros: Widely known for JSON reshaping.
Cons: Expressions yield streams, so a filter or a single param value needs conventions on top., Untyped: a bool filter and a string key are both just output.
Rejected because: CEL gives typed single-value expressions and is what Tekton Triggers and Kubernetes admission use for this job.

Pros: One sender-side notification can drive several playbooks.
Cons: A receiver becomes a root resource of its own, and subscribing across owners needs a new action and policy.
Rejected because: Deferred rather than refused. Senders already allow several notifications per repository, and 1:1 fits the ADR-0037 sidecar model without new authorization.

Keyboard shortcuts

crucible