ADR-0007: Isolation pre-flight — verify the gate isolates the target before spending a loop

Status: accepted | Date: 2026-06-26

Context

#1109 (long-context requests see >1s TTFT overhead in the EPP request path) ran as a loop and produced two valid-but-flat candidates. ADR-0006's profiler showed the EPP is ~24% of one core busy with GC incidental. The EPP's actual work is ~20-30 microseconds per request; the 2.26s TTFT is the ext_proc handler sitting blocked waiting on Envoy's body buffering, data-plane transfer, mock backend, and queueing — none of which the EPP controls. The request_duration metric (~2.26s) measures handler attachment wall-clock time, not compute. The gate is end-to-end and cannot attribute a result to the EPP, so every EPP-side candidate ties in the noise by construction. Root cause is world-scoping (the 220KB-body mock drowns the EPP in Envoy transfer cost), not the EPP.

Decision

Onboarding must verify the gate actually isolates the target and that the target's contribution is material before spending a loop. Three parts: (1) Attribute the gate — before the first scored iteration, measure the target's own contribution to the gated metric using the target's instrumentation; if the target is a sub-percent sliver of the gated number, the loop cannot win, escalate as mis-scoped. (2) Distinguish work from wait — prefer a compute/decision metric or a profile for attribution; never infer "the component is slow" from a duration that includes downstream wait. (3) Re-scope or escalate — a real experiment needs a world where the target's contribution is material. #1109 is therefore escalated (no EPP-side win exists as gated).

Consequences

Positive: a ~10-minute instrumented pre-flight kills an unwinnable run before it spends a paid loop; the autoresearch stack surfaces a mis-scoped problem instead of grinding; the check is generic (any target with self-instrumentation) and slots into the judge-validation gate; it converts "the loop keeps tying in noise" from a mystery into a one-number verdict. Negative/cost: another gate in onboarding (the pipeline must learn to read a target's attribution metric); a per-target choice of "which metric attributes the gate" is not always obvious. Carries forward: empirical case for the world-scoping discipline — the judge can be perfectly frozen and still measure the wrong thing if the world doesn't isolate the target.

Alternatives Considered

Require an isolation pre-flight in onboarding that verifies the gate actually isolates the target and that the target's contribution is material before spending a loop (accepted)

Pros: A ~10-minute instrumented pre-flight kills an unwinnable run before it spends a paid loop, Generic (any target with self-instrumentation) and slots into the judge-validation gate, Converts 'the loop keeps tying in noise' from a mystery into a one-number verdict
Cons: Another gate in onboarding; the pipeline must learn to read a target's attribution metric, Per-target choice of 'which metric attributes the gate' is not always obvious

Skip the pre-flight and rely on the loop to discover isolation failures during iteration (rejected)

Rejected because: Wastes an entire paid loop discovering what a ~10-minute pre-flight can determine — the agent doom-loops against noise

Keyboard shortcuts