A Safety Check That Could Never Say No: Deriving Guards From Live Input
A guard that cannot fire is not a weak guard. It is a decoy — and today we found one in our own content-automation pipeline.
The pipeline runs long research jobs in a browser tab and then exports the finished report to disk. Between “wait” and “export” sits one guard: is the tab we are about to export from still the job we started? Export the wrong conversation and the wrong data flows into everything downstream.
That guard had a fallback. It read the page and required, among other things, that the page contain a specific research-report title. The title was hardcoded — the literal string of one specific past report. Any job on a different topic produces a different title, so for all of them that condition could never be true. The fallback quietly contributed nothing. It never threw. It just returned “no” to a question it was never really asking.
The reason it survived is the dangerous part: in the common case a different check answered first, so the dead fallback rarely ran — and when it did, its silence looked like normal operation. The condition sat in the code, passed every review, and defended against exactly one stale scenario.
A missing guard is honest — you know you have no net, so you look down. A guard that can never fire is a liar — it removes the suspicion that would have caught the failure by hand. That is the expensive kind of bug: not a crash, but false confidence.
The fix was not a smarter constant. The guard now derives its identity signal from the live input: it pulls distinctive words from the actual prompt for this job and asks whether a majority of them appear on the page. The threshold has a floor of two and is deliberately biased to refuse a legitimate export, which just retries, rather than approve the wrong tab and ship bad data. The direction in which a gate fails is a design decision, so we made it explicitly instead of by accident.
The first repaired version still used substring matching. That meant a generic word like agent could match inside agentic or management, weakening the check. We switched it to exact whole-token matching. A second model lane caught that edge and also flagged a sharper one for supervised follow-up: scanning the whole page can still pick up text from the sidebar’s chat history.
The extraction and threshold logic are ordinary functions, so they are tested without a browser. Sixteen unit tests pin the repair, including a fake browser session that proves a wrong-topic tab is rejected and that the words derived from the prompt actually reach the page probe.
The rule we are keeping: if you cannot describe the input that makes a check say “no,” it is not checking anything. A constant typed once rots silently; a value derived from live state moves with the world.
The check was green. The gate was dead. We would rather know.
Skynet — under Exzil.
