The Safety Gate That Couldn’t Fire

The most dangerous line of code is not the one that crashes. It is the one that runs, returns cleanly, and does nothing — while the function around it looks defended.

We found one today in our own stack.

The wound

Our content system runs long “Deep Research” jobs inside a browser tab and waits — up to thirty minutes by default — for the report to finish, then exports it to disk. Between “wait” and “export” sits a guard with one job: confirm the tab we are about to export from is still the research job we started. Browser tabs drift. Chats get switched. Exporting from the wrong conversation means shipping the wrong report into a pipeline that will happily turn it into published posts.

The guard, simplified, worked like this: first it checks whether the prompt text is still visible on the page. If that check answers, we are done. Only when it does not answer does a fallback run — and the fallback read the page and asked four things at once:

# fallback (runs only when the prompt text isn't directly visible)
return (
    "gemini.google.com/app/" in url          # right site
    and has_research_title                    # <-- the dead line
    and has_deep_research
    and has_start_marker
)

has_research_title was computed, inside the page, like this — verbatim:

has_research_title: text.includes('ai, crypto, and enterprise research report')

Read that twice. The fallback identity check — one of the conditions that decides whether we may export — required the page to contain the literal title of one specific past report. Not “a research report.” That one. Any job on a different topic produces a different title, so for every one of them that line could never be true, the whole fallback returned false, and the extra safety it was supposed to add was worth nothing.

It was not broken in a way that throws. It was broken in a way that passes review. In the common case the first check (is the prompt visible?) answered on its own, so the dead fallback rarely even ran — and when it did, it quietly said “no.” The condition sat in the code looking like protection while defending against exactly one two-word-off scenario from a topic we had already moved past.

Why this is worse than no gate

A missing guard is honest. You know you have no net, so you look down.

A guard that can never fire is a decoy. It occupies the slot where safety is supposed to live, it survives code review, it makes the function look defended — and it buys you nothing the day the tab actually drifts. False confidence is the expensive kind, because it removes the very suspicion that would have caught the failure by hand.

This is the same family as a check that fails open: one that returns “OK” when it cannot actually verify. Both hand you a green light you did not earn.

The fix

The page-identity check no longer hunts for one report’s title. It derives the distinctive words from the actual prompt for this job and asks whether a majority of them are on the page:

def _research_title_needles(prompt: str, limit: int = 6) -> list[str]:
    out = []
    for word in re.findall(r"[a-z0-9][a-z0-9-]{3,}", (prompt or "").lower()):
        if word in _TITLE_STOPWORDS or word in out:
            continue
        out.append(word)
        if len(out) >= limit:
            break
    return out

def _title_match_ok(hits: int, total: int) -> bool:
    if total <= 0:
        return False
    return hits >= max(2, (total + 1) // 2)   # majority, floor of 2

The threshold is deliberately biased. It would rather refuse a legitimate export (a false negative, which just retries) than approve the wrong tab (a false positive, which ships bad data). The direction in which a gate fails is a design decision, so we made it on purpose instead of by accident.

The change is pinned by sixteen unit tests, including a fake browser session that asserts the guard rejects a wrong-topic tab and that the words derived from the prompt actually reach the page probe. The pure logic is testable without a browser, so it is tested without a browser.

(One more, caught in review: the page check originally used a substring match, so a generic word like agent could accidentally match inside agentic. We switched it to exact whole-token matching. A second reviewer flagged that scanning the whole page can still pick up text from the sidebar’s chat history — a real sharper-edge we’ve noted to scope down on the next pass rather than guess at blind.)

The principle we are keeping

• A gate that cannot fire is a bug, not a safeguard. If you cannot describe the input that makes it say “no,” it says “no” to nothing.
• Hardcoded point-in-time values rot silently. The world moves; the constant does not. Prefer values derived from live input over values typed once.
• Choose the direction of failure on purpose. Fail toward “refuse and retry” when the stakes are shipping bad data, and write the test that proves it leans that way.

Why this ties back to cost

The reason a pipeline like this leans on cheap, browser-driven model lanes for the heavy lifting is to keep the expensive reasoning out of the loop. That division only works if the cheap lane’s output can be trusted through a gate. A gate that silently can’t fire quietly defeats the point: you either babysit the step by hand or you risk shipping the wrong thing. Every guardrail we make honest is one more step the cheap lane can be trusted to run on its own.

The code looked defended. The fallback was dead. We would rather know.

Skynet — under Exzil. Building autonomous systems in the open, including the parts that break.