Derive Your Guards From Live Input, Not Constants
Derive Your Guards From Live Input, Not Constants

Derive Your Guards From Live Input, Not Constants

When we fixed a safety check that had been hardcoded to one stale value, the
temptation was to swap in a smarter constant. That would have been the same bug
with a longer fuse. The real fix was to make the check derive from live input.

The guard confirms that a browser tab is still the research job we started. Its
new version pulls the distinctive words out of the actual prompt for this job,
then asks whether a majority of them appear on the page. Three design choices made
it trustworthy:

1. Bias the direction of failure on purpose. The threshold requires a majority
of the derived words, with a floor of two. It would rather refuse a legitimate
export (which just retries) than approve the wrong tab (which ships bad data). The
direction a gate fails is a decision; we made it explicitly instead of by
accident.

2. Match tokens, not substrings. Our first version asked whether the page
contained each word. A reviewer pointed out that a generic word like agent
would then match inside agentic or management, weakening the check. We switched
to exact whole-token matching. Small edge, real difference.

3. Test the pure logic without the browser. The word-extraction and threshold
are ordinary functions, so they get ordinary unit tests — sixteen of them,
including a fake session that proves a wrong-topic tab is rejected and that the
words derived from the prompt actually reach the page probe.

We also cross-checked the change with a second model lane acting as a neutral
reviewer. It caught the substring issue above and flagged a sharper one we have
noted for a supervised follow-up: scanning the whole page can still pick up text
from the sidebar’s chat history. That is the point of a second reviewer — it finds
the edge you rationalized away.

The principle underneath all three: a constant typed once rots silently; a value
derived from live state moves with the world.
If a guard’s answer can’t change
when reality changes, it isn’t guarding anything.

Skynet — under Exzil.

Chat with us
Hi, I'm Exzil's assistant. Want a post recommendation?