I Tested Browser Agents Across 288 Conditions. Success Was Not Enough
I built RStackBench because a browser agent can look finished before the requested state is actually true. Across 288 controlled conditions, I measured the gap between visible success and persisted success.
Key Takeaways
- Visible success is not persisted success. The selector-only control showed apparent success on 63.9% of 288 runs but reached the requested persisted postcondition on only 18.4%, producing 131 false completions.
- Waiting and verifying solve different failures. State faults resolved passively from 0/24 at 50 ms to 24/24 by 750 ms, while Verification faults remained 0/24 through 1000 ms without another action.
- The 100% result is a harness check, not an agent score. The benchmark-aware positive control can inspect the known postcondition and perform bounded repair, so its 100% VTS validates recoverability under the benchmark contract rather than open-world agent reliability.
- The model pilot is statistically inconclusive. Observed VTS moved from 16/30 to 21/30 under reliability-oriented prompting, but exact McNemar p = 0.0625 does not cross the conventional 0.05 threshold.
- The scope is intentionally bounded. The study uses three synthetic localhost interaction archetypes, one Windows host, and one installed Chromium build; it is a diagnostic measurement method, not a production-Web leaderboard.
Why I ran this: I kept seeing the same uncomfortable pattern: the interface said the action succeeded, but the state I actually cared about was not always there when I checked. I wanted a benchmark that treated that difference as the main result, not an edge case.
“Saved” is one of the easiest words in a browser to trust. It is also exactly the kind of signal that can fool an evaluator.
I built RStackBench after running into a recurring gap between an interaction that looked finished and a postcondition that was actually true. A button can fire. A success banner can appear. The automation layer can return cleanly. None of those facts, by themselves, prove that the requested state persisted.
The benchmark is my attempt to make that gap measurable. It does not ask whether a browser controller looked competent. It asks a narrower question: when the task changes state, did the application end in the state that was requested?
The result I care about most is not the 100% row in the table below. It is the split between what the interface reports and what the application actually persisted – and the fact that passive waiting repairs one class of failure while another class requires new action.
The result I would not reduce to one score
RStackBench uses three synthetic localhost interface archetypes and five controlled execution layers: Access, Perception, Action, State, and Verification. I generated 288 deterministic task conditions and ran three hand-designed validation scripts over the same conditions, producing 864 Chromium traces.
| Validation script | Runs | Verified Task Success | Apparent Success | False Completions |
|---|---|---|---|---|
| Selector-only control | 288 | 18.4% | 63.9% | 131 |
| Semantic-recovery control | 288 | 66.7% | 100.0% | 96 |
| Benchmark-aware positive control | 288 | 100.0% | 100.0% | 0 |
The selector-only control is the clearest warning about trusting surface success: apparent success appeared on 63.9% of its runs, while only 18.4% reached the requested persisted postcondition. The semantic-recovery control fixed the Access, Perception, and Action layers in this controlled setup, but its short-boundary Verified Task Success remained 66.7% because State and Verification failures are not just targeting problems.
Those percentages are not estimates of how often commercial browser agents fail on the open Web. These are deterministic controls on synthetic applications. Their purpose is diagnostic: to show what an evaluator can miss when it gives credit to a visible success signal instead of the requested postcondition.
Waiting fixed State. It did nothing for Verification.
This is the distinction that changed how I read the benchmark.
A State fault in RStackBench is temporal settling: the interface can announce success before persistence finishes. A Verification fault is different: the success cue can appear while the requested postcondition still requires another action.
To separate those cases, I re-ran all 24 original State tasks and all 24 original Verification tasks at five passive scoring horizons, producing 240 additional Chromium traces.
| Passive scoring horizon | State VTS | Verification VTS |
|---|---|---|
| 50 ms | 0/24 (0.0%) | 0/24 (0.0%) |
| 150 ms | 7/24 (29.2%) | 0/24 (0.0%) |
| 350 ms | 15/24 (62.5%) | 0/24 (0.0%) |
| 750 ms | 24/24 (100.0%) | 0/24 (0.0%) |
| 1000 ms | 24/24 (100.0%) | 0/24 (0.0%) |
By 750 ms, every State condition had resolved without another user action. Verification remained 0/24 even at 1000 ms. That means a State miss at the short scoring boundary is a premature-completion event at a declared horizon, not proof of permanent backend failure. Verification is the stronger failure: time alone does not repair the requested outcome.
Waiting and verifying are therefore not interchangeable reliability techniques. One gives an asynchronous write time to settle. The other asks whether the goal is actually true and acts again when it is not.
The 100% row is a control, not a benchmark crown
The benchmark-aware positive control reaches 100% VTS because I deliberately gave it access to the benchmark’s postcondition signal and bounded repair logic. That is useful, but it is not a learned-agent result and it is not evidence that explicit verification makes open-world agents 100% reliable.
I use that row as a harness check. If a controller that can inspect the known postcondition and repair bounded mismatches still fails, the benchmark itself may be broken. If it succeeds, the result shows that the injected conditions are recoverable under the benchmark contract.
There is also a cost. Relative to semantic recovery, the positive control increased mean actions from 2.50 to 2.83 and mean host-side latency from 289 ms to 640 ms. Those latency values are implementation- and host-specific; I report them descriptively, not as universal performance numbers.
The model pilot did not clear the statistical bar
I also ran a small paired planning pilot on 30 tasks using GPT-5.6 Sol as identified by the locally authenticated Codex interface used during the study. Observed VTS was 16/30 (53.3%) under ordinary prompting and 21/30 (70.0%) under reliability-oriented prompting.
That difference is statistically inconclusive under the conventional 0.05 threshold: the exact two-sided McNemar test gives p = 0.0625.
I am keeping the pilot in the paper because the plans and executions are part of the reproducibility package, not because the result proves that one prompt reliably improves capability. The correct conclusion from this sample is narrower: an observed directional difference exists, but this experiment did not establish statistical significance.
Read the formal paper
I use this page as the interpretation layer: what I tested, what surprised me, and why the result matters. The PDF is the formal research record with the method, equations, tables, figures, threats to validity, reproducibility statement, disclosures, and complete references.
Independent research preprint – August 2026
The manuscript follows IEEE-style scholarly conventions, but it has not been accepted, published, certified, or peer reviewed by IEEE or any other publisher. A future journal submission, if pursued, would be a separate process.
Open or download the PDF preprint | Download the editable Word source
If inline PDF viewing is unavailable in your browser, use the direct PDF link above. I keep the narrative here and the formal research record in the PDF.
Research files and integrity
The paper is published with its editable source and reproducibility package. These hashes bind the public page to the reviewed artifacts.
- PDF preprint – SHA-256
05D8479969353146E8EF612A79E6BC741D34D9B9627154E07627EC4DACD62E22 - Editable Word source – SHA-256
0EA9B402170F68D83FAC5757D780AD502237534040B83CCBCD55BB9F14057FBE - Reproducibility package – SHA-256
29BF647B2FCFA7D7A4C266FFBC12F4417139AAD086312FDEAB82DC00E282DC43
What this study does not claim
The controlled evidence is deliberately narrower than the title of the broader browser-agent problem.
- It does not prove that explicit postcondition verification makes general autonomous browser agents reliable in open-world environments.
- It does not provide a leaderboard or provider ranking for commercial models or browser agents.
- It does not establish population-level failure rates for the Web.
- It does not evaluate mobile browsing, native desktop agents, live anti-bot systems, authentication challenges, or nondeterministic third-party services.
- It spans three synthetic localhost interaction archetypes, one Windows host, and one installed Chromium build.
I would rather narrow the claim than let a clean percentage imply more than the experiment measured.
Reproduce the evidence, not my prose
The reproducibility package contains the deterministic task generators, Chromium benchmark runners, serialized task definitions, all 864 cross-template validation-script traces, the 240-trace scoring-horizon dataset, the exploratory paired planning records and executions, aggregate summaries, figures, reanalysis scripts, and SHA-256 manifests.
The important part is that the headline rates can be regenerated from raw stored traces instead of trusted because they appear in a manuscript. Failed harness-development attempts are kept separate from evaluated outcomes so discarded runs cannot silently become part of the reported dataset.
AI-use and authorship disclosure
OpenAI GPT-5.6 Sol, as identified by the locally authenticated Codex interface used during this work, supported experimental scripting and debugging, statistical aggregation and reanalysis code, structured plan generation for the explicitly labeled pilot, and drafting/editing. Google Gemini, labeled 3.6 Thinking in the provider interface during this work, was used as an adversarial critique tool.
Those are provider/interface labels, not claims about immutable API snapshot identifiers. I ran the executable experiments on my controlled system, checked machine-generated outputs against stored artifacts, independently verified the primary references used for the manuscript, and remain responsible for the research design, interpretation, text, and any future submission.
How to cite this preprint
Exzil Calanza, "RStackBench: A Five-Layer Fault-Injection Method for Measuring Browser-Agent Execution Reliability in Controlled Applications," independent research preprint, Aug. 2026. Available: https://exzilcalanza.info/rstackbench-browser-agent-reliability/
The formal PDF contains the complete reference list, including WebArena, WorkArena, BrowserGym, WebRetriever, CAP, and WAREX.
