# RStackBench reproducibility package

Frozen study date: 2026-08-14

## Scope
RStackBench is a localhost Chromium fault-injection method for separating Apparent Success from Verified Task Success (VTS) across Access, Perception, Action, State, and Verification layers. The final study contains 288 deterministic task conditions across three UI archetypes, 864 validation-script browser traces, 240 scoring-horizon traces, and 60 interpreted GPT-5.6 Sol pilot executions.

No experiment in this package mutates a third-party website or account.

## Recorded host/tool versions
- Google Chrome: 151.0.7922.138
- Node.js: v24.19.0
- Python: 3.14.7
- OpenAI Codex CLI: 0.147.0
- Model used for the frozen planning pilot: gpt-5.6-sol, as reported by Codex CLI

## Core deterministic experiments
From this directory:

1. `node rstack_benchmark.js`
   - Recreates the original configuration-card task set and three validation-script executions.
   - Primary outputs: `results/tasks.json`, `results/runs.jsonl`, `results/summary.json`.

2. `node multitemplate_benchmark.js`
   - Recreates the profile-form and modal-confirmation extensions.
   - Outputs: `results/multitemplate_tasks.json`, `results/multitemplate_runs.jsonl`, `results/multitemplate_summary.json`.

3. `node horizon_sensitivity.js`
   - Recreates the passive scoring-horizon ablation for State and Verification faults.
   - Outputs: `results/horizon_sensitivity_runs.json`, `results/horizon_sensitivity_summary.json`.

4. `python final_evidence_audit.py`
   - Recomputes the final aggregate invariants from frozen raw artifacts and verifies manuscript consistency.
   - Outputs: `final_evidence_audit.json`, `FINAL_REPRODUCIBILITY_MANIFEST.json`.

## GPT-5.6 Sol planning pilot
`run_llm_n30.py` generated 60 paired plans through an authenticated Codex CLI session: 30 ordinary-prompt plans and 30 reliability-oriented plans. `execute_llm_n30.js` interpreted the frozen plans in the controlled Chromium benchmark.

The package includes the frozen `llm_pilot_n30/plan_index.json`, plan JSON files, logs, and `executions.json`. Future calls to a remote model may not reproduce identical text because model routing/snapshots and service behavior can change. The reported paper results can be independently checked without regenerating the plans by reusing the frozen plans and execution records.

## Final headline invariants
- Cross-template task conditions: 288
- Validation-script Chromium traces: 864
- Passive horizon traces: 240
- GPT planning plans: 60
- GPT pilot Chromium executions: 60
- Total browser traces represented in the study: 1,164
- Selector-only VTS: 53/288 = 18.4%; false completions: 131
- Semantic-recovery VTS: 192/288 = 66.7%; false completions: 96
- Verification-aware VTS: 288/288 = 100%; false completions: 0
- State passive-horizon VTS: 0/24 at 50 ms; 24/24 by 750 ms
- Verification passive-horizon VTS: 0/24 through 1000 ms
- GPT pilot ordinary VTS: 16/30 = 53.3%; false completions: 9
- GPT pilot reliability VTS: 21/30 = 70.0%; false completions: 5
- GPT paired exact two-sided McNemar p = 0.0625

## Integrity
`FINAL_REPRODUCIBILITY_MANIFEST.json` stores SHA-256 hashes for core code, data, manuscript, and figures. `final_evidence_audit.json` is generated from the raw records rather than copied from manuscript tables.

## Submission note
The local package is frozen but is not automatically published to a public repository or archive. Public release should use this frozen bundle or a byte-equivalent version so the manuscript can cite a stable artifact identifier.
