Why Accessibility-Tree Node Counts Are Not Machine-Comparable

The observation

A synthetic fixture — 685 elements, comprising 120 interactive controls and 200 paragraphs of fixed copy — was rendered and measured on two Windows hosts. Both ran Chrome 151.0.7922.140, V8 15.1.206.17, DevTools Protocol 1.3. The fixture is deterministic: no randomness, no clock, no network, fixed strings.

The DOM was identical. The actionable subset was identical. The reduction ratio — DOM characters divided by scene characters, 55,945 ÷ 4,579 — was identical to two decimal places on both hosts. Note this is a page-content ratio, not a ratio taken against the accessibility node count: actionable elements divided by raw nodes differs between the hosts precisely because the denominator does. The accessibility node count moved by 26%, and the screenshot payload by roughly 8% — the two figures in that table read off the rendering rather than off the page.

A benchmark that reports the node count as a property of the instrumentation would, on this evidence, conclude that one machine’s perception layer is a quarter larger than the other’s. Nothing of the kind is true.

Method

The natural first hypothesis is browser version, and it was already excluded: the build strings match to the patch level. The second is page content, excluded by identical DOM length and element count.

The third hypothesis — that part of the tree is layout-derived rather than semantic — is testable on a single machine. Chrome’s accessibility tree contains InlineTextBox nodes, which correspond to laid-out runs of text rather than to elements. If layout drives the count, changing the viewport must change it while leaving the DOM untouched.

The fixture was therefore measured at four viewport widths on one host, using Emulation.setDeviceMetricsOverride, with a settle interval before each read so that no tree was captured mid-relayout.

Results

Three findings follow directly.

The raw count is viewport-dependent. Narrowing the window to 800 px added 190 nodes to a page whose DOM did not change by a single character.

The variance is entirely InlineTextBox. The delta in the total and the delta in the inline-box count are the same number at every width. Subtracting them yields 1,244 — invariant across a 3.2× range of viewport widths.

Ignored nodes are not the cause. Only two nodes in the entire tree carry the ignored flag, a population far too small to account for a 440-node gap. This matters because “generic and ignored nodes vary by platform” is the intuitive explanation, and it is measurably wrong here.

The full role histogram at 1280 px accounts for every node: StaticText 440, InlineTextBox 440, generic 240, paragraph 200, LabelText 120, textbox 120, button 120, none 2, RootWebArea 1, main 1. The 240 actionable elements are exactly the 120 textboxes plus the 120 buttons.

A later run on Host B returned TOTAL 1684 / INLINETEXTBOX 440 / SEMANTIC 1244 at an inner window of 929 × 925 — the same semantic figure, reached independently on different hardware at a third viewport size.

What this does not establish

Host B’s original reading of 1,244 was a bare total; its role breakdown was never captured. The arithmetic is suggestive but not decisive, and it is worth stating why: in this tree, three disjoint role sets each happen to total exactly 440 — InlineTextBox, StaticText, and generic plus paragraph. Any one of them going missing produces 1,244. Attributing that first reading specifically to absent inline boxes is an inference, not a measurement.

Viewport width alone does not appear to explain it: the inline-box count floors at 440 and does not move between 1280 px and 2560 px, so reflow across the widths tested never approaches 1,244. That points toward the layer being absent rather than merely reflowed, and it is consistent with Chrome not emitting InlineTextBox nodes without StaticText parents. It is not established. The distinction between what the evidence favours and what was measured is exactly the distinction a benchmark exists to preserve, and the honest answer here is that the first reading’s composition is unknown.

Note also that the two hosts did not agree on everything: screenshot payload sizes differed by roughly 8%. That is consistent with — but not proof of — a rendering difference, since PNG payload size is a function of image content and encoder behaviour, and no pixel comparison was performed. It is what one would expect from two machines whose windows were never pinned to a common size, and it is a second reason to pin them.

Implications for benchmark design

The defect is not in the protocol. getFullAXTree returns exactly what it promises. The defect is in treating one number as though it described the tool, when it describes the tool, the window, and the browser’s active accessibility mode at once — inline text boxes are only computed when something asks for them, so a second consumer attached to the same browser can change the figure.

Three corrections make such a measurement honest:

Pin the viewport, then verify it took. Issuing the override is not enough; read innerWidth back. An override that silently failed produces numbers indistinguishable from a successful one. Pin it only on a tab the harness created — resizing a tab that belongs to an operator is a side effect a benchmark has no business causing.

Separate layout-derived nodes from semantic ones. Report the semantic count as the candidate cross-machine figure and the raw count as diagnostic — candidate because it is the figure that survived every condition tested here, not one proven invariant everywhere. An unqualified field name is what invites the false comparison: a reader who sees ax_nodes reasonably assumes it characterises the instrument.

Scope the comparability claim to specific measurements. A single boolean asserting that a whole report is comparable is almost always false, because latency, screenshot size and node totals do not travel with the same guarantees as element counts. Enumerate which fields survive a move between machines, and say plainly which do not.

Applied here, the DOM length, the actionable count, the scene length and the reduction ratio agreed exactly across both hosts. The semantic count also agreed — but on two hosts, one browser build and one fixture, with the accessibility mode never pinned. That is evidence it is the more stable figure, not a guarantee that it is invariant; a contract that reports it as comparable should pin and report the active accessibility mode alongside it. The raw node total, the inline-box count, the screenshot payload and every timing figure are not comparable.

Reproducing it

The connector, the fixture and the benchmark used above are open source and dependency-free, so the whole measurement is a single file that runs on a machine where nothing may be installed — which is how the second host was measured at all. It is published at skynet-chrome-cdp.

The wider point generalises past accessibility trees. Any figure derived from a rendered page — node totals, screenshot sizes, layout boxes, visible-text extents — is a reading of the environment as much as of the page. The remedy is not a better number. It is stating the conditions under which the number was taken, and refusing to compare two readings taken under conditions that were never made equal.