Abstract

A click is not a click. That sentence started as an engineering intuition and ended as a correction to my own benchmark. A 50+ trial Windows experiment on UIA, CDP, DOM activation, coordinate spaces, occlusion, and what actually proves a click landed. I began this Skynet field study because Windows UI Automation (UIA) looked reliable but slow, while the Chrome DevTools Protocol (CDP) looked like the obvious faster replacement for browser actions. The first experiment seemed to confirm a neat story: UIA could activate Gemini’s Upload & tools control, while raw CDP mouse input failed. That story was wrong. The deeper investigation found that Skynet’s own normal-profile CDP bridge was silently dropping the buttons bitfield from Input.dispatchMouseEvent. After repairing the bridge, adding regression coverage, and rerunning the experiment at N=50, CDP pointer input succeeded 50 out of 50 times and produced trusted pointer/mouse/click events. That reversal became the most useful result of the project: automation accuracy depends on proving every layer between intent and application state, including the transport implementation itself. On this Skynet machine, a cached in-process UIA path removed most of the overhead in the existing managed implementation while preserving fail-closed behavior, while synthetic DOM activation also worked but carried different event fidelity.

Index Terms—application acceptance; browser automation; Chrome DevTools Protocol; coordinate systems; input fidelity; UI Automation; Windows automation

I. Introduction

A click is not a click.

That sentence started as an engineering intuition and ended as a correction to my own benchmark.

I began this Skynet field study because Windows UI Automation (UIA) looked reliable but slow, while the Chrome DevTools Protocol (CDP) looked like the obvious faster replacement for browser actions. The first experiment seemed to confirm a neat story: UIA could activate Gemini’s Upload & tools control, while raw CDP mouse input failed.

That story was wrong.

The deeper investigation found that Skynet’s own normal-profile CDP bridge was silently dropping the buttons bitfield from Input.dispatchMouseEvent. The protocol transaction could complete while the event being delivered was not the event the experiment claimed to test. After repairing the bridge, adding regression coverage, and rerunning the experiment at N=50, CDP pointer input succeeded 50 out of 50 times and produced trusted pointer/mouse/click events.

That reversal became the most useful result of the project: automation accuracy depends on proving every layer between intent and application state, including the transport implementation itself.

II. Background and Related Work

The original article does not contain a dedicated related-work section; its cited platform documentation is retained in References and discussed where it directly supports the measurements.

III. Methodology

The controlled environment

This is a field study on one real Skynet machine, not a universal benchmark.

• Windows 10 Pro N, build 19045
• AMD Athlon X4 870K, 4 logical processors
• 7.9 GB RAM
• NVIDIA GeForce GTX 1660
• Chrome 151.0.7922.138
• one 1920×1080 display
• system DPI: 96
• Chrome window DPI: 96
• scaling: 100%
• devicePixelRatio: 1

The principal reversible target was Gemini’s Upload & tools button in Skynet’s existing signed-in normal Chrome profile.

I did not have 125%, 150%, 200%, mixed-DPI, or multi-monitor hardware states available during this run. Those configurations are therefore explicitly outside the evidence. Microsoft documents that UI Automation bounding rectangles and clickable points use physical coordinates and warns that non-96-DPI clients need explicit DPI handling. [1]

Every counted click required an independent browser-side visible-state postcondition. A transport returning success was never sufficient by itself.

IV. Results

The final datasets were persisted as raw trial-level JSON and SHA-256 hashed. The main combined artifacts are:

TABLE I. Raw evidence and statistical treatment

For the two UIA action paths, the managed route was 56/56 successful in the measured actions. The cached Python/UIA route was 58/59; the single failed attempt returned a typed no_clickable_point result and did not produce a false application success.

The Wilson 95% interval for 56/56 is approximately 93.6%-100%. For 58/59 it is approximately 91.0%-99.7%. Those intervals are a reminder that even a perfect observed run does not prove a universal 100% reliability rate.

Discovery 1: native-window identity is cheapest at creation time

Multiple Chrome windows can have nearly identical titles and similar accessibility trees. Resolving the intended native HWND later by title or mutable controls introduces unnecessary ambiguity.

I tested a simpler binding method:

1. enumerate visible Chrome top-level HWNDs immediately before Skynet creates a dedicated Gemini window;
2. create the leased browser window;
3. enumerate again;
4. take the set difference;
5. accept only an exact single newborn visible Chrome HWND.

Across 50 fresh-window trials, the delta contained exactly one new visible Chrome HWND 50/50 times. Median browser-window creation time was 2197.8 ms, p95 2534.9 ms, and p99 2654.5 ms.

This does not prove that Windows or Chromium will always create exactly one HWND. The production rule should be the opposite of an assumption: if the delta is zero or greater than one, fail closed or use another identity proof.

The useful result is architectural. Window identity can often be bound before mutable page state, titles, menus, prompts, and action controls complicate the problem.

Discovery 2: UIA was not inherently the 2.25-second bottleneck

Skynet’s existing trusted UIA route crossed into a PowerShell/.NET UIA implementation and performed repeated accessibility work. On the final dataset it succeeded in 56/56 measured actions with:

• median: 2250.9 ms
• mean: 2317.1 ms
• standard deviation: 195.6 ms
• p95: 2774.8 ms
• p99: 2928.0 ms

Chromium did not expose an InvokePattern that completed this particular control. The working fallback was ClickablePointForegroundVerified: identify the exact accessible button, obtain a clickable point, verify the target window, deliver physical input, then require a browser postcondition.

I reproduced the same essential action in a persistent Python COM/UIA client using CUIAutomation8 and IUIAutomation2. Across 59 attempts:

• 58 application successes
• one explicit no_clickable_point failure
• median all-attempt latency: 113.9 ms
• mean: 122.3 ms
• standard deviation: 30.5 ms
• p95: 162.1 ms
• p99: 246.0 ms

The correct conclusion is not “Python UIA is twenty times faster than PowerShell UIA.” Both paths use Windows UI Automation. The evidence points to Skynet implementation overhead: process/runtime startup, repeated tree traversal, and cross-process property work.

Microsoft’s UIA documentation supports the mechanism behind that hypothesis: retrieving UIA properties and control patterns requires cross-process calls, and caching them in batch can improve performance. [2]

The production direction is therefore a persistent, narrow UIA client with exact-window scope and carefully cached properties/patterns–not removal of safety checks.

Discovery 3: my first CDP conclusion was invalid

The Chrome DevTools Protocol defines Input.dispatchMouseEvent with separate fields for the event type, viewport-relative CSS-pixel coordinates, button, the buttons bitfield, and click count. The protocol defines Left=1 in buttons. [3]

My original Skynet bridge validated button but silently rebuilt the parameter object without preserving buttons.

That meant this intended sequence:

• mouseMoved, buttons=0
• mousePressed, button=left, buttons=1
• mouseReleased, button=left, buttons=0

was not actually the sequence the bridge could faithfully express.

I treated that as an instrumentation defect, patched it under Skynet’s blast-radius guard, and added tests requiring the exact 0 -> 1 -> 0 state while rejecting unsupported bitfields.

The repaired CDP study then produced:

• 50/50 application successes
• median 1325.0 ms
• mean 1358.4 ms
• standard deviation 90.3 ms
• p95 1550.2 ms
• p99 1648.9 ms
• trusted target events in all 50 trials

The dominant event sequence was:

focus -> pointermove -> mousemove -> pointerdown -> mousedown -> pointerup -> mouseup -> click

Four trials also included pointer-over/mouse-over before movement. All observed target pointer/mouse/click events in the CDP treatment were trusted.

This is the central scientific correction of the study: a transport implementation bug inverted the apparent benchmark result.

A successful protocol receipt does not prove that the client sent the protocol fields it thought it sent. The adapter itself belongs inside the evidence chain.

Discovery 4: DOM activation can succeed with lower input fidelity

As a control, I also used HTMLElement.click() on the exact button.

It succeeded 50/50 times on the application postcondition. Median transaction latency was 1260.1 ms, but its tail was wider: p95 2405.9 ms and p99 2470.3 ms.

The event log showed a focus event followed by the synthetic click. The click itself had isTrusted=false.

That is not a browser anomaly. The WHATWG HTML Standard defines HTMLElement.click() by firing a synthetic pointer event named click with the not-trusted flag set. [4]

This gives us two different kinds of success:

• repaired CDP pointer input: full trusted pointer/mouse sequence plus application acceptance;
• HTMLElement.click(): synthetic activation plus application acceptance.

Both can be useful. They are not equivalent receipts.

Discovery 5: coordinate space must be an explicit contract

CDP mouse coordinates are relative to the main-frame viewport in CSS pixels. [3] UIA GetClickablePoint and bounding rectangles use physical screen coordinates. [1]

I moved the same browser window through ten different positions and sizes. All ten samples were valid at 96 DPI / 100% scaling.

The raw numeric translation from DOM center to UIA center varied with the window geometry, as expected. A transform using the window’s screen position plus Chrome’s outer/inner dimensions predicted the UIA X coordinate with a maximum absolute error of 0 px and the Y coordinate within 8 px in this environment.

That is evidence for a method, not a mixed-DPI guarantee.

A controller should tag coordinates with their coordinate space and reject cross-layer reuse unless a DPI-aware transform has been explicitly established.

Discovery 6: occlusion belongs to the action contract

A physical clickable point is meaningful only if another window is not occupying the pixel.

The first version of my occlusion harness produced an apparently clean result–but it was invalid. I had called CreateWindowExW through ctypes without declaring its 64-bit HWND return type, allowing the handle to be truncated. I discarded that run.

In the corrected run I explicitly declared HWND return types and refused to count a trial until WindowFromPoint proved the topmost helper window was the actual root window over the target pixel.

Under that verified complete occlusion:

• 50 guarded trials were performed;
• UIA returned no clickable point in 50/50;
• the physical action path therefore failed closed in 50/50.

That specific provider behavior is useful, but it should not become a universal assumption. A robust physical-input controller can still use a z-order hit-test as an independent defense before delivering the mouse event.

Discovery 7: browser-local file state was a false delivery receipt

The study also uncovered a different class of automation error in Skynet’s Gemini advisor attachment path.

The extension could construct File objects, assign them to a hidden input[type=file], and read the exact filename and byte size back. That proved local browser input state.

It did not prove Gemini accepted or hydrated the attachment.

The production advisor code then contained a second problem: if provider-readiness proof failed, exact input-file readback could still be credited as successful delivery.

That is now fixed. Gemini attachment delivery fails closed unless provider-ready acceptance is proven.

This generalizes far beyond files. A populated field, selected file, successful CDP command, clicked element, or UIA action is an attempt receipt unless a different surface proves application acceptance.

V. Discussion

The six-layer click-truth model

The experiments support a practical control model in which clicked=true is replaced with six separate proofs.

1. Window identity

Which browser tab and native window are under control? Creation-time correlation can reduce ambiguity for connector-created windows, with fail-closed handling when the identity is not unique.

2. Element identity

Which semantic control is being targeted? Exact accessible name, role/control type, stable identifiers when available, and uniqueness belong before the action.

3. Coordinate-space identity

Are the coordinates viewport CSS pixels, device pixels, logical desktop coordinates, or physical screen coordinates? Never silently cross those boundaries.

4. Transport and fidelity

Was the action a UIA control pattern, a UIA clickable-point physical action, CDP pointer input, DOM click(), synthetic event dispatch, or another mechanism? The receipts should say which.

5. Foreground and occlusion

For physical input, verify the intended native window and pixel ownership immediately before acting. UI state can move between identity resolution and input delivery.

6. Independent application acceptance

After the action, require another surface to prove the result: visible DOM state, URL change, accessibility state, stored server state, or a durable provider-side receipt.

What changed in Skynet because of the study

This was not a paper written after the engineering. The experiments modified the system.

Two production defects were fixed under guarded blast-radius sessions:

1. Gemini advisor attachments can no longer count exact input.files readback as delivery when provider acceptance is absent.
2. The normal-profile CDP bridge now preserves the Input.dispatchMouseEvent.buttons bitfield and rejects unsupported values instead of silently weakening the event.

Current relevant source hashes at the final evidence freeze were:

• skynet_socials_extension_api.py: 593af990058a38a46173c1c0982a9f6eda85bf800c871adda9746dd157bee270
• skynet_connector_advisor.py: e6171b4705c5fe3fc84094fec614b856e5e457b1db7877ca8ddde7d19d340cbf

The final affected regression suite passed 130 tests in 9.69 seconds.

VI. Threats to Validity

Negative results that were not promoted into claims

A scientific engineering report should retain the experiments that failed methodological checks.

During this project I discarded or narrowed several results:

• the original CDP failure, because the bridge stripped the buttons field;
• early UIA loops, because fixed 60 ms sleeps sampled the application before the menu became visible;
• reset checks based on hidden DOM nodes, because node presence did not equal visible menu state;
• the first occlusion run, because the Win32 helper HWND prototype was wrong;
• a planned UIA propagation-latency distribution, because repeated Chromium UIA polling raised a COM subscriber error before a clean distribution was established.

The final article therefore makes no quantitative claim about UIA accessibility-tree propagation latency.

This matters because Chromium’s accessibility tree is a separate representation derived from the DOM and exposed through platform-specific accessibility APIs. [5] That fact supports treating DOM and UIA as separate observation surfaces; it does not entitle us to invent a causal latency number when the experiment did not survive.

Limits and next experiments

This result is deliberately narrower than the first draft.

It does not establish:

• mixed-DPI correctness;
• multi-monitor coordinate correctness;
• universal reliability of HWND birth binding;
• universal UIA-versus-CDP performance;
• UIA accessibility-tree propagation latency;
• behavior on every Chromium version or web framework;
• reliability for controls that expose different UIA patterns.

The next controlled studies should include 125%, 150%, and 200% DPI, mixed-DPI monitor moves, Chrome-native UI controls, controls exposing InvokePattern/TogglePattern/SelectionItemPattern, event-driven UIA cache updates, direct versus extension-mediated CDP, and WebDriver BiDi.

WebDriver BiDi is particularly relevant because the W3C protocol adds bidirectional event streaming rather than relying only on strict request/response automation. [6]

VII. Conclusion

The useful question is not “UIA or CDP?”

The useful questions are:

• Which layer owns identity?
• Which coordinate system is active?
• What transport was actually delivered?
• What fidelity did it have?
• Was the target foreground and unobstructed?
• What independent evidence proves the application accepted it?

On this Skynet machine, a cached in-process UIA path removed most of the overhead in the existing managed implementation while preserving fail-closed behavior. Repaired CDP pointer input was reliable in 50/50 trials and produced trusted event sequences. Synthetic DOM activation also worked but carried different event fidelity. Creation-time HWND correlation was reproducible in 50/50 controlled openings. And two experiments that initially looked like conclusions–the CDP failure and the first occlusion result–were overturned because the instrumentation itself was wrong.

That last point is the result I would keep even if every latency number changed tomorrow:

An automation system is not scientifically trustworthy merely because it records receipts. The receipt chain must prove that the intended protocol fields reached the intended control and that the application accepted the result.

Key Takeaways

• CDP result reversed. Preserving the mouse buttons bitfield changed pointer input from apparent failure to 50/50 application success.
• UIA overhead was mostly implementation overhead. Cached in-process UIA reduced Skynet’s measured median from 2250.9 ms to 113.9 ms, with one explicit fail-closed result.
• Local state is not provider acceptance. The study found and fixed a Gemini attachment path that could over-credit exact browser file-input readback.
• A click needs layered proof. Reliable control separates window identity, element identity, coordinate space, transport fidelity, occlusion, and application acceptance.

References

• [1] Microsoft, Understanding Screen Scaling Issues, Windows UI Automation documentation.
• [2] Microsoft, CacheRequest Class / UI Automation caching, documenting cross-process retrieval and batch caching.
• [3] Chrome DevTools Protocol, Input domain – Input.dispatchMouseEvent.
• [4] WHATWG, HTML Standard – HTMLElement.click() synthetic click activation.
• [5] Chromium, UI Automation and Chrome accessibility-tree architecture documentation.
• [6] W3C, WebDriver BiDi, bidirectional user-agent control and event streaming.