Background hover and screenshot coordinates
后台悬停与截图坐标
Hover-only controls can sometimes be revealed without moving the user's pointer. Treating every hover as a system movement unnecessarily disturbs that pointer. Screenshot pixels also differ from native window points after Retina capture, attachment normalization or model-request resizing; making the model convert them duplicates provider geometry rules.
English
Problem
Hover-only controls can sometimes be revealed without moving the user's pointer. Treating every hover as a system movement unnecessarily disturbs that pointer. Screenshot pixels also differ from native window points after Retina capture, attachment normalization or model-request resizing; making the model convert them duplicates provider geometry rules.
Decision
Native hover first uses a same-window element or ancestor advertising both AXShowAlternateUI and AXShowDefaultUI. Entering its revealed descendants preserves the parent UI; leaving restores it. An unavailable route falls back to the real system pointer. An AX action failure propagates because received effects may be uncertain. The result records the route and whether the system pointer moved. System movement requires the selected foreground app and AppKit's mouse hit test, which excludes input-transparent overlays. No route grants cross-call ownership.
Model-facing coordinate input defaults to pixels in the returned screenshot. relative: true accepts integer thousandths from 0 to 1000 for click, hover, scroll and drag; it does not apply to AX indices. The Host owns the mapping to native window points. An application-wide observation refreshes its inventory; a scoped observation preserves other windows' image mappings. Window resizing invalidates the old image, while movement preserves local coordinates. Relative endpoints remain inside the window and do not depend on screenshot resolution.
LlmAdapter.imageRequestTarget exposes the same dimensions and byte target used by image serialization. DeepSeek and pi-ai share their existing projection functions with this method. Other adapters preserve normalized images unless they declare resizing. Screenshot bytes returned to programs match the request image; its durable source is admitted and stored once. The native PNG metadata is checked against decoded source dimensions before any input mapping is published.
Alternatives considered
PID-directed movement alone does not establish AppKit hover tracking. The continuous-hover observations retain that limitation. Browser-engine or Electron page input can hover without the system pointer, but needs a separately established connection; a generic native app handle does not provide one.
Making all coordinates relative would discard the familiar absolute-pixel interface. Keeping both modes explicit avoids guessing a coordinate's units from its value. Using capture dimensions alone would miss later image resizing; the provider owns its projection formula.
Consequences
The native fixture covers alternate-UI reveal, delayed observation, screenshots, descendant entry, one click and exit while sampling the system pointer. Separate owned Chromium/CDP and Electron-input probes verify CSS hover, trusted input and pointer stability; they do not establish automatic access to arbitrary existing applications. Applications without a usable background route still require system-pointer hover.
Unit and provider tests cover Retina and downscaled images, both coordinate modes, endpoint validation, scoped-window preservation, resize invalidation, decoded-image metadata and one-time image storage. Recorded native-tool and PTC sessions retain the model-facing coordinate contract and hover-route evidence.
中文
问题
部分悬停控件可以在不移动用户鼠标的情况下展开。每次悬停都使用系统鼠标会增加不必要的干扰。Retina 截图、附件归一化和模型请求缩放也会使截图像素不同于窗口 point；让模型换算会重复提供方已有的几何规则。
决策
原生悬停优先寻找同一窗口内同时声明 AXShowAlternateUI 和 AXShowDefaultUI 的元素或祖先。进入其展开区域的子控件时保持父控件展开，离开后恢复。没有可用路径时回退到真实系统鼠标。AX 动作失败直接返回，因为已产生的效果可能不确定。结果记录实际路径及是否移动系统鼠标。系统移动要求所选应用处于前台，并使用 AppKit 的鼠标命中查询排除不接收输入的覆盖窗口。所有路径都遵守跨调用不分配所有权的决定。
面向模型的坐标默认使用返回截图中的像素。click、hover、scroll 和 drag 传入 relative: true 时接受 0～1000 的千分比整数，不适用于 AX 索引。Host 负责换算原生窗口 point。应用级观察刷新窗口清单，单窗口观察保留其他窗口的图片映射。窗口尺寸变化使旧图片失效，移动窗口保留局部坐标。相对坐标的端点保持在窗口内，不随截图分辨率改变。
LlmAdapter.imageRequestTarget 提供与图片序列化相同的尺寸和字节目标。DeepSeek 和 pi-ai 在该方法中复用已有投影函数。其他适配器除非声明缩放，否则保留归一化图片。返回程序的截图字节与模型请求图片一致，持久原图只接入并保存一次。发布输入映射前，原生 PNG 元数据必须与解码后的原图尺寸一致。
考虑过的替代方案
仅向 PID 发送移动事件不能保证触发 AppKit 悬停跟踪，连续悬停观察保留了这一限制。浏览器引擎和 Electron 页面输入可以不移动系统鼠标，但需要单独建立控制连接，普通原生应用句柄不提供这种连接。
统一使用相对坐标会放弃已有的绝对像素接口。显式选择模式可避免按数值猜测单位。只使用截图采集尺寸会遗漏后续图片缩放，因此投影公式由模型提供方负责。
影响
原生 fixture 覆盖替代界面展开、延迟观察、截图、进入子控件、一次点击和移出，并采样系统指针。独立的 Chromium/CDP 和 Electron 输入实验验证 CSS 悬停、可信输入和指针不变，不代表已能自动接入任意现有应用。没有可用后台路径的应用仍需系统鼠标悬停。
单元及提供方测试覆盖 Retina 和缩小后的图片、两种坐标模式、端点校验、单窗口观察保留其他映射、尺寸变化失效、解码尺寸校验及一次图片存储。原生工具和 PTC 的 Session 回放保留模型可见的坐标约定与悬停路径信息。