Native CUA objects independent of evaluator ownership
独立于执行器管理的原生 CUA 对象
Computer-use programs need consistent app and window operations. Coupling those operations to a CUA-specific interpreter makes input handling depend on unrelated choices about variable persistence, imports and cancellation.
Source compatibility: Chinese title uses a compatible non-canonical header
English
Problem
Computer-use programs need consistent app and window operations. Coupling those operations to a CUA-specific interpreter makes input handling depend on unrelated choices about variable persistence, imports and cancellation.
Decision
The shared cua library owns application/window objects, consent, and logged dispatch. cua-macos owns the Swift helper, its build, and its local connection lifecycle. Providers pass a prepared executable and a connection factory to the shared installer, keeping platform code out of object consumers without adding another runtime registry.
The experimental CUA package owns a native-app subset of the Codex-style object interface and selected MIT-licensed macOS behavior adapted from Cua Driver in a Swift helper. A source manifest records the adapted upstream files and local implementation. The helper is locally compiled; the package has no external Cua Driver runtime dependency. The existing provider registration prevents competing computer-use providers within one composition.
The facade accepts an operation dispatcher. Its tool adapter resolves the current tool binding for each operation, preserving normal authorization, logging and image admission. The execution host owns variable persistence and completion serialization. CUA defines no persistence directive, worker or run_code replacement.
Native process ownership follows the shared helper decision, with isolated Agent connections. App handles bind process identity and kernel start time; replacing the process or helper never revives old handles. AX snapshots are scoped to an app and optional window, so independent window observations remain usable. Coordinate actions select an owned window; the Host maps screenshot coordinates to native window-local points. Keyboard input remains app-scoped because macOS delivers it to the app's current focus.
The operation schema owns shared data declarations. Host-only limits and launch continuations remain separate from model parameters, and generated native field/enum tables preserve independent native admission without hand-maintained copies. Native code continues to own system calls, permissions and cancellation. AX formatting consumes immutable observations; the walker retains sole ownership of indices and AX references.
Clipboard paste is distinct from Unicode key input: Return and Tab key events can trigger actions instead of inserting clipboard content. Pasteboard access runs on the AppKit main actor and retains promised data until a reader requests it. AX command completion alone does not establish that the application has fetched the data. Restoration checks ownership before replacing contents; the caller still verifies the application result. After successful command dispatch, pasteReadTimeoutMs bounds waiting for a data request, with a default of 1000 ms. Expiry returns PASTE_NOT_CONSUMED, closes the data provider against late delivery, and restores the clipboard only while still owned. The failure reports no read within that interval; it does not prove the application will never process its queued Paste command. A delayed command may read restored clipboard contents, so callers inspect the target and never retry automatically. Foreground-dependent consumption was observed in the installed ChatGPT/Codex application (com.openai.codex); other applications need their own validation. Guidance presents this as an example and applies recovery instructions to any PASTE_NOT_CONSUMED result.
CUA registers its own Harness tool and returns operation values directly. Screenshot capture requires a resolved image-capable model and the shared attachment service. The tool admits each screenshot once and renders its durable image reference; the object API, combined observation and image sink retain the same CuaScreenshot containing encoded bytes, mimeType, pixelWidth, pixelHeight and windowId from the validated screenshot result. The bridge preserves this metadata without another upload or image emission. Callers use the reported format and dimensions when saving or declaring images.
Alternatives considered
Depending on the complete upstream driver would simplify upgrades but does not meet the requirement for locally owned implementation. Copying browser and other platform code would add unsupported scope. The JSON operation protocol allows another native implementation without changing object consumers.
A CUA-specific persistent evaluator duplicates execution lifecycle policy and prevents independent iteration on PTC. Requiring callers to manage raw app and snapshot tokens alone loses reusable object semantics. The facade and small tool adapter provide those semantics while leaving the evaluator outside this package.
Reusing the MCP result adapter would impose MCP envelopes and wire validation on native CUA operations. Direct Harness tool output and shared attachment admission provide the needed logging and images without a second protocol.
Fixing the screenshot format would require otherwise unnecessary re-encoding and discard the attachment pipeline’s pass-through optimization. A format convention alone still forces callers to guess, and cannot tell them when encoding policy changes. Returning the already computed format and dimensions preserves pass-through and makes each result describe its actual bytes.
Waiting until the full native-operation timeout hides successful command dispatch behind a generic timeout. A separate configurable read budget reports the missing acknowledgement without treating slow applications as unsupported. Requiring every application to be foreground would claim a prerequisite that the available observations do not establish.
Consequences
The package remains opt-in and macOS-only. Observations do not lock the desktop, and input success records dispatch rather than task completion. Callers must verify effects and must not retry uncertain input automatically. Native fixture tests verify actual window state separately from protocol and model-transcript tests. Persistent execution, binary distribution, permission onboarding and real-model task success remain separately owned work.
Source builds retain a content-addressed executable per compiled generation. The shared helper decision owns capture-client identity and cross-Host lifetime. Retained source generations avoid replacing a live executable, while the package payload remains separately published for distribution.
中文
问题
Computer-use 程序需要一致的应用和窗口操作。将这些操作与 CUA 专用解释器绑定，会使输入处理依赖变量持久化、导入和取消等独立设计。
决策
共享的 cua 库负责应用与窗口对象、授权和已记录的派发。cua-macos 负责 Swift helper、构建和本地连接生命周期。提供方将准备好的可执行文件和连接工厂交给共享安装函数，使对象调用方无需包含平台代码，也无需增加另一套运行时注册表。
实验 CUA 包提供 Codex 风格的原生应用对象接口，并在 Swift 辅助进程中改写 Cua Driver 中选取的 MIT 许可 macOS 行为。源码清单记录改写的上游文件和本地实现。辅助进程在本地编译，本包不依赖外部 Cua Driver 运行时。已有的 provider 注册机制防止同一组合中出现竞争 provider。
对象接口接收操作分派器。工具适配模块每次操作都获取当前工具绑定，保留标准授权、日志和图片接入。执行宿主负责变量持久化和最终表达式值的序列化。CUA 不定义持久化指令、执行 worker 或 run_code 替代实现。
原生进程归属遵循共享 helper 决策，Agent 连接保持隔离。应用句柄绑定进程身份及内核启动时间，替换进程或辅助进程不会使旧句柄重新有效。AX 快照按应用和可选窗口保存，不同窗口的观察可以分别使用。坐标操作选择属于该应用的窗口；Host 按截图坐标规则换算为原生窗口内 point。键盘输入保持应用范围，因为 macOS 将其送往应用的当前焦点。
操作 schema 负责共享数据声明。Host 专属限制和启动续接与模型参数分开，生成的原生字段、枚举表保留独立原生校验，同时避免手工维护副本。系统调用、权限和取消仍由原生代码负责。AX 格式化只消费不可变观察数据，遍历器独自负责索引与 AX 引用。
剪贴板粘贴与 Unicode 按键输入分别实现，因为 Return 和 Tab 按键可能触发动作，而不是插入剪贴板内容。剪贴板访问使用 AppKit 主线程，保留延迟提供的数据直到读取方请求。AX 命令完成本身不能证明应用已取得数据。恢复前检查剪贴板归属；调用方仍需验证应用中的结果。 命令成功分派后，pasteReadTimeoutMs 限制等待数据请求的时间，默认 1000 毫秒。到期返回 PASTE_NOT_CONSUMED，关闭数据提供方以阻止迟到的数据交付，并仅在仍持有剪贴板时恢复原内容。该失败只说明这段时间内没有读取，不证明应用以后不会处理已排队的 Paste 命令。迟到的命令可能读取恢复后的剪贴板内容，因此调用方必须检查目标，不能自动重试。已安装的 ChatGPT/Codex 应用（com.openai.codex）已观察到依赖前台才能读取的行为，其他应用需要分别验证。指引将其写为例子，并对任何 PASTE_NOT_CONSUMED 结果提供恢复步骤。
CUA 注册自己的 Harness 工具，直接返回操作结果。截图要求解析到支持图片输入的模型，并提供共享附件服务。工具只接入一次截图，再渲染持久图片引用；对象 API、组合观察和图片输出回调保留同一个 CuaScreenshot，其中的编码 bytes、mimeType、pixelWidth、pixelHeight 和 windowId 来自已校验的截图结果。桥接保留这些元数据，不再次上传或输出图片。调用方落盘或声明图片时使用返回的格式和尺寸。
考虑过的替代方案
依赖完整上游驱动可以简化升级，但不符合本地维护实现的要求。复制浏览器及其他平台代码会引入尚不支持的范围。JSON 操作协议允许替换原生实现，而不改变对象调用方。
CUA 专用持久执行器会重复实现执行生命周期规则，使 PTC 无法独立演进。仅让调用方管理原始应用和快照 token，又会失去可复用的对象语义。对象接口和轻量工具适配模块提供这些语义，并将执行器留在本包之外。
复用 MCP 结果适配器会给原生 CUA 操作引入 MCP 信封和协议校验。直接使用 Harness 工具输出与共享附件接入，即可取得所需的日志和图片能力，无需再经过一种协议。
固定截图格式会要求原本不需要的重新编码，并失去附件处理的原样直通优化。仅约定格式仍要求调用方猜测，也无法告知编码策略的变化。返回已经算出的格式和尺寸，既保留直通优化，也让每个结果说明其实际字节。
等到原生操作的总超时会用一条通用超时掩盖命令已经成功分派的事实。独立、可配置的读取等待上限能报告缺失的读取确认，而不将响应慢的应用认定为不支持粘贴。要求所有应用都必须位于前台，则超出了现有观察能支持的结论。
后果
本包需要显式启用，且只提供 macOS 实现。观察不会锁定桌面，输入成功只记录分派完成，不能证明任务完成。调用方必须验证效果，不能自动重试结果不确定的输入。原生测试通过实际窗口状态验证效果，与协议和模型消息测试分别进行。持久执行、二进制分发、权限引导及真实模型任务成功率是分别负责的工作。
源码构建按编译产物的内容摘要保留可执行文件。共享 helper 决策负责捕获客户端身份与跨 Host 生命期。源码各代副本避免替换运行中的可执行文件，包内产物仍单独发布。