ACP snapshot tests — record-once / replay-deterministic
ACP 快照测试——一次录制 / 确定性回放
Unit tests do not exercise the complete assembled-agent subprocess or its ACP automation wire, while real-API tests are nondeterministic and key-gated. Loader wiring, backend behavior, and protocol output can therefore regress despite green unit coverage, as the [default-export postmortem](../../../../docs/postmortem/0001-acp-default-export-drops-inject.md) demonstrated. The blocker for a full-transcript test is the
English
Problem
Unit tests do not exercise the complete assembled-agent subprocess or its ACP automation wire, while real-API tests are nondeterministic and key-gated. Loader wiring, backend behavior, and protocol output can therefore regress despite green unit coverage, as the default-export postmortem demonstrated.
The blocker for a full-transcript test is the model: the agent's output is driven by a non-deterministic LLM, and a key-gated test that hits the real API on every run is neither deterministic nor CI-runnable. The tier needs the fidelity of a real run with the determinism of a fixture.
Decision
A snapshot test boots the real ACP example, drives its stdio protocol from a deterministic script, and compares normalized output with committed expected outputs. A session log recorded once from the real API supplies all later model streams. The fixture is a projection of the product's persisted JSONL: its header and payloads remain, while body sequence/time envelopes are omitted.
The fixture projects the persisted session JSONL
Each scenario's session.jsonl is harvested from a real run. assistant/chunk events reproduce the model streams; tool, message, and boundary events capture the harness behavior. One ordinary session artifact therefore serves as both replay source and behavioral expected output.
Every committed session-format fixture uses the canonical packed physical layout. The all-row-kinds scenario is mechanically derived from an independent real recording; its test requires every packed storage-row kind and exact event-for-event equality after both fixtures decode, then ordinary replay and log comparison prove that the assembled process consumes and reproduces the layout.
Replay derives the model script from the log
llm-replay short-circuits the provider-agnostic llm/stream waterfall. deriveReplayScript() splits recorded assistant/chunk events at terminal finish chunks and uses (turn, step) changes to reject an unterminated prior call. A compaction/summary with llmStreamCall: true contributes one call at its durable log position: replay reconstructs canonical block boundaries from rawOutput, retains recorded usage when present, and supplies a terminal stop. The marker distinguishes that local call from template or remote summaries whose retained rawOutput did not consume this context's adapter.
The in-memory replay entry honors the full LLM contract
deriveReplayScript produces a list of ReplayEntry, the in-memory unit the replay listener serves positionally:
{ kind: 'chunks', chunks: StreamChunk[] }
| { kind: 'throw', chunks: StreamChunk[], message: string, code: string }
| { kind: 'hang' }
Logs derive chunk entries from finished assistant streams and explicitly marked compaction calls. Pre-stream throws, hangs, and external summarizer calls have no reconstructable local chunk representation, so those scenarios provide replay.override.json. A throw entry may include prefix chunks for mid-stream failure. Explicit overrides avoid inferring adapter behavior from lossy turn-end reasons or provider output alone.
Positional replay, one in-flight stream
Replay is positional and therefore permits only one in-flight model stream per scenario. Concurrent-session snapshots require request-keyed entries. Changed call order requires re-recording, and missing or exhausted fixtures fail loudly.
Recording harvests the log; keyless replay needs a providerless config
Recording runs the scenario with the real llm-deepseek adapter and the JSONL persistence backend configured with persistenceCompression: 'none', then projects the produced .jsonl into the scenario dir. The explicit raw mode keeps harvested logs line-readable while ordinary deployments use the backend's compressed default; eligible chunk runs still use the default packed storage rows. Per-event appends are durable, but the harness shuts the subprocess down gracefully (close stdin → await ctx.dispose()) before harvesting so the final events are flushed. llm-replay itself does no recording — it is replay-only.
Replay uses a cordis.snapshot.yml overlay that replaces the real adapter with llm-replay while retaining the live composition. Recording uses the ordinary config and a harness-supplied persistence root. Replay mode skips .env loading, so a stray API key cannot trigger a live call. See the single-source config Agent Note.
Two outputs: normalize, then compare
A snapshot run asserts two normalized outputs, because the harness's external APIs are distinct:
- The stdout transcript — the framed ACP JSON-RPC responses and committed-message updates an automation client receives. It catches regressions in the transport contract and is compared against a committed
stdout.expected.jsonl. - The re-persisted session JSONL, normalized and compared with
session.jsonl. The same fixture is both replay source and expected log. Prompt and tool bulk are scrubbed; one scenario per header class pins the remaining header sequence. The pin owns readable prompt and tool-schema sidecars by default, or names another pin as either source when the complete sequence is identical, so each distinct sidecar version is committed once. Fixture guards reject duplicate sidecar content, and record/refresh rejects shared claimants that generate different bytes. The original header-pinning rationale is preserved in the header-pinning Agent Note. Override scenarios derive model behavior solely from their sidecar.
The surfaces are complementary: stdout covers the minimal automation wire, while JSONL covers loop, tool, and boundary structure that the wire intentionally omits.
Normalization replaces session, cwd, protocol-id, timestamp, path, and process volatility; fixture projection omits body sequence/time envelopes without changing payload references. Record and refresh also store a generated workspace and its filesystem-resolved aliases as {{cwd}} in the replay fixture, so platform temp roots and random basenames do not affect recordings; authored temp paths and cwd values under an explicit workspaceParent remain literal. Scenarios constrain real bash use to stable commands. The stdout expected output remains wire-shaped JSONL and every raw line must parse as JSON. Ordinary Vitest snapshot updates write only the stdout expected output; the explicit record and refresh modes own replay-fixture writes.
Isolation: normalization now, sandbox later
Tool determinism comes from a generated cwd, scrubbed environment, fresh non-login shell, constrained commands, and normalization. The cwd defaults to the platform temp directory; a scenario can instead supply its parent when temp is an always-writable policy root and the behavior needs an independent project location. Concurrent replay runs own separate cwd, persistence, and fixed-length scenario-keyed spill roots, so one scenario's teardown cannot delete another's in-flight full-output recovery while real-path preview budgets remain stable. This tier does not claim OS confinement. A sandboxed executor can replace the local backend through the existing capability seam if a stronger tier is needed.
The replay plugin is its own package
@deepseek-ai/dsh-llm-replay is a support package rather than example-local glue. It replaces the real adapter by short-circuiting llm/stream with streams reconstructed from JSONL, and its package placement keeps the replay logic under normal coverage gates.
Two subcommands, replay in the default gate
pnpm run test:snapshot replays committed fixtures keylessly; test:snapshot:record uses the real API and rewrites the projected session snapshot and stdout expected output. The same keyless gate discovers repository JSONL by its session header and rejects any fixture that differs from the shared codec's projected canonical packed representation. Missing fixtures fail loud. Every scenario carries input.json, stdout.expected.jsonl, and session.jsonl; no-model cases use a header-only log. replay.override.json is required only for scenarios marked overridden, because its presence replaces derived replay. Fixture guards reject missing, mismatched, and orphaned files. Both commands accept scenario filters.
Alternatives considered
- A hand-authored
llm.jsonof model chunks — the earlier draft; reusing the real session log makes the fixture a genuine product of the system rather than a hand-built mock, and doubles it as a behavioral expected output. - A compulsory replay override for every compaction summary — rejected: the durable summary event already fixes a successful local call's position, complete output, and optional usage. An explicit local-call marker preserves that single-source fixture without inventing a call for template or remote summarizers.
- A byte-level HTTP-record library (Polly/nock/MSW) — rejected: adapter-specific, awkward with streaming SSE, and lower-level than the thing under test.
- Synthesizing throw/cancel entries from
turn/end {kind:'error'|'aborted'}— rejected: it couplesllm-replayto loop-internal turn-closing semantics, and theturn/endreason is lossy (it cannot distinguish a thrown 401 from a finish-error); the explicitreplay.override.jsonsidecar is the cleaner seam. - Copying both request-header sidecars beside every class pin — rejected: prompt and tool-schema composition vary independently, so a change to one shared component would churn byte-identical files across unrelated class pins. Explicit per-component sources retain one structural pin per class without duplicating content.
Consequences
The tier adds reviewed per-scenario input, session, stdout, optional override, and optional workspace fixtures, plus one file for each distinct pinned prompt and tool-schema sequence. Workspace seeds are copied into the generated cwd for both record and replay. In return the tier provides deterministic keyless coverage through the real Loader and tool composition, including an assembled context-overflow recovery whose marked compaction summary supplies the auxiliary call. Most retained scenarios exercise the assembled backend rather than ACP; the automation-only ACP decision keeps that corpus here until it can move to a transport-neutral headless suite without losing coverage.
This Agent Note relates to but does not supersede the proposed determinism Agent Note: that proposal's "universal replay fixture" re-derives session message history after every test (an internal-consistency invariant), whereas these snapshots pin assembled behavior plus the external automation output. They are complementary until the backend corpus moves off ACP.
中文
问题
单元测试不会覆盖组装后的完整 agent（智能体）子进程及其 ACP（Agent Client Protocol）自动化协议格式，而真实 API 测试不具确定性且受密钥门控。因此，即使单元测试覆盖率检查通过，Loader 接线、后端行为和协议输出仍可能回归，默认导出事故复盘（postmortem）已经证明了这一点。
完整 transcript（文本记录）测试的阻塞因素在于模型：agent 的输出由非确定性的 LLM（大语言模型）驱动，而每次运行都命中真实 API 的密钥门控测试既不确定也无法在 CI 中运行。该测试层级需要真实运行的保真度与 fixture（测试前置数据）的确定性兼得。
决策
快照测试会启动真实 ACP 示例，通过确定性脚本驱动其 stdio 协议，并将规范化输出与已提交的预期输出比较。从真实 API 一次记录的会话日志为后续所有模型流提供数据。fixture 是产品持久化 JSONL 的投影：保留 header 与 payload，省略正文序号／时间 envelope。
fixture 投影持久化会话 JSONL
每个场景的 session.jsonl 都从真实运行中采集。assistant/chunk 事件复现模型流；工具、消息和边界事件捕获 harness 行为。因此，一份普通会话产物同时充当回放来源和行为预期输出。
每个签入仓库的会话格式 fixture 都使用规范的打包物理布局。覆盖所有行类型的场景从一份独立的真实录制机械派生；测试要求它包含每一种打包存储行类型，并在两份 fixture 解码后逐事件精确相等；随后，普通回放与日志比较会证明组装后的进程能够消费并复现该布局。
回放从日志推导模型脚本
llm-replay 短路了提供方无关的 llm/stream waterfall（瀑布式事件）。deriveReplayScript() 在终止的 finish 分片处切分已记录的 assistant/chunk 事件，并用 (turn, step) 变化拒绝前一条未终止的调用。携带 llmStreamCall: true 的 compaction/summary 会在其持久日志位置贡献一次调用：回放根据 rawOutput 重建规范块边界，保留已记录的 usage（如有），并提供终止的 stop。该标记将这次本地调用与模板摘要或远程摘要区分开；后两者即使保留了 rawOutput，也未使用此上下文的适配器。
内存中的回放条目遵守完整的 LLM 约定
deriveReplayScript 产出一组 ReplayEntry，即回放监听器按位置服务的内存单元：
{ kind: 'chunks', chunks: StreamChunk[] }
| { kind: 'throw', chunks: StreamChunk[], message: string, code: string }
| { kind: 'hang' }
日志从已结束的 assistant 流和显式标记的压缩（compaction）调用推导分片条目。流开始前的抛出、挂起和外部摘要器调用没有可重建的本地分片表示，因此这些场景提供 replay.override.json。throw 条目可以包含前缀分片以模拟流中途失败。显式覆盖避免了从有损的轮次结束原因或单独的提供方输出推断适配器行为。
位置式回放，单个在途流
回放是位置式的，因此每个场景只允许一个在途模型流。并发会话快照需要按请求键索引的条目。调用顺序变更需要重新录制，fixture 缺失或耗尽时立即报错。
录制采集日志；无密钥回放需要无提供方的配置
记录模式使用真实 llm-deepseek 适配器和配置为 persistenceCompression: 'none' 的 JSONL 持久化后端运行场景，再把生成的 .jsonl 投影到场景目录。显式 raw 模式让采集日志保持逐行可读，而普通部署使用后端的压缩默认值；符合条件的分片连续段仍使用默认的打包存储行。逐事件追加具有持久性，但 harness 会在采集前优雅关闭子进程（关闭 stdin → await ctx.dispose()），以确保最终事件已刷出。llm-replay 本身不执行记录——它只负责回放。
回放使用 cordis.snapshot.yml overlay，以 llm-replay 替换真实适配器，同时保留实际组合。记录使用普通配置和由 harness 提供的持久化根目录。回放模式跳过 .env 加载，因此意外存在的 API 密钥不会触发真实调用。参见单一来源配置 Agent Note。
两个表面：归一化后比对
快照运行断言两个归一化后的表面，因为 harness 的外部表面是不同的：
- stdout transcript——自动化客户端收到的、分帧后的 ACP JSON-RPC 响应与已提交的消息更新。它捕获传输约定的回归，与已提交的
stdout.expected.jsonl比较。 - 重新持久化的会话 JSONL，经过规范化后与
session.jsonl比较。同一 fixture 同时作为回放来源和预期日志。提示词与工具的主体内容会被清理；每种请求头类别由一个场景固定余下的请求头序列。该 pin 默认拥有可读的提示词与工具 schema 伴随文件；当完整的对应序列相同时，也可将另一个 pin 指定为其中任一来源，因此每个不同的伴随文件版本只提交一次。fixture 守卫会拒绝重复的伴随文件内容，录制/刷新会拒绝生成不同字节的共享引用方。最初的请求头固定理由保留在请求头固定 Agent Note中。Override 场景仅从其伴随文件派生模型行为。
两个表面互补：stdout 覆盖精简的自动化协议格式，JSONL 覆盖协议格式有意省略的循环、工具和边界结构。
规范化会替换会话、cwd、协议 id、时间戳、路径和进程易变值；fixture 投影会省略正文序号／时间 envelope，而不修改 payload 引用。录制与刷新还会在回放 fixture 中将生成的 workspace 及其文件系统解析出的别名存储为 {{cwd}}，使平台临时根目录和随机 basename 不影响录制结果；手工编写的临时路径与显式 workspaceParent 下的 cwd 值仍保留字面值。场景把真实 bash 使用限制在稳定命令上。stdout 预期输出仍是符合协议格式的 JSONL，每个原始行都必须可解析为 JSON。普通 Vitest 快照更新只写入 stdout 预期输出；回放 fixture 的写入由显式 record 和 refresh 模式负责。
隔离：当前靠归一化，后续可加沙箱
工具确定性来自生成的 cwd、清理后的环境、全新的非登录 shell、受限命令和规范化。cwd 默认为平台临时目录；当临时目录是始终可写的策略根，而行为需要独立项目位置时，场景可以改为提供其父目录。并发回放运行各自拥有独立 cwd、持久化目录和定长且按场景键区分的 spill 根目录，因此一个场景的清理操作无法删除另一个场景仍在进行的完整输出恢复，同时真实路径预览预算保持稳定。该层不声称提供 OS 级隔离。如果需要更强层级，沙箱执行器可以通过现有能力 seam替换本地后端。
回放插件是独立的包
@deepseek-ai/dsh-llm-replay 是一个支撑包，而非示例本地的胶水代码。它通过用从 JSONL 重建的流短路 llm/stream 来替换真实适配器，其包级放置使回放逻辑处于正常覆盖率门禁之下。
两个子命令，回放在默认门禁中
pnpm run test:snapshot 无需密钥即可回放已提交 fixture；test:snapshot:record 使用真实 API，并重写投影后的会话快照与 stdout 预期输出。同一无密钥门禁会通过 session 头记录发现仓库中的 JSONL，并拒绝与共享编解码器的投影后规范打包表示不同的任何 fixture。缺少 fixture 时会明确报错。每个场景都包含 input.json、stdout.expected.jsonl 和 session.jsonl；不调用模型的情况使用仅含头记录的日志。只有标记为 overridden 的场景才需要 replay.override.json，因为它一旦存在就会取代派生回放。fixture 守卫会拒绝缺失、不匹配和孤立文件。两个命令都接受场景过滤器。
曾考虑的替代方案
- 手工编写包含模型分片的
llm.json——早期草案；复用真实会话日志，使 fixture 成为系统的真实产物而非手工构建的 mock，并让它同时充当行为预期输出。 - 为每个压缩摘要强制提供回放 override——否决：持久摘要事件已经固定成功本地调用的位置、完整输出与可选 usage。显式的本地调用标记保留了这份单一来源 fixture，而不会为模板摘要器或远程摘要器凭空构造调用。
- 字节级 HTTP 录制库（Polly/nock/MSW）：否决。与适配器耦合，处理流式 SSE（Server-Sent Events）时笨拙，且层级低于被测对象。
- 从
turn/end {kind:'error'|'aborted'}合成抛错/取消条目：否决。这会将llm-replay耦合到 loop 内部的轮次关闭语义，且turn/end原因是有损的（无法区分抛出的 401 与 finish-error）；显式的replay.override.json伴随文件是更清晰的 seam。 - 在每个类别 pin 旁复制两个请求头伴随文件：否决。提示词与工具 schema 的组合各自独立变化，因此一个共享组件发生变更，就会使不相关类别 pin 中字节完全相同的文件产生无意义改动。显式的分组件来源可在不重复内容的情况下，为每个类别保留一个结构性 pin。
后果
该测试层为每个场景增加经过评审的输入、会话、stdout、可选 override 和可选 workspace fixture，并为每个不同的已固定提示词序列、每个不同的已固定工具 schema 序列各增加一个文件。记录与回放都会把 workspace seed 复制到生成的 cwd。作为回报，该层通过真实 Loader 和工具组合提供确定性的无密钥覆盖，其中包括一个组装后的上下文溢出恢复场景，其带标记的压缩摘要提供辅助调用。保留下来的大多数场景测试的是组装后的后端而非 ACP；仅面向自动化的 ACP 决策将该语料保留在此处，直至它能够在不损失覆盖的情况下迁移到传输无关的 headless 套件。
本 Agent Note 与拟议的确定性 Agent Note相关，但不取代它：该提案的「通用回放 fixture」在每次测试后重新派生会话消息历史（内部一致性不变量），而这些快照固定组装后的行为与外部自动化输出。在后端语料迁出 ACP 之前，两者相互补充。