Zstandard JSONL session logs
Zstandard JSONL 会话日志
The JSONL persistence backend keeps every `SessionEvent` verbatim, including high-volume `assistant/chunk` records. Raw text makes logs inspectable but spends storage and I/O on repeated JSON keys and model text. Compression must retain the existing append/fsync commit boundary, collision-safe first materialization, crash repair, and metadata-only listing; rewriting a whole compressed file after every turn would disc
English
Problem
The JSONL persistence backend keeps every SessionEvent verbatim, including high-volume assistant/chunk records. Raw text makes logs inspectable but spends storage and I/O on repeated JSON keys and model text. Compression must retain the existing append/fsync commit boundary, collision-safe first materialization, crash repair, and metadata-only listing; rewriting a whole compressed file after every turn would discard those properties.
The encoding also has to remain explicit at the deployment boundary. Snapshot fixtures and external line readers require raw JSONL, while a backend cannot safely guess between compressed and raw artifacts in one root or silently migrate pre-release session data.
Decision
Configuration and suffix ownership
dsh-session-persistence-jsonl accepts compression?: 'zstd' | 'none' and explicitly resolves omission to 'zstd'. Zstandard artifacts end in .jsonl.zstd; 'none' retains the original newline-delimited UTF-8 .jsonl representation. SessionLocation.kind remains 'jsonl', because both encodings carry the same logical record format, and SESSION_FORMAT_VERSION remains 0 under the repository's pre-release reject-without-migration policy.
Each persistence root belongs to one encoding. A one-time discovery preflight rejects any opposite suffix, and targeted load, live-adoption, listing, and materialization paths repeat the relevant suffix check after an initially empty preflight. The error names the incompatible artifact and directs the deployment to the matching configuration or a separate root. There is no migration, dual read, dual write, or extension-based fallback.
Frame and write path
The compressed artifact is a standard concatenation of independent Zstandard frames: one checksummed frame containing exactly the header line, followed by one checksummed frame for every durable append batch. Normal loop batches are turn commits, so frame boundaries preserve the existing persistence checkpoint without making the storage layer depend on turn event types.
Compression uses Node's built-in zstdCompress and zstdDecompress, available at the repository's Node 22.19 floor. The backend enables ZSTD_c_checksumFlag, otherwise accepts Node's defaults, and exposes neither a compression-level knob nor a new dependency. The API is marked experimental by Node, so the Node 22.19, 24, and 26 compatibility gate exercises the exact helper.
First materialization compresses the two initial frames before opening the temporary file, then writes and fsyncs that file. POSIX publishes it through a collision-safe hard link and directory fsync; Windows publishes it without replacement through MoveFileExW(..., MOVEFILE_WRITE_THROUGH). Later batches are compressed before opening the destination and appended at EOF. A caught write or file-sync failure closes the append handle, reopens the log read/write, truncates to the prior byte length, syncs the rollback, and rethrows so the coordinator can retry the unchanged batch on both platforms.
Read, listing, and crash recovery
A frame-boundary scanner reads the standard magic, variable header fields, block headers and payload sizes, and optional checksum trailer. It does not interpret compressed blocks. Complete frames are independently checksum-validated and passed through the large-session restore pipeline, which owns decoder reuse, cooperative yielding, and incremental JSONL scanning. A checksum/decompression failure in any complete frame, a malformed complete-frame JSONL tail, or invalid frame structure is corruption and rejects.
Listing reads in bounded chunks only until the first complete frame is available, validates and decompresses that header frame, and never reads an event frame. The dedicated header frame therefore preserves metadata-only listing even for very large session logs.
EOF inside the final frame is a recoverable torn tail. After the scanner establishes that boundary, a dedicated prefix decoder uses finishFlush: ZSTD_e_flush so Node emits available plaintext without requiring frame or checksum completion; every complete newline-terminated event it emits is retained. Repair truncates from that frame's starting byte and appends one new checksummed frame containing the recovered complete events followed by the coordinator's synthetic tool, step, and turn closers. If the tear occurs before any complete event is decodable, repair drops the partial frame and retains all prior complete frames.
Consumers and verification
The CLI, ACP, and stdio app bundles expose symmetric persistenceCompression pass-through configuration. The web host assembly and ordinary app compositions omit the option and use the compressed default. Snapshot recording and replay compositions select 'none' explicitly because committed fixtures are raw JSONL inputs to replay and normalization.
The shared persistence and coordinator contracts run against both encodings. Backend tests cover standard framing and checksum interoperability, header-only listing, append rollback, encoding mismatch rejection, complete-frame corruption, and final-frame tears through headers, blocks, and checksum trailers. Default runtime, built-bin, headless, ACP, and Python smokes assert the compressed suffix and Zstandard magic or decode the header; raw-content tests opt out explicitly.
Alternatives considered
- One frame per JSONL record — rejected because it multiplies frame headers and checksums for high-volume chunk events and makes a physical boundary unrelated to the durable append batch.
- Rewrite one whole compressed stream after every append — rejected because cost grows with log size and replacement would give up append/fsync rollback and the established collision-safe materialization mechanics.
- Use a streaming compressor across appends — rejected because an interrupted encoder state does not leave independently checksummed append units, complicating bounded listing and frame-start repair.
- Add an external native Zstandard dependency — rejected because the supported Node floor already provides the required codec; another native artifact would enlarge installation and executable-packaging risk without adding a required behavior.
- Expose compression level or keep raw JSONL as the default — rejected because there is no deployment evidence for a second tuning policy, while
'none'preserves the line-readable path for fixtures and integrations that need it.
Consequences
- Ordinary session roots store
.jsonl.zstdand retain append-only, fsync, rollback, and interrupted-turn recovery semantics. - Raw JSONL remains a deliberate configuration, but changing encoding requires a fresh/separate root or selecting the mode that matches existing artifacts.
- One frame per durable batch adds bounded framing/checksum overhead and allows header-only listing plus repair from an exact append boundary.
- External tools must understand concatenated Zstandard frames or consume raw-mode artifacts; generic one-shot Node decompression reads only the first independent frame, so backend reads walk frames through the restore pipeline.
- The implementation depends on Node's experimental built-in Zstandard API without an npm dependency; the supported-version compatibility gate makes drift visible.
中文
问题
JSONL 持久化后端会逐字保留每个 SessionEvent，其中包括数量庞大的 assistant/chunk 记录。原始文本便于检查，但重复的 JSON 键和模型文本会增加存储与 I/O 开销。压缩编码必须保留既有的 append/fsync 提交边界、首次物化时的无冲突发布、崩溃修复以及仅元数据列举；如果每轮都重写整个压缩文件，就会失去这些属性。
编码还必须在部署边界上保持显式。快照 fixture（测试前置数据）与外部逐行读取器需要原始 JSONL，而后端无法在同一根目录中安全猜测压缩产物与原始产物，也不能静默迁移预发布会话数据。
决策
配置与后缀归属
dsh-session-persistence-jsonl 接受 compression?: 'zstd' | 'none'，并将省略值显式解析为 'zstd'。Zstandard 产物使用 .jsonl.zstd 后缀；'none' 保留原有的换行分隔 UTF-8 .jsonl 表示。SessionLocation.kind 仍为 'jsonl'，因为两种编码承载同一逻辑记录格式；按照仓库的预发布拒绝且不迁移策略，SESSION_FORMAT_VERSION 仍为 0。
每个持久化根目录只归属于一种编码。一次性的发现预检会拒绝任何相反后缀，而针对性的加载、活跃采用、列举与物化路径会在最初空目录预检之后再次执行对应后缀检查。错误会指出不兼容产物，并要求部署选择匹配配置或单独根目录。系统不提供迁移、双重读取、双重写入或基于扩展名的兜底。
帧与写入路径
压缩产物是标准独立 Zstandard 帧的串联：第一个带校验和的帧只包含头部行，后续每个持久追加批次各占一个带校验和的帧。正常 agent loop（智能体循环）批次就是轮次提交，因此帧边界保留既有持久化检查点，同时不让存储层依赖轮次事件类型。
压缩使用 Node 内置的 zstdCompress 与 zstdDecompress，仓库最低支持的 Node 22.19 已提供这些 API。后端启用 ZSTD_c_checksumFlag，其余采用 Node 默认值，不公开压缩级别调节项，也不增加依赖。Node 将该 API 标记为实验性，因此 Node 22.19、24 与 26 兼容性门禁会执行同一个辅助实现。
首次物化会在打开临时文件之前压缩两个初始帧，然后写入该文件并执行 fsync。POSIX 通过避免冲突的硬链接和目录 fsync 发布该文件；Windows 通过 MoveFileExW(..., MOVEFILE_WRITE_THROUGH) 在不替换目标文件的情况下发布。后续批次也会先压缩，再打开目标并在 EOF 追加。捕获到写入或文件同步失败时，后端会关闭追加句柄，以读写方式重新打开日志，截断到原有字节长度，同步回滚结果，再重新抛出错误，让协调器能够在两个平台上重试未变化的批次。
读取、列举与崩溃恢复
帧边界扫描器会读取标准魔数、可变头字段、块头与负载长度，以及可选校验和尾部，但不会解释压缩块。完整帧会独立验证校验和，再进入大型会话恢复流水线；该流水线负责复用解码器、协作式让出事件循环和增量扫描 JSONL。任何完整帧的校验和或解压失败、完整帧中畸形的 JSONL 尾部，或者无效帧结构都属于损坏并拒绝加载。
列举只按有界分片读取到第一个完整帧可用为止，验证并解压该头部帧，绝不读取事件帧。因此，即使会话日志很大，专用头部帧仍能维持仅元数据列举。
最终帧内部遇到 EOF 属于可恢复的撕裂尾部。扫描器确定该边界后，专用前缀解码器会使用 finishFlush: ZSTD_e_flush，使 Node 不必等到帧结束或读到完整校验和就能产出已有明文；其中每个完整且以换行结束的事件都会保留。修复从该帧起始字节截断，再追加一个新的带校验和帧，其中依次包含恢复出的完整事件，以及协调器生成的工具、步骤与轮次闭合事件。如果撕裂位置尚不足以解码任何完整事件，修复会丢弃该不完整帧并保留此前全部完整帧。
消费方与验证
CLI（命令行界面）、ACP（Agent Client Protocol）与 stdio 应用包公开对称的 persistenceCompression 透传配置。web 宿主装配与普通应用组合省略该选项并使用压缩默认值。快照录制与回放组合显式选择 'none'，因为提交的 fixture 是回放与规范化过程使用的原始 JSONL 输入。
共享持久化约定与协调器约定会针对两种编码运行。后端测试覆盖标准帧与校验和互操作性、仅头部列举、追加回滚、编码不匹配拒绝、完整帧损坏，以及横跨头部、块和校验和尾部的最终帧撕裂。默认运行时、构建后二进制、headless、ACP 与 Python 冒烟测试会断言压缩后缀与 Zstandard 魔数，或解码头部；读取原始内容的测试则显式退出压缩。
考虑过的替代方案
- 每条 JSONL 记录一个帧——不予采纳，因为它会让大量分片事件各自承担帧头与校验和开销，并让物理边界脱离持久追加批次。
- 每次追加都重写一个完整压缩流——不予采纳，因为成本会随日志大小增长，而且替换操作会放弃追加/fsync 回滚和既有的无冲突物化机制。
- 跨追加使用流式压缩器——不予采纳，因为编码器状态中断后不会留下可独立校验的追加单元，从而使有界列举与按帧起点修复更复杂。
- 增加外部原生 Zstandard 依赖——不予采纳，因为受支持的 Node 最低版本已经提供所需编解码器；另一个原生产物会增加安装与可执行文件打包风险，却不增加必需行为。
- 公开压缩级别或继续默认使用原始 JSONL——不予采纳，因为没有部署证据支持第二种调节策略，而
'none'已为需要逐行读取的 fixture 与集成保留路径。
后果
- 普通会话根目录存储
.jsonl.zstd，并保留仅追加、fsync、回滚与中断轮次恢复语义。 - 原始 JSONL 仍是显式配置，但切换编码需要使用全新或单独根目录，或者选择与既有产物匹配的模式。
- 每个持久批次一个帧会增加有界的帧与校验和开销，同时支持仅头部列举和从精确追加边界开始修复。
- 外部工具必须理解串联的 Zstandard 帧，或者消费原始模式产物；Node 通用的一次性解压只读取第一个独立帧，因此后端读取会通过恢复流水线遍历各帧。
- 实现依赖 Node 的实验性内置 Zstandard API，但不增加 NPM 依赖；受支持版本兼容性门禁会暴露 API 漂移。