JSONL persistence compression latency
JSONL 持久化压缩延迟
Physical persistence optimizations need to reduce retained storage without moving disproportionate work into full writes, reads, or Session forks. The original 105-Session corpus showed that JSONL level-19 compression made full writes and forks more than twice as slow. The decision needs evidence from more varied sessions, including long event streams and payloads outside the original corpus. The expanded corpus cont
English
Problem
Physical persistence optimizations need to reduce retained storage without moving disproportionate work into full writes, reads, or Session forks. The original 105-Session corpus showed that JSONL level-19 compression made full writes and forks more than twice as slow.
The decision needs evidence from more varied sessions, including long event streams and payloads outside the original corpus. The expanded corpus contains 501 real sessions, 16,153,332 logical events, and 2,002,145,570 bytes of serialized event data.
Decision
Storage encoding stays physical and independently decodable
JSONL stores strictly increasing sourceEventSeqs as mixed scalar values and inclusive ranges; other orders remain verbatim. Reading restores the original number[] before exposing an event.
JSONL uses the standard Zstandard level
The JSONL writer keeps one checksummed Zstandard frame per durable append batch but uses the compressor's standard level. Lossless sourceEventSeqs range encoding remains active. Frames stay independently decodable for suffix reads and torn-tail recovery; only the expensive level-19 search is removed.
Expanded benchmark
Each candidate was rebuilt five times from the same 501-session corpus with 512-event append batches. Their order rotates between rounds so every candidate occupies each run position once. Each build runs three complete and suffix-read sweeps. For each displayed metric, the highest and lowest build are discarded and the remaining three values are averaged. Complete and suffix read times cover one sweep over all sessions, and fork time covers all 501 sessions.
| Backend | Stored size | Full write | Full read | Suffix read | Fork |
|---|---|---|---|---|---|
JSONL master | 172.43 MB | 200.902 s | 8.033 s | 24.479 s | 72.670 s |
| JSONL with provenance ranges | 148.15 MB (-14.1%) | 197.281 s (-1.8%) | 7.799 s (-2.9%) | 24.582 s (+0.4%) | 72.308 s (-0.5%) |
| JSONL with provenance ranges and level 19 | 130.22 MB (-24.5%) | 329.442 s (+64.0%) | 7.764 s (-3.3%) | 24.454 s (-0.1%) | 166.177 s (+128.7%) |
Relative to standard-level frames with provenance ranges, level 19 saves another 12.1% of the JSONL bytes but increases full-write time by 67.0% and fork time by 129.8%. Its complete and suffix reads change by -0.4% and -0.5%. The extra search therefore benefits retained size without improving the latency-sensitive operations enough to offset its repeated encoding cost.
Alternatives considered
Keep JSONL level 19. Rejected. On the expanded corpus it saves another 12.1% relative to default-level frames but increases full-write time by 67.0% and fork time by 129.8%, while complete and suffix reads differ by less than 1%. Default-level frames plus provenance ranges retain a 14.1% size reduction relative to master without a material latency regression.
Compress one whole JSONL log as a single frame. Rejected. It improves cross-batch compression but makes suffix reads decompress from the start and removes batch-local torn-tail recovery.
Deduplicate event content. Rejected. Message restatements and tool arguments can be reconstructed only under assumptions that compaction, retries, and pruning may invalidate. Physical compression preserves every event without adding reconstruction semantics.
Consequences
JSONL keeps the low-cost provenance optimization without the level-19 write and fork penalty. The expanded corpus measures a 14.1% retained-size reduction from provenance ranges without a material latency regression.
Related
- JSONL-only first-party Session persistence — owns deletion of the alternative authoritative backend; the archived SQLite compression record retains its historical measurements.
- zstandard-jsonl-session-logs — owns the checksummed frame-per-batch container and the standard compressor-level policy restored here.
中文
问题
物理持久化优化需要减少保留存储，同时不能把不成比例的工作转移到完整写入、读取或 Session fork。原有的 105-Session 语料显示，JSONL level-19 压缩会让完整写入与 fork 耗时增加一倍以上。
该决策需要来自更多样会话的证据，包括长事件流与原语料之外的 payload。扩展后的语料包含 501 个真实会话、16,153,332 个逻辑事件与 2,002,145,570 字节序列化事件数据。
决策
存储编码保持为物理层行为并可独立解码
JSONL 把严格递增的 sourceEventSeqs 存为标量值与闭区间的混合数组，其他顺序保持原样。读取时会在暴露事件前还原原始 number[]。
JSONL 使用 Zstandard 标准级别
JSONL 写入方继续为每个持久 append 批次写入一个带 checksum 的 Zstandard frame，但使用压缩器的标准级别。无损 sourceEventSeqs 区间编码继续生效。各 frame 仍可独立解码，以支持后缀读取与撕裂尾部恢复；只移除昂贵的 level-19 搜索。
扩展基准
每个候选方案都从同一份 501 会话语料独立重建五次，每个 append 批次包含 512 个事件。各轮轮换执行顺序，使每个候选方案在每个运行位置各出现一次。每次重建执行三轮完整读取与后缀读取。下表中的每项指标都去掉最高与最低的一次重建，再平均其余三次。完整读取与后缀读取耗时覆盖对全部会话的一轮扫描，fork 耗时覆盖全部 501 个会话。
| 后端 | 存储大小 | 完整写入 | 完整读取 | 后缀读取 | Fork |
|---|---|---|---|---|---|
JSONL master | 172.43 MB | 200.902 s | 8.033 s | 24.479 s | 72.670 s |
| JSONL + 来源区间 | 148.15 MB (-14.1%) | 197.281 s (-1.8%) | 7.799 s (-2.9%) | 24.582 s (+0.4%) | 72.308 s (-0.5%) |
| JSONL + 来源区间 + level 19 | 130.22 MB (-24.5%) | 329.442 s (+64.0%) | 7.764 s (-3.3%) | 24.454 s (-0.1%) | 166.177 s (+128.7%) |
相对使用来源区间的标准级别 frame，level 19 可再减少 12.1% 的 JSONL 字节，但会让完整写入增加 67.0%、fork 增加 129.8%；完整读取与后缀读取分别变化 -0.4% 与 -0.5%。因此，更深入的搜索只改善保留体积，无法通过延迟敏感操作的收益抵消反复付出的编码成本。
考虑过的替代方案
保留 JSONL level 19。 不予采用。在扩展语料上，它相对默认级别 frame 可再减少 12.1%，却让完整写入增加 67.0%、fork 增加 129.8%，而完整读取与后缀读取的差异都不足 1%。默认级别 frame 配合来源区间后，相对 master 仍能缩小 14.1%，且没有实质性延迟退化。
把整份 JSONL 日志压成单个 frame。 不予采用。该方案可改善跨批次压缩，但后缀读取必须从头解压，也会失去按批次恢复撕裂尾部的能力。
对事件内容去重。 不予采用。消息复述与工具参数只能在依赖重建假设时删除，而 compaction、重试和修剪可能让这些假设失效。物理压缩保留每个事件，不增加重建语义。
后果
JSONL 保留低成本来源优化，同时避开 level-19 的写入与 fork 代价。扩展语料显示，来源区间让保留体积缩小 14.1%，且没有实质性延迟退化。
相关资料
- JSONL-only first-party Session persistence——负责删除另一种权威 backend；已归档 SQLite 压缩记录保留其历史测量。
- zstandard-jsonl-session-logs — 定义带 checksum 的按批次 frame 容器，以及本笔记恢复的标准压缩级别策略。