Performance baselines for tool-heavy backend continuation
工具密集后端续聊的性能基线
Opening one Session does not measure the repeated cost of preparing model requests after a long tool conversation, executing another tool-heavy turn, or discovering multiple inactive fork children. The [Session-opening gate](2026-09-04-session-open-performance-gate.md) covers first history and activation but deliberately stops before new model work. Its text/reasoning workload also lacks historical tool-call argument
English
Problem
Opening one Session does not measure the repeated cost of preparing model requests after a long tool conversation, executing another tool-heavy turn, or discovering multiple inactive fork children. The Session-opening gate covers first history and activation but deliberately stops before new model work. Its text/reasoning workload also lacks historical tool-call arguments and large tool results.
Decision
The agent-continuation benchmark adds three scenario groups, including a shipped-profile variant, without changing product implementations. They use current-generation Zstandard Sessions authored through production append, stream accumulation, and persistence APIs. A separate seed process creates the deterministic source before measurement; each sample copies that source into its private root and starts a fresh compiled plain-Node worker. No recorded Session, ambient repository, network, private Harness home, or deployed GUI supplies input.
The shared history has 800 completed two-step turns, four tool calls per turn, and 2,048-character tool results: 13,600 events and 5,600 conversation messages. Each assistant reply carries reasoning, text, and compact streamed records; tool replies additionally carry fragmented arguments. Fixed timestamps and ids describe the seed. Live synthetic replies use the real loop's clocks and ids without overriding process globals.
| Case | Timed operation | Endpoint |
|---|---|---|
| Request history | After unmeasured cold resume, deliver 40 sequential text-only turns over the tool-heavy history, then flush | Idle Agent with all 40 model requests completed; reports turn and final-flush time separately |
| Tool continuation | Cold resume, 20 sequential turns with eight parallel-safe synthetic tool calls and a final reply per turn, then flush | Idle Agent with 40 model requests and 160 completed tool executions; reports resume, turns, and final flush separately |
| Shipped SDK workflow | Launch built dsh with the sdk-minimal profile, deliver 100 sequential turns with eight real file-view calls per turn, then close the SDK | SDK receives 200 assistant messages and 800 successful file results; includes Loader boot, stdio JSON-RPC, persistence, and shutdown |
| Child catalog | Observe one cold parent catalog twice through the real Session query and subagent services | Two complete 16-child catalogs with parent observations released; each stored fork child inherits 80 tool-heavy turns, and discovery reads only the parent facts |
The tool execution pipeline, request preparation, Session projections required by those services, persistence, and catalog observations remain production code. Only the model adapter and bounded tool body are synthetic. The adapter retains a request counter, not request objects, so the fixture cannot manufacture a growing retention cost. Sequential input means each idle interval belongs to the one request delivered by this worker; it does not generalize idle to a per-message completion API under concurrent input.
The SDK fixture explicitly inserts fs-local and str_replace_editor through its profile patch. This preserves the calibrated file-view workload independently of the minimal profile's shell-only defaults. File reads, timing endpoints, and budgets remain the same.
Five samples report raw wall time, CPU user/system time, peak RSS, endpoint counts, and the minimum, median, and maximum total wall time. Each aggregate report also records CPU models, available parallelism, platform, architecture, and Node/V8 versions after the timed workers exit. Budgets enforce the unrounded median. Continuation additionally measures retained heap against an initialized Host: two explicit GCs separated by an event-loop yield precede and follow the timed operation, while the idle Agent remains reachable. The measured delta therefore includes the resident historical Session and live additions, not just newly appended turns. GC and teardown are outside timing; flush is inside. Request-history retention starts after resume and is diagnostic only. Catalog peak RSS is diagnostic; no retained-heap budget claims to measure already-released parent observations.
The parent bounds every child to 60 seconds, checks timeout, signal, exit, and report independently, awaits process close, and removes private roots after failures. Context and Agent teardown run in finally blocks. Seed processes cannot warm the measured process's caches. Filesystem caches are not forcibly evicted: cold means a fresh process, not cold physical storage.
Calibration evidence
The implementation reference is the PR #3537 merge on Apple M4 Pro, macOS arm64, Node 24.19.0. Two exclusive five-sample runs use the same seed and no product optimization. Durations below are milliseconds; source expectations round above the observed run medians rather than imposing an unimplemented optimization target.
| Case | Run 1 raw totals | Run 2 raw totals | Medians | Historical M4 expectation | Historical scaled budget |
|---|---|---|---|---|---|
| Request history | 209.134, 210.333, 208.959, 236.355, 238.685 | 222.833, 213.911, 208.089, 211.494, 209.137 | 210.333 / 211.494 | 220 | 550 |
| Tool continuation | 358.953, 324.790, 318.861, 320.119, 322.896 | 324.280, 321.952, 340.409, 325.470, 324.312 | 322.896 / 324.312 | 340 | 850 |
| Child catalog | 318.730, 309.006, 311.404, 308.565, 310.105 | 308.670, 310.030, 280.086, 303.084, 284.829 | 310.105 / 303.084 | 320 | 800 |
Continuation retains approximately 22.295 MiB; its source expectation is 23 MiB and its budget is 28.75 MiB. SDK time expectations use the existing calibration helper: 2× shared CI time scale and 1.25× variance headroom. Request history uses the direct hosted expectation in the request-freeze calibration, without the 2× scale. Memory uses only 1.25× headroom. The scale is inherited from the existing lane's calibration, not a new Linux measurement of these cases; CI evidence remains necessary when runner characteristics change. Baseline budgets protect the measured implementation; tighter budgets belong with a measured behavior-preserving fix.
A separate plain-Node request-history CPU profile attributes 132.876 ms of sampled self time to deepFreeze called by buildRequest during a 211.300 ms operation. This identifies repeated traversal of already-frozen history as a focused investigation target, not a proven optimization result. Catalog first/repeat timings separately report the first and second parent-catalog observations.
The shipped SDK variant completes 100 turns, 200 requests, and 800 real file reads. Its five-sample smoke totals are 1,521.773, 1,463.465, 1,689.701, 1,365.485, and 1,417.106 ms (median 1,463.465 ms); a full-suite repeat reports 1,596.183, 1,784.536, 2,120.082, 1,405.365, and 1,355.894 ms (median 1,596.183 ms). Its 1,700 ms reference expectation yields a 4,250 ms CI budget. The repeat also slows the unchanged service cases, so it is validation under variable host load rather than evidence to relax their exclusive calibration. The SDK process receives an allowlisted environment and private home/workspace. A 40-second deadline starts SDK shutdown; every path awaits the same memoized close promise before the outer worker’s 60-second deadline. Profile timing includes boot, all turns, and shutdown, reported separately; no parent-process CPU or heap metric is presented as server memory. The adapter does not serialize requests for an external model provider.
The first Linux x64 CI measurement ran on VM-7-113-ubuntu-ci-10 with Node 24.18.1 (run 34017868081, attempt 1, job 101444810498). The SDK median was 2,753.441 ms against its 4,250 ms budget, and tool-continuation retained-heap median was 22.274 MiB against 28.75 MiB. Request-history and tool-continuation time budgets failed: 785.498 ms against 550 ms and 1,077.285 ms against 850 ms, respectively. The unchanged Session-reopen open phase also failed at 31.6 ms against 30 ms. Attempt 2, job 101447076381 (run 34017868081) passed every benchmark on the same commit and unchanged budgets, but used VM-7-113-ubuntu-ci-29 with Node 24.19.0. The gate runner suppressed successful child output, so that attempt supplies a passing verdict rather than raw medians. The changed runner and Node version prevent attributing the difference solely to contention or claiming stable repeated CI calibration; neither the budgets nor the shared scale are changed on this evidence.
Catalog uses an explicit 900 ms expected CI duration and only the existing 1.25× headroom, yielding 1,125 ms without applying the reference-machine scale again. The standard two-CPU hosted ubuntu-24.04 run 34033336380, job 101487280801 reports five unchanged-catalog totals of 797.374, 883.157, 858.364, 790.569, and 904.579 ms: median 858.364 ms exceeds the historical 800 ms budget. The 320 ms M4 expectation above remains historical evidence, not a CI measurement. This follows the explicit-CI calibration used by Session reopening (50 ms expected CI); shared factors, workloads, timing endpoints, and product implementations remain unchanged. Deterministic controls use the same assertion as the measured verdict: the unrounded recorded median passes 1,125 ms and fails 800 ms, while a synthetic 1,400 ms median fails 1,125 ms. A passing run on a faster host does not calibrate the standard hosted runner.
Tool continuation also uses a 900 ms expected CI duration with 1.25× headroom (1,125 ms). With unchanged implementation, standard two-CPU hosted run 34034524265, job 101490056074 reports totals of 917.007, 892.091, 887.839, 905.659, and 898.252 ms: median 898.252 ms exceeds the historical 850 ms budget. The 340 ms M4 expectation remains historical evidence. The same measured-verdict assertion accepts the recorded unrounded median under 1,125 ms, rejects it under 850 ms, and rejects a synthetic 1,400 ms regression. Workload, timing, product code, and the 28.75 MiB retained-heap budget remain unchanged.
Baseline request history uses a 600 ms expected CI duration with 1.25× headroom (750 ms). With unchanged implementation, standard two-CPU hosted run 34035306987, job 101492163630 reports totals of 618.598, 618.606, 582.035, 582.304, and 581.832 ms: median 582.304 ms exceeds the historical 550 ms budget. The 220 ms M4 expectation remains historical evidence. The same measured-verdict assertion accepts the recorded unrounded median under 750 ms, rejects it under 550 ms, and rejects a synthetic 900 ms regression. This calibrates the unoptimized baseline only; workload, timing, product code, and memory budgets remain unchanged.
Alternatives considered
Repeat existing migration and first-open variants. Rejected: those twelve cases already distinguish read-only preparation from writable publication. These cases use the current generation and begin or continue actual model work, or enumerate a corpus rather than open one Session.
Measure only deriveMessages. Rejected: its incremental cache does not include complete request freezing, adapter dispatch, live append, or persistence. Actual sequential requests protect the cost the Agent pays per step.
Use only unseeded children with warm projection-cache rows. Rejected: that path bypasses body observations and misses the exact inherited-cut requirement of fork children. The original calibration omits the optional projection cache and reports the seeded fallback path. Current catalog reads observe durable parent facts; the recorded child-body timings describe the calibration implementation.
Apply an optimization and its desired budget together with the first measurements. Rejected: a baseline-only layer remains independently mergeable and records the current workload before attribution or implementation changes. Source constants cannot be overridden by environment variables.
Consequences
The lane adds four cases in three scenario groups and twenty measured workers, plus two seed processes. The integrated continuation case spans resume through completed model/tool work and durable flush. The shipped SDK workflow additionally includes profile boot, SDK transport, real file tools, and shutdown; only its model adapter is synthetic. It starts a fresh Session because the public SDK prompt API creates rather than resumes stored identities. Neither path includes network model latency, provider-specific request serialization, optional user plugins, compaction, failed tool results, images, cancellation, or browser rendering. Functional tests retain responsibility for event contents, immutable messages, tool semantics, fork lineage, and read-only versus writable side effects; endpoint counts prevent timing a skipped workload without duplicating those assertions.
This note supplements, rather than supersedes, the Session-opening gate's isolation and calibration rationale. No existing active decision is retired.
中文
问题
打开一个 Session 不能衡量长工具对话后重复准备模型请求、执行更多工具密集轮次或发现多个非活动 fork 子会话的成本。Session 打开门禁覆盖首屏历史和激活，但有意停在新的模型工作开始前。它的文本与推理负载也不包含历史工具调用参数和大型工具结果。
决定
agent-continuation 基准增加三个场景组，包含一个已发布 profile 变体，不修改产品实现。它们通过生产追加、流累积和持久化 API 构造当前代际的 Zstandard Session。独立播种进程在测量前生成确定性源数据；每个样本将其复制到私有根目录，并启动新的已编译纯 Node worker。输入不来自录制 Session、环境仓库、网络、私有 Harness 主目录或已部署 GUI。
共享历史包含 800 个已完成的双步骤轮次，每轮四次工具调用，工具结果为 2,048 字符：共 13,600 个事件和 5,600 条对话消息。每条助手回复携带推理、文本和紧凑流记录；请求工具的回复还携带分片参数。播种数据使用固定时间戳和 id。实时合成回复使用真实循环的时钟和 id，不覆盖进程全局状态。
| 用例 | 计时操作 | 终点 |
|---|---|---|
| 请求历史 | 在不计时的冷恢复后，向工具密集历史顺序提交 40 个纯文本轮次，然后 flush | 空闲 Agent，已完成全部 40 次模型请求；分别报告轮次和最终 flush 时间 |
| 工具续聊 | 冷恢复，顺序执行 20 个轮次，每轮八次可安全并行的合成工具调用和一条最终回复，然后 flush | 空闲 Agent，已完成 40 次模型请求和 160 次工具执行；分别报告恢复、轮次和最终 flush 时间 |
| 已发布 SDK 工作流 | 使用 sdk-minimal profile 启动已构建 dsh，顺序提交 100 个轮次，每轮八次真实文件查看调用，然后关闭 SDK | SDK 收到 200 条助手消息和 800 个成功文件结果；包含 Loader 启动、stdio JSON-RPC、持久化和关闭 |
| 子会话目录 | 通过真实 Session 查询和 subagent 服务，两次观察一个冷 parent 目录 | 两份完整的 16 子级目录，parent 观察已释放；每个存储的 fork 子级继承 80 个工具密集轮次，发现只读取 parent fact |
工具执行管线、请求准备、这些服务所需的 Session 投影、持久化和目录观察均保留生产代码。只有模型适配器和有界工具体是合成的。适配器只保留请求计数，不保留请求对象，因此 fixture（测试前置数据）不会制造不断增长的保留成本。顺序输入使每个空闲区间对应此 worker 提交的唯一请求；这不代表并发输入时可以把空闲状态推广为逐消息完成 API。
SDK fixture 通过 profile patch 显式插入 fs-local 和 str_replace_editor。这使经校准的文件查看负载不依赖极简 profile 只提供 shell 的默认组合。文件读取、计时终点和预算保持不变。
五个样本报告原始壁钟时间、CPU 用户态／内核态时间、峰值 RSS、终点计数及总壁钟时间的最小值、中位数和最大值。每份汇总报告还在计时 worker 退出后记录 CPU 型号、可用并行度、平台、架构以及 Node/V8 版本。预算约束未经舍入的中位数。续聊还相对已初始化 Host 测量保留堆内存：计时操作前后各执行两次显式 GC，中间让出一次事件循环，空闲 Agent 始终可达。因此该增量包含常驻历史 Session 和实时追加，而不只是新轮次。GC 与资源释放不计时；flush 计时。请求历史的内存基线从恢复后开始，只作诊断。目录峰值 RSS 仅作诊断；没有保留堆预算声称衡量已经释放的父会话观察。
父进程为每个子进程设置 60 秒上限，独立检查超时、信号、退出状态和报告，等待进程关闭，并在失败后删除私有根目录。Context 和 Agent 在 finally 中释放。播种进程无法预热被测进程的缓存。不强制清除文件系统缓存：冷指新进程，不指冷物理存储。
校准证据
实现参考为 Apple M4 Pro、macOS arm64、Node 24.19.0 上的 PR #3537 合并结果。两轮独占的五样本运行使用相同播种数据，没有产品优化。下表时间单位为毫秒；源码期望值向上取整至实测各轮中位数以上，而不是施加尚未实现的优化目标。
| 用例 | 第一轮原始总时间 | 第二轮原始总时间 | 中位数 | 历史 M4 期望 | 历史缩放预算 |
|---|---|---|---|---|---|
| 请求历史 | 209.134, 210.333, 208.959, 236.355, 238.685 | 222.833, 213.911, 208.089, 211.494, 209.137 | 210.333 / 211.494 | 220 | 550 |
| 工具续聊 | 358.953, 324.790, 318.861, 320.119, 322.896 | 324.280, 321.952, 340.409, 325.470, 324.312 | 322.896 / 324.312 | 340 | 850 |
| 子会话目录 | 318.730, 309.006, 311.404, 308.565, 310.105 | 308.670, 310.030, 280.086, 303.084, 284.829 | 310.105 / 303.084 | 320 | 800 |
续聊保留约 22.295 MiB；源码期望值为 23 MiB，预算为 28.75 MiB。SDK 时间期望值使用现有校准辅助函数：2× 共享 CI 时间比例和 1.25× 波动余量。请求历史使用请求历史基准中的直接托管期望值，不乘以 2× 比例。内存只使用 1.25× 余量。比例继承现有通道的校准，并非这些用例的新 Linux 实测值；runner 特征变化时仍需 CI 证据。基线预算保护实测实现；更紧预算属于有测量依据且保持行为的修复。
独立的纯 Node 请求历史 CPU profile 在一次 211.300 ms 操作中，将 132.876 ms 采样自身时间归因于 buildRequest 调用的 deepFreeze。这把重复遍历已冻结历史定位为聚焦调查目标，不是已证实的优化结果。目录首次／重复时间分别报告第一次和第二次 parent 目录观察。
已发布 SDK 变体完成 100 个轮次、200 次请求和 800 次真实文件读取。五样本 smoke 总时间为 1,521.773、1,463.465、1,689.701、1,365.485 和 1,417.106 ms（中位数 1,463.465 ms）；完整套件重复运行报告 1,596.183、1,784.536、2,120.082、1,405.365 和 1,355.894 ms（中位数 1,596.183 ms）。1,700 ms 参考期望对应 4,250 ms CI 预算。重复运行中未改变的服务用例也变慢，因此这是可变主机负载下的验证，不是放宽其独占校准预算的依据。SDK 进程使用白名单环境和私有主目录／工作区。40 秒截止时间启动 SDK 关闭；所有路径等待同一个记忆化 close Promise，并早于外层 worker 的 60 秒截止时间。Profile 时间包含启动、全部轮次和关闭，分别报告；不把父进程 CPU 或堆指标当作服务端内存。适配器不为外部模型服务商序列化请求。
首次 Linux x64 CI 测量使用 VM-7-113-ubuntu-ci-10 和 Node 24.18.1（run 34017868081，attempt 1，job 101444810498）。SDK 中位数为 2,753.441 ms，预算为 4,250 ms；工具续聊保留堆中位数为 22.274 MiB，预算为 28.75 MiB。请求历史与工具续聊时间预算失败：分别为 785.498 ms 对 550 ms、1,077.285 ms 对 850 ms。未修改的 Session 重开 open 阶段也以 31.6 ms 对 30 ms 失败。Attempt 2，job 101447076381 (run 34017868081) 在同一提交和未修改预算下通过全部基准，但使用 VM-7-113-ubuntu-ci-29 和 Node 24.19.0。门禁运行器隐藏成功子进程的输出，因此该次运行只提供通过结论，不提供原始中位数。Runner 与 Node 版本同时变化，不能把差异仅归因于资源争用，也不能宣称已获得稳定的重复 CI 校准；这些证据不改变预算或共享比例。
目录用例使用显式的 900 ms CI 期望时间，仅乘现有 1.25× 余量，得到 1,125 ms，不再应用参考机器比例。标准双 CPU 托管 ubuntu-24.04 的 run 34033336380，job 101487280801 报告未修改目录实现的五个总时间为 797.374、883.157、858.364、790.569 和 904.579 ms：中位数 858.364 ms 超出历史 800 ms 预算。上表 320 ms M4 期望保留为历史证据，不是 CI 测量。此方法与 Session 重开使用的显式 CI 校准一致（CI 期望为 50 ms）；共享系数、负载、计时终点和产品实现均不改变。确定性对照与实测判定使用同一断言：未经舍入的录制中位数通过 1,125 ms 并被 800 ms 拒绝，合成的 1,400 ms 中位数则被 1,125 ms 拒绝。更快主机上的通过结果不能校准标准托管 runner。
工具续聊同样使用 900 ms CI 期望时间与 1.25× 余量（1,125 ms）。未修改的实现在标准双 CPU 托管 run 34034524265，job 101490056074 中报告总时间为 917.007、892.091、887.839、905.659 和 898.252 ms：中位数 898.252 ms 超出历史 850 ms 预算。340 ms M4 期望保留为历史证据。与实测判定相同的断言在 1,125 ms 下接受未经舍入的录制中位数，在 850 ms 下拒绝它，并拒绝合成的 1,400 ms 回退。负载、计时、产品代码和 28.75 MiB 保留堆预算均不改变。
基线请求历史使用 600 ms CI 期望时间与 1.25× 余量（750 ms）。未修改的实现在标准双 CPU 托管 run 34035306987，job 101492163630 中报告总时间为 618.598、618.606、582.035、582.304 和 581.832 ms：中位数 582.304 ms 超出历史 550 ms 预算。220 ms M4 期望保留为历史证据。与实测判定相同的断言在 750 ms 下接受未经舍入的录制中位数，在 550 ms 下拒绝它，并拒绝合成的 900 ms 回退。此校准仅针对未优化基线；负载、计时、产品代码和内存预算均不改变。
考虑过的替代方案
重复现有迁移和首次打开变体。 拒绝：现有十二个用例已经区分只读准备与可写发布。这些用例使用当前代际并开始或继续实际模型工作，或者列举语料集合而不是打开单个 Session。
只测 deriveMessages。 拒绝：它的增量缓存不包含完整请求冻结、适配器分发、实时追加或持久化。实际顺序请求保护 Agent 每一步支付的成本。
只使用投影缓存行已预热的无种子子会话。 拒绝：该路径绕过正文观察，遗漏 fork 子会话的精确继承切点要求。原始校准有意不挂载可选投影缓存，报告带种子的回退路径。parent 自有目录的测量观察持久 parent fact；子级正文读取只属于已记录的校准实现。
将优化及其目标预算与首次测量一起应用。 拒绝：纯基线层可以独立合并，并在归因或实现改变前记录当前负载。环境变量不能覆盖源码常量。
后果
通道增加三个场景组中的四个用例、二十个测量 worker 和两个播种进程。集成续聊用例覆盖恢复、完成模型／工具工作及持久化 flush。已发布 SDK 工作流额外包含 profile 启动、SDK 传输、真实文件工具和关闭；只有模型适配器是合成的。它创建新 Session，因为公共 SDK prompt API 创建而非恢复已存储身份。两条路径均不包含网络模型延迟、服务商专属请求序列化、可选用户插件、压缩、失败工具结果、图像、取消或浏览器渲染。功能测试仍负责事件内容、不可变消息、工具语义、fork 谱系以及只读／可写副作用；终点计数防止把跳过的工作当作测量结果，不重复这些断言。
本记录补充而非取代 Session 打开门禁的隔离和校准依据。不退役任何现有活跃决策。