Gate-runner fail-fast
门禁运行器快速失败
The gate scheduler in `scripts/run-gates.ts` runs every independent gate in an aggregate to completion and reports `run-gates: N passed, M failed`. A gate failure does not stop the remaining gates; only gates whose `needs` dependency failed are skipped. On an aggregate that is already red, the remaining gates keep consuming runner time and produce evidence that cannot change the verdict. The largest single cost is th
English
Problem
The gate scheduler in scripts/run-gates.ts runs every independent gate in an aggregate to completion and reports run-gates: N passed, M failed. A gate failure does not stop the remaining gates; only gates whose needs dependency failed are skipped. On an aggregate that is already red, the remaining gates keep consuming runner time and produce evidence that cannot change the verdict. The largest single cost is the instrumented coverage run in the ci-coverage aggregate, which has taken about 27 minutes; in ci-consumers, the Node compatibility smoke runs independently of the build, so it keeps running after a build failure that already settles the verdict.
GitHub Actions provides no native cross-job cancellation: fail-fast applies only inside a matrix, and the all checks passed aggregate settles only after every needed job finishes, so it cannot cancel siblings early. The only in-repository lever is the gate scheduler itself.
Decision
run-gates.ts accepts a fail-fast scheduling option. When enabled, the first blocking gate failure (a gate whose allowFailure is not true) aborts the aggregate: the shared AbortSignal terminates every running gate's process tree, and every not-yet-run gate is recorded as skipped with the error aborted by fail-fast: <label> failed (or aborted by fail-fast: host interruption when the host signal aborted the run). The exit status remains 1, including when a killed child traps the signal and exits zero: such a result carries the abort mark and is recorded skipped, never passed. A gate that settled before the abort took effect keeps its real result, so the summary stays truthful about what produced evidence.
Termination covers the whole tree, not just the direct pnpm wrapper: POSIX signals the detached child's process group (kill(-pid, SIGTERM), escalating unconditionally to SIGKILL after 5 seconds) and additionally signals every transitive descendant read from the process table, so the detached leaves of a nested run-gates (the check:node-compat and check:ci:lint:contracts-ready gates inside ci-consumers) are killed without relying on the inner scheduler's own escalation. The descendant list is primed at spawn and refreshed every 5 seconds while the child runs, each tick merging the fresh enumeration into the live-filtered cache (the enumeration is asynchronous — a slow WMI/CIM call is bounded by its own 10-second timeout and never blocks the gate's output draining or exit handling; an enumeration still in flight when the gate settles is cancelled rather than left holding stdio handles, and a snapshot that settles after the child exited is still merged, because the child may be gone while a grandchild keeps close pending — exactly when the abort needs the list) so a descendant reparented by an exited intermediate stays tracked across ticks; at abort the fresh enumeration is merged into the same list, then re-signalled on the escalation, because the group kill reaps the direct child and reparents its detached descendants, making them unreachable by parent id afterwards; settlement waits until the group and the captured descendants are gone. On the abort path only, a bounded pipe-drain timer force-closes the stdio streams 10 seconds after the abort, so a descendant holding the write ends (uninterruptible I/O included) cannot keep close pending to the job timeout; ordinary runs keep waiting rather than report passed over a live leak. Windows runs taskkill /PID <pid> /T /F immediately, because a taskkill without /F does not terminate console processes, which is what gate commands are; the same process-table enumeration as POSIX supplies a descendant list there, and each captured descendant is also taskkilled, because a taskkill /T rooted at a pid that already exited finds nothing. Windows never reparents, so an exited root's descendants keep it as their parent and remain reachable through the table; the same sampler cadence as POSIX keeps the cache crossing a vanished intermediate's table record (the enumeration is bounded by a 10-second PowerShell timeout so a hung WMI/CIM call cannot stall the abort path). Without this, Windows has no signal forwarding and a wrapper-only kill would orphan the script tree on the shared self-hosted pool. Children are detached into their own POSIX process group only when fail-fast is enabled; ordinary runs keep them in the host group so terminal Ctrl+C still reaches them. Host SIGINT/SIGTERM on a fail-fast run is forwarded to the abort path, so an interrupted or runner-cancelled run drains and kills its gate trees instead of orphaning them.
The option is enabled through DSH_GATE_FAIL_FAST (accepted values: 1 or unset; anything else fails loud through the existing flagEnabled contract) on every run-gates aggregate job in ci.yml: the three blocking Linux jobs (node-24 static, node-24-coverage, node-24-consumers), the Node compatibility matrix (node-compat), and the two native Windows lanes that drive aggregates (windows-build, windows-coverage). scripts/ci-workflow.spec.ts pins the flag on those jobs and pins its absence on windows-observational, so removing it fails the CI gate.
The windows-observational lane stays complete: it is continue-on-error by design and exists to collect as much Windows-native evidence per run as possible, so the first failure must not truncate the rest. The windows Wine lane and windows-native-tests run a single script or Vitest command rather than a run-gates aggregate, so the scheduler option does not apply to them. The master serial standby lanes (serial-linux-selfhosted, serial-windows) and the manual runner benchmarks do not set the flag: they are completeness drills that must execute the full aggregate to prove pool readiness.
Consequences
A red pull-request run ends sooner. The largest saving is in ci-coverage: a failing exempt-heavy gate aborts the multi-minute instrumented coverage gate instead of letting it run out.
The trade-off is diagnostic: one push returns only the first blocking failure instead of the full failure set, so resolving several independent failures may take more push-fix rounds. Killed gates are recorded as skipped with the fail-fast error, so the summary line N passed, M failed, K skipped remains truthful about what produced evidence and what did not. A gate that ignores SIGTERM is force-killed after the 5-second grace. A tree that survives both signals holds the aggregate only while its direct child's stdio stays open; once close fires, the group-liveness poll gives up after 8 seconds and the run settles with a loud gate tree not quiescent warning instead of reporting a clean tree.
Alternatives considered
Cross-job cancellation watchdog. A job that polls sibling conclusions and calls the run-cancel API would stop all lanes on the first failure. It is not native, adds a polling dependency and token surface, and discards the parallel evidence other jobs have already produced. Rejected; fail-fast at the scheduler is orthogonal to the job topology and carries none of that.
Consolidating the three Linux jobs into one check. A single job could fail fast natively, but it would lose the independent runner allocation whose queue-delay overlap is documented in the independent CI consumer build note, and it would make the coverage long tail the tail of the whole job. Rejected; fail-fast applies within the existing job split instead.
Signaling only the direct child. The first implementation sent SIGTERM to the pnpm wrapper and relied on pnpm forwarding it to the script child. A probe confirmed the forwarding on POSIX. Rejected: Windows has no signal forwarding, and a wrapper-only kill orphans the script tree on the shared self-hosted pool; the tree termination above covers both platforms.
中文
问题
scripts/run-gates.ts 中的门禁调度器会把一个聚合流程里的每个独立门禁都跑完,然后报告 run-gates: N passed, M failed。某个门禁失败并不会停止其余门禁;只有 needs 依赖失败的门禁会被跳过。当聚合流程已经确定失败时,剩余门禁仍在消耗运行器时间,产出的证据也无法改变结论。最大的单项成本是 ci-coverage 聚合流程里的插桩覆盖率运行,实测约 27 分钟;在 ci-consumers 中,Node 兼容性冒烟检查与构建相互独立,因此它会在构建失败、结论已定的情况下继续运行。
GitHub Actions 不提供原生的跨作业取消:fail-fast 只作用于矩阵内部,而 all checks passed 聚合流程要等所有必需作业结束后才落定,因此它无法提前取消兄弟作业。仓库内唯一可用的杠杆就是门禁调度器本身。
决策
run-gates.ts 接受一个快速失败调度选项。启用后,首个阻塞性门禁失败(allowFailure 不为 true 的门禁)即中止整个聚合流程:共享的 AbortSignal 终止每个运行中门禁的整个进程树,所有尚未运行的门禁记录为 skipped,错误信息为 aborted by fail-fast: <label> failed(当宿主信号中止运行时为 aborted by fail-fast: host interruption)。退出状态保持为 1——即使被终止的子进程捕获信号并以 0 退出,这类结果也带有中止标记、记为 skipped,绝不记为通过。在中止生效前已结算的门禁保留其真实结果,因此汇总行仍如实反映哪些门禁产出了证据。
终止覆盖整棵进程树,而不只是直接的 pnpm 包装进程:POSIX 向分离子进程所在的进程组发信号(kill(-pid, SIGTERM),5 秒后无条件升级为 SIGKILL),并额外按进程表逐个信号所有传递性后代——嵌套 run-gates(ci-consumers 里的 check:node-compat 与 check:ci:lint:contracts-ready 门禁)分离出去的叶子进程因此不依赖内层调度器自己的升级也能被杀掉。后代列表在 spawn 时预填、随后在子进程运行期间每 5 秒刷新(枚举是异步的——慢的 WMI/CIM 调用以自身 10 秒超时封顶,绝不会阻塞门禁的输出排空或退出处理;门禁结算时仍在途的枚举会被取消,而不是留着持有 stdio 句柄;子进程退出后才结算的快照仍会被合并,因为子进程可能已消失而孙进程仍持有 close 未触发——这正是中止需要该列表的时刻),每个采样 tick 把新枚举结果并入存活过滤后的缓存,被已退出中间进程 reparent 的后代因此跨 tick 仍被跟踪;中止时把新枚举结果并入同一列表,随后在升级时再次信号:组杀会重收直接子进程并 reparent 其分离后代,之后按父 id 不可达,因此结算要等进程组与已捕获后代都消失。仅在中止路径上,有界 pipe-drain 定时器会在中止后 10 秒强制关闭 stdio 流,因此持有写端的后代进程(包括不可中断 I/O)无法让 close 一直挂到作业超时;普通运行保持等待,而不是在有存活泄漏时报告通过。Windows 立即运行 taskkill /PID <pid> /T /F,因为不带 /F 的 taskkill 无法终止控制台进程,而门禁命令正是控制台进程;与 POSIX 相同的进程表枚举在 Windows 上也提供后代列表,每个捕获的后代同样被 taskkill——因为以已退出 pid 为根执行 taskkill /T 找不到任何东西,而 Windows 从不 reparent,已退出根的子孙仍以它为父、通过进程表可达;与 POSIX 相同的采样节奏让缓存跨过已消失中间进程的进程表记录(枚举以 10 秒 PowerShell 超时封顶,挂起的 WMI/CIM 调用不会拖住中止路径)。没有这一步,Windows 上不存在信号转发,只杀包装进程会在共享自托管池上留下孤儿脚本树。只有在启用快速失败时,子进程才会分离到自己的 POSIX 进程组;普通运行保持子进程在宿主进程组内,终端 Ctrl+C 仍能送达它们。快速失败运行上的宿主 SIGINT/SIGTERM 会转发到中止路径,因此被中断或被运行器取消的运行会排干并杀死自己的门禁进程树,而不是留下孤儿。
该选项通过 DSH_GATE_FAIL_FAST 开启(可接受值:1 或未设置;其它值通过既有的 flagEnabled 契约响亮失败),作用于 ci.yml 中所有由 run-gates 驱动的聚合作业:三个阻塞性 Linux 作业(node-24 静态、node-24-coverage、node-24-consumers)、Node 兼容性矩阵(node-compat)以及两个驱动聚合流程的原生 Windows 车道(windows-build、windows-coverage)。scripts/ci-workflow.spec.ts 固定了这些作业上的该标志,并固定 windows-observational 上不存在该标志,移除它会令 CI 门禁失败。
windows-observational 车道保持完整执行:它按设计是 continue-on-error,存在的意义是每次运行尽量收集 Windows 原生证据,因此首个失败不应截断其余部分。windows Wine 车道和 windows-native-tests 运行的是单个脚本或 Vitest 命令,不是 run-gates 聚合流程,因此调度选项不适用于它们。master 串行备用车道(serial-linux-selfhosted、serial-windows)和手动运行器基准测试不设置该标志:它们是完整性演练,必须执行完整聚合流程以证明池的可用性。
结果
红色拉取请求运行会更早结束。最大节省在 ci-coverage:失败的豁免重型门禁会中止多分钟的插桩覆盖率门禁,而不是让它跑完。
代价在诊断侧:一次推送只返回第一条阻塞性失败,而不是完整失败集,因此解决多个独立失败可能需要更多次推送-修复往返。被终止的门禁记录为带快速失败错误的 skipped,因此 N passed, M failed, K skipped 汇总行仍然如实反映哪些门禁产出了证据、哪些没有。忽略 SIGTERM 的门禁会在 5 秒宽限期后被强制终止。同时扛过两种信号仍存活的进程树,只有在直接子进程的 stdio 保持打开时才会拖住聚合流程;一旦 close 触发,进程组存活轮询在 8 秒后放弃,运行以响亮的 gate tree not quiescent 警告结算,而不是报告一棵干净的树。
备选方案
跨作业取消看门狗。 一个轮询兄弟作业结论并调用运行取消 API 的作业可以在首个失败时停止所有车道。它不是原生能力,会增加轮询依赖和令牌暴露面,并丢弃其它作业已并行产出的证据。已拒绝;调度器层面的快速失败与作业拓扑正交,不带来上述任何负担。
把三个 Linux 作业合并成一个检查。 单个作业可以原生快速失败,但会失去独立的运行器分配——其排队延迟重叠的收益记录在独立 CI 消费方构建笔记中——并且会让覆盖率长尾成为整个作业的尾部。已拒绝;快速失败在既有作业拆分内生效即可。
只向直接子进程发信号。 第一版实现向 pnpm 包装进程发送 SIGTERM,依赖 pnpm 把它转发给脚本子进程。探针确认了 POSIX 上的转发。已拒绝:Windows 没有信号转发,只杀包装进程会在共享自托管池上留下孤儿脚本树;上述整树终止覆盖两个平台。