CI failover runbook — hosted pools → in-house pool
CI 故障切换手册 — 托管池 → 自有池
The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) run on the hosted enterprise 32-core pools; the required verdict job that aggregates them (`all checks passed`) runs on standard `ubuntu-latest`; the [native Windows jobs](2026-08-08-native-windows-pull-request-ci.md) run on the hosted `dsh-windows-2025-16c
English
Problem
The three required Linux worker jobs in CI (node 24 / static, node 24 / coverage, node 24 / snapshots and artifacts) run on the hosted enterprise 32-core pools; the required verdict job that aggregates them (all checks passed) runs on standard ubuntu-latest; the native Windows jobs run on the hosted dsh-windows-2025-16core larger runner. When the enterprise pools degrade — jobs queue indefinitely or the enterprise labels vanish — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. Scope: two independent switches, one per platform. DSH_CI_FAILOVER_LINUX recovers an enterprise Linux-pool outage (the three required Linux workers plus the all checks passed verdict); DSH_CI_FAILOVER_WINDOWS recovers a hosted Windows-pool outage (the native Windows jobs). A Linux-pool outage need not retarget Windows jobs and vice versa. The Node compatibility jobs also follow the Linux switch with isolated setup; the verdict's node-24-bench, python-sdk, and python-runtime dependencies stay on standard hosted runners; in a broader GitHub-hosted capacity failure that also takes out the standard pools, those dependencies still block all checks passed. An outage therefore needs a switch any responder with repository write access can throw without merging anything.
Decision
The three primary Linux jobs (node-24, node-24-coverage, node-24-consumers), the three node-compat matrix entries, and all-checks-passed resolve through DSH_CI_FAILOVER_LINUX; the native Windows jobs resolve through DSH_CI_FAILOVER_WINDOWS. A platform switch does not redirect the other platform. Set to selfhosted by a repository writer, the applicable trusted jobs select vm-backup or dsh-win-ci; the blacksmith value routes the participating jobs per the blacksmith failover leg note; unset or any other value retains the workflow-defined hosted fallbacks. Node compatibility jobs require a same-repository, non-fork head and a non-Dependabot author, use isolated runtime setup, and retain the ubuntu-latest fallback under unset and non-special values; the blacksmith branch carries none of those predicates. Under the selfhosted value, Linux failover bounds snapshot concurrency and skips hosted package-cache restores. The verdict follows its workers so it does not remain queued on an unavailable hosted pool. Each switch is writer-manageable repository state, not a merge, so it works while checks are red. The serial / linux (self-hosted standby) and serial / windows (self-hosted standby) lanes re-prove the complete unsharded aggregates on master pushes.
The superseded-CI cancellation policy governs master pushes and manual runs in the same workflow/ref group, including standby drills. Rapid master updates can starve a drill before it reaches a verdict. Use the latest completed standby verdict and check its age and commit before treating it as readiness evidence; a cancelled or merely scheduled run is not proof of readiness.
Release rehearsals share the Linux switch
DSH_CI_FAILOVER_LINUX=selfhosted also routes the credential-free dependency-layout job and both dsh/vendor pack jobs onto vm-backup for eligible same-repository PRs and master pushes. Their release rehearsal reference owns the stricter event eligibility and hosted manual dispatch. This coupling is intentional: keeping the variable set to save release minutes also keeps the eligible main-CI Linux jobs self-hosted. Clearing it returns both workloads to their hosted targets for subsequent runs; publication stays hosted regardless.
What the in-house pool is
vm-backup: one shared VM with multiple always-on systemd-managed runner instances. Registrations share its CPU, memory, and disk; their count is not a count of independent machines. Its image must preinstall the Linux system packages for Playwright Chromium and WebKit; CI downloads the lockfile-selected browsers but never runs apt on this persistent shared host. Check the latest serial / linux (self-hosted standby) run before switching: its aggregate includes browser replay, so a green standby verifies both ordinary capacity and this browser prerequisite.
Windows pool
dsh-win-ci: 32 always-on runner instances (scheduled tasks GH-Runner-01…GH-Runner-32) on the in-house Windows CI server (one 96-core / 580 GB machine). Labels: [self-hosted, dsh-win-ci, windows]. The image must preinstall Node 24, pnpm, Git (with Git Bash on PATH, i.e. C:\Program Files\Git\bin — the bash tool spawns bash by name), PowerShell 7, and enable Developer Mode for symlink support. The general-purpose Windows workspaces and pnpm store must both live on a ReFS volume (F:): those installs pass --package-import-method=clone on ReFS, which needs that volume layout and the @reflink/reflink native module that the system corepack pnpm carries (see the Windows ReFS store note); a rebuilt runner without this layout fails the Windows build gates with TS6231. Check the latest serial / windows (self-hosted standby) run before switching: a green standby verifies the pool can execute check:ci:windows-complete end-to-end.
Switch (any repository writer, ~1 minute, no merge)
The two switches are independent: flip only the one whose platform is degraded.
- Repository Settings → Secrets and variables → Actions → Variables → New repository variable: name
DSH_CI_FAILOVER_LINUX(Linux pool outage) orDSH_CI_FAILOVER_WINDOWS(Windows pool outage), valueselfhosted. - Retrigger the required jobs so they re-resolve their pool. Jobs already queued for the hosted labels do not retarget and cannot be re-run in place, so for the documented indefinite-queue outage, cancel the stuck run and re-run all jobs, or push a new commit; "Re-run failed jobs" only helps once a job has actually failed rather than queued.
- That is the entire switch. Under the
selfhostedLinux failover value the workflow also dropsDSH_SNAPSHOT_MAX_CONCURRENCYto 12 for the shared VM and skips the hosted-path pnpm cache restores because the VM's persistent store serves warm installs. Self-hosted coverage uses four single-worker instrumented partitions and two exempt workers; isolated hosted runners derive worker defaults from available CPUs. The Windows switch selects the native Windows pool and retains explicit worker limits on shared self-hosted runners.
Dependabot exception. Both switches' selfhosted legs deliberately exclude dependabot[bot]: under self-hosted failover, Dependabot PRs stay queued for the hosted pool rather than executing dependency-supplied code on the persistent VMs. A Dependabot PR that remains queued during an outage is expected behavior, not a failed switch; it completes when the hosted pool recovers. The blacksmith value's branches carry no such exclusion, because Blacksmith runners are ephemeral (see the blacksmith failover leg note).
Who can flip the variable. GitHub's API lets any collaborator with write access manage repository variables, so each switch is writer-level, not strictly admin-only. In this repository's trust model that is not an escalation: the runner groups admit all workflows of this private, fork-disabled repository (a deliberate trade to make PR-ref failover possible at all), so any writer could already reach the VMs by pushing a branch workflow. The boundary against untrusted code is repository membership; the variables only route work for members.
Capacity during failover
Capacity includes the master standby, main-CI jobs, and three release-rehearsal jobs for each eligible PR or master push while the Linux switch is set. Each trusted PR also adds three Node compatibility jobs at gate concurrency one, including the build-backed Node 22 leg and cold temporary runtime downloads. The release rehearsal workflows cancel superseded runs within each workflow/ref group under the cancellation policy; different refs can still add concurrent build, pack, and install load. Check current CPU, memory, disk, and queue pressure before extending self-hosted operation; extra registrations on this VM add scheduling slots, not machine resources. Do not infer spare capacity from the standby alone. When host resources permit extra registrations, use an org registration token (org Settings → Actions → Runners → New runner). Clone an existing runner directory excluding its identity files — rsync -a --exclude '.runner*' --exclude '.credentials*' --exclude '_diag' --exclude '_work' <src>/ <dst>/ (the globs also catch .runner_migrated/.credentials_migrated, which GitHub writes on migrated runners and which equally trigger the already-configured refusal) — then run config.sh (copying .runner/.credentials verbatim makes it refuse with "already configured"), and start the listener: sudo ./svc.sh install ubuntu && sudo ./svc.sh start. Registration alone leaves the runner offline; a started service adds a scheduling slot, not CPU or memory.
Switch back
Delete the DSH_CI_FAILOVER_LINUX or DSH_CI_FAILOVER_WINDOWS variable (or set it to any value other than selfhosted or blacksmith). New runs resolve back to their hosted pools. Setting it to blacksmith keeps the jobs on Blacksmith until the value changes. Remove any extra instances that were registered during the incident.
Trust boundary
The variables are writer-manageable repository state; a pull request event itself can neither set them nor read a different value into effect, and the selector expressions live in workflow definitions. Note that under failover, pull_request runs execute the PR merge ref's own workflow definition — the boundary against untrusted code is repository membership (private, forking disabled, Dependabot excluded by the selfhosted legs; the blacksmith legs carry no exclusion), not the variable. Note on runner-group policy: pinning the runner group to the master-ref workflow is incompatible with this failover — the failover jobs, including the Node compatibility matrix, are pull_request runs evaluated from PR merge refs, and a master-pinned group leaves them queued (observed live on 2026-07-27; the group was widened to all workflows of this repository to unblock the switch). A stricter runner-side policy therefore costs PR failover; the shipped posture accepts repository-scoped, all-workflow group access.
Alternatives considered
Merge a workflow change to switch pools. Rejected because the outage that motivates the switch is exactly the state in which no PR can merge: the required checks are the ones failing. A repository variable is writer-manageable state that takes effect on re-run without a merge.
Keep the self-hosted pool always in the required path. Rejected because it trades hosted-pool availability for the in-house VM's, moving a single point of failure rather than adding a fallback. The unset defaults retain hosted targets and the switches provide a reversible, operator-selected self-hosted path; splitting them by platform means an outage on one platform does not retarget the other.
Consequences
Recovering from a hosted-pool outage is flipping the affected platform's variable (any writer) plus a re-run, with no merge on the critical path. The cost is a second runner topology per platform to keep working: master pushes schedule the standby lanes, but only completed verdicts establish readiness under the cancellation policy, and the snapshot-concurrency and cache-restore branches in ci.yml carry a selfhosted leg (Linux only) that must stay in step with the hosted leg. Splitting the switch by platform adds one more variable to manage but bounds the blast radius of each switch to the jobs of a single platform.
中文
问题
CI 中三个必需的 Linux 工作作业（node 24 / static、node 24 / coverage、node 24 / snapshots and artifacts）运行在托管的企业级 32 核池上；聚合它们的必需判定作业（all checks passed）运行在标准 ubuntu-latest 上；原生 Windows 作业运行在托管的 dsh-windows-2025-16core 大型运行器上。当企业池发生故障——作业无限排队或企业标签消失——所有开启的拉取请求都无法合并，而"合并一个修复"这一常规恢复手段本身正被那些无法运行的必需检查死锁。适用范围：两个独立开关，每个平台一个。DSH_CI_FAILOVER_LINUX 恢复企业级 Linux 池故障（三个必需的 Linux 工作作业加 all checks passed 判定作业）；DSH_CI_FAILOVER_WINDOWS 恢复托管 Windows 池故障（原生 Windows 作业）。Linux 池故障无需重定向 Windows 作业，反之亦然。Node 兼容性作业也通过隔离设置跟随 Linux 开关；判定作业的 node-24-bench、python-sdk 和 python-runtime 依赖仍留在标准托管运行器上；若更大范围的 GitHub 托管容量故障连标准池一并击倒，这些依赖仍会阻塞 all checks passed。因此故障需要一个任何具备仓库写权限的响应者都能在不合并任何代码的情况下触发的开关。
决策
三个主要 Linux 作业（node-24、node-24-coverage、node-24-consumers）、三个 node-compat 矩阵条目和 all-checks-passed 通过 DSH_CI_FAILOVER_LINUX 解析；原生 Windows 作业通过 DSH_CI_FAILOVER_WINDOWS 解析。一个平台的开关不会重定向另一个平台。仓库写者将变量设为 selfhosted 时，适用的可信作业选择 vm-backup 或 dsh-win-ci；blacksmith 取值按 blacksmith 故障切换支路笔记 路由参与切换的作业；未设置或任何其它值保留工作流定义的托管回退。Node 兼容性作业要求同仓库且非 fork 的头部以及非 Dependabot 作者，使用隔离运行时设置，并在未设置与非特殊值下保留 ubuntu-latest 回退；blacksmith 分支不带上述任何条件。在 selfhosted 取值下，Linux 故障切换会限制快照并发，并跳过托管软件包缓存恢复。判定作业跟随工作作业，避免继续在不可用的托管池排队。每个开关都是写者可管理的仓库状态而非一次合并，因此在检查失败时仍然有效。serial / linux (self-hosted standby) 与 serial / windows (self-hosted standby) 通道在 master 推送上重新验证完整的未分片聚合流程。
被取代 CI 的取消策略 管理同一工作流/引用组内的 master 推送和手动运行，包括热备演练。master 快速更新可能让演练因反复被取消而始终无法得出结论。判断就绪状态时，使用最近一次已完成的热备结论，并核对其时间和提交；已取消或仅被调度的运行不构成就绪证据。
发布演练共用 Linux 开关
DSH_CI_FAILOVER_LINUX=selfhosted 还会将符合条件的同仓库 PR 和 master 推送中的无凭据依赖布局作业与 dsh/vendor 两个打包作业路由到 vm-backup。发布演练说明 负责更严格的事件准入规则及保留托管的手动触发。这种耦合是有意的：持续设置变量来节省发布分钟，也会让符合条件的主 CI Linux 作业持续使用自托管。清除变量会让两类负载的后续运行返回各自的托管目标；发布操作始终保留托管。
自有池是什么
vm-backup：一台共享虚拟机，运行多个常驻 systemd 管理的运行器实例。注册实例共享 CPU、内存和磁盘；实例数量不代表独立机器数量。其镜像必须预装 Playwright Chromium 和 WebKit 所需的 Linux 系统软件包；CI 会下载锁文件选定的浏览器，但绝不在这台持久化共享主机上运行 apt。切换前先看 serial / linux (self-hosted standby) 最近一次运行：其聚合流程包含浏览器回放，因此绿色热备同时验证常规容量和这项浏览器先决条件。
Windows 池
dsh-win-ci：公司内部 Windows CI 服务器（一台 96 核 / 580 GB 机器）上 32 个常驻运行器实例（计划任务 GH-Runner-01…GH-Runner-32）。标签：[self-hosted, dsh-win-ci, windows]。镜像必须预装 Node 24、pnpm、Git（Git Bash 在 PATH 上，即 C:\Program Files\Git\bin——bash 工具按名称 spawn bash）、PowerShell 7，并为符号链接支持启用开发人员模式。通用 Windows 通道的工作区与 pnpm store 必须都位于 ReFS 卷（F:）上：这些安装步骤在 ReFS 上传递 --package-import-method=clone，这需要该卷布局以及系统 corepack pnpm 携带的 @reflink/reflink 原生模块（见 Windows ReFS store note）；没有此布局的重建运行器会在 Windows 构建门禁阶段以 TS6231 失败。切换前先看 serial / windows (self-hosted standby) 最近一次运行：绿色热备验证该池能端到端执行 check:ci:windows-complete。
切换步骤（任何具备写权限的协作者，约 1 分钟，无需合并）
两个开关相互独立：只切换发生故障的那个平台。
- 仓库 Settings → Secrets and variables → Actions → Variables → New repository variable：名称
DSH_CI_FAILOVER_LINUX（Linux 池故障）或DSH_CI_FAILOVER_WINDOWS（Windows 池故障），值selfhosted。 - 重新触发必需作业，使其重新解析运行器池。已经为托管标签排队的作业不会重定向，也无法原地 re-run，因此对于本手册所述的无限排队故障，应取消卡住的运行并 re-run all jobs，或推送一个新提交；“Re-run failed jobs”只有在作业真正失败（而非仍在排队）时才有用。
- 切换到此完成。在
selfhosted的 Linux 故障切换取值下，工作流还会把DSH_SNAPSHOT_MAX_CONCURRENCY降为 12，以限制共享虚拟机上的争抢，并跳过托管路径的 pnpm 缓存恢复，因为虚拟机的持久 store 会直接提供热安装。自托管覆盖率使用 4 个单 worker 插桩分区与 2 个豁免 worker；独占的托管运行器根据可用 CPU 计算默认 worker 数。Windows 开关选择原生 Windows 运行器池，并在共享的自托管运行器上保留显式 worker 限制。
**Dependabot 例外。**两个开关的 selfhosted 腿都刻意排除 dependabot[bot]：自托管故障切换期间，Dependabot 拉取请求继续在托管池排队，而不是把依赖项提供的代码放到持久化虚拟机上执行。故障期间 Dependabot PR 持续排队是预期行为而非切换失败；托管池恢复后它会自行完成。blacksmith 取值下的分支不带此类排除，因为 Blacksmith 运行器是临时的（见 blacksmith 故障切换支路笔记）。
**谁能扳动这个变量。**GitHub 的 API 允许任何具有写权限的协作者管理仓库变量，因此每个开关实际是写者级而非严格的管理员级。在本仓库的信任模型下这并不构成升权：runner group 接纳本私有、禁 fork 仓库的全部工作流（这是让 PR 引用的故障切换得以成立的刻意取舍），因此任何写者本就可以通过推送分支工作流触达这台虚拟机。抵御不可信代码的边界是仓库成员资格；变量只是为成员路由工作。
切换期间的容量
Linux 开关启用期间，容量需覆盖 master 热备、主 CI 作业，以及每个符合条件的 PR 或 master 推送的三个发布演练作业。每个可信 PR 还会增加三个门禁并发度为一的 Node 兼容性作业，包括需要构建的 Node 22 条目和冷临时运行时下载。发布演练工作流依据取消策略取消各工作流/引用组内被取代的运行；不同引用仍可能增加并发构建、打包和安装负载。延长自托管运行前，检查当前 CPU、内存、磁盘和队列压力；同一虚拟机上新增注册只增加调度槽位，不增加机器资源。不能只依据热备负载推断空闲容量。主机资源允许增加注册实例时，使用组织级注册 token（组织 Settings → Actions → Runners → New runner）。复制现有 runner 目录时必须排除身份文件——rsync -a --exclude '.runner*' --exclude '.credentials*' --exclude '_diag' --exclude '_work' <src>/ <dst>/（通配同时排除 .runner_migrated/.credentials_migrated——GitHub 会在迁移过的运行器上写入这些文件，它们同样会触发 already-configured 拒绝）——再跑 config.sh（原样拷贝 .runner/.credentials 会使其以 "already configured" 拒绝），然后启动监听器：sudo ./svc.sh install ubuntu && sudo ./svc.sh start。仅注册不会上线；启动服务增加的是调度槽位，而非 CPU 或内存。
切回
删除 DSH_CI_FAILOVER_LINUX 或 DSH_CI_FAILOVER_WINDOWS 变量（或改为 selfhosted 与 blacksmith 之外的任何值），新的运行即解析回各自的托管池。设为 blacksmith 会让作业留在 Blacksmith，直到该值改变。若故障期间追加注册过实例，将其移除。
信任边界
这些变量是写者可管理的仓库状态；pull_request 事件本身既不能设置它们，也不能让不同的值生效，选择器表达式存在于工作流定义中。需要注意：故障切换期间，pull_request 运行执行的是 PR merge 引用自带的工作流定义——抵御不可信代码的边界是仓库成员资格（私有、禁 fork、Dependabot 由 selfhosted 腿排除；blacksmith 腿不带排除），而非该变量。关于 runner group 策略的说明：把 runner group 绑定到 master 引用的工作流与本故障切换机制不兼容——包括 Node 兼容性矩阵在内的故障切换作业是从 PR merge 引用求值的 pull_request 运行，master 绑定的组会让它们持续排队（2026-07-27 实际故障中亲历；当时将组放宽为本仓库全部工作流才疏通了切换）。更严格的运行器侧策略以牺牲 PR 故障切换为代价；当前采用的形态是仓库范围、全工作流的组访问。
曾考虑的替代方案
通过合并一次工作流改动来切换池。 否决，因为触发切换的故障状态恰恰是任何 PR 都无法合并的状态：必需检查正是失败的那些。仓库变量是写者可管理的状态，重跑即生效，无需合并。
让自托管池长期处于必需路径中。 否决，因为这是拿托管池的可用性去换自有虚拟机的可用性，只是搬移了单点故障而非增加回退。未设置变量时默认保留托管目标，开关提供由运维人员选择、可逆的自托管路径；按平台拆分意味着一个平台的故障不会重定向另一个平台。
后果
从托管池故障中恢复只需切换受影响平台的变量（任何写者可设）加一次重跑，关键路径上没有合并。代价是每个平台都要维护第二套运行器拓扑：master 推送会调度热备通道，但依据取消策略，只有已完成的结论才能证明就绪状态；而 ci.yml 中的快照并发与缓存恢复分支带有一条 selfhosted 支路（仅 Linux），必须与托管支路保持同步。按平台拆分开关多了一个需要管理的变量，但把每个开关的影响范围限定在单个平台的作业上。