Blacksmith as a failover leg for CI pools
Blacksmith 作为 CI 池的故障切换支路
The [failover runbook](2026-07-26-ci-failover-runbook.md) routes degraded enterprise-pool traffic onto the in-house `vm-backup` and `dsh-win-ci` pools, which are proven but bounded standbys. Blacksmith sells GitHub-hosted-style runners on demand, and its migration wizard proposed replacing the repository's `runs-on` labels wholesale, which would put credential-holding, benchmark, and spec-pinned jobs on third-party i
English
Problem
The failover runbook routes degraded enterprise-pool traffic onto the in-house vm-backup and dsh-win-ci pools, which are proven but bounded standbys. Blacksmith sells GitHub-hosted-style runners on demand, and its migration wizard proposed replacing the repository's runs-on labels wholesale, which would put credential-holding, benchmark, and spec-pinned jobs on third-party infrastructure as the default. The wholesale replacement is not acceptable as a default: it breaks the workflow contract specs, distorts benchmark and performance-budget measurements, and moves trust-boundary jobs without a decision record.
Decision
DSH_CI_FAILOVER_LINUX and DSH_CI_FAILOVER_WINDOWS additionally accept the value blacksmith. When any repository writer sets it, the platform's participating jobs retarget onto Blacksmith's hosted runners at each job's configured vCPU size; any other value (or unset) keeps today's runner, and selfhosted keeps routing to the in-house pools. Like the existing values this is writer-manageable repository state — no merge, re-run the affected jobs — and it costs money only while set, which is why it is an explicit outage or experiment choice, never the default.
Participating jobs under blacksmith: the three enterprise Linux workers, the node-compat legs, the all-checks-passed verdict, and the three Windows jobs in ci.yml, using the per-job runner sizes declared there; expected-filenames; the sandbox bwrap leg; and the two manual benchmark matrices, whose 4-32 vCPU tiers map to Blacksmith equivalents while the 64/96-core rows keep the fleet labels because Blacksmith offers no such tier. The blacksmith branches carry no Dependabot exclusion: Blacksmith runners are ephemeral, so the persistent-VM rationale that keeps Dependabot on the hosted pool does not apply. The python-runtime builder stays hosted in full: its PR and master callers (ci.yml's and ci-master.yml's python-runtime jobs) pass real-API credentials (DEEPSEEK_API_KEY_EXTERNAL) into the reusable workflow, its macOS and arm64 single-exe legs need GitHub's images, and its wheel output feeds the PyPI publication chain. The docs-pages workflow is a tag-dispatched release publication, so it stays hosted too.
Everything else keeps its own runner under every value: jobs pinned to standard hosted runners by workflow spec (node-24-bench, python-sdk, the master Wine gate, release-publish and release-vendor-publish, the Cloudflare preview, the weighted-approval pair); jobs that hold credentials (the real-API e2e lanes, the issue App-token jobs, the npm and PyPI publishers); jobs that need a host capability Blacksmith lacks (the Landlock legs and their pack verification, the Seatbelt leg, the linux-arm64 single-exe leg); and the release-shaped chains — the npm publishers, the PyPI publishers, the node-addon-system release chain, and the Pages deployment workflow — which stay entirely hosted because their artifacts feed published registry output. The release rehearsal jobs in release.yml and release-vendor.yml (dependencies, pack) keep their existing pre-change routing unchanged: they carry only the pre-existing selfhosted branch, so under blacksmith they fall back to their hosted ubuntu-24.04 default exactly as under an unset variable, and they gain no blacksmith branch.
Alternatives considered
Adopting the migration wizard wholesale. Rejected because it breaks the workflow contract specs, changes benchmark semantics and performance-budget calibration, and moves credential-holding and trust-boundary jobs to third-party infrastructure as the default; the earlier wizard migration (closed PR #3053) was rejected for cost and the same review findings.
Keeping the in-house pools as the only failover target. Rejected as the sole option because they are a bounded standby; Blacksmith adds elastic capacity for the outage and experiment cases where the fleet itself is the degraded resource.
Consequences
The blacksmith legs stay dormant until the value is set, so they are exercised only during an actual failover or an explicit experiment. The mirror trial (PR #3841, closed) ran the participating set hardcoded onto the Blacksmith labels and confirmed that blacksmith-4vcpu-ubuntu-2404, blacksmith-16vcpu-ubuntu-2404, and blacksmith-16vcpu-windows-2025 schedule and execute. Blacksmith registers its runners as self-hosted, so under the blacksmith value the node-compat legs run their toolcache-isolation and isolated-installation verification steps (gated on runner.environment == 'self-hosted') rather than skipping them; all three legs passed on the trial image (run 34322356689). The same trial surfaced one environment finding that this change fixes: while expanding **, node 24.13's internal fs.glob lstat-probes <matched>/<next segment> for a symlinked file and throws ENOTDIR instead of skipping (run 34319926270), failing verify-md-wrap inside check:ci:static on Blacksmith until repo-files.ts replaced globSync with a dirent walker; the fixed static lane is green on the same image. The trigger is the lstat-probe of a path beneath a matched symlink, not ** over symlinked trees in general: the other globSync ** call sites that scan symlink-bearing trees stayed green on the same run and needed no change. The benchmark matrices' 8 and 32 vCPU labels (ubuntu-2404 and windows-2025) were not part of the trial and are inferred by naming symmetry from the verified 4 and 16 vCPU labels; confirm with a manual dispatch before first use. The Landlock, Seatbelt, and linux-arm64 exclusions still rest on the prose dispatch measurements of the earlier migration attempt (closed PR #3053), recorded here as a known gap; the trial could not exercise those legs by design. Workflow specs pin the new branches — ci-workflow.spec.ts asserts the Blacksmith labels appear in the extended selectors, and ci-compatible-selfhosted.spec.ts evaluates the blacksmith routing mode — so a future rewrite of the selectors fails the unit gates. Default cost posture is unchanged: nothing runs on Blacksmith unless the variable says so.
中文
问题
故障切换手册把降级的企业池流量路由到自有的 vm-backup 与 dsh-win-ci 池——它们是经过验证但容量有界的备用池。Blacksmith 按需出售类似 GitHub 托管的运行器，其迁移向导提议整体替换仓库的 runs-on 标签，这会把持有凭据、基准与 spec 钉死的作业默认放到第三方基础设施上。整体替换不适合作为默认：它会打破 workflow 契约 spec、扭曲基准与性能预算测量，并在没有决策记录的情况下搬动信任边界作业。
决策
DSH_CI_FAILOVER_LINUX 与 DSH_CI_FAILOVER_WINDOWS 额外接受值 blacksmith。任何仓库写者设置它时，该平台参与切换的作业按各作业配置的 vCPU 档位重定向到 Blacksmith 托管运行器；任何其它值（或未设置）保持今天的运行器，selfhosted 继续路由到自有池。与既有取值一样，这是写者可管理的仓库状态——无需合并，重跑受影响的作业即可——并且只在设置期间产生费用，这正是它作为明确的故障或实验选择、而非默认的原因。
blacksmith 下参与切换的作业：ci.yml 中三个企业级 Linux 工作作业、node-compat 各腿、all-checks-passed 判定作业与三个 Windows 作业，使用该工作流为各作业声明的运行器规格；expected-filenames；sandbox 的 bwrap 腿；以及两个手动基准矩阵——其 4-32 vCPU 档位映射到 Blacksmith 等价档，64/96 核行保留自有池标签，因为 Blacksmith 没有对应档位。blacksmith 分支不带 Dependabot 排除：Blacksmith 运行器是临时的，因此把 Dependabot 留在托管池的持久虚拟机理由不适用。python-runtime 构建器整体保持托管：其 PR 与 master 调用方（ci.yml 与 ci-master.yml 的 python-runtime 作业）会把真实 API 凭据（DEEPSEEK_API_KEY_EXTERNAL）传入该可复用工作流，其 macOS 与 arm64 单文件分支需要 GitHub 镜像，且其 wheel 产物供给 PyPI 发布链。docs-pages 工作流是 tag 派发的发布，因此同样保持托管。
其余作业在任何取值下都保留自己的运行器：被 workflow spec 钉死在标准托管运行器上的作业（node-24-bench、python-sdk、master 的 Wine 门禁、release-publish 与 release-vendor-publish、Cloudflare preview、weighted-approval 一对）；持有凭据的作业（真实 API 的 e2e 通道、issue App token 作业、npm 与 PyPI 发布作业）；需要 Blacksmith 缺失的宿主能力的作业（Landlock 各腿及其打包校验、Seatbelt 腿、linux-arm64 单文件分支）；以及发布形态的链——npm 发布、PyPI 发布、node-addon-system 发布链与 Pages 部署工作流——它们整体保持托管，因为其产物供给已发布的 registry 输出。release.yml 与 release-vendor.yml 中的发布演练作业（dependencies、pack）保持改动前既有的路由不变：它们只带既有的 selfhosted 分支，因此在 blacksmith 取值下与变量未设置时一样回落到托管的 ubuntu-24.04 默认，并不新增 blacksmith 分支。
备选方案
整体采纳迁移向导。否决，因为它打破 workflow 契约 spec、改变基准语义与性能预算校准，并默认把持有凭据与信任边界的作业搬到第三方基础设施；更早的向导迁移（已关闭的 PR #3053）因成本与同样的 review 结论被否决。
只把自有池作为唯一故障切换目标。作为唯一选项被否决，因为它是容量有界的备用；Blacksmith 为「故障对象正是自有池本身」的故障与实验场景提供弹性容量。
后果
blacksmith 各腿在有人设置该值前保持休眠，因此只在真实故障切换或明确实验期间被使用。镜像试运行（PR #3841，已关闭）把参与集硬编码到 Blacksmith 标签上跑过，确认 blacksmith-4vcpu-ubuntu-2404、blacksmith-16vcpu-ubuntu-2404 与 blacksmith-16vcpu-windows-2025 可调度可执行。Blacksmith 将其运行器注册为 self-hosted，因此在 blacksmith 取值下 node-compat 各腿会真实执行 toolcache 隔离与隔离安装校验步骤（以 runner.environment == 'self-hosted' 为条件），而非全部跳过；三条腿都在试运行镜像上通过（run 34322356689）。同一试运行还暴露了一个由本改动修复的环境发现：在 ** 展开中遇到 symlink 文件时，node 24.13 的内部 fs.glob 会 lstat-probe <matched>/<下一段> 并抛 ENOTDIR 而非跳过（run 34319926270），使 check:ci:static 内的 verify-md-wrap 在 Blacksmith 上失败，直到 repo-files.ts 用 dirent walker 替换 globSync；修复后的 static 通道在同一镜像上转绿。触发条件是 probe 已匹配 symlink 之下的路径，而非笼统的「** 扫过含 symlink 的树」：同一 run 中其它扫描含 symlink 树的 globSync ** 调用点保持绿色，无需改动。两个基准矩阵的 8 与 32 vCPU 标签（ubuntu-2404 与 windows-2025）不在试运行范围内，是按已实测的 4/16 vCPU 标签命名对称推定的；首次使用前请用一次手动 dispatch 确认。Landlock、Seatbelt 与 linux-arm64 的排除仍基于更早迁移尝试（已关闭的 PR #3053）的散文式 dispatch 实测，此处记录为已知缺口；试运行按设计无法触发这几条腿。workflow spec 钉住了新分支——ci-workflow.spec.ts 断言扩展后的 selector 中出现 Blacksmith 标签，ci-compatible-selfhosted.spec.ts 求值 blacksmith 路由模式——因此未来重写 selector 会使单元门禁变红。默认成本姿态不变：除非变量指明，否则没有任何东西跑在 Blacksmith 上。