Experimental voice input with explicit recognition providers
使用显式识别 Provider 的实验性语音输入
Dictation must preserve user control over when a coding task starts. Local recognition brings model files, a native runtime and CPU work; merely installing or enabling a UI contribution must not allocate these resources. Later cloud providers need a selection mechanism that does not silently change where recordings are processed.
Source compatibility: Chinese title uses a compatible non-canonical header
English
Problem
Dictation must preserve user control over when a coding task starts. Local recognition brings model files, a native runtime and CPU work; merely installing or enabling a UI contribution must not allocate these resources. Later cloud providers need a selection mechanism that does not silently change where recordings are processed.
Decision
Five public packages under packages/experimental/ compose one optional, default-disabled voice input bundle. Service Definition, SenseVoice Provider and Remote Consumer are separate because provider deployment and browser transport evolve independently. The browser UI dynamically mounts its generated Remote contribution. Stable API Remotes do not acquire an experimental dependency.
Provider ids select exact registrations. Resolution captures an instance, and deregistration closes admission, cancels and joins accepted work. There is no provider fallback: a local failure cannot authorize uploading speech to a cloud service. A new cloud provider supplies its own credentials and explicit processing location through the same registry.
Recordings and transcripts remain transient until the user submits ordinary text. Recognition does not start an agent turn or create an audio Session event. The input facade accepts asynchronous plain text against a captured draft revision, preserving reference chips and undo history. Changed drafts retain the recognized text for explicit insertion instead of replacing later edits. Session changes and plugin disposal invalidate late results and release recording resources. Capture failures notify the current recording activity immediately, independently of resource-close latency. Cancellation, failure and plugin withdrawal join one release promise, so ownership lasts until the AudioContext closes.
SenseVoice ONNX and Silero VAD run in one managed CPU subprocess. The native sherpa-onnx package includes ONNX Runtime; explicit preparation only downloads and verifies models; subsequent calls reuse the warm model until the configured idle deadline. Every activation verifies cached assets and restores complete caches to standby without starting the worker. Readiness therefore follows disk contents rather than the lifetime of a Provider instance. Recordings wait in the bounded queue during wake-up. The worker checks readiness at execution, so queued work cannot implicitly download resources invalidated by an earlier cache inspection. Inspection uses the same owned task as preparation, so cancellation and disposal join it. Disabling the provider joins its work and process exit while retaining reusable disk assets. Native browser recording and resampling avoid an additional audio conversion executable. The authenticated private loopback protocol carries complete PCM recordings, allowing cancellation to terminate the worker without sharing a long-running inference with another request.
The Host owns preparation independently of the view that started it. An ordered step list survives view remounts and records completion only as execution advances. Recognition weights, VAD and final verification have separate operations; downloads use pinned byte counts, while other work reports elapsed time. Preparation status and progress stay in bundle details through plugins.bundle.config, keyed by package name. The generic plugins.bundle.activation slot carries explicit enablement intent and detail navigation; the voice occupant waits for Host cache inspection and offers installation guidance only for unprepared local models. Enabling an already prepared provider does not prompt or download. A single composer activity Slot expands the voice controls while preserving the draft editor and submit action. The context meter sits below the composer; recording and transcription hide the meter along with ordinary toolbar controls until the activity closes. Shared primitives provide buttons, disclosure and status markers; only measured waveform drawing belongs to the voice UI. Recordings start on click and transcribe on Stop, with feedback inside the activity. Persistent preferences remain in plugin settings.
Alternatives considered
Always loading the model on plugin activation would make a UI toggle incur substantial startup time and memory even without dictation. Python/FunASR/PyTorch adds interpreter provisioning, dependency installation and disk cost; completed ONNX artifacts and platform-native packages serve CPU dictation. End users never export models. Browser-native speech recognition does not provide a uniform local-processing guarantee. Streaming voice interaction would require partial-result revision, turn arbitration and bidirectional audio semantics that complete-recording dictation does not need.
Consequences
First use needs download time and disk space. Local means the Host machine, including a remote Host. A cancelled active inference loses its warm worker and the next request reloads it. Optional preparation paths support existing offline environments without silently changing defaults.
Verification covers exact-provider routing, registration teardown, bounded audio parsing, microphone release, stale-draft and Session-switch handling, worker queue/deadline recovery, and verified download publication. Real local inference uses the same managed worker as the product. Browser composition verifies optional enablement and draft insertion; ordinary Session replay verifies that only the submitted text reaches the model.
Real Node 24 and Electron 44 children complete FP32 and INT8 inference. Electron copies VAD output to avoid external-buffer restrictions. Three runs of the 4.2-second public Chinese sample agree; this short sample is not a complete accuracy evaluation. Desktop microphone requests are restricted to audio from the primary application window, with macOS authorization handled by the operating system. Windows, Linux and signed installers still require platform acceptance.
INT8 is the default to reduce download and storage cost; precision: fp32 selects the reference weights. Public English and concatenated bilingual samples show word differences between precisions, so quantization is not treated as accuracy-equivalent. A roughly 101-second repeated-speech sample completes in both runtimes; its observed post-request RSS is not a peak-memory bound.
中文
问题
听写必须保留用户对编码任务启动时机的控制。本地识别带来模型文件、原生运行时和 CPU 工作；仅安装或启用 UI 贡献不应分配这些资源。之后的云端 Provider 需要明确的选择机制，不能悄悄改变录音的处理位置。
决策
packages/experimental/ 下的五个公开包组合成一个默认禁用的可选语音输入 Bundle。服务定义、SenseVoice Provider 和 Remote 消费者分开，因为 Provider 部署与浏览器传输独立演化。浏览器 UI 动态挂载其生成的 Remote 贡献。稳定 API Remotes 不引入实验性依赖。
Provider id 选择确切的注册。解析捕获实例，注销先拒绝新任务，再取消并等待已接收的任务。不存在 Provider 回退：本地失败不能授权将语音上传至云端服务。新增云端 Provider 通过同一注册表提供自己的凭据配置与显式处理位置。
录音与转写在用户提交普通文字前都是临时数据。识别不启动智能体轮次，也不创建音频 Session 事件。输入门面按照捕获的草稿版本接收异步纯文本，保留引用芯片与撤销历史。草稿变化时保留识别文字，等待显式插入，而非替换之后的编辑。切换 Session 和释放插件会使迟到结果失效并释放录音资源。采集失败立即通知当前录音活动，不受资源关闭耗时影响。取消、失败和插件撤销共同等待同一个释放 Promise，所有权持续到 AudioContext 关闭。
SenseVoice ONNX 与 Silero VAD 在一个托管 CPU 子进程中运行。原生 sherpa-onnx 包包含 ONNX Runtime；显式准备只下载并校验模型；后续调用复用预热模型，直到配置的空闲截止时间。每次启用都会校验缓存资源，完整缓存恢复为待唤醒状态，不启动工作进程。就绪状态因此取决于磁盘内容，而非 Provider 实例的存活时间。录音在唤醒期间进入有界队列等待。工作进程在执行时检查就绪状态，因此较早的缓存检查使资源失效后，排队任务也不能隐式下载资源。检查复用准备任务的生命周期，取消与释放都会等待检查结束。禁用 Provider 等待其任务和进程退出，同时保留可复用的磁盘资源。浏览器原生录音与重采样避免增加音频转换可执行文件。带认证的私有环回协议传输完整 PCM 录音，取消时可以终止工作进程，而不与另一请求共享长时间推理。
Host 独立于发起视图拥有准备任务。有序步骤列表在视图重新挂载后保留，仅随着执行推进记录完成。识别模型、VAD 与最终校验分别准备；下载使用固定字节数，其他工作报告等待时间。准备状态和进度通过以包名为 key 的 plugins.bundle.config 显示在详情页。通用 plugins.bundle.activation Slot 传递显式启用意图和详情导航；语音贡献等待 Host 缓存检查，只为尚未准备的本地模型提供安装引导。启用已准备的 Provider 不弹窗也不下载。单个输入框活动 Slot 展开语音控件，同时保留草稿编辑器和提交按钮。空闲时上下文用量按钮位于输入框下方；录音和转写期间，用量按钮与普通工具栏控件一起隐藏，关闭活动后恢复。按钮、折叠与状态标记复用共享原语，语音 UI 只负责实测波形绘制。点击开始录音，停止后转写，反馈留在活动栏内。持久化偏好仍在插件设置中。
考虑的替代方案
在插件启用时加载模型，会让一个 UI 开关在尚未听写时就产生大量启动时间和内存占用。Python/FunASR/PyTorch 增加解释器、依赖安装与磁盘成本；ONNX 成品和平台原生包满足 CPU 听写。用户端不执行模型导出。浏览器原生语音识别不能提供一致的本地处理保证。流式语音交互需要部分结果修订、轮次仲裁与双向音频语义，而完整录音听写无需这些机制。
影响与验证
首次使用需要下载时间和磁盘空间。本地指 Host 所在机器，包括远程 Host。取消运行中的推理会丢失预热进程，下次请求重新加载。可选准备路径支持已有离线环境，不会悄悄改变默认行为。
验证覆盖精确 Provider 路由、注册释放、有界音频解析、麦克风释放、过期草稿和 Session 切换处理、工作进程队列与截止时间恢复，以及经过校验的下载发布。真实本地推理使用与产品相同的托管工作进程。浏览器组合验证可选启用与草稿插入；普通 Session 回放验证只有提交的文字进入模型。
Node 24 与 Electron 44 的真实子进程均完成 FP32 和 INT8 识别。Electron 的 VAD 输出使用复制缓冲区，避免 external buffer 限制。4.2 秒公开中文样本的三次识别结果一致；该短样本不代表完整准确率评测。桌面麦克风请求限制为主应用窗口的音频请求，macOS 权限由系统处理。Windows、Linux 真实机器及签名后的安装包仍需平台验收。
INT8 以降低下载和存储成本为默认值，precision: fp32 选择参考权重。公开英文与拼接的中英双语样本显示两种精度存在用词差异，因此不将量化视为准确率等效。约 101 秒的重复语音样本在两种运行时均完成识别；请求后的 RSS 观测不构成峰值内存上限。