Skip to content

Implement dictation end to end (spec 0001) - #8

Merged
dimastatz merged 2 commits into
mainfrom
feat/dictation
Oct 7, 2026
Merged

dimastatz merged 2 commits into
mainfrom
feat/dictation

Conversation

@dimastatz

Copy link
Copy Markdown
Owner

Implements spec 0001 (overview) on top of spec 0002 (whisper-flow as the transcription engine). Dictation now works end to end: microphone → whisper-flow → editor.

What's in it

Spec 0001 component Where
Audio capture (sidecar, AU-1..4) src/audio/recorder.ts: SoX, then arecord (Linux) or ffmpeg; or whisperCode.recorderCommand. 2048-byte frames, sent unbuffered
whisper-flow client (TR-1..4, CF-1..3, ER-1) src/whisperflow/: /ready check, version and protocol check, ws/wss URL, x-api-key from SecretStorage, start and stop frames on 1.2.0+
Symbol index (goal 2) src/dictation/workspaceVocabulary.ts + src/text/vocabulary.ts: compound identifiers from open editors and document symbols, ranked, sent as Whisper's prompt, and respelled in final text ("use memo" → useMemo)
Context detector (goal 3) src/text/context.ts: comment (per-language markers, ignoring strings), commit message, prompt
Post-processor (goal 3, RS-6) src/text/format.ts: final text only; sentence case in comments, no trailing period on commit subjects
Editor integration (RS-2..5, ER-4) src/dictation/editorSink.ts + src/extension.ts: dimmed pending range replaced on each partial; committed on final, stop, user edit, cursor move or editor switch; selection replaced; status bar idle/connecting/recording/error; Retry
Session lifecycle (ER-2, ER-3) src/dictation/session.ts, no vscode import

Also: consent once per non-local URL (CF-4), a warning for non-local http:// (CF-5), and the commands Set API Key / Clear API Key.

Decisions worth reviewing

  • whisper-flow 1.2.0 features are used when available. 1.2.0 (protocol_version: 1) adds start (vocabulary prompt, model) and stop (finalize on stop), which settles two open questions in spec 0002. 1.1.0 still works (TR-4); there the last partial is committed as-is on stop.
  • Bad key vs. full server: uvicorn answers a close-before-accept with HTTP 403, never close codes 1008/1013 (checked against a real server). classifyRejectedHandshake probes POST /transcribe_pcm_chunk: a 401 means the key is wrong, anything else means the server is full. The fake server reproduces the 403.
  • Recorder: SoX first, as in the spec's open question, with ffmpeg/arecord fallbacks and a setting. No Python on the client.
  • No editor focused: final text goes through the type command, so it lands in whatever input has focus (for example the chat or SCM box). Partials are skipped there.
  • New runtime dependency: ws (custom headers on the WebSocket upgrade). npm audit --omit=dev is clean.

Testing

  • npm run check passes: 65 tests; coverage 98.6% lines, 96.3% branches.
  • TS-2: src/test/fakeServer.ts is a fake whisper-flow over real HTTP and WebSocket that replays events recorded from whisper-flow 1.2.0, including non-monotonic partials, the 403 rejections, and close codes 1011/4000/1000. fakeRecorder.ts replaces the microphone.
  • Manual smoke test against a real whisper-flow 1.2.0 (tiny.en) with speech from say, streamed in real time: partials replace each other, the prompt makes Whisper output useMemo, stop finalizes the last words, and a bad key or full server shows the right error. ffmpeg -f avfoundation -i :default captured the real microphone on macOS.

Not verified / follow-ups

  • RS-5 (one undo step per phrase) still needs a manual check. Edits use undoStopBefore on the first edit of a segment and undoStopAfter only on the final. The undo command does nothing in the test window (no OS focus), so no test asserts it.
  • The full extension hasn't been run against a live microphone in VS Code (F5); only the pieces were tested.
  • Chat-input detection uses URI schemes (vscode-chat*), which is a guess; that open question in spec 0001 is still open.
  • TS-3 (optional Docker end-to-end CI job) isn't included.
  • whisper-flow bug found: if a client disconnects while a segment is being transcribed, session.stop() sends on the closed socket and raises (Unexpected ASGI message 'websocket.send'), so the session never leaves sessions. With WF_MAX_SESSIONS set, the server eventually refuses every connection. To fix in whisper-flow.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NcUyc2475ugBRGPm7yQMzy

Streams microphone audio from a sidecar recorder to a whisper-flow server
and inserts partial and final transcripts into the editor.

- whisper-flow client: /ready and version checks, ws streaming, stop/flush,
  API key from SecretStorage, handshake rejections classified as bad key or
  server full (uvicorn sends both as HTTP 403)
- Recorder sidecar: sox, arecord or ffmpeg, or whisperCode.recorderCommand
- Editor sink: dimmed pending range replaced on each partial, committed on
  final, stop, user edit or cursor move
- Workspace vocabulary sent as Whisper's prompt and applied to final text
- Context-aware formatting for comments and commit messages
- Consent for non-local servers, TLS warning, four-state status bar, Retry
- Fake whisper-flow server and recorder for tests; coverage 96%+

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NcUyc2475ugBRGPm7yQMzy
Comment thread src/test/server.test.ts
const info = await checkReady("http://localhost:8181", "k", (url, init) => {
assert.strictEqual(url, "http://localhost:8181/ready");
headers = init.headers;
return respond(READY)(url, init);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants