Repository navigation
Implement dictation end to end (spec 0001) - #8
Merged
Merged
Conversation
Streams microphone audio from a sidecar recorder to a whisper-flow server and inserts partial and final transcripts into the editor. - whisper-flow client: /ready and version checks, ws streaming, stop/flush, API key from SecretStorage, handshake rejections classified as bad key or server full (uvicorn sends both as HTTP 403) - Recorder sidecar: sox, arecord or ffmpeg, or whisperCode.recorderCommand - Editor sink: dimmed pending range replaced on each partial, committed on final, stop, user edit or cursor move - Workspace vocabulary sent as Whisper's prompt and applied to final text - Context-aware formatting for comments and commit messages - Consent for non-local servers, TLS warning, four-state status bar, Retry - Fake whisper-flow server and recorder for tests; coverage 96%+ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NcUyc2475ugBRGPm7yQMzy
| const info = await checkReady("http://localhost:8181", "k", (url, init) => { | ||
| assert.strictEqual(url, "http://localhost:8181/ready"); | ||
| headers = init.headers; | ||
| return respond(READY)(url, init); |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NcUyc2475ugBRGPm7yQMzy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements spec 0001 (overview) on top of spec 0002 (whisper-flow as the transcription engine). Dictation now works end to end: microphone → whisper-flow → editor.
What's in it
src/audio/recorder.ts: SoX, thenarecord(Linux) orffmpeg; orwhisperCode.recorderCommand. 2048-byte frames, sent unbufferedsrc/whisperflow/:/readycheck, version and protocol check,ws/wssURL,x-api-keyfrom SecretStorage,startandstopframes on 1.2.0+src/dictation/workspaceVocabulary.ts+src/text/vocabulary.ts: compound identifiers from open editors and document symbols, ranked, sent as Whisper'sprompt, and respelled in final text ("use memo" →useMemo)src/text/context.ts: comment (per-language markers, ignoring strings), commit message, promptsrc/text/format.ts: final text only; sentence case in comments, no trailing period on commit subjectssrc/dictation/editorSink.ts+src/extension.ts: dimmed pending range replaced on each partial; committed on final, stop, user edit, cursor move or editor switch; selection replaced; status bar idle/connecting/recording/error; Retrysrc/dictation/session.ts, novscodeimportAlso: consent once per non-local URL (CF-4), a warning for non-local
http://(CF-5), and the commands Set API Key / Clear API Key.Decisions worth reviewing
protocol_version: 1) addsstart(vocabulary prompt, model) andstop(finalize on stop), which settles two open questions in spec 0002. 1.1.0 still works (TR-4); there the last partial is committed as-is on stop.classifyRejectedHandshakeprobesPOST /transcribe_pcm_chunk: a 401 means the key is wrong, anything else means the server is full. The fake server reproduces the 403.ffmpeg/arecordfallbacks and a setting. No Python on the client.typecommand, so it lands in whatever input has focus (for example the chat or SCM box). Partials are skipped there.ws(custom headers on the WebSocket upgrade).npm audit --omit=devis clean.Testing
npm run checkpasses: 65 tests; coverage 98.6% lines, 96.3% branches.src/test/fakeServer.tsis a fake whisper-flow over real HTTP and WebSocket that replays events recorded from whisper-flow 1.2.0, including non-monotonic partials, the 403 rejections, and close codes 1011/4000/1000.fakeRecorder.tsreplaces the microphone.tiny.en) with speech fromsay, streamed in real time: partials replace each other, the prompt makes Whisper outputuseMemo, stop finalizes the last words, and a bad key or full server shows the right error.ffmpeg -f avfoundation -i :defaultcaptured the real microphone on macOS.Not verified / follow-ups
undoStopBeforeon the first edit of a segment andundoStopAfteronly on the final. Theundocommand does nothing in the test window (no OS focus), so no test asserts it.vscode-chat*), which is a guess; that open question in spec 0001 is still open.session.stop()sends on the closed socket and raises (Unexpected ASGI message 'websocket.send'), so the session never leavessessions. WithWF_MAX_SESSIONSset, the server eventually refuses every connection. To fix in whisper-flow.🤖 Generated with Claude Code
https://claude.ai/code/session_01NcUyc2475ugBRGPm7yQMzy