Skip to content

feat(asr): add local Whisper fallback and desktop voice input - #290

Open
IRONICBo wants to merge 3 commits into
socai-io:mainfrom
IRONICBo:feat/local-whisper-small-asr
Open

feat(asr): add local Whisper fallback and desktop voice input#290
IRONICBo wants to merge 3 commits into
socai-io:mainfrom
IRONICBo:feat/local-whisper-small-asr

Conversation

@IRONICBo

@IRONICBo IRONICBo commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • bundle a sherpa-onnx OfflineWhisper worker using the fixed Whisper small multilingual int8 model and Silero VAD
  • use the same ASR router for Desktop microphone input and site video/audio transcription
  • route signed-in accounts with an active paid subscription and remaining credits to managed cloud ASR
  • route logged-out, unpaid, expired, depleted, BYOK, and wallet-check-failure cases to local Whisper small
  • expose transcribe_audio to every account so the selected cloud/local route is actually usable

Desktop voice input

  • add microphone recording to the Desktop composer using mono 16 kHz PCM WAV with strict RIFF, size, format, alignment, and two-minute validation
  • append successful transcripts to the active composer without submitting them automatically
  • display the backend-selected route and live local model download progress
  • fall back to local ASR only when cloud access is rejected before upload (401, 402, or 403), avoiding duplicate provider work or billing
  • make the persistent local worker cancellation-safe so an interrupted request cannot leave a stale response for the next transcription

Model lifecycle and packaging

  • keep the model fixed to Whisper small; there is no low, medium, or high selector
  • download the pinned 376,129,181-byte model only on first local use, verify byte lengths and SHA-256 digests, then install atomically
  • serialize installation across processes, clean stale staging directories, and reuse one persistent helper process
  • include socai-asr beside socai in Desktop sidecars and macOS/Windows CLI release archives
  • document source installation of both socai and socai-asr

Scope

Validation

  • cargo test -p socai-core --lib: 100 passed, 0 failed, 2 ignored
  • cargo test -p socai_app --lib: 3 passed, 0 failed
  • cargo check --workspace --all-targets
  • cargo check -p socai-asr
  • pnpm --dir app build
  • git diff --check
  • isolated source installation produced executable socai and socai-asr binaries in the same bin directory
  • a real 20-second Douyin audio clip completed local Whisper small transcription and returned a Chinese transcript
  • two read-only Codex review passes completed; cloud-access race, source packaging, worker cancellation, route display, progress display, and repository-test-policy findings were addressed

@vercel

vercel Bot commented Sep 3, 2026

Copy link
Copy Markdown

@IRONICBo is attempting to deploy a commit to the socai Team on Vercel.

A member of the Team first needs to authorize it.

@IRONICBo IRONICBo changed the title feat(asr): add bundled local Whisper small fallback feat(asr): add local Whisper fallback and desktop voice input Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant