Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
132 changes: 132 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
name: CI

on:
pull_request:
push:
branches: [main]

concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true

jobs:
test:
runs-on: ubuntu-22.04
timeout-minutes: 20
env:
# Qt without a display server (UiEventBridge tests create a QCoreApplication)
QT_QPA_PLATFORM: offscreen
steps:
- uses: actions/checkout@v4

- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
libportaudio2 libasound2 \
libgl1-mesa-dev libegl1-mesa-dev libxkbcommon0 libfontconfig1 \
libdbus-1-3 libxcb-xinerama0 libxcb-cursor0 libxcb-shape0 \
libxcb-icccm4 libxcb-keysyms1 libxcb-render-util0 libxcb-image0 \
libnss3

- name: Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: 'pnpm'

- name: Install Node dependencies
run: pnpm install --frozen-lockfile

- name: Setup uv
uses: astral-sh/setup-uv@v4
with:
enable-cache: true
cache-dependency-glob: 'uv.lock'

- name: Setup Python environment
run: |
uv python install 3.12
uv venv --python 3.12 .venv
uv sync

- name: Version consistency
run: node scripts/version.mjs check

- name: Lint
run: pnpm run lint

- name: Typecheck
run: pnpm run typecheck

- name: Backend tests
# test_transcription is excluded — it downloads a whisper model.
# Clipboard tests self-skip headless (no DISPLAY/WAYLAND_DISPLAY).
run: uv run -p .venv pytest src-pyloid/tests/ -q --ignore=src-pyloid/tests/test_transcription.py

# Build verification without installers — catches PyInstaller spec drift
# (new modules/data files not bundled) before it surfaces on a release tag.
# Runs in parallel with `test` so test feedback isn't delayed. Free on
# public repos; no artifacts uploaded (dist/ is hundreds of MB and the
# release workflow's workflow_dispatch covers "build me this branch").
build:
runs-on: ubuntu-22.04
timeout-minutes: 25
env:
QT_QPA_PLATFORM: offscreen
steps:
- uses: actions/checkout@v4

- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
libgl1-mesa-dev libegl1-mesa-dev libxkbcommon0 libfontconfig1 \
libdbus-1-3 libxcb-xinerama0 libxcb-cursor0 libxcb-shape0 \
libxcb-icccm4 libxcb-keysyms1 libxcb-render-util0 libxcb-image0 \
libnss3 libasound2

- name: Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20
cache: 'pnpm'

- name: Install Node dependencies
run: pnpm install --frozen-lockfile

- name: Setup uv
uses: astral-sh/setup-uv@v4
with:
enable-cache: true
cache-dependency-glob: 'uv.lock'

- name: Setup Python environment
run: |
uv python install 3.12
uv venv --python 3.12 .venv
uv sync

- name: Build application
run: pnpm run build

- name: Smoke test - binary exists and shared libraries resolve
run: |
test -x ./dist/VoiceFlow/VoiceFlow || { echo "::error::dist/VoiceFlow/VoiceFlow missing or not executable"; exit 1; }
MISSING=$(ldd ./dist/VoiceFlow/VoiceFlow 2>&1 | grep "not found" || true)
if [ -n "$MISSING" ]; then
echo "::error::Missing shared libraries:"
echo "$MISSING"
exit 1
fi
echo "Build OK: $(du -sh dist/VoiceFlow | cut -f1) bundle, all shared libraries resolved."
43 changes: 43 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,19 @@ jobs:
fi
echo "version=$VERSION" >> "$GITHUB_OUTPUT"

- name: Verify version consistency
run: |
node scripts/version.mjs check
# On tag builds, the tag must match the committed version — catches
# tagging the wrong commit before 30 minutes of build time is spent.
if [[ "${{ github.event_name }}" == "push" ]]; then
PKG=$(node -p "require('./package.json').version")
if [[ "$PKG" != "${{ steps.version.outputs.version }}" ]]; then
echo "::error::Tag v${{ steps.version.outputs.version }} does not match package.json version $PKG — bump with scripts/version.mjs and re-tag."
exit 1
fi
fi

- name: Build application
run: pnpm run build

Expand All @@ -98,6 +111,24 @@ jobs:
rm -f ./dist/VoiceFlow/_internal/libasound*
echo "Removed bundled PortAudio + ALSA (will use system libraries at runtime)"

- name: Verify audio library layout
run: |
# Regression guard for the recurring "no mics in release build" bug
# (v1.3.x–v1.5.0): the bundled libasound/libportaudio MUST be gone...
LEFTOVER=$(find ./dist/VoiceFlow/_internal -maxdepth 1 \( -name 'libasound*' -o -name 'libportaudio*' \) | head -5)
if [ -n "$LEFTOVER" ]; then
echo "::error::Bundled audio libraries still present (would break mic enumeration on non-Ubuntu distros): $LEFTOVER"
exit 1
fi
# ...while PyAV's renamed copy MUST remain — libavdevice lists
# libasound-cfbebb71.so.2.0.0 in NEEDED and the app crashes at
# startup without it (different SONAME, coexists with system lib).
if ! ls ./dist/VoiceFlow/_internal/av.libs/libasound-* >/dev/null 2>&1; then
echo "::error::PyAV's av.libs/libasound-* is missing — startup will fail with ImportError. The cleanup step deleted too much."
exit 1
fi
echo "Audio library layout OK (system libasound/portaudio at runtime, PyAV copy intact)"

- name: Clear executable stack flags
run: |
# python-build-standalone builds ship libpython with GNU_STACK RWE,
Expand Down Expand Up @@ -183,6 +214,18 @@ jobs:
fi
echo "version=$VERSION" >> "$GITHUB_OUTPUT"

- name: Verify version consistency
shell: bash
run: |
node scripts/version.mjs check
if [[ "${{ github.event_name }}" == "push" ]]; then
PKG=$(node -p "require('./package.json').version")
if [[ "$PKG" != "${{ steps.version.outputs.version }}" ]]; then
echo "::error::Tag v${{ steps.version.outputs.version }} does not match package.json version $PKG — bump with scripts/version.mjs and re-tag."
exit 1
fi
fi

- name: Build application
run: pnpm run build

Expand Down
75 changes: 65 additions & 10 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

VoiceFlow is a cross-platform voice-to-text paste utility built with Pyloid (Python desktop framework using PySide6/Qt WebEngine) and React. Users hold a hotkey to record audio, release to transcribe using faster-whisper, and the text is automatically pasted at the cursor. Supports Windows, Linux (Wayland/X11), and macOS.

As of v1.6.0, VoiceFlow also ships a **Meeting Mode** — long-form recording with mic + system-loopback capture, async transcription, and LLM-generated summaries. This is a separate feature surface from the push-to-talk paste flow, kept self-contained under `services/recording/` so it can evolve (or be extracted) without touching PTT code.

## Commands

```bash
Expand All @@ -26,28 +28,44 @@ pnpm run build:installer # Windows (.exe via Inno Setup)
pnpm run build:installer:linux # Linux (.tar.gz + .AppImage)
pnpm run build:installer:macos # macOS (.dmg)

# Run Python tests
cd VoiceFlow && uv run -p .venv pytest src-pyloid/tests/
# Run Python tests (fast suite — excludes test_transcription, which downloads a model)
pnpm run test
pnpm run test:all # includes test_transcription

# Run single test file
uv run -p .venv pytest src-pyloid/tests/test_transcription.py -v

# Everything CI runs (lint + typecheck + tests + version consistency)
pnpm run check

# Version bump — updates all five version files (package.json, voiceflow.iss,
# pyproject.toml, constants.ts, uv.lock); never edit them by hand
pnpm run version:bump X.Y.Z
pnpm run version:check

# Run frontend only (for UI development)
pnpm run vite

# Lint frontend
# Lint / typecheck frontend
pnpm run lint
pnpm run typecheck
```

CI (`.github/workflows/ci.yml`) runs lint, typecheck, version check, and the fast
test suite on every PR and push to main. The release workflow
(`.github/workflows/release.yml`) additionally refuses tags that don't match
package.json and verifies the Linux artifact's audio-library layout
(bundled libasound/libportaudio removed, PyAV's av.libs copy intact).

## Architecture

### Backend (src-pyloid/)

Python backend using Pyloid framework with PySide6:

- **main.py** - Application entry point. Creates Pyloid app, tray icon, main dashboard window, and recording popup window. Sets up UI callbacks connecting backend events to popup state changes.
- **server.py** - RPC server using `PyloidRPC`. Exposes methods (`get_settings`, `update_settings`, `get_history`, etc.) that frontend calls via `pyloid-js` RPC.
- **app_controller.py** - Singleton controller orchestrating all services. Handles hotkey activate/deactivate flow: start recording -> stop recording -> transcribe -> paste at cursor -> save to history.
- **server.py** - RPC server using `PyloidRPC`. Exposes PTT methods (`get_settings`, `update_settings`, `get_history`, etc.) plus the full Meeting Mode surface (`meetings.list_audio_sources`, `meetings.start`, `meetings.pause`, `meetings.resume`, `meetings.stop`, `meetings.transcribe`, `meetings.summarize`, `meetings.get_llm_config`, `meetings.test_llm_connection`, etc.) that frontend calls via `pyloid-js` RPC.
- **app_controller.py** - Singleton controller orchestrating all services. Handles hotkey activate/deactivate flow: start recording -> stop recording -> transcribe -> paste at cursor -> save to history. Also constructs and owns the `MeetingsController` (exposed as `controller.meetings`) and runs an unfinished-recording recovery sweep on startup.

**Services (src-pyloid/services/):**
- `audio.py` - Microphone recording using sounddevice, streams amplitude for visualizer
Expand All @@ -56,21 +74,37 @@ Python backend using Pyloid framework with PySide6:
- `clipboard.py` - Clipboard operations and paste-at-cursor using pyautogui
- `settings.py` - Settings management with defaults
- `database.py` - SQLite database for settings and history (stored at ~/.VoiceFlow/VoiceFlow.db)
- `logger.py` - Domain-based logging with hybrid format `[timestamp] [LEVEL] [domain] message | {json}`. Supports domains: model, audio, hotkey, settings, database, clipboard, window. Configured with 100MB log rotation.
- `model_manager.py` - Whisper model download/cache management using huggingface_hub. Provides download progress tracking (percent, speed, ETA), cancellation via CancelToken, daemon thread execution, and `clear_cache()` to delete only VoiceFlow's faster-whisper models.
- `logger.py` - Domain-based logging with hybrid format `[timestamp] [LEVEL] [domain] message | {json}`. Supports domains: model, audio, hotkey, settings, database, clipboard, window, plus Meeting-Mode domains (recording, transcribe, summary, llm). Configured with 100MB log rotation.
- `model_catalog.py` - Single source of truth for the Whisper model catalog (names, sizes, HF repo IDs). Imported by `model_manager`, `transcription`, and `settings`; the frontend gets repo IDs over RPC (`get_model_info` → `repoId`) rather than keeping its own copy. Never re-declare the model list elsewhere.
- `model_manager.py` - Whisper model download/cache management using huggingface_hub. Owns the single-flight background download session (`start_download` / `cancel_download` / `get_download_status` — the RPC surface), download progress tracking (percent, speed, ETA), cancellation via CancelToken, and `delete_model()` / `clear_cache()`. Model *loading* lives in `TranscriptionService` (the only load path, since it resolves the user's device preference).

**Meeting Mode services (src-pyloid/services/recording/):**
Self-contained per `docs/adr/0003-meeting-mode-isolation.md`; do not call these from the PTT path and vice versa.
- `controller.py` - `MeetingsController` — the feature's facade. All RPC handlers go through this object. Emits `recording-state`, `recording-transcribe-progress`, and `recording-summarize-progress` events to the frontend via the emitter installed by `main.py`.
- `recorder.py` - Long-form recorder with pause/resume, segmented WAV writing, and clock tracking. Sources are fixed at `start()` and cannot change mid-recording.
- `audio_source.py` - Enumerates available mic + loopback devices for the UI device picker.
- `loopback_linux.py` / `loopback_pulse.py` / `loopback_windows.py` - Platform-specific system-audio capture (PulseAudio/PipeWire on Linux, WASAPI loopback on Windows).
- `clock.py` - Monotonic recording clock that survives pause/resume.
- `llm.py` - LLM client with preset + custom-endpoint support; used by summary/title generation.
- `summary.py` / `title.py` - LLM-driven summary and auto-title generation for finished recordings.
- `secrets.py` - API-key storage for LLM providers (kept out of the main settings table).
- `export.py` - Exports a recording's transcript/summary to text formats.
- `recovery.py` - On startup, sweeps recordings left in `recording` / `paused` state from a previous (crashed) session and rolls them forward.
- `audio_scheme.py` / `audio_scheme_handler.py` - Custom Qt `audio://` URL scheme so the WebEngine `<audio>` element can stream recording WAVs from disk without a server.

### Frontend (src/)

React 18 + TypeScript + Vite frontend:

- **App.tsx** - Hash-based routing between `/popup`, `/onboarding`, and `/dashboard`. Checks model cache on startup and shows recovery modal if model is missing.
- **lib/api.ts** - RPC wrapper using `pyloid-js` to call Python backend methods. Includes model management APIs (`getModelInfo`, `startModelDownload`, `cancelModelDownload`).
- **lib/types.ts** - TypeScript interfaces for Settings, HistoryEntry, Stats, Options, ModelInfo, DownloadProgress
- **pages/** - Popup (recording indicator), Onboarding (includes model download step), Dashboard
- **lib/api.ts** - RPC wrapper using `pyloid-js` to call Python backend methods. Includes model management APIs (`getModelInfo`, `startModelDownload`, `cancelModelDownload`) plus the full `recordings*` / meetings RPC surface.
- **lib/types.ts** - TypeScript interfaces for Settings, HistoryEntry, Stats, Options, ModelInfo, DownloadProgress, plus meeting types: `Recording`, `RecordingSegment`, `RecorderState`, and LLM config types.
- **pages/** - Popup (recording indicator), Onboarding (includes model download step), Dashboard. Dashboard uses React Router for sub-routes: `history`, `meetings`, `meetings/record`, `meetings/:id`, `settings`.
- **components/** - Feature components plus shadcn/ui components in `components/ui/`
- `ModelDownloadProgress.tsx` - Download progress UI with progress bar, speed, ETA, and retry support
- `ModelDownloadModal.tsx` - Dialog wrapper for model downloads triggered from settings
- `ModelRecoveryModal.tsx` - Startup modal for missing model recovery
- `meetings/` - Meeting Mode UI: `MeetingsListPage`, `MeetingRecorderPage`, `MeetingDetailPage`, `MeetingImportDialog`, `MeetingRecorderContext` (cross-route recorder state), `AudioPlayer`, `LevelMeter`, `StatusLine`, `TranscriptView`, `SummaryView`, `RetranscribeDialog`, `LLMSettingsSection`, `MeetingsSettingsSection`.

### Frontend-Backend Communication

Expand Down Expand Up @@ -124,6 +158,27 @@ For transparent popup windows on Windows:
6. On completion, model is cached in huggingface cache directory
7. Turbo model uses `mobiuslabsgmbh/faster-whisper-large-v3-turbo` (same as faster-whisper internal mapping)

### Meeting Mode (long-form recording)

Separate from the PTT paste flow. Entrypoint: `controller.meetings` (`MeetingsController`). See `docs/adr/0001-stereo-channel-layout-for-recordings.md` for the on-disk audio layout decision.

1. UI calls `meetings.list_audio_sources()` to populate the device picker (mic + loopback).
2. `meetings.start(mic_device_id, loopback_device_id)` opens up to two simultaneous capture streams. Sources are fixed at start — they cannot be added/removed mid-recording.
3. Audio is written to a WAV file under `~/.VoiceFlow/recordings/`:
- Two active sources → **stereo 16 kHz PCM16**, mic on **L**, loopback on **R** (kept separate on purpose; enables future speaker diarization with no ML — see ADR 0001).
- One active source → mono 16 kHz PCM16.
4. `meetings.pause()` / `meetings.resume()` use a monotonic `Clock` to track real recording time; segments are stitched into one logical recording.
5. `meetings.stop()` finalizes the WAV and persists metadata. Recording rows live in the same SQLite DB but in their own table.
6. Transcription is **async and on-demand**: `meetings.transcribe(id)` runs faster-whisper in a daemon thread and emits `recording-transcribe-progress` events. Long jobs do not block the RPC channel (see fix `dc04d29`).
7. After transcription, `meetings.summarize(id, prompt)` calls the configured LLM provider (preset or custom endpoint) to produce an AI summary, and `title.py` auto-generates a title. LLM config and API keys live in `services/recording/llm.py` + `secrets.py`, not in the main `settings` table.
8. Audio playback in the detail page uses a custom Qt `audio://` URL scheme (`audio_scheme.py`) so the WebEngine can stream the WAV directly without an HTTP server.
9. On startup, `recovery.py` rolls forward any recordings left in `recording` / `paused` state from a crashed previous session.

**Platform quirks**:
- Linux loopback uses PulseAudio/PipeWire monitor sources (`loopback_pulse.py` / `loopback_linux.py`).
- Windows loopback uses WASAPI. Must open the loopback stream at the device's native channel count (`max_output_channels`) — opening at a forced channel count fails on many devices (fixes `96b0b73`, `13d45be`).
- Pyloid validates window IDs on every RPC roundtrip from a background thread; long-running meeting RPCs work around this (see `135fdd7`).

## Key Patterns

- **Singleton controller**: `get_controller()` returns singleton `AppController` instance
Expand Down
18 changes: 18 additions & 0 deletions docs/adr/0001-stereo-channel-layout-for-recordings.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Stereo channel layout for two-source Recordings

When a Recording captures both a mic and a system-loopback source, the WAV is
written as stereo 16 kHz PCM16 with the **mic on the left channel (ch 0)** and
the **loopback on the right channel (ch 1)** — deliberately *not* mixed down to
mono. Keeping the sides separate means "who said what" (you vs. them) is
recoverable later by channel, enabling speaker attribution with zero ML
diarization. A single-source Recording is mono.

## Consequences

- Sources are fixed at `start()`; a source cannot be added or removed
mid-recording, because the channel count of the WAV is decided up front.
- In stereo mode the recorder pads a starved side with silence (starvation
detection in `recorder.py`) so the channels stay time-aligned — alignment is
the property that makes per-channel attribution trustworthy.
- Playback of a stereo Recording sounds hard-panned (you fully left, them
fully right). That is accepted, not a bug.
Loading
Loading