Skip to content

WHISPER_PROTOCOL.md

Latest commit

 

History

History
397 lines (305 loc) · 7.8 KB

WHISPER_PROTOCOL.md

File metadata and controls

397 lines (305 loc) · 7.8 KB
Raw

Whisper Bridge Protocol Reference

Quick reference for the C++ ↔ Python subprocess communication protocol.

Subprocess Launch

Command:

python3 aether_transcriber_bridge.py <model> <sample_rate> <chunk_duration>

Example:

python3 aether_transcriber_bridge.py base 16000 3.0

Arguments:

  • model: Whisper model name (tiny, base, small, medium, large)
  • sample_rate: Audio sample rate in Hz (typically 16000)
  • chunk_duration: Seconds of audio per transcription (typically 3.0)

Data Flow

stdin (C++ → Python)

Format: Raw int16 PCM audio stream

Byte Stream: [s1_lo][s1_hi][s2_lo][s2_hi][s3_lo][s3_hi]...
             └─ int16 ─┘ └─ int16 ─┘ └─ int16 ─┘

Properties:

  • Type: int16_t (signed 16-bit)
  • Range: -32768 to 32767
  • Endianness: Little-endian
  • Channels: 1 (mono)
  • No framing, no headers

C++ Code:

std::vector<int16_t> audio_data = {100, 200, 300, ...};
write(stdin_fd, audio_data.data(), audio_data.size() * sizeof(int16_t));

Python Code:

chunk_bytes = sys.stdin.buffer.read(8192)
chunk_samples = np.frombuffer(chunk_bytes, dtype=np.int16)
audio_float = chunk_samples.astype(np.float32) / 32768.0

stdout (Python → C++)

Format: Line-delimited UTF-8 text

play some music\n
open the browser\n
what time is it\n

Properties:

  • Encoding: UTF-8
  • Delimiter: \n (newline)
  • One transcription per line

Python Code:

print(text, flush=True)  # Must flush!

C++ Code:

std::string line;
std::getline(stdout_stream, line);

stderr (Python → C++)

Format: Line-delimited log messages

Loading Whisper model: base...
Model loaded successfully (sample_rate=16000, chunk_duration=3.0s)
READY
Transcribed: play some music
Error: ...

Purpose: Debugging, status updates, error messages

Startup Sequence

  1. C++ launches subprocess

    fork() + execvp("python3", "script.py", "base", "16000", "3.0")

  2. Python loads model

    model = whisper.load_model("base")
sys.stderr.write("Model loaded successfully\n")

  3. Python signals ready

    print("READY", flush=True)

  4. C++ starts streaming audio

    write(stdin_fd, audio_data, size)

  5. Python accumulates and transcribes

    while len(buffer) >= chunk_size:
    result = model.transcribe(buffer[:chunk_size])
    print(result["text"], flush=True)

  6. C++ reads transcriptions

    std::string text;
if (readText(text)) {
    publish(text);
}

Buffering Strategy

Python Side

audio_buffer = np.array([], dtype=np.int16)

while True:
    # Read chunk from stdin
    chunk = sys.stdin.buffer.read(8192)
    samples = np.frombuffer(chunk, dtype=np.int16)
    
    # Append to buffer
    audio_buffer = np.concatenate([audio_buffer, samples])
    
    # Process when full
    if len(audio_buffer) >= chunk_size:
        audio_chunk = audio_buffer[:chunk_size]
        audio_buffer = audio_buffer[chunk_size:]
        
        # Transcribe
        audio_float = audio_chunk.astype(np.float32) / 32768.0
        result = model.transcribe(audio_float)
        print(result["text"], flush=True)

Key Points:

  • Accumulate samples until chunk_size reached
  • Process in fixed-size chunks (e.g., 3 seconds = 48000 samples @ 16kHz)
  • Keep remainder in buffer for next iteration

C++ Side

// Write audio continuously (non-blocking)
bool writeAudio(const std::vector<int16_t>& audio_data) {
    write(stdin_fd, audio_data.data(), audio_data.size() * sizeof(int16_t));
}

// Read transcriptions asynchronously
bool readText(std::string& text) {
    if (stdout_buffer.empty()) return false;
    text = stdout_buffer.pop_front();
    return true;
}

Key Points:

  • Write audio as soon as available (real-time)
  • Read transcriptions asynchronously (polling or callback)
  • No blocking on either side

Error Handling

Broken Pipe (EPIPE)

Cause: Subprocess crashed or closed stdin

C++ Detection:

if (errno == EPIPE) {
    // Subprocess died
    running_ = false;
    if (auto_restart) restartSubprocess();
}

Python Prevention:

try:
    chunk = sys.stdin.buffer.read(8192)
    if not chunk:
        break  # EOF - parent closed pipe
except KeyboardInterrupt:
    break

EOF (End of File)

Cause: Parent closed stdin (graceful shutdown)

Python Handling:

chunk = sys.stdin.buffer.read(8192)
if not chunk:
    sys.stderr.write("EOF received, exiting\n")
    break

Subprocess Crash

C++ Detection:

int status;
pid_t result = waitpid(pid, &status, WNOHANG);
if (result == pid) {
    // Process exited
    if (WIFEXITED(status)) {
        int code = WEXITSTATUS(status);
        // Handle exit code
    }
}

Auto-restart:

if (auto_restart && restart_count < max_restarts) {
    restartSubprocess();
}

Performance Tuning

Chunk Duration

Trade-off: Latency vs Accuracy

Duration Latency Accuracy Use Case
1.0s Low Lower Interactive commands
3.0s Medium Good General use (default)
5.0s High Better Dictation, long phrases

Configuration:

// In whisper_bridge.cpp
const char* args[] = {
    "python3", "script.py", "base", "16000",
    "1.0",  // ← Change this
    nullptr
};

Model Selection

Trade-off: Speed vs Accuracy

Model Size Speed Accuracy RAM
tiny 39M Fast 70% 1GB
base 74M Fast 80% 1GB
small 244M Medium 85% 2GB
medium 769M Slow 90% 5GB
large 1550M Very Slow 95% 10GB

Configuration:

// In whisper_bridge.cpp
const char* args[] = {
    "python3", "script.py",
    "tiny",  // ← Change this
    "16000", "3.0", nullptr
};

Buffer Size

Read size: 8192 bytes = 4096 samples @ int16

Calculation:

Sample Rate: 16000 Hz
Chunk Duration: 3.0 seconds
Chunk Size: 16000 * 3.0 = 48000 samples
Bytes: 48000 * 2 = 96000 bytes

Read iterations: 96000 / 8192 ≈ 12 reads per chunk

Debugging

Enable Verbose Logging

Python:

sys.stderr.write(f"Read {len(chunk_bytes)} bytes\n")
sys.stderr.write(f"Buffer size: {len(audio_buffer)} samples\n")
sys.stderr.write(f"Transcribing chunk of {len(audio_chunk)} samples\n")
sys.stderr.flush()

C++:

RCLCPP_DEBUG(logger, "Wrote %zu bytes to subprocess", bytes_written);
RCLCPP_DEBUG(logger, "Read transcription: %s", text.c_str());

Monitor Pipes

Check if subprocess is reading:

# In container
lsof -p <pid> | grep pipe

Check buffer sizes:

# In container
cat /proc/<pid>/fdinfo/0  # stdin
cat /proc/<pid>/fdinfo/1  # stdout

Test Manually

Simulate C++ behavior:

# Generate test audio (1 second of 440Hz tone)
python3 -c "
import numpy as np
samples = (np.sin(2 * np.pi * 440 * np.linspace(0, 1, 16000)) * 32767).astype(np.int16)
samples.tofile('/tmp/test.raw')
"

# Feed to subprocess
cat /tmp/test.raw | python3 aether_transcriber_bridge.py base 16000 3.0

Reference Implementation

Minimal C++ Writer

#include <unistd.h>
#include <vector>

void writeAudio(int fd, const std::vector<int16_t>& audio) {
    const uint8_t* data = reinterpret_cast<const uint8_t*>(audio.data());
    size_t size = audio.size() * sizeof(int16_t);
    write(fd, data, size);
}

Minimal Python Reader

import sys
import numpy as np

while True:
    chunk = sys.stdin.buffer.read(8192)
    if not chunk:
        break
    samples = np.frombuffer(chunk, dtype=np.int16)
    # Process samples...

Protocol Version: 1.0
Last Updated: 2025-10-10
Maintained by: Kiro