From dcaa34306b6a58a9b85a3203a6633fc7bc2f3df8 Mon Sep 17 00:00:00 2001 From: Roni bhakta <77425964+ronibhakta1@users.noreply.github.com> Date: Fri, 3 Jul 2026 15:42:35 +0530 Subject: [PATCH] docs: update README and API documentation for clarity and accuracy --- README.md | 43 ++-- docs/API.md | 557 +++++++++++++++------------------------------------- 2 files changed, 187 insertions(+), 413 deletions(-) diff --git a/README.md b/README.md index 67d640b..ef3b516 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Designed to pair with [Readium Speech](https://github.com/readium/speech) and an | **Providers** | PocketTTS (v1) · Kokoro, ElevenLabs, Azure (planned) | | **Languages** | English · French · Italian · German · Spanish · Portuguese | | **Formats** | MP3 · WAV · Opus | -| **Word boundaries** | Schema ready; supported when provider supplies timing data | +| **Word boundaries** | Schema ready, not yet populated by any provider | | **Deployment** | Docker · CPU-only · single named volume for model weights | --- @@ -45,11 +45,17 @@ curl -s -X POST http://localhost:8000/v1/synthesize \ ### Production ```bash -make start # detached, restarts automatically on crash or reboot +make start # detached — docker compose --profile nginx up -d make stop # stop all containers -make logs # tail container logs +make logs # tail app logs (nginx logs: docker compose logs -f nginx) ``` +Reachable at `http://:8080` — nginx publishes host port `8080`, plain HTTP only, no TLS termination. Put a TLS-terminating load balancer in front for a real deployment. + +`restart: unless-stopped` is set on the **app** container only; the nginx sidecar has no restart policy (`docker-compose.yml`), so it won't come back on its own after a crash or host reboot. + +nginx also rate-limits `/v1/synthesize` (2 req/s, burst 4) and caps connections per IP — a `503` from behind nginx under load is a plain nginx error page, not the app's Problem Details JSON. + --- ## Setup wizard @@ -200,7 +206,7 @@ Binary audio with `Content-Type: audio/mpeg` (or `audio/wav`, `audio/ogg`). } ``` -`boundaries` is `null` when the provider does not support word timing. Check `voice.boundary` before requesting — if `false`, the response will always return `null`. +`boundaries` is always `null` today — no provider populates timing marks yet. Every voice reports `boundary: false`; check it before setting `boundary: true` to skip a wasted round trip. Word boundary fields mirror the [Web Speech API `boundary` event](https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesisUtterance/boundary_event): `charIndex` and `charLength` index into the original `text`; `elapsedTime` is seconds from audio start. @@ -214,20 +220,24 @@ Word boundary fields mirror the [Web Speech API `boundary` event](https://develo **Errors:** -All errors return a consistent shape: +All errors are [RFC 9457 Problem Details](https://www.rfc-editor.org/rfc/rfc9457) (`Content-Type: application/problem+json`): ```json -{ "error": { "code": "voice_not_found", "message": "...", "detail": null } } +{ "type": "https://readium.org/speech-server/error#voice_not_found", "title": "Voice Not Found", "status": 404, "detail": "Voice 'urn:unknown' not found." } ``` -| Status | Code | Cause | +| Status | Type suffix | Cause | |---|---|---| | 400 | `validation_failed` | Empty or whitespace text | | 404 | `voice_not_found` | Voice URI not registered | | 413 | `payload_too_large` | Text exceeds `MAX_TEXT_LENGTH` (default 2000 chars) | -| 415 | `unsupported_format` | `format` value not in `mp3`, `wav`, `opus` | -| 422 | — | Request schema invalid (Pydantic detail) | -| 503 | — | Models not yet loaded | +| 422 | `validation_failed` | Request schema invalid (Pydantic detail) | +| 502 | `provider_error` | Provider or ffmpeg failed | +| 503 | `service_not_ready` | `/readyz` only — models not loaded, ffmpeg missing, or a provider unhealthy | + +Behind production nginx, a `503`/`429` can also come from nginx's own rate/connection limits — those are plain nginx error pages, not `application/problem+json`. + +Full field-by-field reference, including every request/response field and what's not implemented yet: [`docs/API.md`](docs/API.md). --- @@ -241,13 +251,16 @@ Run `make configure` to generate `.env`, or run `bash scripts/configure.sh` dire | `HF_TOKEN` | _(empty)_ | HuggingFace token. Optional — prevents rate-limiting on first-run model downloads | | `WORKERS` | `1` | Uvicorn worker processes. Each loads a full copy of every active language model | | `MAX_CONCURRENT_SYNTHESES` | `2` | Max parallel CPU inference jobs per worker | -| `API_KEY_ENABLED` | `false` | Require `X-API-Key` header on all routes | -| `API_KEY` | _(empty)_ | Key value when `API_KEY_ENABLED=true` | +| `API_KEY_ENABLED` | `false` | Reserved — validated at startup but **not yet enforced** on any route | +| `API_KEY` | _(empty)_ | Reserved, same caveat | | `LOG_LEVEL` | `INFO` | `DEBUG` · `INFO` · `WARNING` · `ERROR` | | `PORT` | `8000` | Listen port | | `MAX_TEXT_LENGTH` | `2000` | Maximum characters per synthesis request | | `FFMPEG_BIN` | `ffmpeg` | Path to ffmpeg binary (bundled in the Docker image) | | `POCKET_DEFAULT_VOICE` | `alba` | Default voice when none is specified | +| `ENABLED_PROVIDERS` | `pocket` | Comma-separated provider ids to register at startup. Only `pocket` exists today | +| `DEFAULT_PROVIDER` | `pocket` | Must be one of `ENABLED_PROVIDERS` — validated at startup | +| `DOMAIN` | _(empty)_ | Required when `APP_ENV=production` — used for `TrustedHostMiddleware` and nginx `server_name` | **RAM estimate:** `WORKERS × active languages × ~240 MB` @@ -323,9 +336,9 @@ Client | Provider | Status | Notes | |---|---|---| | PocketTTS | Current| CPU · 6 languages · 156 voices (26 identities × 6 languages) | -| Kokoro | 📆 Comming soon | Referenced, not vendored (IP cleanliness) | -| ElevenLabs | 📆 Comming soon | Proxied · word boundaries supported | -| Azure Speech | 📆 Comming soon | Proxied · word boundaries supported | +| Kokoro | Planned | Referenced, not vendored (IP cleanliness) | +| ElevenLabs | Planned | Proxied · word boundaries supported | +| Azure Speech | Planned | Proxied · word boundaries supported | --- diff --git a/docs/API.md b/docs/API.md index 6d82186..579b7fc 100644 --- a/docs/API.md +++ b/docs/API.md @@ -1,469 +1,230 @@ -# Readium Speech Server — API Reference & Design Notes +# API Reference -Everything implemented in the `feature/pocket-tts` branch. Covers the full -request/response contract, field-by-field meaning, and the reasoning behind -each design decision. +HTTP API for the Readium Speech Server: list voices, synthesize speech. Implements the [Readium Speech](https://github.com/readium/speech) `ReadiumSpeechUtterance` / `ReadiumSpeechVoice` contract plus server-specific extensions, called out below. ---- - -## Table of Contents - -1. [POST /v1/synthesize — Request](#post-v1synthesize--request) -2. [POST /v1/synthesize — Response (audio)](#post-v1synthesize--response-audio) -3. [POST /v1/synthesize — Response (boundary)](#post-v1synthesize--response-boundary) -4. [GET /v1/voices — Response](#get-v1voices--response) -5. [The `id` field and UUID v7](#the-id-field-and-uuid-v7) -6. [Word Boundaries — deep dive](#word-boundaries--deep-dive) -7. [Language Filtering](#language-filtering) -8. [Provider Capabilities](#provider-capabilities) -9. [Future: MathML](#future-mathml) +Base path: `/v1`. All bodies are `application/json` unless noted. --- -## POST /v1/synthesize — Request - -```json -{ - "id": "urn:uuid:019f178c-cc7c-7bb3-a39b-d185f43d3cc4", - "text": "Ceci est un test.", - "ssml": false, - "language": "fr", - "voice": "urn:readium:tts:pocket:fr-estelle", - "prev_utterance": "La nuit était sombre.", - "next_utterance": "La pièce était froide.", - "publication_id": "urn:isbn:9780000000000", - "boundary": true, - "output": { - "format": "mp3", - "bitrate": 64, - "sample_rate": null, - "speed": 1.0, - "pitch": null - } -} -``` - -### Top-level fields - -| Field | Type | Required | Default | Meaning | -|---|---|---|---|---| -| `id` | `string \| null` | No | `null` | Client-generated utterance identifier. UUID v7 URN. Used for correlation, idempotency, future caching. Currently received but not used by server. | -| `text` | `string` | **Yes** | — | The text to synthesize. Max 2000 chars (configurable via `MAX_TEXT_LENGTH`). Must not be empty/whitespace. | -| `ssml` | `boolean` | No | `false` | If `true`, `text` contains SSML markup. PocketTTS strips tags before synthesis — no SSML-aware prosody today. | -| `language` | `string \| null` | No | `null` | Language hint (`"fr"`, `"en"`). Used for provider routing in future multi-language requests. | -| `voice` | `string` | **Yes** | — | `voiceURI` from the voices list. Must exactly match a registered voice. Returns 404 if not found. | -| `prev_utterance` | `string \| null` | No | `null` | Text of the sentence before this one. Passed to provider for prosody context (natural speech flow). PocketTTS ignores today — reserved for Kokoro/commercial providers. | -| `next_utterance` | `string \| null` | No | `null` | Text of the sentence after this one. Same purpose. | -| `publication_id` | `string \| null` | No | `null` | Server extension. Identifies the ebook/document. Future: scopes the in-memory cache so utterances from different books don't collide. | -| `boundary` | `boolean` | No | `false` | If `true`, response is JSON with base64 audio + word timing marks instead of binary audio. See [boundary section](#post-v1synthesize--response-boundary). | - -### `output` object +## Contents -`output` is **optional** — omit it entirely to get mp3 at defaults. - -Why nested? The Readium Speech API spec groups audio parameters under `output` -to distinguish them from utterance properties (`text`, `language`, `voice`). -Flat structure would mix "what to say" with "how to encode it". - -| Field | Type | Default | Meaning | -|---|---|---|---| -| `format` | `"mp3" \| "wav" \| "opus"` | `"mp3"` | Output audio format. `wav` = raw PCM in RIFF container (no ffmpeg needed, fastest). `mp3` and `opus` go through ffmpeg. | -| `bitrate` | `integer \| null` | `null` | kbps for mp3/opus encoding. `null` = ffmpeg default (~128 kbps mp3, ~64 kbps opus). Ignored for `wav`. | -| `sample_rate` | `integer \| null` | `null` | Output sample rate. `null` = native model rate (24000 Hz for PocketTTS). Resampling via ffmpeg if set differently. | -| `speed` | `float` | `1.0` | Playback speed multiplier. `0.5` = half speed, `2.0` = double. PocketTTS ignores today (logs warning). Passed through for providers that support it. | -| `pitch` | `float \| null` | `null` | Pitch adjustment. PocketTTS ignores today. Passed through for providers that support it. | +- [Authentication](#authentication) +- [Errors](#errors) +- [Health](#health) +- [`GET /v1/voices`](#get-v1voices) +- [`POST /v1/synthesize`](#post-v1synthesize) +- [Not implemented](#not-implemented) --- -## POST /v1/synthesize — Response (audio) - -When `boundary: false` (default). Returns **binary audio** directly. - -``` -HTTP 200 OK -Content-Type: audio/mpeg (mp3) - audio/wav (wav) - audio/ogg (opus) -Content-Disposition: attachment; filename=speech.mp3 -``` - -Body = raw audio bytes. No JSON wrapper. - -### Error responses +## Authentication -All errors return JSON: +None enforced today. `API_KEY_ENABLED` / `API_KEY` exist in [config](../README.md#configuration) but no middleware checks them yet — every route is open regardless of the setting. Don't rely on it. -```json -{ - "error": { - "code": "voice_not_found", - "message": "Voice 'urn:unknown' not found.", - "detail": null - } -} -``` - -| Status | `error.code` | Cause | -|---|---|---| -| 400 | `validation_failed` | Empty or whitespace `text` | -| 404 | `voice_not_found` | `voice` URI not in registry | -| 413 | `payload_too_large` | `text` exceeds `MAX_TEXT_LENGTH` | -| 422 | *(Pydantic detail)* | Schema error (wrong type, missing `voice`, invalid `format`) | +In production, nginx sits in front (see [README → Production](../README.md#production)) and rate-limits `/v1/synthesize` (2 req/s, burst 4) plus per-IP connection caps — unrelated to app auth, but the only request throttling that exists. --- -## POST /v1/synthesize — Response (boundary) +## Errors -When `boundary: true`. Returns **JSON** regardless of `output.format`. +All errors are [RFC 7807 Problem Details](https://datatracker.ietf.org/doc/html/rfc7807) (`Content-Type: application/problem+json`): ```json { - "audio": "UklGRiQAAABXQVZFZm10IBAAAA...", - "format": "mp3", - "boundaries": [ - { "name": "word", "charIndex": 0, "charLength": 4, "elapsedTime": 0.0 }, - { "name": "word", "charIndex": 5, "charLength": 3, "elapsedTime": 0.31 }, - { "name": "word", "charIndex": 9, "charLength": 2, "elapsedTime": 0.52 }, - { "name": "word", "charIndex": 12, "charLength": 5, "elapsedTime": 0.68 } - ] + "type": "https://readium.org/speech-server/error#voice_not_found", + "title": "Voice Not Found", + "status": 404, + "detail": "Voice 'urn:unknown' not found.", + "instance": "urn:uuid:3f7a2e10-..." } ``` -### Response fields - -| Field | Type | Meaning | -|---|---|---| -| `audio` | `string` | Base64-encoded audio in the requested `output.format`. Decode to bytes to play. | -| `format` | `string` | Format of the encoded audio (`"mp3"`, `"wav"`, `"opus"`). | -| `boundaries` | `array \| null` | Word timing marks. **`null`** = provider does not support word boundaries. **`[]`** = supported but no marks produced. **`[{...}]`** = marks present. | - -### Why `null` not `[]` for unsupported? - -`[]` is ambiguous — could mean "supported but no words" (edge case) or "not supported." -`null` is an unambiguous sentinel: this provider cannot produce timing marks. -Client does one null-check, no invented boolean flags needed. - -``` -boundaries: null → feature not available for this voice/provider -boundaries: [] → available but nothing produced (shouldn't happen in practice) -boundaries: [...] → available + marks -``` - -### `TimingMark` fields - -Mirrors the **Web Speech API `SpeechSynthesisEvent`** boundary event fields exactly. -This is intentional — Readium clients already handle Web Speech API events; same field -names = zero translation layer. - -| Field | Type | Web Speech API equivalent | Meaning | -|---|---|---|---| -| `name` | `"word" \| "sentence"` | `event.name` | Type of boundary. Currently always `"word"`. | -| `charIndex` | `integer` | `event.charIndex` | Character offset in the original `text` string where this word starts. | -| `charLength` | `integer` | `event.charLength` | Character count of the word. `text.slice(charIndex, charIndex + charLength)` = the word. | -| `elapsedTime` | `float` | `event.elapsedTime` | Seconds from audio start when this word begins. | - -**Why no `text` field in the mark?** The original `text` is already in the request. -Client reconstructs the word: `text.substring(charIndex, charIndex + charLength)`. -Embedding the word string would duplicate data and create mismatch risk if text is -preprocessed differently. +`instance` is the request's `X-Request-Id` (also returned as a response header on every request, success or failure — use it to correlate with server logs). -**Why no `end` field?** End time = next mark's `elapsedTime` (or total audio duration -for the last word). Web Speech API omits it for the same reason. +Pydantic schema errors (`422`) additionally carry an `errors` array (raw Pydantic error list). -### Boundary support by provider - -| Provider | `supports_boundaries` | Notes | +| Status | `type` suffix | Raised when | |---|---|---| -| PocketTTS | `false` | `generate_audio()` returns PCM only — no timing data | -| ElevenLabs (future) | `true` | Returns character-level `alignment` arrays — aggregate to words | -| Azure Speech (future) | `true` | `WordBoundary` SDK events with tick offsets — convert to seconds | -| Web Speech API | `true` (browser-native) | Native `boundary` events | +| 400 | `validation_failed` | `text` is empty or whitespace-only | +| 404 | `voice_not_found` | `voice` URI not in the registry | +| 413 | `payload_too_large` | `text` exceeds `MAX_TEXT_LENGTH` | +| 422 | `validation_failed` | Request body fails schema validation (wrong type, missing required field, invalid enum value) | +| 502 | `provider_error` | Provider or ffmpeg failed (bad voice state, generation error, encode failure) | +| 503 | `service_not_ready` | `/readyz` only — models not loaded, ffmpeg missing, or a provider reports unhealthy | -For ElevenLabs: their response has `characters[]`, `character_start_times_seconds[]`, -`character_end_times_seconds[]`. Walk the arrays, emit a new `TimingMark` each time -a space or punctuation boundary is crossed. +`unsupported_format` (415), `rate_limited` (429), and `provider_timeout` (504) are declared in `app/api/errors.py` for future providers but no current code path raises them — a `429`/`503` seen in production is nginx's own rate/connection limit, a plain nginx error page, not this JSON shape. --- -## GET /v1/voices — Response - -```json -[ - { - "source": "json", - "label": "Alba (English)", - "name": "pocket-en-alba", - "originalName": "alba", - "voiceURI": "urn:readium:tts:pocket:en-alba", - "language": "en", - "gender": "female", - "quality": "normal", - "pitchControl": false, - "preloaded": true, - "provider": "pocket", - "engineVoiceId": "alba", - "sampleRate": 24000, - "mimeTypes": ["audio/mpeg", "audio/wav", "audio/ogg"], - "boundary": false - } -] -``` - -Null-valued optional fields (`localizedName`, `altNames`, `altLanguage`, `otherLanguages`, `multiLingual`, `children`, `pitch`, `rate`, `nativeID`, `note`) are omitted from the response. - -### Field groups - -**Readium `ReadiumSpeechVoice`-aligned fields** (standard — client expects these): +## Health -| Field | Meaning | -|---|---| -| `source` | Always `"json"` for server-hosted voices. `"browser"` = Web Speech API voice (client-side only). | -| `label` | Human-readable display name. Used in UI pickers. | -| `name` | Unique identifier for this voice within Readium ecosystem. | -| `originalName` | Raw engine voice ID as provided by the TTS engine. | -| `voiceURI` | **The key field** — send this in `SynthesizeRequest.voice`. Globally unique URI. | -| `language` | Language code (`"en"`, `"fr"`). Matches PocketTTS model names. Used for filtering and matching to book language. | -| `gender` | `"male"`, `"female"`, `"neutral"`, or `null`. | -| `quality` | `"veryLow"`, `"low"`, `"normal"`, `"high"`, `"veryHigh"`. PocketTTS voices = `"normal"`. | -| `preloaded` | `true` = model weights are in the image / downloaded to the weights volume, ready immediately. | -| `pitchControl` | `true` = provider accepts `output.pitch`. PocketTTS = `false`. | -| `pitch` / `rate` | Recommended defaults for this voice, if the engine specifies any. Usually `null`. | - -**Server extension fields** (not in Readium spec — added by this server): - -| Field | Meaning | -|---|---| -| `provider` | Which TTS backend serves this voice. `"pocket"` today. `"kokoro"`, `"elevenlabs"`, `"azure"` later. | -| `engineVoiceId` | Raw voice ID passed to the engine internally. Not for client use — opaque. | -| `sampleRate` | Native PCM sample rate of the model output in Hz. PocketTTS = `24000`. | -| `mimeTypes` | Audio formats this voice can produce. All voices support `["audio/mpeg", "audio/wav", "audio/ogg"]`. | -| `boundary` | **`true`** = this voice's provider supports word timing marks. Send `boundary: true` in synthesis requests. **`false`** = marks unavailable, response will have `boundaries: null`. | - -### Why `boundary` on the voice? - -Client checks `voice.boundary` **before** sending the synthesis request. -If `false`, client doesn't set `boundary: true` — saves a round trip and avoids -getting `null` back. If a client ignores the flag and sends `boundary: true` anyway, -the response carries `boundaries: null` to explain why marks are absent. +| Method | Path | Description | +|---|---|---| +| `GET` | `/healthz` | Liveness. Always `200 {"status": "ok"}` once the process is up. | +| `GET` | `/readyz` | Readiness. `200` once models are loaded, ffmpeg is on `PATH`, and every registered provider reports healthy; `503` otherwise. | --- -## The `id` field and UUID v7 - -### Anatomy of the `id` - -``` -"id": "urn:uuid:019f178c-cc7c-7bb3-a39b-d185f43d3cc4" - ───┬─── ───┬─── ─────────────────────────────── - │ │ UUID itself (128 bits, 32 hex chars + 4 dashes) - │ └──── URN namespace: UUID (IANA-registered) - └───────────── URN scheme: a name, not a network address -``` - -**`urn:`** — Uniform Resource Name (RFC 8141). Persistent global identifier that -doesn't resolve to a URL. Unlike `https://`, it just *names* something uniquely and -permanently. Good for IDs that need to survive across systems without implying a -network location. - -**`uuid:`** — The IANA-registered URN namespace for UUIDs (RFC 4122). - -### UUID v7 - -The example ID uses **UUID version 7** (RFC 9562, 2024). +## `GET /v1/voices` ``` -019f178c-cc7c-7bb3-a39b-d185f43d3cc4 -───────────────────────────────────── -First 48 bits = Unix timestamp in milliseconds -019f178c cc7c → 0x019f178ccc7c → 1750697878652 ms → 2025-06-23 ~17:57 UTC - -Version nibble = 7 (the "7" in "7bb3") - -Remaining bits = random +GET /v1/voices +GET /v1/voices?language=fr +GET /v1/voices?provider=pocket +GET /v1/voices?offset=0&limit=20 ``` -UUID version comparison: - -| Version | Year | Sortable? | Notes | +| Param | Type | Default | Description | |---|---|---|---| -| v1 | 1997 | Yes | Encodes MAC address — privacy risk | -| v4 | 2003 | No | Pure random — most common today | -| v7 | 2024 | **Yes** | Time-ordered + random — preferred for new systems | - -v7 is preferred because database indexes on UUIDs don't fragment (sorted insert order), -and you can decode the creation timestamp from the ID itself. - -### What the server does with `id` today - -**Nothing.** Field is received, parsed into `Utterance.id`, never used. - -### What it's intended for - -1. **Correlation** — synthesis requests can return out of order. Client matches - audio response to book position using `id`. - -2. **Cache key** — `(publication_id, id, voice, format)` → cached audio bytes. - Same sentence + same voice = return cached bytes, skip synthesis. Defined in - Phase 3 / `design.md §17` as in-memory LRU cache. - -3. **Idempotency** — client retries failed request with same `id` → server detects - duplicate, returns cached result, doesn't synthesize twice. - -4. **Distributed tracing** — log `id` alongside errors, duration, provider name. - "Why did this utterance fail?" traceable across log lines. - ---- - -## Word Boundaries — deep dive - -### Why boundaries matter - -Readium reads ebooks aloud. As audio plays, the UI highlights the word currently -being spoken. To highlight correctly it needs to know: "at 0.31 seconds into this -audio clip, the third word starts." That's a timing mark. - -Without boundaries: audio plays but nothing highlights. Reader loses their place. +| `language` | string | — | Filter by BCP-47 language prefix (`en`, `fr`, ...) | +| `provider` | string | — | Filter by provider id (`pocket`) | +| `offset` | int ≥ 0 | `0` | Voices to skip | +| `limit` | int ≥ 1 | none | Max voices to return | -### The Web Speech API model - -The browser's built-in TTS (`SpeechSynthesis`) fires a `boundary` event for each -word. Event fields: - -``` -charIndex integer Character offset in the utterance string -charLength integer Length of the word -elapsedTime float Seconds since speech started -name string "word" or "sentence" -``` +Response: `200`, `Voice[]`. Null-valued optional fields are omitted (`response_model_exclude_none`). -MDN reference: https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesisUtterance/boundary_event +Headers: `X-Total-Count` (matches before pagination), `X-Offset`, `X-Limit` (omitted when `limit` unset). -Our `TimingMark` is a direct mapping. When a Readium client already handles Web Speech -API boundary events, it handles our server response with zero changes. - -### The ElevenLabs model - -ElevenLabs stream-with-timestamps returns character-level arrays: +### `Voice` ```json { - "alignment": { - "characters": ["H","e","l","l","o"," ","w","o","r","l","d"], - "character_start_times_seconds": [0.0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, ...], - "character_end_times_seconds": [0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, ...] - } + "source": "json", + "label": "Alba (English)", + "name": "pocket-en-alba", + "originalName": "alba", + "voiceURI": "urn:readium:tts:pocket:en-alba", + "language": "en", + "gender": "female", + "quality": "normal", + "pitchControl": false, + "preloaded": true, + "provider": "pocket", + "engineVoiceId": "alba", + "sampleRate": 24000, + "mimeTypes": ["audio/mpeg", "audio/wav", "audio/ogg"], + "boundary": false } ``` -ElevenLabs reference: https://elevenlabs.io/docs/api-reference/text-to-speech/stream-with-timestamps - -To convert to `TimingMark` array: walk `characters[]`, track `charIndex` from the -original text, emit a new mark each time a space or punctuation boundary is crossed, -set `elapsedTime` from `character_start_times_seconds[i]`. +**`ReadiumSpeechVoice`-aligned:** -### PocketTTS limitation - -`pocket_tts.TTSModel.generate_audio(state, text)` returns a single `torch.Tensor` -of PCM samples. The entire sentence is synthesized in one pass. No per-word timing -information comes back from the model. - -Options to work around (not implemented): -- **Word-by-word synthesis** — split text on spaces, synthesize each word, measure - sample count → duration. Very slow (N model calls instead of 1). Approximate - (prosody changes at word boundaries vs. sentence context). -- **Forced alignment** — run a separate forced-alignment model (e.g. `montreal-forced-aligner`) - on the audio + transcript after synthesis. Adds latency, heavy dependency. - -Current behaviour: `boundary: true` with a PocketTTS voice returns `boundaries: null`. -`Voice.boundary = false` tells client not to ask. +| Field | Type | Notes | +|---|---|---| +| `source` | `"json" \| "browser"` | Always `"json"` — every voice here is server-hosted | +| `label` | string | Display name | +| `name` | string | Unique identifier within the Readium ecosystem | +| `originalName` | string | Raw engine voice id | +| `voiceURI` | string | Send this as `SynthesizeRequest.voice` | +| `language` | string | BCP-47 | +| `localizedName`, `altNames`, `altLanguage`, `otherLanguages`, `multiLingual`, `children`, `nativeID`, `note` | — | Not populated by any current provider — always omitted | +| `gender` | `"male" \| "female" \| "neutral" \| null` | | +| `quality` | `"veryLow"…"veryHigh" \| null` | PocketTTS voices are always `"normal"` | +| `pitchControl` | bool | `true` = provider accepts `output.pitch`. PocketTTS = `false` | +| `pitch`, `rate` | float or null | Recommended defaults, if the engine specifies any — currently always `null` | +| `preloaded` | bool | `true` = model weights already resident, ready without a cold-start delay | + +**Server extensions (not in `ReadiumSpeechVoice`):** + +| Field | Type | Notes | +|---|---|---| +| `provider` | string | Backend serving this voice — `"pocket"` today | +| `engineVoiceId` | string | Opaque, internal — not for client use | +| `sampleRate` | int | Native PCM rate in Hz (`24000` for PocketTTS) | +| `mimeTypes` | string[] | Always `["audio/mpeg", "audio/wav", "audio/ogg"]` | +| `boundary` | bool | `true` = this voice's provider fills `boundaries` on synthesis. Check before setting `boundary: true` on the request — a `false` voice always gets `boundaries: null` back | --- -## Language Filtering - -### How it works - -`LANGUAGES` environment variable = comma-separated BCP-47 prefixes: +## `POST /v1/synthesize` +```json +{ + "id": "urn:uuid:019f178c-cc7c-7bb3-a39b-d185f43d3cc4", + "text": "Ceci est un test.", + "ssml": false, + "language": "fr", + "voice": "urn:readium:tts:pocket:fr-estelle", + "prev_utterance": "La nuit était sombre.", + "next_utterance": "La pièce était froide.", + "publication_id": "urn:isbn:9780000000000", + "boundary": true, + "output": { + "format": "mp3", + "bitrate": 64, + "sample_rate": null, + "speed": 1.0, + "pitch": null + } +} ``` -LANGUAGES=en,fr -``` - -PocketTTS declares: -```python -supported_languages = frozenset({"en", "fr", "it", "de", "es", "pt"}) -``` - -`active_languages()` returns the **intersection**: -``` -{"en", "fr", "it", "de", "es", "pt"} ∩ {"en", "fr"} = {"en", "fr"} -``` - -`list_voices()` filters to voices whose `language` BCP-47 prefix is in the active set: -- `"en"` → included -- `"fr"` → included -- `"it"` → excluded - -### RAM impact -Each language model is ~240 MB. Loading 6 languages = ~1.4 GB RSS. -Default `LANGUAGES=en` = ~240 MB. Select only what you need. +Only `text` and `voice` are required; everything else defaults as shown. -### FakeProvider and language filtering +| Field | Type | Default | Notes | +|---|---|---|---| +| `id` | string \| null | `null` | Client-generated UUID v7 URN. Parsed, logged, **not otherwise used** — no caching or idempotency yet ([roadmap](#not-implemented)) | +| `text` | string | — | Max `MAX_TEXT_LENGTH` chars (2000 default). Rejected if empty/whitespace after trim | +| `ssml` | bool | `false` | PocketTTS strips tags before synthesis (regex `<[^>]+>` removal) — no SSML-aware prosody | +| `language` | string \| null | `null` | Hint only; voice resolution is by `voiceURI`, not `language` | +| `voice` | string | — | Must exactly match a `voiceURI` from `/v1/voices`. 404 if not found | +| `prev_utterance` / `next_utterance` | string \| null | `null` | Accepted, passed into `SynthesisParams`; PocketTTS ignores both | +| `publication_id` | string \| null | `null` | Accepted, currently unused (reserved for future cache scoping) | +| `boundary` | bool | `false` | `true` → JSON response with base64 audio + timing marks instead of raw binary | +| `output.format` | `"mp3" \| "wav" \| "opus"` | `"mp3"` | `wav` bypasses ffmpeg (fastest); `mp3`/`opus` are ffmpeg-encoded | +| `output.bitrate` | int \| null | `null` | kbps for `mp3`/`opus`; ffmpeg default (~128 mp3, ~64 opus) if unset; ignored for `wav` | +| `output.sample_rate` | int \| null | `null` | **Accepted but not applied** — output is always the model's native rate (24000 Hz for PocketTTS). No resampling happens today | +| `output.speed` | float | `1.0` | **Accepted but ignored** by PocketTTS (logged at debug level) | +| `output.pitch` | float \| null | `null` | **Accepted but ignored** by PocketTTS | -`FakeProvider.supported_languages = frozenset()` (empty = language-agnostic). -`active_languages()` returns empty frozenset when `supported_languages` is empty. -`list_voices()` returns all fake voices unfiltered. +### Response — audio (`boundary: false`, default) -This means tests are never affected by `LANGUAGES` config — fake voices always appear. +`200`, binary body, `Content-Type: audio/mpeg | audio/wav | audio/ogg`, `Content-Disposition: inline; filename=speech.`. ---- +### Response — boundary (`boundary: true`) -## Provider Capabilities +`200`, JSON regardless of `output.format`: -Every provider declares its capabilities as class variables. Adding a new provider -never touches routes, synthesizer, or voice catalog — only the provider class itself -and the registry. - -```python -class TTSProvider(ABC): - id: ClassVar[str] - supported_languages: ClassVar[frozenset[str]] = frozenset() # empty = all - supports_boundaries: ClassVar[bool] = False +```json +{ + "audio": "UklGRiQAAABXQVZFZm10IBAAAA...", + "format": "mp3", + "boundaries": [ + { "name": "word", "charIndex": 0, "charLength": 4, "elapsedTime": 0.0 }, + { "name": "word", "charIndex": 5, "charLength": 3, "elapsedTime": 0.31 } + ] +} ``` -| Capability | Class var | Effect | +| Field | Type | Notes | |---|---|---| -| Language scope | `supported_languages` | `list_voices()` filters automatically | -| Word boundaries | `supports_boundaries` | `Voice.boundary` set at load time; response `boundaries` is `null` vs array | +| `audio` | string | Base64, encoded in `output.format` | +| `format` | string | Echoes the requested/default format | +| `boundaries` | `TimingMark[] \| null` | `null` = voice's provider doesn't support timing (`Voice.boundary == false`) — currently true for **every** voice, since PocketTTS never populates this | -When a future provider supports boundaries: +### `TimingMark` -```python -class ElevenLabsProvider(TTSProvider): - id = "elevenlabs" - supported_languages = frozenset({"en", "fr", "de", ...}) - supports_boundaries = True # ← flip this +Mirrors the [Web Speech API `boundary` event](https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesisUtterance/boundary_event) field-for-field, so a client that already handles native `SpeechSynthesis` events needs no translation layer. - async def synthesize(self, params) -> AudioResult: - ... - return AudioResult(pcm=pcm, sample_rate=44100, boundaries=marks) # ← populate this -``` +| Field | Type | Meaning | +|---|---|---| +| `name` | `"word" \| "sentence"` | Always `"word"` today | +| `charIndex` | int | Offset of the word's first char in the request's `text` | +| `charLength` | int | Word length — `text[charIndex:charIndex+charLength]` | +| `elapsedTime` | float | Seconds from audio start | -Voices for that provider automatically get `boundary: true` in the voice list. -Synthesis response carries real marks. No other code changes. +No `end` field (next mark's `elapsedTime`, or total duration for the last word, covers it) and no `text` field (client already has the source text). --- -## Future: MathML - -MathML markup in `text` is currently passed through as-is (treated as plain text, -tags will be spoken literally). This is a known gap. +## Not implemented -When **MathCAT** is integrated: -- Detect MathML in `text` (similar to `ssml: true` detection — look for `` root tag) -- MathCAT converts MathML → spoken natural language string -- Pass the resulting string to the TTS engine as plain text +Things the schema or config surfaces but the server doesn't actually do yet: -No API shape change needed. `text` stays `string`. MathML support is a -pre-processing concern inside the provider or synthesizer, not a schema concern. +- **Auth** — `API_KEY_ENABLED` is validated at startup but never enforced on requests. +- **Word boundaries** — no provider populates `TimingMark`s. Every voice reports `boundary: false`. +- **`output.speed` / `output.pitch` / `output.sample_rate`** — accepted, validated, silently ignored by PocketTTS. +- **`id` / `publication_id`** — parsed, not used for caching, idempotency, or dedup. +- **SSML** — tags are stripped, not interpreted. No prosody control. +- **MathML** — passed through as plain text; equations get spoken as raw markup. +- **Providers beyond PocketTTS** — Kokoro, ElevenLabs, Azure are referenced in the provider interface but not wired into `_build_registry()`. See [README → provider roadmap](../README.md#provider-roadmap).