You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
transcript.language is written straight through from the provider's verbose_json response, which returns a human-readable English language name, not a language code. The input side of the same feature demands BCP-47. The column is therefore unusable for filtering or grouping.
Evidence
Two rows produced by the same code path on the same machine:
Meanwhile perima transcribe --language validates a BCP-47 subset:
--language <LANGUAGE>
Optional BCP-47 language hint (e.g. `en`, `es`, `zh-Hans`).
Loose RFC-5646-subset validation: 2-3 letter language with optional
4-letter script and optional 2-letter or 3-digit region.
So the feature accepts en on the way in and stores English on the way out.
Impact
A multilingual library stores English, Russian, Spanish… as free text. Filtering "show me my Russian recordings" requires matching both the code and every provider's spelling.
The value is provider-dependent, not just verbose: OpenAI and Groq do not guarantee identical casing or naming, and a custom OpenAI-compatible endpoint can return anything. Two backends transcribing the same file can disagree on the stored value.
Round-tripping is broken: the stored language cannot be fed back as a --language hint for re-transcription, because it fails the BCP-47 validator.
Where
The adapter deserializes the provider field verbatim (crates/transcribe/src/openai_compat.rs):
No normalization happens between there and the transcript row.
Options
Normalize in the adapter — map the provider's name to a BCP-47 code before constructing the transcript, so the column holds exactly one vocabulary. Needs a name→code mapping; unic-langid is already contemplated for the --language validator (see Umbrella: ASR / transcription v2 follow-ups #176), so the dependency may land anyway.
Store both — add language_raw for the provider's string and keep language normalized. Preserves provider fidelity for debugging; costs a column and a migration.
Document the column as free text and do the normalization at query time. Cheapest now, pushes the problem onto every future consumer (viewer, search facet, filter UI).
Option 1 looks right: the column already has a single documented meaning, and every consumer will otherwise reimplement the same mapping.
Note
Not covered by the umbrella #176. Its BCP-47 item (unic-langid-based validation for --language) concerns input validation only; this is about output normalization and is independent of whether that item lands.
Found while verifying a live 7m30s Groq transcription end-to-end (124 segments, correct timestamps) — everything else about the row was correct.
Summary
transcript.languageis written straight through from the provider'sverbose_jsonresponse, which returns a human-readable English language name, not a language code. The input side of the same feature demands BCP-47. The column is therefore unusable for filtering or grouping.Evidence
Two rows produced by the same code path on the same machine:
languagewhisper-large-v3-turbo, real audioEnglishhello.wav)enFrom the application log:
Meanwhile
perima transcribe --languagevalidates a BCP-47 subset:So the feature accepts
enon the way in and storesEnglishon the way out.Impact
English,Russian,Spanish… as free text. Filtering "show me my Russian recordings" requires matching both the code and every provider's spelling.languagecannot be fed back as a--languagehint for re-transcription, because it fails the BCP-47 validator.Where
The adapter deserializes the provider field verbatim (
crates/transcribe/src/openai_compat.rs):No normalization happens between there and the
transcriptrow.Options
unic-langidis already contemplated for the--languagevalidator (see Umbrella: ASR / transcription v2 follow-ups #176), so the dependency may land anyway.language_rawfor the provider's string and keeplanguagenormalized. Preserves provider fidelity for debugging; costs a column and a migration.Option 1 looks right: the column already has a single documented meaning, and every consumer will otherwise reimplement the same mapping.
Note
Not covered by the umbrella #176. Its BCP-47 item (
unic-langid-based validation for--language) concerns input validation only; this is about output normalization and is independent of whether that item lands.Found while verifying a live 7m30s Groq transcription end-to-end (124 segments, correct timestamps) — everything else about the row was correct.