Skip to content

transcript.language stores provider free-text ("English"), not a BCP-47 code #194

Description

@utof

Summary

transcript.language is written straight through from the provider's verbose_json response, which returns a human-readable English language name, not a language code. The input side of the same feature demands BCP-47. The column is therefore unusable for filtering or grouping.

Evidence

Two rows produced by the same code path on the same machine:

source stored language
Groq whisper-large-v3-turbo, real audio English
wiremock fixture (hello.wav) en

From the application log:

transcription completed request_uuid=019fbe1d… segment_count=1   language=Some("en")
transcription completed request_uuid=019fbe35… segment_count=124 language=Some("English")

Meanwhile perima transcribe --language validates a BCP-47 subset:

--language <LANGUAGE>
        Optional BCP-47 language hint (e.g. `en`, `es`, `zh-Hans`).
        Loose RFC-5646-subset validation: 2-3 letter language with optional
        4-letter script and optional 2-letter or 3-digit region.

So the feature accepts en on the way in and stores English on the way out.

Impact

  • A multilingual library stores English, Russian, Spanish… as free text. Filtering "show me my Russian recordings" requires matching both the code and every provider's spelling.
  • The value is provider-dependent, not just verbose: OpenAI and Groq do not guarantee identical casing or naming, and a custom OpenAI-compatible endpoint can return anything. Two backends transcribing the same file can disagree on the stored value.
  • Round-tripping is broken: the stored language cannot be fed back as a --language hint for re-transcription, because it fails the BCP-47 validator.

Where

The adapter deserializes the provider field verbatim (crates/transcribe/src/openai_compat.rs):

#[derive(serde::Deserialize)]
struct VerboseTranscription {
    language: String,
    duration: f32,
    #[serde(default)]
    segments: Vec<VerboseSegment>,
}

No normalization happens between there and the transcript row.

Options

  1. Normalize in the adapter — map the provider's name to a BCP-47 code before constructing the transcript, so the column holds exactly one vocabulary. Needs a name→code mapping; unic-langid is already contemplated for the --language validator (see Umbrella: ASR / transcription v2 follow-ups #176), so the dependency may land anyway.
  2. Store both — add language_raw for the provider's string and keep language normalized. Preserves provider fidelity for debugging; costs a column and a migration.
  3. Document the column as free text and do the normalization at query time. Cheapest now, pushes the problem onto every future consumer (viewer, search facet, filter UI).

Option 1 looks right: the column already has a single documented meaning, and every consumer will otherwise reimplement the same mapping.

Note

Not covered by the umbrella #176. Its BCP-47 item (unic-langid-based validation for --language) concerns input validation only; this is about output normalization and is independent of whether that item lands.

Found while verifying a live 7m30s Groq transcription end-to-end (124 segments, correct timestamps) — everything else about the row was correct.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/transcribeTouches the perima-transcribe crate or transcription pipelinepriority/mediumDefault priority — address in normal flowtype/bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions