Skip to content

Move punctuation cleanup policy out of TypeScript #16

Description

@ctoth

Problem

punctuationCleanup remains a hardcoded normalization policy island. The YAML pipeline names the step, but the actual pause-punctuation set, lexical apostrophe protection, placeholder protocol, and strip policy are still embedded in TypeScript regex code.

Evidence

  • public/rules/frontends/qlatt-english/normalization-pipeline.yaml has punctuation_cleanup as type: builtin, handler: punctuationCleanup.
  • src/g2p/text-normalize.ts implements punctuationCleanup with a private placeholder ("\x00") and hardcoded punctuation regexes for , . ? ! ; :.
  • The same punctuation marks are already declarative transcription data in public/rules/frontends/qlatt-english/frontend.yaml under transcription.punctuation_tokens, so there is drift risk between normalization and transcription.

Acceptance criteria

  • The pause-generating punctuation inventory is loaded from declarative config, not duplicated as a regex literal in punctuationCleanup.
  • Lexical apostrophe preservation and strip policy are represented as explicit config or a generic named normalizer whose behavior is table-driven.
  • Add tests proving a config change to punctuation tokens affects normalization/tokenization consistently.
  • Remove the hardcoded [,.?!;:] policy from src/g2p/text-normalize.ts.
  • npm.cmd run typecheck:core passes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions