Skip to content

DESIGN.md

Latest commit

 

History

History
101 lines (80 loc) · 4.57 KB

DESIGN.md

File metadata and controls

101 lines (80 loc) · 4.57 KB
Raw

captionsmith design

Status: implemented, validated, and published publicly as thesatvik/captionsmith@0.1.1 on the Prime Intellect Environments Hub on 2026-08-30.

Objective

captionsmith is a native verifiers.v1 taskset for evaluating and training models to turn word-timed ASR transcripts into valid, readable SRT subtitles. It is designed as a portfolio-quality Prime Intellect Environments Hub package: self-contained, procedurally generated, deterministic, and useful for RLVR.

No FrameOS customer transcript or other private production data is included.

Task contract

Each task gives the model a JSON list of words with start/end seconds and an explicit constraint profile. The model emits only SRT and must jointly choose:

  • cue boundaries and millisecond timestamps;
  • one- or two-line layouts;
  • complete, in-order coverage of the source words;
  • durations, reading speed, line lengths, and inter-cue gaps.

Default constraints are 42 characters per line, two lines per cue, 17 CPS, 1–6 second cue duration, and an 83 ms minimum gap. These defaults are inspired by public Netflix timed-text guidance, but this package is not affiliated with Netflix and is not a complete implementation of its delivery specification.

Package architecture

captionsmith/
  pyproject.toml
  README.md
  LICENSE
  captionsmith/
    __init__.py     # exports CaptionsmithTaskset
    gen.py          # seeded generator + hidden feasible reference solver
    srt.py          # strict parser and canonical formatter
    taskset.py      # native v1 data/config/task/taskset and rewards

The taskset is infinite and reproducible for a fixed seed. The generator makes ASR-like word timings from license-clean sentence components, adds configurable pauses/disfluencies/punctuation, and rejects examples that do not have a perfect reference solution. A 2,000-attempt limit turns impossible custom profiles into an actionable error rather than an infinite loop.

The hidden reference uses dynamic programming. A greedy splitter can create a valid early cue but strand a too-short final cue; searching complete paths avoids that failure. References are generated only for feasibility and validate and are never serialized into task data or shown in prompts.

Deterministic scoring

Malformed, non-sequential, or overlapping SRT cannot be aligned and gets zero on all rewards. Parseable output receives a weighted score:

Reward Weight Meaning
coverage 0.25 LCS token F1 for preserving every word in order without additions
timing 0.25 starts/ends aligned to the words assigned to each cue
constraints 0.30 line count/length, duration, CPS, and gaps
segmentation 0.20 natural or forced boundaries and no dangling connectors

Longest-common-subsequence alignment is deliberate: it handles repeated words, omissions, and insertions deterministically without an LLM judge. Punctuation and capitalization are ignored for token alignment because the prompt permits the model to add them.

The first version intentionally has no judge-based naturalness reward. This keeps results reproducible, removes external API dependence from grading, and makes rewards directly usable for reinforcement learning. A future optional judge should be treated as an experiment, not silently mixed into this score.

Validation evidence

  • Wheel and source distribution build successfully with Hatchling.
  • Ruff formatting and lint checks pass.
  • Native model-free validation passes 50/50 generated tasks; every hidden reference parses and scores exactly 1.0 on all four rewards.
  • The built wheel separately passes 10/10 gold tasks in a clean environment against the released verifiers==0.3.1 package.
  • The artifact downloaded from Prime's public package index separately installs and passes another 10/10 gold tasks.
  • Adversarial checks confirm malformed SRT scores zero, missing/invented words reduce coverage, and shifted timestamps reduce timing.
  • Five-task, three-rollout calibration runs separate Qwen3.6 27B (0.93 mean), a local quantized Gemma 4 8B (0.64), and GPT-OSS 120B with low reasoning (0.31). The protocol, component scores, caveats, and public evaluation records are in BENCHMARKS.md.
  • Prime Hosted Training completed a one-step Laguna XS 2.1 GRPO smoke run using the public 0.1.1 environment. The bounded 15-step run is in progress.

Post-release work

  1. Record the bounded Laguna training run's pre/post evaluation and checkpoints.
  2. Use training results and model failure samples to evaluate whether a future difficulty curriculum or optional naturalness judge is justified.