captionsmith

captionsmith is a deterministic Verifiers v1 taskset for evaluating and training language models to convert word-timed ASR transcripts into valid, readable SRT subtitles.

The model receives words with start/end seconds and must decide:

• which words belong in each subtitle cue;
• where the one- or two-line break belongs;
• the cue's millisecond start and end timestamps;
• how to satisfy timing, reading-speed, line-length, and gap constraints together.

No customer transcripts or FrameOS production data are included. Tasks are generated procedurally from a fixed seed and are feasibility-checked before they are emitted.

Install the public Hub release:

prime env install thesatvik/captionsmith@0.1.1

Why this task

Subtitle layout is a useful structured-output benchmark because a valid answer requires exact text preservation, temporal arithmetic, global segmentation, and multiple interacting constraints. It is also directly usable for reinforcement learning: every reward is deterministic and requires no LLM judge or external API.

The default profile is based on public Netflix timed-text guidance:

• 42 characters per line and at most two lines;
• 17 characters per second for English subtitle templates, counting spaces and punctuation;
• a two-frame gap at 24 fps (0.083s) and cue durations within a conservative 1–6s subset of Netflix's timing guidance.

This environment is inspired by those public constraints; it is not affiliated with Netflix and does not claim to reproduce Netflix's complete delivery spec.

Scoring

A malformed or overlapping SRT file cannot be aligned and receives zero on all four rewards. Valid output receives a weighted sum in [0, 1]:

Reward	Weight	What it measures
coverage	0.25	LCS token F1: every source word retained in order, with no invented words
timing	0.25	Cue starts/ends aligned to the words assigned to that cue
constraints	0.30	Lines, characters, duration, reading speed, and inter-cue gaps
segmentation	0.20	Boundaries at pauses/sentence ends/forced splits; no dangling connector words

parse_ok, cue_count, and max_cps are recorded as diagnostic metrics and do not affect reward.

The source/output alignment uses longest-common-subsequence matching, so repeated words, isolated omissions, and invented tokens are handled deterministically.

Development

From a checkout of this repository:

uv sync
uv run eval captionsmith --dry-run -n 3
uv run validate captionsmith --only-gold -n 50 --runtime.type subprocess

Run a local endpoint-backed evaluation with a tool-free harness:

uv run eval captionsmith \
  -m openai/gpt-oss-20b -n 5 -r 1 \
  --env.agent.harness.id null \
  --env.agent.runtime.type subprocess \
  --client.base-url https://api.groq.com/openai/v1 \
  --client.api-key-var GROQ_API_KEY \
  --sampling.reasoning-effort low \
  --sampling.temperature 0.2 \
  --sampling.max-tokens 1000

Configuration

The taskset is infinite and deterministic for a fixed seed; every evaluation must set -n. Important overrides include:

--env.taskset.seed
--env.taskset.min-words / --env.taskset.max-words
--env.taskset.disfluency-rate
--env.taskset.punctuation-ratio
--env.taskset.max-chars-per-line
--env.taskset.max-lines
--env.taskset.cps-cap
--env.taskset.min-gap-s
--env.taskset.min-cue-s / --env.taskset.max-cue-s

If a custom constraint profile cannot produce a feasible task after 2,000 attempts, loading fails with an actionable error instead of hanging forever.

Validation evidence

• 50/50 generated tasks pass the native model-free validate command.
• The built wheel also passes 10/10 gold tasks in a clean environment using the released verifiers==0.3.1, independently of the development monorepo.
• The published Hub artifact independently installs as thesatvik/captionsmith@0.1.1 and passes another 10/10 gold validation run.

The package accepts Prime Hosted Training's bundled native-v1 development build (verifiers==0.2.2.dev6) but clean installations resolve the current stable verifiers==0.3.1. The upper bound prevents unreviewed development releases from being selected automatically.

• The hidden reference solution scores 1.0 on every reward for every validated task.
• Malformed SRT scores 0.0 across all rewards.
• Missing or invented words reduce token coverage; shifted timings reduce timing reward.
• A 5-task × 3-rollout calibration separates Qwen3.6 27B (0.93 mean), a local quantized Gemma 4 8B (0.64), and GPT-OSS 120B with low reasoning (0.31). See BENCHMARKS.md for reward components, protocol, limitations, and public Prime evaluation records.
• Prime Hosted Training completed a one-step Laguna XS 2.1 GRPO smoke run using the public environment, with valid rollouts and non-zero reward variation.

Source and license

Hub: thesatvik/captionsmith · Source: github.com/theSatvik/captionsmith · Author: @theSatvik

MIT © Satvik Shrivas