Add ntt-complex.format_ast benchmark scaffold

[vmendelev]

Add ntt-complex.format_ast benchmark scaffold

Summary

This PR adds the bootstrap ntt-complex benchmark group and its first subtest, ntt-complex.format_ast.

format_ast evaluates a complex2 task: audio speech translation plus strict output formatting. It renders existing FLEURS and CoVoST2 AST/ST examples into three required output formats:

• json_object
• srt_single_cue
• markdown_table

The evaluator validates the requested structure, extracts the translated text, and scores the extracted translation with the standard audio translation BLEU path.

Implementation

Added:

• nemo_skills/dataset/ntt-complex/prepare.py
• nemo_skills/dataset/ntt-complex/ntt_complex_eval.py
• nemo_skills/dataset/ntt-complex/ntt_complex_metrics.py
• nemo_skills/dataset/ntt-complex/README.md
• tests/test_ntt_complex_format_ast.py

The prepare script accepts source manifests from both naming layouts used across the current code and canary-dev references:

• fleurs/st/test.jsonl or fleurs/ast/test.jsonl
• covost2/st/test.jsonl or covost2/ast/test.jsonl

Validation

Local tests on branch codex/ntt-complex-format-ast:

python -m compileall nemo_skills/dataset/ntt-complex tests/test_ntt_complex_format_ast.py
python -m pytest -s tests/test_ntt_complex_format_ast.py

Result:

• 3 passed

Draco/IAD data preflight:

• Workflow: ntt-complex-format-ast-prepare-preflight-v4-20260618
• Slurm id: 10226192
• Config override: /home/vmendelev/.cache/saferun/runner-packages/runner-cluster-configs/cluster_configs/v0.7.1
• Generated manifest: /lustre/fsw/portfolios/llmservice/users/vmendelev/workspace/code/nemo-run/ntt-complex-eval/eval-format-ast/skills_data/ntt-complex/format_ast/test.jsonl
• Rows: 1200
• Source balance: fleurs=600, covost2=600
• Format balance: json_object=400, markdown_table=400, srt_single_cue=400

Runtime Evaluation Status

Target checkpoint:

/lustre/fs12/portfolios/llmservice/users/pzelasko/results/speechlm-2026h1/0615-h50-r3-thd-id-riva20-ast-suno-cbias10-iad-resumable-buckfix-4node/eval-step-7200

The latest successful startup eval was:

• Workflow: ntt-complex-format-ast-piotr-step7200-eval-v25-open-asr-startup-20260618
• Slurm app: 10228227
• Image: /lustre/fs12/portfolios/llmservice/users/pzelasko/containers/open-asr-nemotron-omni.sqsh
• Scope: one node, one GPU, ++max_samples=8
• State: Runner success, exit_code=0
• Result: 8 rows generated; format_valid=0/8; format_ast_is_correct=0/8

The full eval completed successfully:

• Workflow: ntt-complex-format-ast-piotr-step7200-eval-v26-full-20260618
• Slurm app: 10228387
• Output: /lustre/fsw/portfolios/llmservice/users/vmendelev/workspace/code/nemo-run/ntt-complex-eval/eval-format-ast/evals/piotr_step7200_format_ast_v26_full/eval-results/ntt-complex.format_ast/output.jsonl
• Result: 1200 rows generated; format_valid=0/1200; format_ast_is_correct=0/1200; average BLEU 0.2322
• By format: json_object 0/400 valid (missing_json_object), markdown_table 0/400 valid (too_few_markdown_rows), srt_single_cue 0/400 valid (too_few_srt_lines)
• By source dataset: covost2 0/600 valid, average BLEU 0.2450; fleurs 0/600 valid, average BLEU 0.2194

latest/iad.yaml must not be used for this work because it is an unfilled template. The active workflows use the rendered v0.7.1 config override above.

Hosted Nemotron Omni comparison:

• Probe workflow v2: ntt-complex-format-ast-hosted-nemotron-omni-api-probe-v2-20260618
• Probe Slurm app: 10228637
• Backend/model: hosted NVIDIA Inference API, nvidia/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning
• Probe result: 6 rows generated; errors=0; format_valid=6/6; format_ast_is_correct=2/6; average BLEU 0.1997
• Full workflow: ntt-complex-format-ast-hosted-nemotron-omni-api-full-v1-20260618
• Full Slurm app: 10228702
• Full output: /lustre/fsw/portfolios/llmservice/users/vmendelev/workspace/code/nemo-run/ntt-complex-eval/eval-format-ast/evals/hosted_nemotron_omni_format_ast_full_v1/eval-results/ntt-complex.format_ast/output.jsonl
• Full result: 1200 rows generated; errors=0; format_valid=1124/1200; format_ast_is_correct=120/1200; average BLEU 0.1118
• By format: json_object 392/400 valid and 44/400 correct; markdown_table 332/400 valid and 33/400 correct; srt_single_cue 400/400 valid and 43/400 correct.

PR Handoff

Suggested base:

NVIDIA-NeMo/Skills:codex/ntt-smoke

Suggested head:

NVIDIA-NeMo/Skills:codex/ntt-complex-format-ast

Branch commit:

e3ce0839a17fa53ab20bb77d63e9a1d7daf2d0bb

Browser PR URL:

https://github.com/NVIDIA-NeMo/Skills/pull/1491

Qwen Omni comparison status:

• Hosted API probe: ntt-complex-format-ast-hosted-qwen-omni-model-probe-v1-20260618, Slurm app 10228800
• Candidate ids tested: Qwen/Qwen3-Omni-30B-A3B-Instruct, qwen/qwen3-omni-30b-a3b-instruct, Qwen/Qwen3-Omni-30B-A3B-Thinking, qwen/qwen3-omni-30b-a3b-thinking
• Result: all hosted candidates returned HTTP 401 key_model_access_denied with the available runner key.
• Self-host discovery: internal docs identify Qwen/Qwen3-Omni-30B-A3B-Instruct and vllm_omni.OmniLLM, but bounded Draco/IAD cache checks did not find the documented Granary chain script, a Qwen3-Omni HF snapshot, or a usable vllm_omni runtime/package.
• Required next input: an authorized hosted key or a concrete Draco/IAD self-host runtime path.