[WIP]Add CS-FLEURS code-switched ASR benchmark with Mixed Error Rate scoring#1494
Open
Slyne wants to merge 12 commits into
Open
[WIP]Add CS-FLEURS code-switched ASR benchmark with Mixed Error Rate scoring#1494Slyne wants to merge 12 commits into
Slyne wants to merge 12 commits into
Conversation
New benchmark group nemo_skills/dataset/cs-fleurs/ (paper arXiv 2509.14161, HF byan/cs-fleurs) covering the read / mms / xtts-test1 / xtts-test2 test sets as ASR-only sub-benchmarks. Reuses the existing audio evaluator's Multilingual-ASR path (same as fleurs): per-language CER for scriptio-continua matrix languages (cmn/jpn/kor/tha/...), WER otherwise; subset_for_metrics is the code-switched pair. prepare.py downloads from HF and roots audio at NEMO_SKILLS_AUDIO_ROOT (default /data) like librispeech-pc/musan. Includes a group score aggregator and a docs section. No existing files modified beyond an appended docs section. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: slyned <slyned@nvidia.com>
_download_subset re-fetched from HuggingFace on every prepare run, which could stall/hang when the (large) audio corpus was already present. Check for a complete local copy (metadata + matching .wav count, or metadata only under --no-audio) and return early, re-downloading only when incomplete. Signed-off-by: slyned <slyned@nvidia.com>
- Exclude cs-fleurs from gpu-tests/test_eval.py auto-prepare (HF download, mirrors the existing fleurs group exclusion). - Add network-free unit tests for the custom scoring helpers: pair parsing, CER-vs-WER language decision, and entry-weighted group aggregation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: slyned <slyned@nvidia.com>
Add cs-fleurs (own dataset; distinct from fleurs) to the DATASETS list, matching how contextual-earnings22 is listed. Reword the GPU exclusion comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: slyned <slyned@nvidia.com>
Code-switched utterances mix scriptio-continua scripts (counted by character) with space-delimited scripts (counted by word) in one sentence, so a single per-utterance CER/WER switch mis-scores them. Add a true MER path: - audio.py: mixed_segment() tokenizes mixed text (Han/kana/Hangul/Thai/Lao/ Myanmar/Khmer split into units, Latin runs kept as words), with optional grapheme-cluster mode for combining-mark scripts (Thai/Myanmar). evaluate_mer() normalizes -> segments -> word-level edit distance, reported under wer* keys. Wired into the Multilingual-ASR branch behind extra_fields["use_mer"], which defaults False so fleurs and all other datasets are unchanged. - cs-fleurs/prepare.py: set use_mer=True; mer_grapheme=True only when a side is a combining-mark script (Thai/Lao/Myanmar/Khmer). - cs-fleurs/languages.py: GRAPHEME_CLUSTER_LANGS + uses_grapheme(). - audio_score.py: emit zero-valued summed metrics instead of dropping them. - tests: mixed_segment coverage + grapheme tests (test_mer_segmentation.py), uses_grapheme + helper-fallback + zero-sum tests (test_cs_fleurs.py). Not yet runtime-validated: evaluate_mer needs jiwer (runs on CI/cluster), and the Thai/Myanmar diacritic-normalization interaction needs GPU eval check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: slyned <slyned@nvidia.com>
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds the CS-FLEURS code-switched ASR benchmark as a new dataset group with four subsets ( ChangesCS-FLEURS Benchmark Integration
Sequence Diagram(s)sequenceDiagram
participant CLI as prepare.py main()
participant HF as HuggingFace Hub
participant PrepFunc as prepare_cs_fleurs()
participant Langs as languages.py
participant Record as _build_record()
participant Evaluator as evaluate_sample()
participant MER as evaluate_mer()
CLI->>PrepFunc: call with data_dir, subsets
PrepFunc->>HF: snapshot_download(metadata.jsonl + audio)
HF-->>PrepFunc: local files
PrepFunc->>Record: iterate metadata rows
Record->>Langs: split_pair(), uses_cer(), uses_grapheme()
Langs-->>Record: matrix/embedded codes, scoring flags
Record-->>PrepFunc: manifest record with use_mer, mer_grapheme
PrepFunc-->>CLI: write test.jsonl per subset
Note over Evaluator,MER: At evaluation time
Evaluator->>Evaluator: read use_mer from extra_fields
Evaluator->>MER: evaluate_mer(ref, hyp, grapheme=mer_grapheme)
MER->>MER: mixed_segment(text) → tokens
MER->>MER: _wer_with_counts(tokens)
MER-->>Evaluator: wer* metrics dict
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/evaluation/speech-audio.md`:
- Around line 811-831: The CS-FLEURS documentation section is missing expected
results examples for tested models, which is required per the coding guidelines
for new benchmarks. Add a new section after the output structure showing example
evaluation commands (similar to how the prepare_data command is shown) and
include expected results output from at least one model you have tested with on
this benchmark. The expected results should demonstrate what the evaluation
output looks like, including any metrics or scores produced by the evaluation
run.
- Around line 824-831: The fenced code block displaying the data directory
structure is missing a language specification tag, which violates the
markdownlint MD040 rule. Add the `text` language identifier to the opening fence
of the code block that contains the directory structure output (the block
showing `<data_dir>/cs-fleurs/` and its subdirectories), changing the opening
from just triple backticks to ```text triple backticks.
In `@nemo_skills/dataset/cs-fleurs/audio_score.py`:
- Around line 63-70: The aggregation code is using .get() with default values
for required metric fields (num_entries, gen_seconds, success_rate, avg_tokens,
no_answer), which silently defaults to 0 when keys are missing and can hide
upstream contract breaks. Replace all the .get() calls with direct dictionary
access using square brackets (for example, metrics["num_entries"] instead of
metrics.get("num_entries", 0), metrics["gen_seconds"] instead of
metrics.get("gen_seconds", 0), metrics["success_rate"] instead of
metrics.get("success_rate", 0.0), metrics["avg_tokens"] instead of
metrics.get("avg_tokens", 0.0), and metrics["no_answer"] instead of
metrics.get("no_answer", 0.0)) to fail fast when required keys are missing.
In `@nemo_skills/dataset/cs-fleurs/languages.py`:
- Around line 106-127: The code currently fails silently when encountering
malformed or unknown language codes. In the split_pair function, add validation
to ensure the language parameter has the correct format after splitting, raising
an error for malformed input. In the get_lang_name and get_iso1 functions,
replace the .get() dictionary lookups with direct bracket notation access to
CS_FLEURS_LANGUAGES so that KeyError is raised immediately for unknown language
codes instead of silently returning fallback values, ensuring bad language codes
are caught early in the evaluation pipeline.
In `@nemo_skills/dataset/cs-fleurs/prepare.py`:
- Around line 45-47: The file nemo_skills/dataset/cs-fleurs/prepare.py has
formatting issues that are causing ruff-format to fail in the CI pipeline. Run
the ruff-format tool on this file to automatically reformat it according to the
project's formatting standards, then commit the formatted changes to resolve the
pre-commit check failure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: c9fa19ce-210c-4f8d-a6af-6223eec72341
📒 Files selected for processing (14)
docs/evaluation/speech-audio.mdnemo_skills/dataset/cs-fleurs/__init__.pynemo_skills/dataset/cs-fleurs/audio_score.pynemo_skills/dataset/cs-fleurs/languages.pynemo_skills/dataset/cs-fleurs/mms/__init__.pynemo_skills/dataset/cs-fleurs/prepare.pynemo_skills/dataset/cs-fleurs/read/__init__.pynemo_skills/dataset/cs-fleurs/xtts-test1/__init__.pynemo_skills/dataset/cs-fleurs/xtts-test2/__init__.pynemo_skills/evaluation/evaluator/audio.pytests/gpu-tests/test_eval.pytests/test_cs_fleurs.pytests/test_datasets.pytests/test_mer_segmentation.py
- languages.py: add aze/swh (the two dataset codes missing from the table); make get_lang_name/get_iso1 fail fast (KeyError) on unknown codes instead of silently degrading, now that the table covers every CS-FLEURS language. Intentional None (languages with no ISO 639-1, e.g. ceb/yue) is preserved. - audio_score.py: bracket-access the required num_entries (fail loudly if absent); presence-gate the optional generation metrics (success_rate / no_answer / gen_seconds) instead of fabricating 0.0 for pure-ASR runs. - prepare.py: ruff-format (collapse _lang_spec to one line). - docs: add 'text' language tag to the directory-structure code block (MD040). - tests: update get_iso1/get_lang_name tests to assert fail-fast on unknown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: slyned <slyned@nvidia.com>
Adds reference MER numbers (full read subset, 5818 utterances, greedy) from an external multilingual model, scored via the benchmark's own MER path, satisfying the CONTRIBUTING requirement for example evaluation results. Overall MER 25.75%. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: slyned <slyned@nvidia.com>
Reformat the Whisper large-v3 baseline from a markdown table into a ```text 'Example output' block, matching the ContextASR-Bench / Contextual Earnings-22 sections in the same doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: slyned <slyned@nvidia.com>
Author
|
@KunalDhawan could you help with the unit test issue: https://github.com/NVIDIA-NeMo/Skills/actions/runs/28053390144/job/83049524775?pr=1494 |
Slyne
commented
Jun 23, 2026
| **Scoring — Mixed Error Rate (MER).** Because each utterance mixes two scripts, a single per-utterance CER-or-WER choice mis-scores it (e.g. whole-utterance CER over a `cmn-eng` sentence scores the embedded English at the character level). Instead, records set `use_mer=true` and the audio evaluator computes a Mixed Error Rate: scriptio-continua scripts (Han / kana / Hangul / Thai / Lao / Myanmar / Khmer) are counted by character and space-delimited scripts (e.g. Latin) by word, within the same utterance, then scored with the standard word-level edit distance. This matches the standard MER used for Mandarin-English code-switching (e.g. SEAME). The mixed figure is reported in the headline `wer*` column (a parallel pure-CER column is still emitted). `subset_for_metrics` is the code-switched pair (e.g. `ara-eng`), giving a per-pair breakdown; the group score module adds a per-test-set and overall entry-weighted figure. Scoring uses the same multilingual normalization as `fleurs` (case-insensitive, punctuation/diacritics removed). MER only affects CS-FLEURS — it is gated behind `use_mer`, which monolingual benchmarks (including `fleurs`) never set, so their CER/WER behavior is unchanged. | ||
|
|
||
| !!! warning "Known limitation: Thai/Myanmar diacritic stripping (reviewers' input welcome)" | ||
| The shared multilingual normalization removes Unicode combining marks (category `Mn`, and turns `Mc` into spaces) as "diacritics". This is correct for Latin accents but **destroys meaning-bearing vowel and tone marks in Thai/Myanmar** (e.g. `ที่` → `ท`), so CER/MER for those languages is computed over consonant skeletons and under-counts real errors. This is **pre-existing** — it affects the plain `use_cer` path and `fleurs` too, not just MER. For combining-mark scripts the prepare step sets `mer_grapheme=true` so grapheme-cluster segmentation *would* keep marks together, but it is currently a no-op because normalization strips the marks first. Properly fixing this needs script-aware normalization (preserve `Mn`/`Mc` for Thai/Lao/Myanmar/Khmer/Indic) in the shared evaluator, which would change `fleurs` numbers too — deferred pending maintainer guidance. Affects only ~695 of ~49k CS-FLEURS records (Thai + Myanmar). |
Author
There was a problem hiding this comment.
Hi @KunalDhawan could you help with this issue? Need some input here. whether we want to change our normalization pipeline.
Collaborator
There was a problem hiding this comment.
re the unit test issue: #1494 (comment)
The unit-tests job is dying at the Login to GHCR step (~13s in, before pytest runs), because fork PRs don't get the registry secrets to pull the test image. Could you raise the PR from a branch instead of a fork please?
Collaborator
There was a problem hiding this comment.
re the scoring issue: thanks for flagging! I'd keep the current approach for this PR: the gated preserve_marks (for MER) + lower_nopunct (for the paper-comparable CER), both behind use_mer, is the right scope, it's correct for CS-FLEURS and leaves fleurs/covost2 untouched. So no change needed here.
On the broader question: yes, the shared multilingual normalizer stripping Mn/Mc is genuinely wrong for abugida scripts (Thai/Lao/Myanmar/Khmer/Indic) and under-reports errors in fleurs too, not just MER. But making it script-aware globally would change our existing fleurs baselines, so I don't want to fold that into a benchmark-addition PR. Let's do it as a separate change with explicit re-baselining + a note in the docs.
The CS-FLEURS paper reports 'case insensitive and unpunctuated character error rate'. Add a mark-preserving normalization mode (lower_nopunct: lowercase + drop punctuation, keep Mn/Mc combining marks) and route the parallel CER column through it for cs-fleurs (gated on use_mer) so its cer* numbers are comparable to the paper. Monolingual benchmarks (fleurs/covost2) keep the existing multilingual normalization unchanged. MER (wer*) is unaffected. Tests: lower_nopunct preserves Thai/Myanmar marks, lowercases, de-punctuates, and differs from no_tn_itn (which strips marks). Docs updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: slyned <slyned@nvidia.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_mer_segmentation.py`:
- Around line 134-136: In the test_preserves_myanmar_marks method, the assertion
string contains a Thai combining mark (U+0E48) instead of a Myanmar combining
mark, which defeats the purpose of testing Myanmar mark preservation. Replace
the Thai mark with a Myanmar combining mark such as U+1037 to ensure the test
actually validates that Myanmar combining marks are properly preserved during
normalization with lower_nopunct mode.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: e75f0292-a58b-4065-910b-4ad0817797c6
📒 Files selected for processing (3)
docs/evaluation/speech-audio.mdnemo_skills/evaluation/evaluator/audio.pytests/test_mer_segmentation.py
🚧 Files skipped from review as they are similar to previous changes (2)
- nemo_skills/evaluation/evaluator/audio.py
- docs/evaluation/speech-audio.md
Comment on lines
+134
to
+136
| def test_preserves_myanmar_marks(self): | ||
| assert self._norm()("ကျော่.", mode="lower_nopunct") == "ကျော่" | ||
|
|
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
test_preserves_myanmar_marks is currently validating a Thai mark, not a Myanmar mark.
Line 135 uses ่ (Thai U+0E48), so this test does not actually assert Myanmar combining-mark preservation. Replace it with a Myanmar combining mark (for example \u1037) to match the test intent.
Proposed fix
def test_preserves_myanmar_marks(self):
- assert self._norm()("ကျော่.", mode="lower_nopunct") == "ကျော่"
+ assert self._norm()("ကျော\u1037.", mode="lower_nopunct") == "ကျော\u1037"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_mer_segmentation.py` around lines 134 - 136, In the
test_preserves_myanmar_marks method, the assertion string contains a Thai
combining mark (U+0E48) instead of a Myanmar combining mark, which defeats the
purpose of testing Myanmar mark preservation. Replace the Thai mark with a
Myanmar combining mark such as U+1037 to ensure the test actually validates that
Myanmar combining marks are properly preserved during normalization with
lower_nopunct mode.
Add a preserve_marks option to MultilingualTextNormalizer (keeps category-M combining marks while still removing symbols/punctuation, via NFC). Enable it for cs-fleurs MER (gated on use_mer) so MER counts Thai/Myanmar vowel and tone marks and grapheme-cluster segmentation is effective, while num2words and other multilingual steps still apply. Monolingual benchmarks (fleurs/covost2) keep the existing mark-stripping behavior. Both MER (preserve_marks) and the CER column (lower_nopunct) now preserve marks for cs-fleurs. Tests + docs updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: slyned <slyned@nvidia.com>
Refresh the Whisper large-v3 Example output with the current evaluator's numbers (mark-preserving MER + paper-comparable CER per pair; overall MER 24.51, CER 16.47 / ~18.0 macro vs paper ~19.8). De-duplicate the CER-normalization detail that overlapped the marks note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: slyned <slyned@nvidia.com>
Add a strip_whitespace option to evaluate_cer (removes all whitespace before scoring, so CER counts only non-space characters) and enable it for cs-fleurs (gated on use_mer). This matches the CS-FLEURS paper's character error rate, where spaces are not counted; default False keeps existing behavior, so fleurs/covost2 are unchanged. Resolves the apparent CER mismatch: with whitespace excluded, the macro-averaged read CER is 19.54% vs the paper's ~19.8% (the prior gap was spaces-counted + micro-vs-macro averaging). Updated baseline numbers and tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: slyned <slyned@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds CS-FLEURS (paper), a massively multilingual code-switched extension of FLEURS (52 languages, 113 code-switched pairs), as an ASR benchmark group, plus a Mixed Error Rate (MER) scoring path in the audio evaluator.
Subtasks (ASR only):
cs-fleurs.read(human-read, the paper's intended benchmark),cs-fleurs.mms(concatenative MMS-TTS),cs-fleurs.xtts-test1,cs-fleurs.xtts-test2(generative XTTS-v2).Why MER
Each utterance mixes two languages — and often two scripts — in one sentence. A single per-utterance CER-or-WER choice mis-scores it (e.g. whole-utterance CER over a
cmn-engsentence scores embedded English at the character level). MER counts scriptio-continua scripts (Han/kana/Hangul/Thai/Lao/Myanmar/Khmer) by character and space-delimited scripts by word within the same utterance, then scores with the standard word-level edit distance — the convention used for Mandarin-English code-switching (e.g. SEAME).Changes
nemo_skills/dataset/cs-fleurs/— new benchmark group:prepare.py(HF download + manifest build, with a download-skip guard for completed subsets),languages.py(language metadata, CER/grapheme policy),audio_score.py(per-test-set + overall aggregation), per-subtask configs.nemo_skills/evaluation/evaluator/audio.py— addsmixed_segment()(multi-script tokenizer, with optional grapheme-cluster mode for combining-mark scripts),_grapheme_clusters(), andevaluate_mer(). Wired into theMultilingual-ASRbranch.docs/evaluation/speech-audio.mdCS-FLEURS section + MER explanation + a known-limitation note.tests/test_cs_fleurs.py,tests/test_mer_segmentation.py; registerscs-fleursintest_datasets.pyand excludes it from GPU auto-prepare (heavy HF download).No impact on other datasets
MER is gated behind
extra_fields["use_mer"], which only CS-FLEURS sets. The evaluator change is a prependedif use_mer:branch — the existinguse_cer/WER logic is byte-identical.fleursandcovost2(the otherMultilingual-ASRconsumers) never set the flag, so their scoring is unchanged.cs-fleurs/audio_score.pyis private to this benchmark.How to run
Known limitation (reviewer input welcome)
The shared multilingual normalization removes Unicode combining marks (
Mn) as "diacritics" — correct for Latin accents, but it strips meaning-bearing vowel/tone marks in Thai/Myanmar (e.g.ที่→ท), so CER/MER for those languages is computed over consonant skeletons. This is pre-existing (affects the plainuse_cerpath andfleurstoo), not introduced here, and affects only ~695 of ~49k records. A proper fix needs script-aware normalization in the shared evaluator (would changefleursnumbers), so it's deferred pending maintainer guidance. Details in the docs.Testing
evaluate_merscoring — passing.Summary by CodeRabbit
Release Notes