feat(speech): add AppTek Call-Center Dialogues ASR benchmark#1486
Open
pzelasko wants to merge 4 commits into
Open
feat(speech): add AppTek Call-Center Dialogues ASR benchmark#1486pzelasko wants to merge 4 commits into
pzelasko wants to merge 4 commits into
Conversation
Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>
Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>
pzelasko
force-pushed
the
speech/apptek-callcenter-dialogues
branch
from
June 15, 2026 15:29
0bb1cdb to
50b9afd
Compare
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds the AppTek Call-Center Dialogues ASR benchmark to NeMo Skills. This includes a word-mapping normalization module integrated into the existing audio evaluator, a dataset preparation script with HuggingFace download support, a standalone Parakeet v3 reproduction recipe with direct and Silero VAD transcription modes, evaluator tests, and updated documentation. ChangesAppTek Call-Center Dialogues ASR Benchmark
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@nemo_skills/evaluation/evaluator/audio.py`:
- Around line 453-457: The normalize_apptek_callcenter_text function call
silently defaults is_prediction to False when the kwarg is missing, allowing
callers to inadvertently treat predictions as references. Replace the
.get("is_prediction", False) pattern with explicit access to kwargs (e.g.,
kwargs["is_prediction"]) so the code fails with a clear KeyError if the caller
forgets to provide this required parameter, forcing explicit specification
rather than silent misbehavior.
In `@recipes/apptek-callcenter-dialogues/reproduce_parakeet_v3.py`:
- Around line 441-443: The --skip-transcribe and --force-segment flags should
not be permitted when used with --mode direct, but currently they are silently
ignored in that case. After the parse_args() call in the function that returns
the parser, add validation logic that checks if these mode-incompatible flags
are set when --mode is set to "direct". If they are, raise an error with a
descriptive message explaining that these flags are not compatible with direct
mode, rather than allowing the program to proceed silently with unused
arguments. This ensures the code fails fast and alerts users to their incorrect
command-line arguments.
- Around line 345-359: In the score_predictions function, line 358 uses
prediction_by_file.get(file_name, "") which silently treats missing predictions
as empty strings, masking transcription failures. Replace this .get() call with
direct dictionary access using prediction_by_file[file_name] so that a KeyError
is raised when a file is missing, properly surfacing the broken run instead of
silently reporting misleading metrics.
- Around line 74-100: The code uses fallback patterns with `.get()` and default
values like "unknown" in the `_row_accent_code` function (line 87) and
`_manifest_audio_path` function (lines 91-97) that allow malformed rows to
silently pass through instead of failing immediately. Replace the silent
fallbacks by removing the "unknown" default in `_row_accent_code` so missing
accent codes cause an error, and in `_manifest_audio_path` replace the `.get()`
calls with direct dictionary key access that will raise a KeyError if required
keys like "audio_filepath" or message structure keys are missing. This ensures
data quality issues are caught immediately with clear errors rather than
silently corrupting downstream processing.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 51b24e6d-daab-408a-b697-7bdd1ab8c8b2
📒 Files selected for processing (11)
docs/evaluation/speech-audio.mdnemo_skills/dataset/apptek-callcenter-dialogues/__init__.pynemo_skills/dataset/apptek-callcenter-dialogues/prepare.pynemo_skills/evaluation/evaluator/apptek_callcenter.pynemo_skills/evaluation/evaluator/apptek_callcenter_word_mappings.pynemo_skills/evaluation/evaluator/audio.pyrecipes/apptek-callcenter-dialogues/README.mdrecipes/apptek-callcenter-dialogues/reproduce_parakeet_v3.pytests/gpu-tests/test_eval.pytests/test_audio_evaluator.pytests/test_datasets.py
Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>
Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migration note
Supersedes #1443. Recreated from the
NVIDIA-NeMo/Skillsbranchspeech/apptek-callcenter-dialoguesso repository CI can run.Summary
Adds the AppTek Call-Center Dialogues ASR benchmark, sourced from
apptek-com/apptek_callcenter_dialogueson HuggingFace.The benchmark covers long-form English call-center audio: 1,746 split-channel WAVs, roughly 129 hours, across 14 accents and 16 domains.
What Changed
apptek-callcenter-dialoguesdataset package and preparation CLI.--audio-prefixsupport.apptek_callcenterASR normalization that follows the official scoring protocol:recipes/apptek-callcenter-dialogues/.PR Stack
This PR contains only the AppTek benchmark work.
Validation
python -m py_compileon changed Python files.Follow-Up
Summary by CodeRabbit
apptek_callcentertext normalization for end-to-end ASR scoring and improved WER/CER handling (including empty references).