Integrate Dynamic WER/CER Evaluation Hooks in Seq2SeqTrainer

[purvanshjoshi]

Technical Overview

Evaluating speech adaptation using only the training loss can lead to overfitting on accent patterns. We need to calculate and track the Word Error Rate (WER) and Character Error Rate (CER) on accented test sets at regular evaluation intervals.

Affected Modules

• File: src/train.py
• Target Classes: Seq2SeqTrainer and its evaluation hooks.

Acceptance Criteria

1. Automated Computation: Calculates WER and CER metrics automatically at each designated eval_steps interval.
2. Performance Constraints: Metric evaluation must not increase total step duration by more than 10%.
3. Accurate Tracking: Decodes model tokens back to clean string predictions to calculate error rates, ignoring special tokens.

Proposed Implementation Approach

Implement a metric calculation function and register it within the trainer constructor:

import evaluate
import numpy as np

wer_metric = evaluate.load("wer")
cer_metric = evaluate.load("cer")

def compute_metrics(pred):
    pred_ids = pred.predictions
    label_ids = pred.label_ids
    
    # Replace -100 to ignore pad tokens
    label_ids[label_ids == -100] = tokenizer.pad_token_id
    
    pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
    label_str = tokenizer.batch_decode(label_ids, skip_special_tokens=True)
    
    wer = wer_metric.compute(predictions=pred_str, references=label_str)
    cer = cer_metric.compute(predictions=pred_str, references=label_str)
    
    return {"wer": wer, "cer": cer}

Severity & Priority

• Severity: Medium (Improves validation monitoring)
• Priority: P2

[Saksham-dev-del]

Hi @purvanshjoshi,

I would like to work on this issue. My plan is to implement a compute_metrics function in src/train.py that:

Loads the wer and cer metrics via the evaluate library.
Replaces -100 label placeholders with the tokenizer's pad token before decoding.
Decodes predictions and labels with skip_special_tokens=True, then computes WER and CER.
Registers this function with Seq2SeqTrainer so metrics compute automatically at each eval_steps interval.

I will also benchmark the added evaluation overhead to confirm it stays within the 10% step-duration constraint specified in the acceptance criteria.

Could you please assign this issue to me? I will submit a PR for review once complete.

[purvanshjoshi]

Hi @Saksham-dev-del,

Thank you for a well-scoped proposal — this is one of the strongest plans we've received. Your approach maps cleanly to the acceptance criteria, and we particularly appreciate that you explicitly committed to benchmarking the added evaluation overhead against the 10% step-duration ceiling; that constraint is easy to overlook. We're happy to approve this approach and assign the issue to you.

A few refinements to fold into your implementation:

1. Masking without mutation: Our collator (DataCollatorSpeechSeq2SeqWithPadding in src/utils.py) already replaces pad tokens with -100. Your plan to swap -100 back to pad_token_id before decoding is correct, but do it on a copy (np.where on pred.label_ids, never in-place) so you don't corrupt the underlying evaluation tensors.

2. Generation mode: predict_with_generate=True and generation_max_length=225 are already set in src/train.py, so predictions arrive as generated token sequences, not raw logits — your decode path should expect token IDs.

3. Text normalization: For Indian-accented content, normalize casing, whitespace, and diacritics (idempotently) before computing WER/CER so the metrics reflect linguistic errors rather than formatting differences. Keep the helper documented and reusable.

4. Dependency pinning: Add evaluate (or jiwer/torchmetrics) to requirements.txt with a pinned version so metric implementations don't silently drift between runs.

5. Overhead evidence: For the 10% constraint, measure per-eval wall time with and without compute_metrics on the same eval set and report the delta in the PR description.

One question for you: how do you intend to handle fully-padded reference batches (where the entire label sequence is -100)? Should those be filtered out of the reference set before metric computation, or do you expect the collator's padding behavior to prevent that case in practice?

We've assigned the issue to you. As discussed in our replies on the other threads, this is one of your two assignments (the other is #6). Please raise the PR against src/train.py when ready and feel free to ask if anything is ambiguous.