Establish Upfront Dataset Manifest and Path Verification Layer

[purvanshjoshi]

Technical Overview

Currently, the training script parses dataset manifests and validates path alignments inside the training thread pool during data collation. If manifest paths are broken or files are missing, the pipeline crashes late in the process, wasting significant compute allocation.

Affected Modules

• File: src/train.py & src/utils.py

Acceptance Criteria

1. Immediate Verification: The training script verifies dataset manifests and local audio paths immediately on launch.
2. Diagnostic Errors: Detailed path failures are logged indicating exactly which manifest lines are malformed.
3. Zero Waste: The training script terminates before allocating GPU tensors or loading models if verification fails.

Proposed Implementation Approach

Add a lightweight, fast-pass manifest validator function prior to importing heavy models:

def verify_manifests(manifest_paths):
    import json
    for path in manifest_paths:
        if not os.path.exists(path):
            raise FileNotFoundError(f"Manifest path not found: {path}")
        with open(path, "r") as f:
            for i, line in enumerate(f):
                entry = json.loads(line)
                if "audio_filepath" not in entry or not os.path.exists(entry["audio_filepath"]):
                    raise FileNotFoundError(f"Audio file referenced on line {i+1} in {path} does not exist: {entry.get('audio_filepath')}")

Severity & Priority

• Severity: Medium (Reduces debug turnaround times)
• Priority: P2

[Saksham-dev-del]

Hi @purvanshjoshi,

I would like to work on this issue. I will add a verify_manifests function to src/train.py/src/utils.py that runs immediately on launch, before any model or GPU tensor allocation, and:

Confirms each manifest path exists.
Parses each manifest line and validates the referenced audio_filepath exists.
Raises detailed errors identifying the exact failing line and file.

Could you please assign this issue to me? I will submit a PR for review once complete.