Skip to content

Add SongEval and shared model caching - #86

Merged
ftshijt merged 1 commit into
wavlab-speech:mainfrom
ftshijt:agent/songeval-shared-model-cache
Jul 21, 2026
Merged

Add SongEval and shared model caching#86
ftshijt merged 1 commit into
wavlab-speech:mainfrom
ftshijt:agent/songeval-shared-model-cache

Conversation

@ftshijt

@ftshijt ftshijt commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add production-ready SongEval support with pinned upstream assets, optional dependencies, local/offline configuration, audio validation, and stable result keys
  • make --cache_folder consistently configure utterance and corpus metrics in both scorer entry points
  • share model caches by backend for Hugging Face, Whisper, ESPnet, Torch Hub, and NeMo while preserving explicit metric cache_dir overrides
  • route cache settings through existing model-backed metrics including ARECHO, NeMo WER, NOMAD, PAM, Qwen2-Audio, Qwen2.5-Omni, speaking rate, SQUIM, and VAD
  • document the shared-cache layout and add focused regression coverage

Motivation

Versa exposed cache options in several places, but the main scorer did not apply --cache_folder to ordinary utterance metrics and some pretrained-model loaders ignored metric cache configuration. This could duplicate large checkpoints or place downloads in user-specific default locations. SongEval also needed a reliable setup path for both its predictor and MuQ encoder.

This change makes a mounted cache root reusable across runs and machines while keeping unrelated metric artifacts isolated.

Validation

  • 132 passed, 1 skipped across metric definitions, base metric/pipeline coverage, SongEval, shared-cache routing, NOMAD, PAM cache forwarding, and speaking-rate tests
  • Black checks pass for all changed Python files
  • bash -n tools/install_songeval.sh
  • git diff --check
  • wheel build with pip wheel . --no-deps --no-build-isolation
  • real SongEval + MuQ CPU inference:
    VERSA_RUN_REAL_MODEL_TESTS=1 python -m pytest test/test_metrics/test_songeval.py::test_songeval_real_model_inference -vv -s
    (1 passed in 43.04 seconds)

The real-model test remains opt-in because it downloads approximately 8.5 GB
of MuQ assets in addition to the SongEval checkpoint.

@ftshijt
ftshijt marked this pull request as ready for review July 20, 2026 23:44
@ftshijt
ftshijt merged commit fbe2e89 into wavlab-speech:main Jul 21, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant