Vocal2Midi

Vocal2Midi is a Windows-first desktop tool and inference pipeline for turning vocal audio into lyric-aligned MIDI, USTX, VSQX, and supporting editing artifacts. macOS and Linux use the CPU path for ONNX models, while Apple Silicon can use Metal for the llama.cpp decoder.

The current runtime is ONNX-first:

• llama.cpp is used for the Qwen decoder and runs on Metal, Vulkan, or CPU.
• ONNX models default to DirectML and fall back to CPU when DirectML is unavailable.
• The main user-facing entrypoint is the Fluent GUI in app_fluent.py.

Highlights

• End-to-end vocal-to-MIDI workflow in one project
• Chinese, Japanese, and English lyric handling
• GUI workflow for interactive use
• Batch slice + ASR CLI for folder processing
• Portable-folder packaging flow for Windows distribution
• ONNX-based inference stack for ASR, alignment, note extraction, and RMVPE

What Vocal2Midi Does

At a high level, the hybrid pipeline looks like this:

audio
  -> optional RMVPE pitch curve
  -> slicing
  -> ASR
  -> lyric matching / .lab generation
  -> HubertFA forced alignment
  -> GAME note extraction
  -> quantization
  -> export

There is also a no-lyrics path:

audio
  -> optional RMVPE pitch curve
  -> slicing
  -> GAME pitch-only extraction
  -> export

Runtime Stack

Component	Current backend	Location
Qwen3-ASR	ONNX Runtime + llama.cpp	inference/qwen3asr_dml/
RomajiASR	ONNX Runtime	inference/romaji_asr/
HubertFA	ONNX Runtime	inference/HubertFA/
GAME	ONNX Runtime	inference/game/
RMVPE	ONNX Runtime	inference/API/rmvpe_api.py
Device normalization	DirectML / CPU helpers	inference/device_utils.py

Some public function names still contain _torch for compatibility, but the active backend is ONNX-based.

Repository Layout

application/   application-layer orchestration and config objects
docs/          architecture notes and supporting docs
experiments/   local model directories
gui/           PyQt5 + qfluentwidgets desktop UI
inference/     ASR, alignment, pitch extraction, slicing, quantization, export
scripts/       batch CLI and portable build helpers
tests/         automated tests

Quick Start

1. Install dependencies

Use your preferred Python environment, then install:

pip install -r requirements.txt

The main runtime dependencies are:

• onnxruntime-directml on Windows, or onnxruntime on macOS/Linux
• PyQt5
• PyQt-Fluent-Widgets
• librosa
• soundfile
• mido
• pyopenjtalk

An environment.yml file is also included as a reference environment snapshot.

2. Prepare model folders

By default, the GUI expects models in these locations:

Component	Default path
GAME	experiments/GAME-1.0.3-medium-onnx
HubertFA	experiments/1218_hfa_model_new_dict
Qwen3-ASR	experiments/Qwen3-ASR-1.7B-dml
Japanese mora ASR (RomajiASR)	experiments/romajiASR
RMVPE	experiments/RMVPE/rmvpe.onnx

You can change these paths in the GUI settings panel.

3. Launch the GUI

For a normal developer environment:

python app_fluent.py

GUI Workflow

The GUI is the main way to use Vocal2Midi interactively. It lets you:

• choose model paths
• pick the runtime device
• set slicing mode and slice length bounds
• choose language and lyric output mode
• provide optional reference lyrics
• export MIDI, USTX, VSQX, text, CSV, chunk audio, and alignment artifacts

The application-layer job entrypoint is run_auto_lyric_job() in application/pipeline.py, which dispatches into the hybrid inference pipeline in inference/pipeline/auto_lyric_hybrid.py.

Language Behavior

Chinese

• Qwen3-ASR provides text transcription.
• The lyric matcher and G2P path prepare .lab content for HubertFA.
• Lyrics can be exported in Hanzi or pinyin-oriented forms depending on mode.

Japanese

In the main hybrid lyric pipeline:

• romaji and kana lyric modes use the dedicated mora ASR path
• if the output mode is romaji, the pipeline uses mora ASR output directly
• if the output mode is kana, the pipeline converts matched mora output to kana for display
• if reference lyrics are provided, the reference text is processed through pyopenjtalk, converted to kana mora tokens, then converted again to romaji mora tokens for matching

This keeps Japanese lyric matching consistent with the mora-based ASR path instead of routing through the old phoneme-ASR forced-alignment branch.

The current Japanese mora / romaji ASR integration in this repository is based on Xiantaidu/RomajiASR, the separate Japanese singing ASR project used for the experiments/romajiASR model path and the inference/romaji_asr/ runtime integration.

English

• Qwen3-ASR transcribes with the English language prompt; CJK bleed-over, digits, and punctuation are stripped so only dictionary-friendly words remain.
• The word lyric output mode assigns one note per English word.
• After HubertFA, multi-syllable words are split into per-syllable time chunks (e.g. impossible -> im/poss/ible) that are fed to GAME as align units: the word lyric lands on the first note, later syllable positions are marked +, and melisma/转音 notes inside a syllable fall back to -.
• If reference lyrics are provided, they are matched word-by-word against the ASR output before alignment.
• HubertFA aligns through the DiffSinger CMU dict (ds_cmudict-07b.txt) shipped inside the HubertFA model folder; words missing from the dictionary are warned and skipped.
• Breath detection (AP) is enabled for English, mirroring the Chinese path.

Runtime Device Rules

Visible device options in the current UI are:

• dml
• cpu

Notes:

• dml is the default ONNX device on Windows; macOS/Linux default to CPU
• if DirectML is unavailable, ONNX Runtime falls back to CPU
• legacy cuda values are still accepted by some public interfaces, but they are normalized to dml
• on macOS, llama.cpp automatically prefers Metal when its shared library was built with Metal; it falls back to CPU if model loading fails

On macOS and Linux, the GUI exposes the CPU device by default because DirectML is a Windows-only execution provider. The split Qwen ONNX encoder can be selected with metal on Apple Silicon, while its int4 decoder remains on CPU; a GGUF Qwen decoder can independently use llama.cpp Metal.

macOS / Linux

The repository does not ship model assets or compiled llama.cpp libraries. On macOS or Linux, install the platform-marked dependencies and run the Unix model setup script:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
bash download_models.sh
python app_fluent.py

The setup script builds or locates the platform-specific llama.cpp shared libraries. On Apple Silicon it enables the Metal backend for GGUF Qwen decoding; the split ONNX model defaults to CPU because DirectML is Windows-only. If a Metal-enabled library cannot load, the decoder retries on CPU.

The bundled Qwen3-ASR archive is the split ONNX format (encoder, decoder_init, and decoder_step). On macOS, --device metal lets the encoder try CoreMLExecutionProvider; the int4 decoder graphs currently run on CPU because CoreML cannot compile decoder_step reliably on M4. If a GGUF Qwen model is provided instead, the llama.cpp decoder uses the compiled Metal backend.

Slicing

The user-facing slice duration settings currently support:

• minimum slice length: 0 to 60 seconds
• maximum slice length: 0 to 60 seconds

Current defaults:

• minimum: 8.0 seconds
• maximum: 22.0 seconds

Validation rules:

• slice_max_sec must be greater than 0
• slice_min_sec must be less than or equal to slice_max_sec

Batch Slice + ASR CLI

For folder-based batch ASR processing:

python scripts/slice_asr_cli.py <input_dir> <output_dir> \
  --asr-model experiments/Qwen3-ASR-1.7B-dml \
  --device dml \
  --language zh

This CLI is focused on:

• scanning input audio files
• slicing audio or bypassing slicing
• running local Qwen3-ASR
• saving chunk audio and .lab outputs
• optionally saving JSON timing / ASR metadata

Supported input extensions currently include:

• .wav
• .m4a
• .mp3

Useful options:

--no-slice              bypass slicing and send the whole file to ASR
--asr-batch-size        ASR batch size
--file-batch-size       number of audio files per batch
--rmvpe-model           enable RMVPE-assisted smart slicing
--rmvpe-batch-size      RMVPE batch size
--keep-model            keep the ASR runtime alive across the batch
--keep-rmvpe            keep the RMVPE runtime alive across the batch
--save-json             save slice timing and ASR outputs as JSON
--no-recursive          scan only the top level
--no-skip-existing      force reprocessing of existing outputs

Japanese whole-file example:

python scripts/slice_asr_cli.py input output \
  --asr-model experiments/Qwen3-ASR-1.7B-dml \
  --device dml \
  --language ja \
  --no-slice

On Apple Silicon, replace --device dml with --device metal to enable the CoreML encoder path with a CPU decoder fallback.

Windows Setup Scripts

The repository also includes Windows helper scripts:

• install.bat
• run.bat

These are useful for a smaller distribution model where the user downloads or initializes the runtime on first setup rather than receiving a fully bundled python/ folder.

Export Formats

Depending on the selected workflow, Vocal2Midi can export:

• .mid
• .ustx
• .vsqx
• .txt
• .csv
• TextGrid
• chunk .wav files
• .lab
• ASR matching logs

Project Notes

• The repository has already migrated away from the earlier Torch-heavy runtime design for the main inference path.
• Some historical function names remain for compatibility.
• Model assets are expected to exist locally under experiments/ or another user-provided path.
• The codebase is still being cleaned up in places, so you may still see a few legacy names or UI strings from earlier iterations.

License

The overall Vocal2Midi repository is distributed under the Apache License 2.0. See LICENSE.

Third-party components, vendored code, model assets, dictionaries, and other embedded materials may also carry their own original licenses, notices, or attribution requirements. Those original notices remain applicable to the corresponding materials. See ACKNOWLEDGEMENTS.md and any embedded license files for details.

Development and Testing

The repo includes a focused automated test suite under tests/.

Examples:

python -m pytest tests/test_auto_lyric_hybrid_pipeline.py
python -m pytest tests/test_asr_api.py tests/test_game_api.py tests/test_rmvpe_api.py
python -m pytest tests/test_device_selection.py tests/test_hubertfa_decoder.py

For architecture details, see docs/architecture.md.

Related Files

• Main GUI entrypoint: app_fluent.py
• Architecture notes: docs/architecture.md
• Third-party credits: ACKNOWLEDGEMENTS.md
• License: LICENSE