Skip to content
BranchesTags

Repository files navigation

GAME ggml backend

Ask DeepWiki

Languages: English | 中文

Native C++ inference for the GAME singing-voice-to-MIDI model, built on ggml. Runs on CPU, Metal (default on Apple Silicon), CUDA, or Vulkan. Drop-in replacement for python infer.py extract with no Python dependency at runtime.

Origin & Credits

This work grew out of KCKT0112/GAME-ggml (native C++/ggml GAME inference — CPU/Metal/CUDA/Vulkan) and its web variant KCKT0112/web-game (Web Generative Adaptive MIDI Extractor). We build on the model, the operator set, and the weight layout first ported there, and retain the GAME PyTorch original as the parity reference.

Highlights

  • End-to-end CLI — WAV in, MIDI/TXT/CSV out (mirrors Python extract).
  • Small footprint — ~50 MB GGUF for the 1.0-medium checkpoint, ~50 M params.
  • Fast startup — Metal binary-archive patch keeps first-run latency under a second on Apple Silicon. (The patch is kept in-repo at cmake/patches/ggml-metal-binary-archive.{patch,md}; see the .md for design and re-apply notes.)
  • Third-party integration — clean PIMPL C++ API; add_subdirectory and link game_ggml::game_ggml.
  • Parity-tested — full pipeline output matches the PyTorch reference bit-for-bit when the same RNG numbers are injected.

Architecture

waveform (44100 Hz mono)
      │
      ▼
   MelExtractor (pocketfft STFT + librosa-compatible mel)
      │ mel [T, 80]
      ▼
   Encoder (EBFBackbone, 4 layers, dim=128)
      │ x_seg, x_est  each [T, 128]
      ▼
   D3PM loop (1 step default; --nsteps 8 for higher quality)
     ├─ remove_mutable_boundaries (stochastic)
     ├─ Segmenter (EBFBackbone, 8 layers; noise/time/lang embeddings)
     └─ decode_soft_boundaries (local-max)
      │ regions [T]  +  N
      ▼
   Estimator (JEBFBackbone, 4 layers; joint attention, mixed RoPE)
      │ pool_logits [N, 257]
      ▼
   Gaussian-blurred pitch decode → notes (offset, duration, pitch, voiced)

Build

cmake -S . -B build \
      -DCMAKE_BUILD_TYPE=Release \
      -DGAME_GGML_BUILD_TESTS=ON
cmake --build build -j

Options:

Option Default Meaning
GAME_GGML_METAL ON (Apple only) Enable Metal backend
GAME_GGML_CUDA OFF Enable CUDA backend
GAME_GGML_VULKAN OFF Enable Vulkan backend
GAME_GGML_BUILD_CLI ON Build game_ggml_cli
GAME_GGML_BUILD_TESTS OFF Build GoogleTest suite

Convert a PyTorch checkpoint

pip install -r scripts/requirements.txt
python scripts/convert_pt_to_gguf.py \
    --model-dir GAME-pt-1.0-medium \
    -o game_medium.gguf

The script reads model.pt + config.yaml + lang_map.json from the given directory and writes a single GGUF file containing all 671 tensors (FP32) and 74 metadata KV pairs.

Inspect the result:

./build/bin/game_ggml_cli inspect game_medium.gguf

Run inference

./build/bin/game_ggml_cli extract input.wav \
    -m game_medium.gguf \
    --output-formats mid,txt,csv \
    --output-dir out/ \
    --tempo 120 \
    --seed 42

CLI → infer.py extract option mapping

CLI flag Python equivalent
-m / --model -m
-l / --language -l (takes numeric id — use inspect to see the mapping)
--output-formats --output-formats
--output-dir --output-dir
--tempo --tempo
--seg-threshold --seg-threshold
--seg-radius --seg-radius (in frames)
--est-threshold --est-threshold
--t0 / --nsteps --t0 / --nsteps (ggml defaults to --nsteps 1; Python defaults to 8)
--seed (new) — 0 pulls a random seed from the OS
--pitch-format --pitch-format
--round-pitch --round-pitch
--rng-replay <path> (new) — replay D3PM random numbers from a file for bit-exact parity with PyTorch

Performance

Measured on Apple M4 (macOS, 16-core Apple Silicon), 3 runs per side under /usr/bin/time -l, fresh subprocess each run (so peak RSS is clean). Input: 28.wav — 216.0 s mono, resampled to 44.1 kHz.

Both sides consume the exact same random-number stream via --rng-replay (see scripts/align_demo.py), so the note lists are bit-exact aligned.

Default settings (--nsteps 1)

                          min      mean       max
PyTorch wall (s)         5.99      6.27      6.71   (MPS via Lightning)
ggml    wall (s)         3.05      3.06      3.06   (Metal, default binary)
Speedup                                    2.05 ×

PyTorch peak RSS      980.8 MB  981.2 MB  981.7 MB
ggml    peak RSS      334.1 MB  334.4 MB  334.8 MB
Memory ratio                                2.93 ×

PyTorch notes: 471  ┐
ggml    notes: 471  ├── matched 1-to-1, max |Δpitch| = 0.000 semitone
                     ┘

Real-time factor: 70.6× (ggml) vs 34.4× (PyTorch).

Higher quality (--nsteps 8)

                          min      mean       max
PyTorch wall (s)        17.73     18.05     18.65
ggml    wall (s)         9.11      9.28      9.62
Speedup                                    1.94 ×

Real-time factor drops to 23.3× (ggml) / 11.97× (PyTorch), but segmentation quality is marginally higher (+9 notes recovered in this clip).

Per-stage breakdown (ONNX-aligned)

Run with GAME_GGML_PROFILE=1 to print a per-chunk breakdown. Numbers below use --nsteps 8 so the segmenter share is visible:

encoder     ~0.17 s  (~16%)   waveform → x_seg/x_est  (mel + spec_proj + 4× EBF)
segmenter   ~0.79 s  (~78%)   x_seg → boundaries       (8× D3PM sampling steps)
estimator   ~0.06 s  (~ 6%)   x_est + regions → notes  (4× JEBF + joint attn)

Segmenter dominates because D3PM loops it --nsteps times. Default 1 keeps it cheap; bump to 4 or 8 for higher quality at linear cost.

DBCache (multi-step acceleration, default on)

For --nsteps > 1, a cross-step DBCache is on by default on every backend (threshold 0.25, front blocks 1, warmup 1): when the segmenter's front-block residual between consecutive D3PM steps is below the threshold, the tail blocks are skipped and the previous step's tail delta is reused — a near-lossless approximation (~0.2–0.3 cents pitch drift, no note-count change in the ablation) that cuts nsteps=8 segmenter wall time roughly in half.

On GPU backends (Vulkan/Metal/CUDA) it is also on by default: the device-side cache decision (B) removed the per-step host round-trip that used to regress quantized-weight graphs, so the same 0.25 threshold applies everywhere. Set --cache-threshold 0 to disable.

  • Tuning: --cache-threshold <float> (auto / 0 = off / 0.25...), --cache-fn-blocks <int>, --cache-warmup <int>.
  • Robustness knobs (borrowed from cache-dit/edge-dit.cpp): a reuse WINDOW (--cache-window-start/end, fractions of the loop — first/last steps stay full compute), UCache-style accumulated-error gate (--cache-error-decay + --cache-error-limit), and a consecutive-hit cap (--cache-max-continuous). All default to the per-step-threshold behavior.
  • With --nsteps 1 the cache is automatically disabled even if a threshold is set, keeping the fused single-graph path with zero overhead.
  • --cache-bn-blocks (recompute the last N tail blocks on a hit) is implemented but not recommended: the extra host round-trip between the middle and back slices reintroduces the fused-vs-split fp drift and regressed note count (33→32 on the test clip); keep 0.

Backend × weights guide

Local measurements (10s clip, seed 42; RTX 2070 Vulkan vs CPU AVX2; all configs produce identical note output):

EP weights nsteps=1 nsteps=8
CPU Q8_0 5.9s 14.0s（DBCache on）
Vulkan (warm) F32 0.39s 0.65s
Vulkan (warm) Q8_0 0.42s 1.09s
  • GPU backends are 10–20× faster than CPU here once warm — use -full (F32) packs when a discrete GPU is available; Q8 is a close second if RAM is limited.
  • CPU benefits from Q8 for memory, at essentially the same speed — use the -q8 pack on CPU.
  • Cold start: the first inference on a GPU compiles shaders (Vulkan/Metal). Metal uses an in-repo binary-archive PSO cache; Vulkan now persists a disk-backed VkPipelineCache (cmake/patches/ggml-vulkan-pipeline-cache.*, GGML_VK_PIPELINE_CACHE_PATH, default under the user cache dir), so later launches load precompiled PSOs instead of recompiling — no reliance on driver-level caches.

OpenUtau integration (.oudep)

Packages for OpenUtau are distributed as .oudep archives (a zip of game_ggml_cli + game_medium.gguf + config.json + oudep.yaml) from the GitHub Releases. Install the package for your platform; OpenUtau unpacks it itself.

OpenUtau build Package to install
Current (new serve-API protocol) game_ggml-<platform>.oudep or -q8 — from the latest release (e.g. v0.1.3). This package speaks the current serve protocol (src/cli/main.cpp, game_ggml_cli serve).
Early (old API spec) old_game_ggml-<platform>.oudep — only shipped on the v0.1.0 release.

If your (older) OpenUtau build fails to talk to the engine, you are on the old API spec — download the old_-prefixed .oudep from v0.1.0, not the newest release.

Platforms: windows-x64-vulkan, linux-x64-vulkan, macos-arm64-metal, macos-x64-metal (old-prefixed set).

Reproducing the benchmark

# 1. Resample to 44.1 kHz / mono (if not already)
python3 -c "
import librosa, soundfile as sf
y, _ = librosa.load('28.wav', sr=44100, mono=True)
sf.write('/tmp/28_44100.wav', y, 44100, subtype='PCM_16')"

# 2. Capture PyTorch's D3PM RNG stream (also produces a reference MIDI)
python3 scripts/align_demo.py /tmp/28_44100.wav \
    -m GAME-pt-1.0-medium/model.pt \
    -g game_medium.gguf \
    --cli build/bin/game_ggml_cli \
    -l zh -o /tmp/align_out

# 3. Run the 3-per-side subprocess-isolated benchmark
python3 scripts/benchmark_align.py /tmp/28_44100.wav \
    -m GAME-pt-1.0-medium/model.pt \
    -g game_medium.gguf \
    --cli build/bin/game_ggml_cli \
    --rng /tmp/align_out/align_rng.bin \
    -l zh -o /tmp/bench_out --runs 3

Using as a third-party library

add_subdirectory(path/to/game.cpp)

add_executable(my_app main.cpp)
target_link_libraries(my_app PRIVATE game_ggml::game_ggml)
// main.cpp
#include <game_ggml/model.h>
#include <vector>

int main() {
    auto model = game_ggml::Model::load("game_medium.gguf");
    std::vector<float> waveform = /* ... load 44100 Hz mono ... */;

    game_ggml::InferParams params;
    params.language = 4;   // from lang_map: { "zh": 4 }
    params.seed     = 42;
    // DBCache (segmenter cross-step reuse; affects nsteps>1 only).
    // -1 = auto (0.25 on all backends); 0 = off; >0 = explicit threshold.
    params.db_cache_threshold = 0.25f;
    params.db_cache_fn_blocks = 1;
    params.db_cache_warmup    = 1;
    // Optional robustness knobs (defaults match --cache-* CLI flags above):
    // params.db_cache_window_start = 0.0f; params.db_cache_window_end = 1.0f;
    // params.db_cache_err_decay = 0.0f;    params.db_cache_max_cont = 0;
    // params.db_cache_bn_blocks = 0;

    auto result = model.infer(waveform.data(), waveform.size(), params);
    for (const auto & n : result.notes) {
        if (!n.voiced) continue;
        printf("  %.2fs + %.2fs : %.2f\n",
               n.offset_seconds, n.duration_seconds, n.pitch_midi);
    }
    // Cache hit/miss counters for this inference (0/0 when cache disabled).
    printf("  dbcache hits=%d misses=%d\n",
           result.db_cache_hits, result.db_cache_misses);
}

The public header <game_ggml/model.h> uses PIMPL; consumers never transitively include any ggml header.

See examples/external_consumer/ for a minimal standalone CMake project that builds against the library.

Tests

ctest --test-dir build --output-on-failure

The suite has 37 tests covering:

  • Backend initialisation
  • GGUF I/O round-trip
  • Every op (RMSNorm, Linear, LayerScale, Embedding, GLU-FFN, CgMLP, RoPE in all three modes, Attention, PAC, EBF block)
  • Encoder / Segmenter / Estimator end-to-end vs PyTorch reference dumps
  • D3PM 8-step loop bit-exact with injected RNG (tolerates ≤ 2/100 boundary flips from Metal FP32 drift)
  • Mel spectrogram vs lib.feature.mel.StretchableMelSpectrogram
  • Slicer (short-clip + split-on-silence)
  • MIDI writer (SMF type-0 structure)
  • Text writers (TXT + CSV, note-name formatting)
  • Full pipeline bit-exact E2E

Reference dumps are generated by python scripts/dump_reference.py --category all. Dumps are gitignored — regenerate them as part of CI.

Known limitations (v1)

  • 44100 Hz mono WAV only — other sample rates raise InvalidWav. Resampling is deliberately out-of-scope to keep the footprint small.
  • FP32 and Q8_0 weights — the converter emits FP32 GGUF by default, or Q8_0 via --quant-config (see CI prepare-model). All backends optimise for the quantized layout; results are near-lossless vs FP32.
  • Only the shipped 1.0-medium config branch is supported. The estimator rejects split attention, learned pool merger, region_token_num > 1, and use_region_bias=true at load time with a clear NotImplemented message.
  • Batch size 1 per inference call — matches infer.py extract. For parallel streams hold multiple Model instances.
  • Metal FP32 precision — expected ~1e-3 per matmul; at boundary decoding this can flip one frame out of every few hundred vs the CPU reference.

Dependencies

Everything is fetched at configure time by CMake; nothing is vendored. Source trees live under build/_deps/<name>-src/ after the first configure.

Dependency Version pin License SPDX identifier
ggml v0.19.0 tag (temporary anchor — see AGENT.md "ggml version gate") MIT MIT
pocketfft commit 32424d20 on cpp branch BSD-3-Clause BSD-3-Clause
dr_libs commit 243e26ff on master Public Domain / MIT-0 (dual) Unlicense OR MIT-0
GoogleTest v1.14.0 tag (tests only) BSD-3-Clause BSD-3-Clause

Each upstream LICENSE file is preserved under build/_deps/<name>-src/LICENSE* after download. To update a dependency, change its GIT_TAG (or ggml's URL/URL_HASH) in cmake/Dependencies.cmake and reconfigure.

License

MIT — same as the parent GAME project. Redistributions should also carry the upstream license notices listed in the table above.

About

C++ inference for the “GAME” singing-voice-to-MIDI model.

Resources

Readme
License
Activity

Stars

11 stars

Watchers

0 watching

Forks

1 fork
Report repository

Releases

Packages

Contributors

Languages