Merge pull request 0xShug0#86 from lapy/fix/loader-catalog-sync-orphans

Keep package catalog in sync with registered loaders

Author: 0xShug0

.github/workflows/linux-build.yml

@@ -37,6 +37,11 @@ jobs:
       - name: Checkout
         uses: actions/checkout@v4
 
+      - name: Check loader/catalog sync
+        run: |
+          python3 tools/check_loader_catalog_sync.py --self-test
+          python3 tools/check_loader_catalog_sync.py
+
       - name: Install dependencies
         run: |
           sudo apt-get update

.github/workflows/mac-build.yml

@@ -26,6 +26,11 @@ jobs:
       - name: Checkout
         uses: actions/checkout@v4
 
+      - name: Check loader/catalog sync
+        run: |
+          python3 tools/check_loader_catalog_sync.py --self-test
+          python3 tools/check_loader_catalog_sync.py
+
       - name: Configure
         run: |
           cmake -S . -B "$BUILD_DIR" \

.github/workflows/windows-build.yml

@@ -23,6 +23,12 @@ jobs:
       - name: Checkout
         uses: actions/checkout@v4
 
+      - name: Check loader/catalog sync
+        shell: pwsh
+        run: |
+          python tools/check_loader_catalog_sync.py --self-test
+          python tools/check_loader_catalog_sync.py
+
       - name: Build audiocpp_cli
         shell: pwsh
         run: |

CONTRIBUTING.md

@@ -20,6 +20,14 @@ Please check the supported model table in [README.md](README.md) before starting
 
 If you want to add support for a model family that is already listed, please focus on improving the existing implementation instead of opening a duplicate port.
 
+When a loader is registered (or parked), keep the **package catalog** in sync. Installable `ModelPackage` entries must not advertise families that `audiocpp_cli --list-loaders` does not expose. Follow the checklist in [docs/maintainers/loader_and_catalog.md](docs/maintainers/loader_and_catalog.md) and run:
+
+```bash
+python3 tools/check_loader_catalog_sync.py --self-test
+python3 tools/check_loader_catalog_sync.py
+```
+
+Do not leave a live Hugging Face `SnapshotSource` for a loader that is commented out of `registry.cpp` — mark it `UnsupportedSource` (or remove it) and update the README package table.
 Good follow-up work for existing model families includes:
 
 - Better CLI or server examples

README.md

@@ -90,7 +90,7 @@ Community model ports live under `community_models` to make the ownership bounda
 | **outetts** | TTS, voice cloning | en, ar, zh, nl, fr, de, it, ja, ko, lt, ru, es, pt, be, bn, ka, hu, lv, fa, pl, sw, ta, uk | Mirek [@mirek190](https://github.com/mirek190) | Llama-OuteTTS-1.0-1B TTS and voice cloning support |
 | **vietneu_tts** | TTS, voice cloning | vi, en | Phuoc [@phuocnguyen90](https://github.com/phuocnguyen90) | [VieNeu-TTS-v3-Turbo](vietneu_tts.md) TTS and voice cloning support |
 
-WIP: Higgs Audio v3 TTS 4B, Fish Audio S2 Pro.
+WIP (loaders not registered in this release tree — catalog entries are `UnsupportedSource`): Kokoro 82M bf16, Higgs Audio v3 TTS 4B, Parakeet TDT 0.6B v3, Fish Audio S2 Pro. See [docs/maintainers/loader_and_catalog.md](docs/maintainers/loader_and_catalog.md).
 
 PocketTTS language selection is a model-load option. When the model path points at the PocketTTS root, the loader uses `english` unless you pass `--load-option language=<name>`. Kyutai's normal non-English PocketTTS releases are smaller distilled language models intended for the fast PocketTTS path. The `_24l` variants are larger 24-layer, undistilled preview models that can sound better but are slower. Kyutai currently publishes French only as `french_24l`, not as a normal distilled `french` language directory, so French is not listed as a normal PocketTTS language here.
 
@@ -299,7 +299,8 @@ Useful CLI features:
 - `--help` with `--task` shows task-oriented help
 - `--help` with `--model <path>` and optional `--family <family>` shows model-owned request, session, and load options
 - `--inspect` prints discovered configs, weights, and capabilities
-- `--list-loaders` prints registered model families
+- `--list-loaders` prints registered model families (`--json` for the machine-readable contract)
+- `python tools/model_manager.py list --json` prints installable packages; keep it synced with loaders ([docs/maintainers/loader_and_catalog.md](docs/maintainers/loader_and_catalog.md))
 - `--batch-text-file <txt>` runs one offline request per non-empty line
 - `--batch-text-dir <dir>` runs one offline request per `.txt`, `.md`, or `.json` file, normalizing each file as one paragraph
 - `--batch-audio-dir <dir>` runs one offline request per `.wav`
@@ -370,7 +371,7 @@ The CLI also exposes the runtime loader catalog with `audiocpp_cli --list-loader
 
 Recommended top-level install packages:
 
-`Yes` means Hugging Face has a ready-to-use repo that the framework can download as-is. `No` means the tool must assemble, convert, or post-process files before the framework can use them.
+`Yes` means Hugging Face has a ready-to-use repo that the framework can download as-is. `No` means the tool must assemble, convert, or post-process files before the framework can use them. Packages whose loaders are not registered in this release tree are listed as **Unavailable** (see [docs/maintainers/loader_and_catalog.md](docs/maintainers/loader_and_catalog.md)).
 
 | Package id | Model | HF ready-to-use repo |
 |---|---|---|
@@ -379,12 +380,13 @@ Recommended top-level install packages:
 | `citrinet_asr` | Citrinet ASR converted layout | No |
 | `heartmula` | HeartMuLa | No |
 | `higgs_audio_stt` | Higgs Audio STT | No |
-| `higgs_audio_v3_tts_4b` | Higgs Audio v3 TTS 4B | **Yes** |
+| `higgs_audio_v3_tts_4b` | Higgs Audio v3 TTS 4B | Unavailable (loader not in this tree) |
 | `htdemucs` | HTDemucs | No |
 | `hviske_asr` | Hviske ASR | **Yes** |
 | `irodori_tts_500m_v3` | Irodori-TTS 500M v3 | No |
 | `irodori_tts_600m_v3_voice_design` | Irodori-TTS 600M v3 VoiceDesign | No |
-| `kokoro_82m_bf16` | Kokoro 82M bf16 | **Yes** |
+| `index_tts2` | IndexTTS-2 | **Yes** |
+| `kokoro_82m_bf16` | Kokoro 82M bf16 | Unavailable (loader not in this tree) |
 | `marblenet_vad` | MarbleNet VAD converted layout | No |
 | `mel_band_roformer` | Mel-Band RoFormer MLX | **Yes** |
 | `miocodec_25hz_44k_v2` | MioCodec 25Hz 44.1kHz v2 | No |
@@ -397,7 +399,7 @@ Recommended top-level install packages:
 | `nemotron_asr` | Nemotron ASR | **Yes** |
 | `omnivoice` | OmniVoice | **Yes** |
 | `outetts_1_0_1b` | OuteTTS 1.0 1B with IBM DAC codec and Qwen3-aligned voice cloning | No |
-| `parakeet_tdt_0_6b_v3` | Parakeet TDT 0.6B v3 | **Yes** |
+| `parakeet_tdt_0_6b_v3` | Parakeet TDT 0.6B v3 | Unavailable (loader not in this tree) |
 | `pocket_tts` | PocketTTS | **Yes** |
 | `qwen3_asr_0_6b` | Qwen3 ASR 0.6B | **Yes** |
 | `qwen3_asr_1_7b_hf` | Qwen3 ASR 1.7B HF | **Yes** |
@@ -413,6 +415,7 @@ Recommended top-level install packages:
 | `stable_audio_3_small_sfx` | Stable Audio 3 Small SFX | **Yes** |
 | `supertonic_3` | Supertonic 3 | **Yes** |
 | `vevo2` | Vevo2 | No |
+| `vietneu_tts_v3_turbo` | VieNeu-TTS v3 Turbo | **Yes** |
 | `vibevoice_1_5b` | VibeVoice 1.5B | No |
 | `vibevoice_7b` | VibeVoice 7B | No |
 | `vibevoice_asr` | VibeVoice ASR | No |

docs/maintainers/loader_and_catalog.md

@@ -0,0 +1,106 @@
+# Maintaining Loaders and the Package Catalog
+
+Integrators (CLI users, servers, and UIs such as Studio) treat two exports as
+**authoritative**:
+
+1. **Runtime loaders** — `audiocpp_cli --list-loaders --json`
+2. **Install packages** — `python tools/model_manager.py list --json`
+
+Those surfaces must stay in sync. A package that is installable in the catalog
+but whose `family` is missing from `--list-loaders` looks available to users and
+then fails at runtime or in search/install UIs.
+
+## The rule
+
+For every **installable, standalone** `ModelPackage`:
+
+| Field | Must match |
+|---|---|
+| `ModelPackage.family` | The loader family string advertised by the C++ loader |
+| `model_specs/<family>.json` | Present when the family uses package-spec loading |
+| `registry.cpp` entry | Uncommented `make_<family>_loader()` (or the family's actual factory name) |
+| README package table | Lists the package; use **Unavailable** when not installable |
+
+Dependency / subcomponent packages (`standalone=False`, with
+`parent_package_id`) do **not** need their own loader.
+
+Registered loaders that ship as bundled assets (no downloadable package) are
+allowed. List them in `BUNDLED_LOADERS_WITHOUT_PACKAGE` inside
+`tools/check_loader_catalog_sync.py`.
+
+If a loader is not ready for this release tree:
+
+1. Keep it **commented out** in `src/framework/runtime/registry.cpp`, and
+2. Mark matching catalog packages as `UnsupportedSource(reason=...)`, **or**
+   remove them from `CATALOG`, and
+3. Mark the README package row **Unavailable**.
+
+Do **not** leave a live `SnapshotSource` for a commented-out loader.
+
+Optional catalog↔registry family renames for parked stubs go in
+`PARKED_FAMILY_ALIASES` in the sync check (collapse to one id when re-enabling).
+
+## Checklist: adding a model family
+
+1. Implement `include/engine/models/<family>/` (or `community_models/`) with a
+   loader that overrides `advertised_capabilities()` so tasks/endpoints are
+   explicit.
+2. Register it in `src/framework/runtime/registry.cpp` (include +
+   `available_loaders` entry). Prefer the factory name
+   `make_<family>_loader()` so the id matches the advertised family.
+3. Add `model_specs/<family>.json` when the family needs package-spec discovery.
+4. Add one or more `ModelPackage` entries in `tools/model_manager.py`:
+   - Set `family="<family>"` explicitly when the package id does not strip cleanly
+     to the loader id.
+   - Set `tasks=(...)` when defaults would be ambiguous.
+   - Use `standalone=False` + `parent_package_id` for tokenizers / subcomponents.
+5. Update README supported-model / package tables.
+6. Run:
+
+```bash
+python3 tools/check_loader_catalog_sync.py --self-test
+python3 tools/check_loader_catalog_sync.py
+# after building:
+build/.../bin/audiocpp_cli --list-loaders --json
+python3 tools/model_manager.py list --json
+```
+
+Confirm the new family appears in `--list-loaders` and that installable packages
+for that family set `"family"` to the same string.
+
+## Checklist: parking or removing a family
+
+1. Comment out the include and `make_*_loader()` entry in `registry.cpp`.
+2. Convert related **standalone** packages to `UnsupportedSource` with a reason
+   that names the missing loader and points at this doc (or delete them).
+3. Leave `family=` / `tasks=` on unsupported entries if useful for history.
+4. Update README so the package row says **Unavailable**.
+5. Run `python3 tools/check_loader_catalog_sync.py`.
+
+## Family id consistency
+
+Pick **one** family string and use it everywhere:
+
+- C++ loader / `advertised_capabilities()`
+- `make_<family>_loader()` naming (when practical)
+- `model_specs/<family>.json`
+- `ModelPackage.family`
+- README “Supported Models” family column
+
+Integrators match on the string; aliases are not implied unless listed in
+`PARKED_FAMILY_ALIASES` for currently parked stubs.
+
+## CI
+
+`tools/check_loader_catalog_sync.py` runs in GitHub Actions on Linux/macOS/Windows
+builds. It:
+
+- Parses active vs commented `make_*_loader()` calls in `registry.cpp`
+- Compares them to installable standalone packages from `model_manager.py`
+- Cross-checks the README recommended package table
+- Does **not** require a compiled binary
+
+```bash
+python3 tools/check_loader_catalog_sync.py --self-test
+python3 tools/check_loader_catalog_sync.py
+```

docs/usage.md

@@ -11,7 +11,7 @@ audiocpp_cli --task <task> --family <family> --model <model-dir> --backend <back
 | Option | Values | Default | Meaning |
 |---|---|---:|---|
 | `--task` | `gen`, `tts`, `clon`, `vc`, `svc`, `s2s`, `asr`, `align`, `vad`, `diar`, `sep`, `vdes` | required | User task. |
-| `--family` | model family name | required | Selects the model implementation. |
+| `--family` | model family name | required | Selects the model implementation. Must match a registered loader (`audiocpp_cli --list-loaders`). |
 | `--model` | local model directory | required | Path to local model assets. |
 | `--backend` | `cpu`, `cuda`, `vulkan`, `metal`, `best` | `cpu` | Inference backend. |
 | `--mode` | `offline`, `streaming` | `offline` | Run mode. Most models are offline. |

src/framework/runtime/registry.cpp

@@ -4,7 +4,9 @@
 #include "engine/framework/assets/model_package.h"
 #include "engine/framework/io/config.h"
 #include "engine/framework/io/filesystem.h"
-// Development registry entries from Share/AudioCPP that are not present in this release tree yet:
+// Parked loaders (sources not in this release tree). When commenting these out,
+// also mark matching ModelPackage entries UnsupportedSource — see
+// docs/maintainers/loader_and_catalog.md and tools/check_loader_catalog_sync.py.
 // #include "engine/models/higgs_tts/loader.h"
 // #include "engine/models/kokoro_tts/loader.h"
 // #include "engine/models/parakeet_tdt/loader.h"
@@ -242,7 +244,8 @@ ModelRegistry make_registry_from_config(
 
 ModelRegistry make_default_registry(const std::optional<std::filesystem::path> & config_path) {
     const std::vector<std::shared_ptr<IVoiceModelLoader>> available_loaders = {
-        // Development registry entries from Share/AudioCPP that are not present in this release tree yet:
+        // Parked loaders — keep catalog packages UnsupportedSource while these stay commented.
+        // See docs/maintainers/loader_and_catalog.md.
         // engine::models::kokoro_tts::make_kokoro_tts_loader(),
         // engine::models::higgs_tts::make_higgs_tts_loader(),
         // engine::models::parakeet_tdt::make_parakeet_tdt_loader(),