GGUF_do_not_need_an_external_model_specs_folder - #53
Conversation
Compile model_specs/*.json into engine_runtime at CMake generation time so CLI and server deployments do not need an external model_specs directory. This applies to both safetensors packages and supported GGUF models: GGUF weights and sidecars remain model data, while the matching package layout is carried by the binary. Add an explicit --model-spec-override escape hatch for CLI workflows and server deployments, including server-wide and per-model JSON configuration. Built-in specs remain the default and per-model server overrides take precedence.
|
@mirek190 Good catch. I will take a look. |
|
Even a safetensor models had that problem. Currently that config can be straight in the folder with safetensor model. |
|
@mirek190 Thanks, the motivation makes sense. I'd like to split this into two smaller PRs though. I’m less sure about embedding all model_specs into the audio.cpp binary. One workflow I care about is contributor-driven model support. Suppose a contributor adds a new GGUF package under a different GGUF layout. In the current design, the package layout is runtime data: they can contribute or ship an updated model_specs/.json, and the existing binary can use that new layout without recompilation. Of course, the current spec-based mechanism is still preliminary. Longer term, I’d like this to become more robust: audio.cpp could automatically discover/match package specs instead of relying only on a fixed built-in spec for a family. For example, it could search a spec directory and match by family/layout/version metadata. Maybe a build flag would be a good compromise here. What about making embedded specs optional, e.g. AUDIOCPP_DEPLOYMENT_BUILD=ON/OFF? With it off, audio.cpp keeps today’s behavior and loads model_specs/.json from disk. With it on, CMake embeds the specs into the binary for portable deployments. |
|
For GGUF models, you can load a separate configuration file from a specified path, such as model_specs/*.json, by using the --model-spec-override option. Without this option, the configuration is loaded in the following order: From the GGUF model metadata. By default, the GGUF build tool updated in this commit will not build a GGUF model when a model_specs/*.json configuration is required but cannot be found. The tool searches for this configuration in the model_specs directory and in the SafeTensors model configuration. Also AUDIOCPP_DEPLOYMENT_BUILD=ON/OF is added. Read bellow how it works. Changes - continuation for gguf workground: 1. Package specification embedded in GGUF - exampleNewly converted GGUF files store: audiocpp.model_spec.version = 1 The embedded JSON describes:
The GGUF reader validates:
Therefore, a new standalone GGUF normally needs only: application/ No external model_specs directory is required. 2. Runtime package-spec resolutionCLI and server now resolve the package specification in this order: --model-spec-override Important details:
The registry now passes the actual model path into package-spec resolution, allowing it to inspect GGUF metadata before choosing a 3. Converter and runtime catalogs are separatedPreviously, PR #53 compiled all model_specs/*.json files into every runtime binary. The update separates this into two cases. audiocpp_gguf converterThe converter always carries a compiled package-spec catalog. This means you can copy: audiocpp_gguf.exe and convert a model without keeping the audio.cpp source-tree model_specs directory beside the converter. The converter uses that catalog to select and embed the correct specification into the generated GGUF. CLI and serverCLI/server compile the fallback catalog only when built with: cmake -S . -B build -DAUDIOCPP_DEPLOYMENT_BUILD=ON The option is off by default because new GGUF models already contain their own spec. The deployment catalog remains useful for:
4. New converter optionsThe converter now supports: --family --model-spec and --model-spec-override are aliases in the converter. Example: audiocpp_gguf.exe 5. Automatic specification discovery during conversionThe converter searches in this order:
config.json can provide:
Recognized keys include: audiocpp_model_spec 6. Improved automatic family detectionThe converter can infer known audio.cpp families from upstream model_type values, including:
Qwen3 Forced Aligner detection specifically checks: thinker_config.model_type = qwen3_forced_aligner This prevents Qwen3-ASR 1.7B from being incorrectly detected as the forced aligner. 7. Conversion validationBefore writing the GGUF, the converter now validates:
This prevents successfully producing a GGUF that audio.cpp cannot subsequently load. 8. Default sidecar behaviorStandalone GGUF creation remains the default. If sidecar embedding is enabled but the converter cannot find any sidecars, conversion now fails instead of silently creating a The error tells you to provide: --root or explicitly choose: --no-sidecars --sidecar and --no-sidecars cannot be combined. 9. Tensor-only GGUF behaviorA GGUF created with --no-sidecars still contains:
It does not contain required tokenizer/configuration/processor files. Deployment therefore looks like: model-directory/ The loader was updated so a tensor-only GGUF can use external sidecars from its surrounding directory. Previously, GGUF package 10. Generic tensor archive escape hatchThe converter normally refuses to create a GGUF without a valid package spec. For a generic tensor archive that is not intended to run as an audio.cpp model, you can use: --allow-missing-model-spec The converter tries to find a spec, prints a warning if it cannot, and creates the archive without embedded model-spec metadata. This option is not recommended for deployable models. 11. Improved inspection outputRunning: audiocpp_gguf.exe --inspect model.gguf now additionally reports: embedded_sidecars=true It continues reporting tensor count, scalar count, and namespaces. 12. Documentation updatesThe following were updated:
They now document:
What this update does not do
I think I have too much free time today... |
Store the resolved package spec and family in GGUF metadata so new standalone and tensor-only GGUF files do not depend on an external model_specs directory. Resolve runtime specs in explicit-override, embedded-GGUF, deployment-catalog, then external-discovery order. Keep the converter catalog portable, validate required namespaces and sidecars before writing, and document deployment and legacy GGUF behavior.
|
@mirek190 Sorry, I somehow missed your long update earlier :( Thanks a lot for taking the time to write everything up. I’ll go through it and test the changes tomorrow. |
|
@mirek190 Merged! Kudos to you for putting this together! A few changes:
|
|
no problem |
…del-specs GGUF_do_not_need_an_external_model_specs_folder
2 participants
Summary
This PR makes model package specifications portable without forcing every runtime binary to carry every model layout.
Runtime package-spec resolution
An explicit override is authoritative. If it is invalid, loading fails instead of silently falling through to another source.
Deployment behavior
A normal new standalone GGUF deployment can therefore be:
No external model_specs folder is needed for that GGUF.
Converter behavior
The converter finds a package spec in this order:
The selected spec is embedded as versioned audiocpp.model_spec metadata. The converter rejects missing required namespaces and sidecars before producing a deployable model. --allow-missing-model-spec remains an explicit escape hatch for generic tensor archives that are not expected to load as audio.cpp models.
Compatibility
Validation
The MinGW CLI executable link still encounters the repository's existing WinMain entry-point mismatch; all changed CLI sources compile, and the supported Windows build/check remains covered by CI.