Version: v0.5.3 · Platform: macOS, Apple Silicon
Problem
/v1/models returns {"data": []}. The server loads a model on demand from the id in each request, so there is nothing loaded to enumerate — which is a reasonable design, but it breaks the OpenAI-compatible discovery contract that clients rely on.
The failure mode is bad: every call fails with the client's equivalent of "no model available" while the server works perfectly if you name a model explicitly. Nothing in the error points at discovery, so the natural conclusion is that the server is broken.
Why it matters
Discovery isn't a nicety for a class of clients — it's how they classify what they have. A harness that needs "a chat model and an embedding model" reads /v1/models, sorts them, and configures itself. With an empty list it can't distinguish "no models" from "models exist but aren't enumerable", so it reports the former.
Suggestion
Enumerate what the server could serve, not only what is currently resident — the set of cached models it would accept (it already passes local_files_only=True to HuggingFace, so that set is knowable from the local cache). Failing that, a documented, non-empty response distinguishing "load-on-demand: name your model" from "no models" would let a client give an accurate message.
Workaround (for anyone hitting this)
Name the models explicitly in client config and skip discovery. Also note the server does not download: an unknown id fails with a confusing "outgoing traffic has been disabled" even with a working connection. Pre-download with hf download <id> — the hf CLI ships inside the uv tool env.
Filed alongside two related findings (image_url schema rejection, MLX_VLM_ONLY_MODELS routing gate).
Version: v0.5.3 · Platform: macOS, Apple Silicon
Problem
/v1/modelsreturns{"data": []}. The server loads a model on demand from the id in each request, so there is nothing loaded to enumerate — which is a reasonable design, but it breaks the OpenAI-compatible discovery contract that clients rely on.The failure mode is bad: every call fails with the client's equivalent of "no model available" while the server works perfectly if you name a model explicitly. Nothing in the error points at discovery, so the natural conclusion is that the server is broken.
Why it matters
Discovery isn't a nicety for a class of clients — it's how they classify what they have. A harness that needs "a chat model and an embedding model" reads
/v1/models, sorts them, and configures itself. With an empty list it can't distinguish "no models" from "models exist but aren't enumerable", so it reports the former.Suggestion
Enumerate what the server could serve, not only what is currently resident — the set of cached models it would accept (it already passes
local_files_only=Trueto HuggingFace, so that set is knowable from the local cache). Failing that, a documented, non-empty response distinguishing "load-on-demand: name your model" from "no models" would let a client give an accurate message.Workaround (for anyone hitting this)
Name the models explicitly in client config and skip discovery. Also note the server does not download: an unknown id fails with a confusing "outgoing traffic has been disabled" even with a working connection. Pre-download with
hf download <id>— thehfCLI ships inside the uv tool env.Filed alongside two related findings (
image_urlschema rejection,MLX_VLM_ONLY_MODELSrouting gate).