Skip to content

Add the two endpoints Hugging Face needs to list Chutes as an inference provider - #189

Open
fstandhartinger wants to merge 1 commit into
chutesai:mainfrom
fstandhartinger:hf-inference-provider-endpoints
Open

Add the two endpoints Hugging Face needs to list Chutes as an inference provider#189
fstandhartinger wants to merge 1 commit into
chutesai:mainfrom
fstandhartinger:hf-inference-provider-endpoints

Conversation

@fstandhartinger

Copy link
Copy Markdown
Contributor

Adds the two endpoints Hugging Face requires to list Chutes as an Inference Provider. Both are additive — no existing response field changes shape, no existing caller needs touching.

Context: I audited our live API against HF's requirements and we're closer than expected. The chutesai Hub org is already on their Team plan (their prerequisite), and our inference already passes their automated validation as-is — I measured TTFT at 0.98–2.75s across 8 models streaming against their 5s threshold, and tool calling and JSON-schema structured output both work. These two gaps are the only hard blockers.

1. /v1/models pricing aliases

HF reads pricing.input / pricing.output (USD per million tokens) to power their provider comparison table and their :cheapest routing policy. We emit pricing.prompt / pricing.completion, the OpenRouter naming. Both key pairs are now present with identical values.

Same spot backfills context_length from max_model_len for the two models that don't report it (unsloth/Mistral-Nemo-Instruct-2407-TEE, Nemotron-3-Nano-Omni-30B-TEE), since HF wants it on every listed model.

2. POST /partners/huggingface/billing

HF polls this once a minute with the request IDs it routed to us, using the same bearer auth as inference, and bills its users whatever we report.

Costs come from the per-invocation billed balance already persisted in invocations.metrics->>'b', so there is no second pricing path to keep in sync — HF users are charged exactly the standard Chutes rate, including any overrides and discounts that applied. Scoped to the authenticated user so one partner key can never read another account's costs.

Two behaviours in there are load-bearing rather than stylistic, both straight from HF's spec:

  • Unknown request IDs are omitted, never returned as 0. HF replaces its placeholder cost with whatever we return and never asks again, so answering 0 for an invocation that simply hasn't been written yet would bill it as free, permanently. Omitted IDs get retried every minute for ~30 minutes.
  • An entirely unknown batch returns {"requests": null}, HF's documented "no data yet, retry later" signal.

3. Inference-Id header (optional)

Mirrors X-Chutes-InvocationID onto an Inference-Id response header in build_response_headers, and exposes it via CORS. HF accepts a custom header name, so this part is genuinely optional — it's one line at the single choke point and saves a negotiation round. Happy to drop it if you'd rather not add a header.

Routing note

The billing endpoint lands on default_router, so it is reachable at https://api.chutes.ai/partners/huggingface/billing and not on llm.chutes.ai — the MEGALLM branch in main.py captures every POST to that host. That matches HF's spec, which explicitly allows the billing URL to sit on a different domain from the inference base URL as long as the auth pattern is the same.

Testing

Syntax-checked only; I don't have a local DB/Redis to run the suite against. The billing query follows the existing text() pattern used for invocations elsewhere in api/chute/util.py. Worth a look at the INTERVAL lookback window (currently 2 hours, chosen because HF gives up on a request after ~30 minutes) and at whether the invocation_id/completed_at index selection is what you'd want at that batch size (HF sends up to 10,000 IDs per call).

Nothing is pending on HF's side — we haven't contacted them yet — so there's no time pressure on this.

…ce provider

HF's provider registration has exactly two hard requirements we don't meet today.
Both are covered here, additively — no existing response field changes shape and
no existing caller has to be touched.

1. /v1/models pricing aliases. HF reads `pricing.input` / `pricing.output` (USD
   per million tokens) to power their provider comparison table and the
   `:cheapest` routing policy. We emit `pricing.prompt` / `pricing.completion`
   (OpenRouter naming). Both key pairs are now present with identical values.
   Also backfills `context_length` from `max_model_len` for the two models that
   don't report it (Mistral-Nemo, Nemotron-3-Nano-Omni), since HF wants it on
   every listed model.

2. POST /partners/huggingface/billing. HF polls this once a minute with the
   request IDs it routed to us and bills its users whatever we report. Costs come
   from the per-invocation billed balance already persisted in
   `invocations.metrics->>'b'`, so there is no second pricing path to keep in
   sync — HF users are charged exactly the standard Chutes rate. Scoped to the
   authenticated user, same bearer auth as inference.

   Two details in there are load-bearing rather than stylistic. Unknown request
   IDs are omitted rather than returned as 0: HF replaces its placeholder with
   whatever we return and never asks again, so a 0 for an invocation that just
   hasn't been written yet would bill it as free forever. An entirely unknown
   batch returns `{"requests": null}`, HF's documented retry signal. Both give
   us the ~30 minutes of retries HF allows.

Also mirrors X-Chutes-InvocationID onto an `Inference-Id` response header in
build_response_headers, and exposes it via CORS. HF accepts a custom header name,
so this is optional — but it costs one line at the single choke point and makes
the integration zero-config on their side.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant