Add the two endpoints Hugging Face needs to list Chutes as an inference provider - #189
Open
fstandhartinger wants to merge 1 commit into
Open
Conversation
…ce provider
HF's provider registration has exactly two hard requirements we don't meet today.
Both are covered here, additively — no existing response field changes shape and
no existing caller has to be touched.
1. /v1/models pricing aliases. HF reads `pricing.input` / `pricing.output` (USD
per million tokens) to power their provider comparison table and the
`:cheapest` routing policy. We emit `pricing.prompt` / `pricing.completion`
(OpenRouter naming). Both key pairs are now present with identical values.
Also backfills `context_length` from `max_model_len` for the two models that
don't report it (Mistral-Nemo, Nemotron-3-Nano-Omni), since HF wants it on
every listed model.
2. POST /partners/huggingface/billing. HF polls this once a minute with the
request IDs it routed to us and bills its users whatever we report. Costs come
from the per-invocation billed balance already persisted in
`invocations.metrics->>'b'`, so there is no second pricing path to keep in
sync — HF users are charged exactly the standard Chutes rate. Scoped to the
authenticated user, same bearer auth as inference.
Two details in there are load-bearing rather than stylistic. Unknown request
IDs are omitted rather than returned as 0: HF replaces its placeholder with
whatever we return and never asks again, so a 0 for an invocation that just
hasn't been written yet would bill it as free forever. An entirely unknown
batch returns `{"requests": null}`, HF's documented retry signal. Both give
us the ~30 minutes of retries HF allows.
Also mirrors X-Chutes-InvocationID onto an `Inference-Id` response header in
build_response_headers, and exposes it via CORS. HF accepts a custom header name,
so this is optional — but it costs one line at the single choke point and makes
the integration zero-config on their side.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the two endpoints Hugging Face requires to list Chutes as an Inference Provider. Both are additive — no existing response field changes shape, no existing caller needs touching.
Context: I audited our live API against HF's requirements and we're closer than expected. The
chutesaiHub org is already on their Team plan (their prerequisite), and our inference already passes their automated validation as-is — I measured TTFT at 0.98–2.75s across 8 models streaming against their 5s threshold, and tool calling and JSON-schema structured output both work. These two gaps are the only hard blockers.1.
/v1/modelspricing aliasesHF reads
pricing.input/pricing.output(USD per million tokens) to power their provider comparison table and their:cheapestrouting policy. We emitpricing.prompt/pricing.completion, the OpenRouter naming. Both key pairs are now present with identical values.Same spot backfills
context_lengthfrommax_model_lenfor the two models that don't report it (unsloth/Mistral-Nemo-Instruct-2407-TEE,Nemotron-3-Nano-Omni-30B-TEE), since HF wants it on every listed model.2.
POST /partners/huggingface/billingHF polls this once a minute with the request IDs it routed to us, using the same bearer auth as inference, and bills its users whatever we report.
Costs come from the per-invocation billed balance already persisted in
invocations.metrics->>'b', so there is no second pricing path to keep in sync — HF users are charged exactly the standard Chutes rate, including any overrides and discounts that applied. Scoped to the authenticated user so one partner key can never read another account's costs.Two behaviours in there are load-bearing rather than stylistic, both straight from HF's spec:
0. HF replaces its placeholder cost with whatever we return and never asks again, so answering0for an invocation that simply hasn't been written yet would bill it as free, permanently. Omitted IDs get retried every minute for ~30 minutes.{"requests": null}, HF's documented "no data yet, retry later" signal.3.
Inference-Idheader (optional)Mirrors
X-Chutes-InvocationIDonto anInference-Idresponse header inbuild_response_headers, and exposes it via CORS. HF accepts a custom header name, so this part is genuinely optional — it's one line at the single choke point and saves a negotiation round. Happy to drop it if you'd rather not add a header.Routing note
The billing endpoint lands on
default_router, so it is reachable athttps://api.chutes.ai/partners/huggingface/billingand not onllm.chutes.ai— the MEGALLM branch inmain.pycaptures every POST to that host. That matches HF's spec, which explicitly allows the billing URL to sit on a different domain from the inference base URL as long as the auth pattern is the same.Testing
Syntax-checked only; I don't have a local DB/Redis to run the suite against. The billing query follows the existing
text()pattern used forinvocationselsewhere inapi/chute/util.py. Worth a look at theINTERVALlookback window (currently 2 hours, chosen because HF gives up on a request after ~30 minutes) and at whether theinvocation_id/completed_atindex selection is what you'd want at that batch size (HF sends up to 10,000 IDs per call).Nothing is pending on HF's side — we haven't contacted them yet — so there's no time pressure on this.