Skip to content

Fix HuggingFace usage extraction: defaultapi_flavor and generic huggingface provider - #452

Closed
adtyavrdhn wants to merge 13 commits into
mainfrom
fix/324-huggingface-extraction
Closed

Fix HuggingFace usage extraction: defaultapi_flavor and generic huggingface provider#452
adtyavrdhn wants to merge 13 commits into
mainfrom
fix/324-huggingface-extraction

Conversation

@adtyavrdhn

@adtyavrdhn adtyavrdhn commented Jul 7, 2026

Copy link
Copy Markdown
Member

This pull request was posted by Codex Desktop using gpt-5.6-sol on behalf of David.

Part of #324 (gaps 2 and 3). Unblocks the HuggingFace migration in pydantic/pydantic-ai#4818.

What this fixes

  • Generates both an implicit default extractor and the existing explicit chat extractor for every routed huggingface_* provider. Keeping both flavors preserves existing callers while making the library default work.
  • Adds an extraction-only huggingface provider for Hugging Face's server-side auto-routed /v1 endpoint, where the downstream provider is intentionally unknown. It has no models or fallback pricing because assigning a downstream price without provider identity would be ambiguous. Its explicit empty model path makes the public Python API return usage with model=None instead of attempting an impossible model lookup; JavaScript returns the equivalent model: null.
  • Maps required prompt/completion totals and optional nested OpenAI-compatible cache, cache-write, audio, and reasoning details. Reasoning is recorded as output_reasoning_tokens, while output_tokens remains the aggregate completion total.
  • Adds an extraction-only sync command so existing generated Hugging Face provider files stay aligned even when a provider disappears from the live model listing. Generated v2/package artifacts and the golden usage corpus were regenerated through the repository generators.

The implementation adopts #327's generator-root-cause analysis and its backward-compatible “duplicate default and chat” design option. Both #327 and this PR are by @adtyavrdhn; #327 remains open as the original plan/design record.

Evidence

The generic extractor deliberately does not reinterpret Together's provider-specific top-level cached_tokens: HF's formal schema does not define it, response shapes vary by routed provider, and mapping both top-level and nested forms would be additive in the current data-driven contract and could double-count. The exact recorded Together shape is pinned in Python and JavaScript tests to make that behavior explicit.

Validation

  • make build
  • make lint
  • make typecheck
  • make test
  • npm run ci
  • focused Python extractor/source/dataset tests
  • focused JavaScript extractor/dataset tests
  • tests/dataset/extract_usages.py (golden corpus up to date)
  • git diff --check

Review in cubic

The generated huggingface_* providers only exposed api_flavor 'chat',
so extract_usage with the library default flavor raised ValueError.
Duplicate the copied OpenAI chat extractor as the implicit default
flavor, keeping 'chat' for backward compatibility.
HuggingFace defaults to server-side auto routing, in which case no
routed provider URL is available and provider resolution found
nothing. Add a hand-written huggingface provider with the same
OpenAI-chat-shaped extractors. Its api_pattern matches only the /v1
router path so routed huggingface_* providers keep winning URL
matches. It has no priced models, so the public extract_usage model
lookup raises LookupError; callers extracting via the provider object
directly are unaffected.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 21 files

Not reviewed (too large): tests/dataset/usages.json (~12,384 lines), packages/js/src/data.ts (~428 lines), packages/python/genai_prices/data.py (~266 lines), prices/data_slim.json (~2 lines), prices/data.json (~2 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.

Re-trigger cubic

adtyavrdhn and others added 6 commits July 7, 2026 22:23
…dence

The generic provider is extraction-only: with server-side auto
routing the serving provider is unknown, so it has no per-model
prices and price calculation always fails. JS matches api_pattern
with unanchored RegExp.test, so the routed-vs-generic resolution
precedence is now pinned in the JS tests as well.
…xtraction

# Conflicts:
#	README.md
#	packages/js/src/__tests__/extractUsage.test.ts
#	prices/data.json
#	prices/data_slim.json
#	prices/src/prices/source_huggingface.py
#	tests/dataset/usages.json
#	tests/test_extract_usage.py
…xtraction

# Conflicts:
#	prices/new_data/v2/data.json
#	prices/new_data/v2/data_slim.json
Comment thread prices/providers/huggingface.yml Outdated
…xtraction

# Conflicts:
#	prices/new_data/v2/data.json
#	prices/new_data/v2/data_slim.json

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 14 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread packages/js/src/__tests__/dataset.test.ts Outdated
@dsfaccini dsfaccini changed the title Fix HuggingFace usage extraction: default api_flavor and generic huggingface provider Fix HuggingFace usage extraction: defaultapi_flavor and generic huggingface provider Aug 26, 2026
@Kludex

Kludex commented Aug 28, 2026

Copy link
Copy Markdown
Member

Closing in favor of #634, which implements the accepted default api_flavor behavior for generated Hugging Face and OVHcloud providers on the current codebase. Thank you for working on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants