Add price variants for batch APIs and service tiers - #547
Conversation
Add `batch_prices`, an optional sibling of `prices` on a model with the same
shape, holding the rates a provider charges for requests made through its batch
API. Callers opt in with `calc_price(..., batch=True)`,
`calcPrice(usage, id, {batch: true})` or the CLI's `--batch`.
`batch_prices` overrides `prices` key by key, so only the keys whose rate
actually changes need listing. Batch discounts are not uniform per unit:
Anthropic halves every token rate but publishes no batch rate for web searches,
Google bills batch cache hits at the standard cache rate on most models, Groq's
batch rate ignores cache status entirely, and xAI's discount is 20% rather than
50%. An omitted key keeping its standard rate is exactly right for those.
Data covers 126 models across anthropic, openai, google, mistral, groq and
x-ai, from each provider's published batch rates.
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 28 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
An independent re-verification pass against each provider's live pricing pages turned up three problems in the first commit: - Groq's Batch API accepts five chat models (openai/gpt-oss-20b, openai/gpt-oss-120b, llama-3.3-70b-versatile, llama-3.1-8b-instant, meta-llama/llama-guard-4-12b), not all 29 in the file. Batch prices on the other 24, mostly decommissioned models, described a discount that cannot be obtained. - 15 mistral.yml entries match no model the Mistral API currently serves; they exist to match third-party price sources and are not batchable either. - grok-4.20-multi-agent's standard rates come from an OpenRouter sync and do not match xAI's published $1.25/$0.20/$2.50, so a 0.8x derivation of them was wrong in absolute terms. Dropped rather than propagated; correcting the standard rates belongs in its own change. Also adds the batch rates that were missed: gemini-embedding-001 and gemini-embedding-2, which publish their own batch column, and gpt-3.5-turbo, gpt-4 and gpt-4-turbo, whose batch rows are exactly half their standard ones.
|
Ran an independent re-verification pass over every number in this PR against the providers' live pricing pages (one auditor per provider, each re-deriving the rates from scratch rather than checking my arithmetic). Pushed 9a5f56c with the corrections. Summary of what it found, including things that turn out to be pre-existing. Fixed in this PR
Everything else was verified correct. For OpenAI the auditor pulled the unrounded SSR data behind the pricing page rather than the rendered 2-dp cells, and all 32 blocks matched cell for cell. Pre-existing issues found on the way (not addressed here)Worth their own change - flagging rather than fixing, since correcting first-party standard rates has different review implications:
One open judgement callAnthropic's batch page says "All usage is charged at 50% of the standard API prices", but the batch pricing table lists only token rates and the FAQ scopes it to "input tokens, output tokens, and any special tokens". I have left |
The mistral.yml half of "Restrict batch prices to models the provider will actually batch" was left out of that commit, so the published data no longer matched the provider YAML it is generated from and the build hook rewrote it.
There was a problem hiding this comment.
All reported issues were addressed across 8 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Records where each set of batch rates comes from and why some models have none: Groq's eligible-model list, Google's standard-rate cache hits, xAI's 20%, and Anthropic's undiscounted web searches. Also dates the two Gemini embedding models, whose standard and batch rates were both checked against the pricing page today.
There was a problem hiding this comment.
All reported issues were addressed across 9 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
A parity sweep over all 91 models with batch prices (8 timestamps x 4 usage shapes, both engines) found no arithmetic drift, but three gaps around it: - JS treated a JSON `null` in batch_prices differently from Python: Python falls back to the standard rate for that key, JS wrote the null into the merged prices and then threw on it, and a null `batch_prices` threw outright. Reachable through caller-supplied provider data, which the published feed's `exclude_none` does not cover. - `batch_prices` could omit a dated change that `prices` makes, which would charge today's batch rate against requests made before it - the batch equivalent of overwriting a `prices` block. The build now requires `batch_prices` to repeat every constraint `prices` uses. - `make collapse-models` compared only `prices` when merging a child model into its parent, so it could drop or misapply batch rates. No model pairs are affected today. Also adds `--batch` to the JS CLI, which the Python CLI already had.
|
Second verification round, this time on the engine rather than the numbers: a parity harness ran every one of the 91 models with batch prices through both implementations at 8 timestamps straddling each constraint boundary in the data, against 4 usage shapes (above and below tier thresholds, cache read, cache write + 1h), batch and standard - 5824 rows, no arithmetic drift between Python and JS. It also confirmed the overlay does not mutate the shared bundled Three gaps around the arithmetic did turn up, fixed in 4930f40 and 81d95a5:
Also added One more pre-existing item for the list above, latent rather than live: Python's |
An empty `batch_prices` list made both engines resolve a conditional list with nothing in it, raising IndexError in Python and TypeError in JS instead of falling back to the standard prices. The build rejects an empty list, so this was only reachable through caller-supplied provider data.
There was a problem hiding this comment.
1 issue found across 11 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="prices/src/prices/collapse.py">
<violation number="1" location="prices/src/prices/collapse.py:36">
P3: Equivalent empty batch overrides now block model collapsing: `None` and `ModelPrice()` both use standard prices, but compare unequal here. Normalize empty batch overrides before comparing so collapse continues to remove semantically identical models.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
`batch_prices: {}` was accepted and meant nothing, which also made two otherwise
identical models compare unequal in `collapse_provider`. An empty list was
already rejected; an empty mapping now is too, so the only way to say "charge
the standard prices" is to omit the field. The runtime keeps treating an empty
overlay as a no-op, since caller-supplied data is not built here.
`batch_prices` could only ever say one thing. Replace it with `price_variants`,
a list of entries that each name the pricing context they apply to:
prices:
input_mtok: 5
output_mtok: 25
web_searches_kcount: 10
price_variants:
- when: { service_tier: batch }
prices:
input_mtok: 2.5
output_mtok: 12.5
`when` draws on a fixed set of parameters named after the fields providers
report them in: `service_tier` for the mutually exclusive rate cards (batch
today, flex and priority when we have rates for them), `speed` for Anthropic's
fast mode, `inference_geo` for data residency. Anthropic and Groq both report
`service_tier: "batch"` in their usage payloads, so a caller can pass what the
provider told them straight through rather than translating it.
Callers select a variant with `price_context` / `priceContext`; `batch=True` is
kept as shorthand for `{'service_tier': 'batch'}`, as is the CLIs' `--batch`.
Resolution is unchanged otherwise: the first matching variant's prices override
the standard ones key by key, and dated variants resolve exactly as `prices` do.
This stays a new optional field on the model, so released clients ignore it
rather than rejecting the feed - which is what lets the vocabulary grow without
a v3 contract. Adding `when` to the existing `prices` list could not: an old
client reads an entry with no `constraint` as unconditionally active and
silently misprices every request.
|
Refactored to generalise this beyond batch, per review feedback that a batch-only field forecloses flex/priority. prices:
input_mtok: 5
output_mtok: 25
web_searches_kcount: 10
price_variants:
- when: { service_tier: batch }
prices:
input_mtok: 2.5
output_mtok: 12.5
Three things worth recording about the shape, since they were not obvious going in.
price = extracted_usage.calc_price(price_context={'service_tier': response['usage']['service_tier']})This did not need a v3 feed. The suggestion was to publish v3 rather than let v2 compatibility dictate the shape, and that is right in general - but a new field is invisible to released clients, so the vocabulary can grow additively. What genuinely cannot go in v2 is Stacking is still unsolved, by any of the designs considered. Of the four axes in #429, only data residency is a true multiplier (1.1x on all token categories, stacks with batch); fast mode is documented as not stacking with batch, so it belongs on the same pick-one axis. Neither a tier map nor The remaining point from that review - making clients tolerant of schema additions - is a separate PR and I would suggest doing it next. Today an unknown constraint shape makes both released clients reject the entire feed permanently, and an unknown key inside a known constraint makes JS reject everything while Python silently misprices. The fix is small (~4 files: classify unrecognised entries instead of throwing, skip them in resolution, fix the fallback so it never lands on one, and close the Python Data and behaviour are unchanged - same 91 models, same verified rates, the real-provider batch payloads still price identically. Python 787 tests at 100% coverage, JS 1859, lint and typecheck clean on both. |
Kludex
changed the title
There was a problem hiding this comment.
All reported issues were addressed across 29 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Resolution flattened every matching variant into one dated list, so a request matching both a general and a more specific variant got whichever was listed last - the opposite of the documented first-match rule, and a difference that only shows up once a second `when` parameter carries data. Both engines now pick the first matching `when`, then resolve that group's dated entries as before. Also from review: group variants by a type-aware key so `1` and `'1'` cannot collapse into one group at build time, narrow `PriceVariant.when` in the JS types to the parameter set the build already enforces, and rebuild the conditional-resolution fixture so it is a shape `make build` would accept.
`grok-4.20` matched only that exact alias, but the batch-eligible IDs xAI documents are `grok-4.20-0309-reasoning` and `grok-4.20-0309-non-reasoning`, neither of which resolved to any model at all - so the batch variant added here could never apply to a real batch request. Both now match, with their `x-ai/`-prefixed forms. Also updates the provider `price_comments` that still named `batch_prices`.
There was a problem hiding this comment.
1 issue found across 6 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/js/src/types.ts">
<violation number="1" location="packages/js/src/types.ts:47">
P2: An optional `when` member accepts `undefined`, which the matcher treats as matching an absent context field. A custom variant built from an optional provider field can therefore override prices for `{}` or unrelated context payloads; reject undefined `when` values before matching (or validate them at the public-data boundary).</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Narrowing `PriceVariant.when` to a partial record let its values be undefined,
and the matcher compared an undefined expectation against an absent context
field and called that a match - so `when: {service_tier: undefined}` matched
every context that did not set a service tier. Python had the same hole via a
`None` value, reachable from caller-supplied provider data in both.
An expected value that is not set now never matches, so the variant is skipped
and the standard prices apply. Both `Object.entries` and the declared value type
hide that `undefined`/`None` can be there at all, so the matchers now say so in
their signatures rather than trusting the narrower inferred type.
An empty `when` matched every request that supplied any pricing context at all, because "every parameter matches" is vacuously true with no parameters. The build already requires at least one, but caller-supplied provider data does not go through it, and both engines had the hole. A variant naming no parameter now matches nothing, so the standard prices apply.
|
I think we should consider dropping the public The shorthand presents batch as a provider-independent capability, but it only changes the result for models that currently have a matching
It also makes the library's synthetic The explicit context API is enough to ship and validate the data model in this PR. A provider-independent convenience flag can be added later once we have decided the no-match behavior and the canonical meaning of batch across providers. If the shorthand stays, I think it at least needs an observable diagnostic when no batch variant matched; silently producing the standard price makes incomplete coverage indistinguishable from verified standard pricing. |
Per review: a provider-independent `batch=True` presents batch as a capability
of the library rather than of the data, but it only changes the result where a
`when: {service_tier: batch}` variant exists. Everywhere else it silently
returns the standard price, which makes four different states look identical -
the model bills the same in batch, the provider has no batch API, nobody has
researched its rates yet, or model matching and context spelling failed.
It also fixed the library's synthetic `service_tier: batch` as if it were the
providers' wire value, which only Anthropic and Groq actually report; OpenAI
returns `default`, Gemini `standard`, Mistral nothing. Shipping the shorthand
would make deciding the no-match behaviour, or separating request processing
mode from provider service tier, a public API break later.
Callers now pass `price_context={'service_tier': 'batch'}` explicitly, which is
enough to ship and validate the data model. `--batch` becomes
`--price-context service_tier=batch` on both CLIs, so they reach the mechanism
rather than a convenience wrapper over it.
|
Applied in 43467b5 - the Your point about the four collapsed states is the one that decided it for me. It also applies to the explicit API, but much less sharply: One judgement call worth flagging, since it is adjacent to what you asked for rather than part of it. Deleting Both CLIs agree, including the failure path: The no-match diagnostic you raised is still unbuilt, deliberately - it needs the same decision about what "no variant matched" should mean, and I would rather that land with the client-tolerance work than be guessed at here. 795 Python tests at 100% coverage, 1862 JS, everything else unchanged. |
There was a problem hiding this comment.
All reported issues were addressed across 13 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Passing the same parameter twice silently kept the last value, so a typo or a pair of conflicting flags produced a plausible price rather than a complaint. Two flags at the same level have no principled winner, so both CLIs now say so.
There was a problem hiding this comment.
All reported issues were addressed across 3 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
`parameter in context` also finds inherited Object.prototype members, so a first `--price-context constructor=...` was rejected as a duplicate, and assigning `__proto__` set the prototype instead of a key. The context object now has a null prototype and the guard checks own properties only.
2 participants
Adds prices that depend on how a request was served, and the batch-API rates to go in them. Closes part of #429, and is the price-data half of what #115 and #390 ask for.
Shape
price_variantsis an optional sibling ofpriceson a model. Each entry names the pricing context it applies to and the prices that replace the standard ones:whendraws on a fixed parameter set named after the fields providers report them in:service_tierfor the mutually exclusive rate cards (batchtoday,flexandpriorityonce we have verified rates),speedfor Anthropic's fast mode,inference_geofor data residency. Anthropic and Groq both returnservice_tier: "batch"in their usage payloads, so a caller can pass what the provider told them straight through.A variant overrides
priceskey by key, so only the keys whose rate actually changes need listing. That matters because discounts are not uniform per unit:web_searches_kcountAn omitted key keeping its standard rate is exactly right for all of those. A model with no
batch_pricesis charged its standard rates.Callers select a variant explicitly:
batch=True/batch: trueis shorthand for the batch tier, as is--batchon both CLIs.price_contextis also accepted byExtractedUsage.calc_price()andModelInfo.calc_price().Why a new field rather than
whenon the existingpriceslistA general request-context mechanism could also go on the existing
ConditionalPriceentries, which is what #445 and #447 do. That shape cannot ship to v2. I tested each variant against the released clients by serving it as a mutateddata.jsontogenai-prices==0.1.1and@pydantic/genai-prices@0.1.1through their real ingest paths:ConditionalPricewith noconstraintSince
prices/new_data/v2/data.jsongoes live to those clients on merge, a new field is the only shape that is safe - and it is what lets thewhenvocabulary grow later without a v3 contract. Publishing v3 instead was considered: v3 is already spoken for by the phase-2 spec (a{units, providers}wrapper for auto-updating units) which freezes v2 at cutover, so attaching tier pricing to it would mean either delaying this or forcing v3 early.Stacking is worth being explicit about: of the four axes in #429, only data residency is a true multiplier (1.1x on all token categories). Fast mode is documented as not stacking with batch, so it sits on the same pick-one axis as the service tiers. No design considered here expresses a multiplier - a tier map and
when/valuesboth fall back to enumerating the cross product with pre-multiplied absolutes. Residency wants a separate multiplier concept, which is additive on top of this.Verification
Ran real batch jobs against five providers and checked the computed price against each published rate table. All matched exactly:
Those payloads are pinned as fixtures in
tests/test_price_variants.pyand mirrored inpackages/js/src/__tests__/priceVariants.test.ts, covering the discount ratio per provider, fall-through for undiscounted units, type-strictwhenmatching, and independent resolution of dated variants.Worth noting for #429: batch mode cannot be auto-detected in general. Anthropic and Groq report
service_tier: "batch", but OpenAI reports"default", Gemini reportsserviceTier: "standard"and Mistral reports null - sobatchhas to stay a caller-supplied flag.Data coverage
91 models:
openai35,anthropic18,google18,mistral13,groq5,x-ai2. Every rate comes from the provider's published batch table, and models the provider will not accept for batch processing are left withoutbatch_pricesso they fall back to standard rates (Groq's Batch API takes five chat models; 15mistral.ymlentries match nothing the Mistral API serves).Deliberately left out, since each needs its own verification pass rather than an assumption: AWS Bedrock and Azure batch, Claude on Vertex/Bedrock, Together (only six named models are discounted, the rest bill at standard rates), Cohere (no published discount at all), OpenAI's audio/realtime/embeddings rows and gpt-5.3/codex/chat-latest, which have no batch pane, and OpenAI's fine-tuned rows, whose batch pane columns do not align unambiguously with the standard pane.
Checks
make lint,make typecheck,make test(787 passed, 100% coverage),npm run ci(1859 passed) andnpm run lintall pass;make buildis idempotent.AI Disclaimer
This PR was developed with the assistance of either Claude or Codex. I've reviewed and verified the changes.