AI DIAL Generic RAG Backend version
latest
What is the problem this feature will solve?
_channel_completion (src/generic_rag/app/chat_completion.py:83-89) reports usage with
response.set_usage(cb.prompt_tokens, cb.completion_tokens), via LangChain's get_openai_callback().
set_usage writes only the single-total usage field — no model field.- Clients can't tell which model generated the answer, only a raw token count.
- Doesn't scale if generation ever involves more than one model (router, multi-step, vision + text
step): a single total would blend costs with no way to attribute tokens per model.
What is the feature you are proposing to solve the problem?
- Track usage per model during generation using
langchain_core.callbacks.get_usage_metadata_callback()
(UsageMetadataCallbackHandler, langchain-core >= 0.3.49) instead of get_openai_callback() — it
keys usage_metadata by model in a dict and sums repeat calls to the same model, rather than
collapsing everything into one flat total.
- Call
response.add_usage_per_model(model=..., prompt_tokens=..., completion_tokens=...) once per
distinct model, instead of (or in addition to) set_usage.
- Caveat:
get_usage_metadata_callback() keys by whatever model name the LLM response itself
reports, which may not match the DIAL Core deployment name — DIAL Core can route a deployment to
an upstream endpoint that reports a different model identifier than the deployment name. If that
turns out to matter (i.e. usage needs to be attributed by DIAL Core's deployment name, not the
upstream model name), reuse TokenUsageByModelsCallback from the statgpt-backend repo instead
(statgpt/common/utils/callbacks.py:122-195)
— it captures the deployment name at call-start time rather than from the response. Note: that
callback's fallback chain for recovering the name beyond its primary capture has unclear
motivation and is probably simplifiable — no need to carry it over as-is.
What alternatives have you considered?
No response
AI DIAL Generic RAG Backend version
latest
What is the problem this feature will solve?
_channel_completion(src/generic_rag/app/chat_completion.py:83-89) reports usage withresponse.set_usage(cb.prompt_tokens, cb.completion_tokens), via LangChain'sget_openai_callback().set_usagewrites only the single-totalusagefield — nomodelfield.step): a single total would blend costs with no way to attribute tokens per model.
What is the feature you are proposing to solve the problem?
langchain_core.callbacks.get_usage_metadata_callback()(
UsageMetadataCallbackHandler, langchain-core >= 0.3.49) instead ofget_openai_callback()— itkeys
usage_metadataby model in a dict and sums repeat calls to the same model, rather thancollapsing everything into one flat total.
response.add_usage_per_model(model=..., prompt_tokens=..., completion_tokens=...)once perdistinct model, instead of (or in addition to)
set_usage.get_usage_metadata_callback()keys by whatever model name the LLM response itselfreports, which may not match the DIAL Core deployment name — DIAL Core can route a deployment to
an upstream endpoint that reports a different model identifier than the deployment name. If that
turns out to matter (i.e. usage needs to be attributed by DIAL Core's deployment name, not the
upstream model name), reuse
TokenUsageByModelsCallbackfrom thestatgpt-backendrepo instead(statgpt/common/utils/callbacks.py:122-195)
— it captures the deployment name at call-start time rather than from the response. Note: that
callback's fallback chain for recovering the name beyond its primary capture has unclear
motivation and is probably simplifiable — no need to carry it over as-is.
What alternatives have you considered?
No response