Multi-objectives: Implement helper to add token usage tracking and validation in objectives/guide#78
Open
doxav wants to merge 2 commits into
Open
Conversation
…ives and guides - Added `UsageTrackingLLM` class to wrap LLMs and record token usage. - Introduced `TokenUsageAugmentingGuide` to integrate token metrics into existing guides. - Enhanced `ObjectiveConfig` to include required metrics validation. - Implemented tests for required metrics in objectives and token usage in guides. - Created JSON output for multi-objective token usage results.
chinganc
reviewed
Jun 10, 2026
| return ranks | ||
|
|
||
|
|
||
| def _prepare_score_dicts( |
Member
There was a problem hiding this comment.
The name and docstring of this method is ambiguous.
Can we rename to capture the optimization or best selection aspect?
chinganc
reviewed
Jun 10, 2026
| return "" if content is None else str(content) | ||
|
|
||
|
|
||
| class TokenUsageAugmentingGuide(Guide): |
Member
There was a problem hiding this comment.
should we later make this as a mixin so we can create subclass with this feature dynamically?
chinganc
reviewed
Jun 10, 2026
| class UsageTrackingLLM: | ||
| """Wrap an LLM and record token usage for the current execution context. | ||
|
|
||
| The wrapper reads OpenAI-compatible usage metadata when available: |
Member
There was a problem hiding this comment.
We should check and throw warning when non compatible backends are used.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
UsageTrackingLLMclass to wrap LLMs and record token usage.TokenUsageAugmentingGuideto integrate token metrics into existing guides.ObjectiveConfigto include required metrics validation.