Skip to content

chore: deprecate TokenCounterCallback and the hardcoded TOKEN_PRICES table #472

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lzhao8956-glitch wants to merge 1 commit into
base: main
Choose a base branch
from
Open

chore: deprecate TokenCounterCallback and the hardcoded TOKEN_PRICES table #472

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 27 additions & 2 deletions packages/valory/skills/task_execution/utils/benchmarks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,40 @@
# limitations under the License.
#
# ------------------------------------------------------------------------------
"""Benchmarking for tools."""
"""Benchmarking for tools.

.. note::
``TokenCounterCallback`` is **deprecated**. The hardcoded ``TOKEN_PRICES``
table is unsustainable as new models are released and conflicts with the
mech's role as a thin intermediary between clients and tools. Tool authors
should report their own pricing in the tool package instead. This module
remains importable for backwards compatibility and will emit
``DeprecationWarning`` on instantiation; it will be removed in a future
release.
"""

import logging
import warnings
from typing import Any, Callable, Dict, Optional, Union

PRICE_NUM_TOKENS = 1000

_DEPRECATION_MSG = (
"TokenCounterCallback is deprecated and the hardcoded TOKEN_PRICES table "
"will be removed. Tool packages should report their own pricing rather "
"than relying on this central registry. See issue #319 for context."
)


class TokenCounterCallback:
"""Callback to count the number of tokens used in a generation."""
"""Callback to count the number of tokens used in a generation.

.. deprecated::
Use the tool package's own pricing metadata instead. This callback and
its ``TOKEN_PRICES`` table are no longer the right place for model
pricing -- they go stale as new models ship and grow unboundedly. See
issue #319 for the rationale and the planned replacement.
"""

TOKEN_PRICES = {
"gpt-3.5-turbo": {"input": 0.0005, "output": 0.0015},
Expand Down Expand Up @@ -60,6 +84,7 @@ class TokenCounterCallback:

def __init__(self) -> None:
"""Initialize the callback."""
warnings.warn(_DEPRECATION_MSG, DeprecationWarning, stacklevel=2)
self.actual_model: Optional[str] = None
self.cost_dict: Dict[str, Union[int, float]] = {
"input_tokens": 0,
Expand Down