Harden estimator/CLI, 4x tests (43->202), 4x demos (5->20)#5
Open
cognis-digital wants to merge 1 commit into
Open
Harden estimator/CLI, 4x tests (43->202), 4x demos (5->20)#5cognis-digital wants to merge 1 commit into
cognis-digital wants to merge 1 commit into
Conversation
Depth pass on tokenmeter: real error handling + fixes, ~4.7x the tests, and 4x the runnable demo scenarios. Public API is unchanged. Fixes / hardening (tokenmeter/core.py, tokenmeter/cli.py): - estimate() now rejects negative input_tokens / output_tokens with a clear ValueError instead of silently producing negative (nonsensical) costs. - add_model() validates its inputs: blank name, non-numeric or negative prices, and non-positive context windows are rejected up front, so a mispriced model can no longer corrupt every downstream estimate. - count_tokens() tolerates None and raises TypeError on non-str input rather than crashing deep in the regex. - CLI maps ValueError to a clean `error: ...` message and exit code 2 (was an uncaught traceback), and guards the compare path against an empty ranking. - Define TOOL_NAME / TOOL_VERSION in core so the documented `from tokenmeter.core import TOOL_NAME, TOOL_VERSION` path resolves instead of silently falling back in __init__. Tests: 43 -> 202. New tests/test_core_edge.py, test_cli_edge.py, test_identity.py, test_demos_behavior.py cover tokenizer boundaries, pricing validation, budget-gate boundaries, model-compare ties/subsets/empty, aggregation, every CLI subcommand x format x exit code, stdin, malformed input, and per-demo behavior. Demos: 5 -> 20 runnable scenarios (chat history, window guard, diff review, few-shot tax, model cascade, agent budget, prompt diet, RAG top-k sweep, portfolio rollup, output-length forecast, library audit, custom pricing, JSON pipeline, cost-regression guard, monthly forecast). run_all.py and docs/DEMOS.md updated; each exits 0 and prints a cost figure offline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A depth pass on tokenmeter: real error handling and bug fixes, ~4.7x the tests, and 4x the runnable demo scenarios. The public API is unchanged.
Fixes / hardening
estimate()rejected nothing. Negativeinput_tokens/output_tokensproduced negative (nonsensical) costs. It now raises a clearValueError. Via the CLI (count -o -5) this had been an uncaught traceback; the CLI now maps it toerror: ...and exit code 2.add_model()accepted garbage. Blank names, non-numeric or negative prices, and non-positive context windows were silently stored — one mispriced model then corrupts every downstream estimate and thecontext_used_pctdivision. All are now rejected up front.count_tokens()toleratesNoneand raisesTypeErroron non-strinput instead of crashing inside the regex.compareCLI guards against an empty ranking (would haveIndexErrord onests[0]).TOOL_NAME/TOOL_VERSIONare now defined incore, so the documentedfrom tokenmeter.core import TOOL_NAME, TOOL_VERSIONpath actually resolves instead of silently falling back in__init__.Tests: 43 -> 202
New
tests/test_core_edge.py,test_cli_edge.py,test_identity.py,test_demos_behavior.pycover tokenizer boundaries (empty/None/unicode/ceildiv/digit grouping), pricing validation, budget-gate boundaries (exactly-at-limit, multi-violation), model compare (ties/subsets/empty/unknown), aggregation (empty/generator/rounding), every CLI subcommand x format x documented exit code, stdin piping, malformed input, and per-demo behavior/idempotence.Demos: 5 -> 20
Fifteen new offline, audience-varied scenarios: chat-history cost, context-window guard, AI diff review, few-shot tax, model cascade/router, agent step budget, prompt diet, RAG top-k sweep, multi-model portfolio, output-length forecast, prompt-library audit, custom/self-hosted pricing, JSON pipeline, cost-regression guard, and a full-month forecast.
demos/run_all.pyanddocs/DEMOS.mdupdated. Each exits 0 and prints a cost figure with no network/keys.