mcp: token-cost benchmark in the README + reproducible script - #26
Merged
Conversation
Adds a Token cost section to mcp/README.md and mcp/benchmark_tokens.py. Handing an agent a raw web page is expensive: the Web scraping Wikipedia article is 68,240 tokens of HTML and Nike's homepage is 353,000 (tiktoken cl100k_base). A built-in web fetch summarizes readable pages cheaply and clears anti-bot on many sites, but returns nothing on JS-rendered pages and some hard walls, where the agent falls back to raw HTML and still fails. The Fortress MCP returns bounded clean output there (285 tokens for the JS quotes page, about 700 for Nike instead of a 403). benchmark_tokens.py counts the tokens each path returns for a list of URLs and flags which ones a naive client is blocked on. Docs plus one script; no engine or server code changed.
Member
|
reproduced and works well on my device. lgtm! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Token cost section to
mcp/README.mdand a reproduciblemcp/benchmark_tokens.py.What it shows
Handing an agent a raw web page is expensive. The "Web scraping" Wikipedia article is 68,240 tokens of HTML; Nike's homepage is 353,000 (tiktoken cl100k_base). A built-in web fetch summarizes readable pages cheaply and clears anti-bot on many sites, but returns nothing on JavaScript-rendered pages and some hard walls, where the agent falls back to raw HTML and still fails. The Fortress MCP returns bounded clean output there: 285 tokens for the JS quotes page, about 700 for Nike instead of a 403.
Reproduce
benchmark_tokens.pycounts the tokens each path returns for a list of URLs and flags which ones a naive client is blocked on. Measured on one residential IP; numbers vary by IP and site.Docs plus one script. No engine or server code changed.