feat(dashboard): estimate token count and cost from an uploaded text file

[FrancescoGnan]

Context

The dashboard Calculator prices a workload from four token counts the user types in by hand. The CLI does the same through --in / --out. In both cases the hard part is left to the user: nobody knows upfront how many tokens a given document is worth.

A concrete example: "what would it cost to have a model summarise this book?" is currently a two-tool question — one tool to count, tokenpricing to price.

Proposal

Add an optional entry point to the dashboard Calculator: upload a text file, get an estimated token count, and have it feed the existing input-token field. The cost math already in place then answers the question with no further changes.

Supported formats: .txt, .md, .csv, .json and plain-text source files, plus .pdf.

Why the dashboard and not the SDKs

libraries/python/AGENTS.md, libraries/typescript/AGENTS.md, AGENTS-docs/contribution-checklist.md and skills/tokenpricing/SKILL.md all state that token counting is out of scope for the libraries, on the grounds that tokenizers differ across providers and estimates presented as counts are unreliable.

I agree with that reasoning and this proposal does not challenge it. The libraries stay untouched. The dashboard is a separate, experimental service whose Calculator is already labelled "Estimated cost", so an explicitly-labelled estimate belongs there and nowhere else.

Design

• A pure estimateTokens(text) function in src/lib/estimate.ts, returning a central value and an uncertainty range, plus the method used.
• Character-to-token heuristic rather than a real tokenizer: no runtime dependency, and no false precision. A real tokenizer would be exact for one model family and wrong for every other one, which is the very problem the libraries' policy avoids.
• Text extraction in src/lib/extract.ts. Plain-text formats via File.text(). PDF via a dynamic import(), so the parser never enters the main bundle — relevant because the production bundle is already 764 kB and trips Vite's 500 kB warning.
• UI built only from the existing components/ui/ primitives, wired into calculator.tsx with a minimal diff.

Acceptance criteria

• Uploading a supported file fills the input-token field with an estimate
• The result is presented as an estimate, with the range and the method visible in the UI — never as an exact figure
• Unsupported formats produce a clear message, not a crash
• The PDF parser is absent from the main bundle
• typecheck, lint and build stay clean; no new lint warnings
• The estimation function is covered by unit tests
• Changes are confined to services/dashboard/

Out of scope

• Any change to the Python or TypeScript SDKs
• Real tokenizers (tiktoken and equivalents)
• .docx, .epub, OCR of scanned PDFs
• Output-token estimation, which depends on the prompt and not on the file

Notes

While setting up the baseline I noticed a pre-existing lint warning in src/components/explorer.tsx:193 (react-hooks/incompatible-library, from TanStack Table's useReactTable). Unrelated to this proposal — flagging it in case it is not already known.

Happy to adjust scope or approach if you have suggestions.