Repository navigation
fix: rename citation precision/recall to inclusion; report real cached-token counts - #4
Merged
Merged
Conversation
…ecall The metric compares the papers a run includes with each golden's expectedPapers; it never looks at the citations in the draft. The README presented it as "the citations it does surface are accurate". - Dashboard labels, descriptions and formulas now say "Inclusion recall/precision" and "included" instead of "cited". Stored metric keys (citation_recall / citation_precision) are unchanged. - README: the cite_check bullet no longer attributes 97%/74% to citation accuracy; it says what is measured and that the figure is averaged over the 9 goldens with an inclusion score (per the committed /evals screenshot), and points to the limitations. - evals/README.md: new "Known limitations" section. 7 of the 17 V1 goldens (000, 004, 007, 008, 014, 015, 016) seed no distractor papers, so inclusion precision is 1.0 by construction on those. - metrics.ts: doc comments on citationRecall/citationPrecision. - live e2e smoke regex matches the new label. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-coded 0 runLLM returned cacheReadInputTokens: 0 on every path. The AI SDK (ai 6) already maps each provider's cached-token field into usage.inputTokenDetails.cacheReadTokens (Anthropic cache_read_input_tokens, OpenAI prompt_tokens_details / input_tokens_details.cached_tokens, Mistral num_cached_tokens, Gemini cachedContentTokenCount; Groq leaves it unset), so read it from there, defaulting to 0. The claude-agent path still reports 0 because it has no per-call usage. No prompt caching is added; Thoth sets no cache_control. The value is a subset of inputTokens, so the token-badge comment that said cache reads are not counted toward the budget is corrected. Adds two unit tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two honesty fixes.
1. "Citation precision" measures paper selection (O06)
citationPrecision/citationRecall(lib/eval/metrics.ts, called fromscripts/run-evals.ts) compare the paper ids a run includes with the golden'sexpectedPapers. They never look at the citations in the draft. The README said "the citations it does surface are accurate — citation precision 97%".Changes:
/evalsdashboard: labels are now "Inclusion recall" / "Inclusion precision"; the descriptions and formulas say "included" instead of "cited"; the precision definition says it does not check the draft's citations and is 100% by construction when a golden's corpus has no distractor papers. The headline banner says what is measured.cite_checkbullet no longer attributes 97% / 74% to citation accuracy. It says the papers Thoth includes are compared with each question's expected list, that this is not a check of citations, and links to the new limitations section. The Proof row and the dashboard description use the new names. The screenshot alt text notes the screenshot still shows the old labels.evals/README.md: new "Known limitations" section. 7 of the 17 V1 goldens (000, 004, 007, 008, 014, 015, 016) seed no distractor papers, so inclusion precision is 1.0 by construction on those (checked by comparingpapers[].idwithexpectedPapersin everyevals/golden/*.yaml).citation_recall,citation_precision) in the DB,eval-results.jsonand the regression script are unchanged.metrics.tsgets doc comments noting the display names.Number of goldens behind 97% / 74%: the README previously placed these figures next to "7 of 18 populated at this commit". The committed screenshot
docs/assets/media/03-evals.png, which shows the same 97% / 74%, lists 9 V1 goldens with a precision/recall score (000, 001, 002, 004, 005, 007, 008, 009, 010). Their per-question values average to 97% and 74%. The "7 of 18" is the dashboard's "goldens with data at the latest commit" badge. The README now says the figures are averaged over 9 goldens, and "populated at this commit" now reads "have data at the latest sweep's commit". Four of those 9 (000, 004, 007, 008) have no distractor papers. The live dashboard currently returns 500 (database down), so these numbers come from the screenshot and could not be checked live.2. Cached input tokens were always 0 (O10)
runLLMreturnedcacheReadInputTokens: 0on every path. Inai@6.0.191,generateObjectreturnsusage.inputTokenDetails.cacheReadTokens, and each provider package fills it from its own field. Checked in the installed packages:@ai-sdk/anthropic:cache_read_input_tokens@ai-sdk/openai:prompt_tokens_details.cached_tokens(chat) /input_tokens_details.cached_tokens(responses)@ai-sdk/mistral:num_cached_tokens, falling back toprompt_tokens_details.cached_tokens@ai-sdk/google:cachedContentTokenCount@ai-sdk/groq: always unsetrunLLMnow reads that field and defaults to 0. Theclaude-agentpath still returns 0 because it reports no per-call usage. This change adds no prompt caching, and the repo sets nocache_control. This was the smaller option: removing the field would have touched theRunStepcolumn, two UI components and about a dozen test files.The AI SDK counts cached tokens inside
inputTokens. The token badge's doc comment said cache reads are not counted toward the budget, which is wrong once real values come through, so the comment is corrected. Budget behaviour is unchanged.Verification
pnpm typecheck: cleanpnpm lint: cleanpnpm test: 95 files passed, 1 skipped; 691 tests passed, 2 skipped. Includes 2 newrunLLMtests for the cached-token mapping./evalspage change was typechecked but not rendered. I have not seen a non-zero cached-token value from a real provider.🤖 Generated with Claude Code