Repository navigation
Add honest-numbers framing banner to /evals - #2
Merged
Merged
Conversation
Adds a short framing banner at the top of the public eval dashboard so the real, published eval numbers read as eval rigor, not failure. Additive only — one <section> element inserted as the first child of the page container, no logic, data-fetching, or component changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughA new "HONEST-NUMBERS FRAMING" explanatory section is inserted at the start of the evals dashboard page content. The section provides context about interpreting the dashboard's automated scores, including notes on golden set scope, CI regeneration behavior, and metric interpretation. ChangesDashboard numbers framing
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~5 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/evals/page.tsx`:
- Around line 120-130: The paragraph in app/evals/page.tsx currently contains
hardcoded KPIs (18, 97%, 74%, 38%, 7 of 18); replace those literals with the
computed values from the component/page state/props (e.g., use the existing
metrics variables such as totalGoldens, precisionPct, recallPct,
faithfulnessPct, and goldensWithData or their actual names in this component) so
the displayed copy reflects the live totals shown elsewhere on the page; update
the rendering in the component that emits that paragraph to interpolate those
state values (format percentages and "X of Y" counts consistently) rather than
embedding fixed numbers.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
Comment on lines
+120
to
+130
| These are real, automated scores over a versioned 18-question golden | ||
| set — not a curated highlight reel. They are regenerated in CI and | ||
| published with the last-run date, so a regression here is a public, | ||
| falsifiable signal. Headline: citation precision 97% and recall 74% | ||
| on the goldens scored so far. Faithfulness (38%) comes from | ||
| cite_check, which scores each cited claim strictly against its source | ||
| paper and counts anything not clearly grounded as unsupported — a | ||
| deliberately conservative bar. Some axes are still maturing: discovery | ||
| and screening are v2 and being calibrated, and 7 of 18 goldens have | ||
| data at this commit. Current focus: lifting faithfulness and golden | ||
| coverage. |
There was a problem hiding this comment.
Replace hardcoded KPI values with computed values from the page state.
Line 120 through Line 130 embeds fixed numbers (18, 97%, 74%, 38%, 7 of 18) that can quickly diverge from the actual metrics rendered below. On a public “honest numbers” page, this creates contradictory and misleading output.
Proposed fix
- These are real, automated scores over a versioned 18-question golden
+ These are real, automated scores over a versioned {totalYamls || "?"}-question golden
set — not a curated highlight reel. They are regenerated in CI and
published with the last-run date, so a regression here is a public,
- falsifiable signal. Headline: citation precision 97% and recall 74%
- on the goldens scored so far. Faithfulness (38%) comes from
+ falsifiable signal. Headline: citation precision {Math.round((aggregate.citation_precision ?? 0) * 100)}% and recall {Math.round((aggregate.citation_recall ?? 0) * 100)}%
+ on the goldens scored so far. Faithfulness ({Math.round((aggregate.claim_faithfulness ?? 0) * 100)}%) comes from
cite_check, which scores each cited claim strictly against its source
paper and counts anything not clearly grounded as unsupported — a
deliberately conservative bar. Some axes are still maturing: discovery
- and screening are v2 and being calibrated, and 7 of 18 goldens have
+ and screening are v2 and being calibrated, and {completedInLatestSweep} of {totalYamls || "?"} goldens have
data at this commit. Current focus: lifting faithfulness and golden
coverage.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@app/evals/page.tsx` around lines 120 - 130, The paragraph in
app/evals/page.tsx currently contains hardcoded KPIs (18, 97%, 74%, 38%, 7 of
18); replace those literals with the computed values from the component/page
state/props (e.g., use the existing metrics variables such as totalGoldens,
precisionPct, recallPct, faithfulnessPct, and goldensWithData or their actual
names in this component) so the displayed copy reflects the live totals shown
elsewhere on the page; update the rendering in the component that emits that
paragraph to interpolate those state values (format percentages and "X of Y"
counts consistently) rather than embedding fixed numbers.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a short framing banner so the real, published eval numbers read as rigor, not failure. Additive only — no logic changes. Part of the personal-brand polish (F08).
Summary by CodeRabbit