Skip to content

Add honest-numbers framing banner to /evals #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ahmedEid1 merged 1 commit into from
May 29, 2026
Merged

Add honest-numbers framing banner to /evals #2

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions app/evals/page.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,24 @@ export default async function EvalsPage() {

return (
<main id="main" className="max-w-6xl mx-auto px-6 py-16">
{/* HONEST-NUMBERS FRAMING ──────────────────────────────────────── */}
<section className="mb-10 max-w-3xl bg-[oklch(1_0_0)] border border-[var(--thoth-rule)] rounded-lg p-6">
<p className="eyebrow mb-3">How to read these numbers</p>
<p className="text-sm text-[var(--thoth-ink)] leading-relaxed">
These are real, automated scores over a versioned 18-question golden
set — not a curated highlight reel. They are regenerated in CI and
published with the last-run date, so a regression here is a public,
falsifiable signal. Headline: citation precision 97% and recall 74%
on the goldens scored so far. Faithfulness (38%) comes from
cite_check, which scores each cited claim strictly against its source
paper and counts anything not clearly grounded as unsupported — a
deliberately conservative bar. Some axes are still maturing: discovery
and screening are v2 and being calibrated, and 7 of 18 goldens have
data at this commit. Current focus: lifting faithfulness and golden
coverage.
Comment on lines +120 to +130

@coderabbitai coderabbitai Bot May 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Replace hardcoded KPI values with computed values from the page state.

Line 120 through Line 130 embeds fixed numbers (18, 97%, 74%, 38%, 7 of 18) that can quickly diverge from the actual metrics rendered below. On a public “honest numbers” page, this creates contradictory and misleading output.

Proposed fix 
-          These are real, automated scores over a versioned 18-question golden
+          These are real, automated scores over a versioned {totalYamls || "?"}-question golden
           set — not a curated highlight reel. They are regenerated in CI and
           published with the last-run date, so a regression here is a public,
-          falsifiable signal. Headline: citation precision 97% and recall 74%
-          on the goldens scored so far. Faithfulness (38%) comes from
+          falsifiable signal. Headline: citation precision {Math.round((aggregate.citation_precision ?? 0) * 100)}% and recall {Math.round((aggregate.citation_recall ?? 0) * 100)}%
+          on the goldens scored so far. Faithfulness ({Math.round((aggregate.claim_faithfulness ?? 0) * 100)}%) comes from
           cite_check, which scores each cited claim strictly against its source
           paper and counts anything not clearly grounded as unsupported — a
           deliberately conservative bar. Some axes are still maturing: discovery
-          and screening are v2 and being calibrated, and 7 of 18 goldens have
+          and screening are v2 and being calibrated, and {completedInLatestSweep} of {totalYamls || "?"} goldens have
           data at this commit. Current focus: lifting faithfulness and golden
           coverage.
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/evals/page.tsx` around lines 120 - 130, The paragraph in
app/evals/page.tsx currently contains hardcoded KPIs (18, 97%, 74%, 38%, 7 of
18); replace those literals with the computed values from the component/page
state/props (e.g., use the existing metrics variables such as totalGoldens,
precisionPct, recallPct, faithfulnessPct, and goldensWithData or their actual
names in this component) so the displayed copy reflects the live totals shown
elsewhere on the page; update the rendering in the component that emits that
paragraph to interpolate those state values (format percentages and "X of Y"
counts consistently) rather than embedding fixed numbers.

</p>
</section>

{/* EDITORIAL HEADER ────────────────────────────────────────────── */}
<header className="mb-14 max-w-3xl">
<p className="eyebrow">Public Eval Dashboard</p>
Expand Down