Skip to content

refactor(site): condense homepage to hero, scoreboard, evaluation #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hiyouga wants to merge 1 commit into
base: main
Choose a base branch
from
Open

refactor(site): condense homepage to hero, scoreboard, evaluation #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
205 changes: 83 additions & 122 deletions site/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@
<span>RAG Bench Essential</span>
</a>
<div class="nav-links">
<a href="#benchmark">Benchmark</a>
<a href="#results">Results</a>
<a href="#evaluation">Evaluation</a>
<a href="https://github.com/Prism-Shadow/rag-bench-essential">GitHub</a>
Expand All @@ -45,10 +44,10 @@
<main id="top">
<section class="hero shell">
<div class="eyebrow"><span class="status-dot"></span> Open evaluation package · Fixed 15</div>
<h1>Measure whether an agent can finish the <span>whole job.</span></h1>
<h1>Fifteen real data tasks, judged by an <span>external hard gate.</span></h1>
<p class="hero-copy">
Fifteen production-shaped cases across long documents, spreadsheets, SQLite, cross-source analysis,
evidence binding, and client-ready delivery. Public payloads stay isolated from external truth.
Long documents, spreadsheets, SQLite, cross-source analysis, evidence binding, and client-ready
delivery. The agent never sees the answer key — every verdict is produced outside its workspace.
</p>
<div class="hero-actions">
<a class="button button-primary" href="#results">Explore results <span>→</span></a>
Expand All @@ -57,115 +56,103 @@ <h1>Measure whether an agent can finish the <span>whole job.</span></h1>
<div class="metric-strip" aria-label="Benchmark summary">
<div><strong>15</strong><span>fixed cases</span></div>
<div><strong>8</strong><span>published settings</span></div>
<div><strong>4</strong><span>hard-gate layers</span></div>
<div><strong>100%</strong><span>external scoring</span></div>
</div>
</section>

<section id="benchmark" class="section shell">
<div class="section-heading">
<div>
<p class="kicker">THE BENCHMARK</p>
<h2>Small roster. High diagnostic value.</h2>
</div>
<p>
Each case is selected for a concrete capability boundary—not a formatting trick or a toy join.
</p>
</div>
<div class="capability-grid">
<article class="capability-card feature-card">
<span class="card-index">01</span>
<h3>Agentic retrieval</h3>
<p>Find the right evidence in long PDFs, OCR-heavy contracts, large corpora, and mixed business documents.</p>
<div class="mini-tags"><span>LongDA</span><span>DocVQA</span><span>DCI</span></div>
</article>
<article class="capability-card">
<span class="card-index">02</span>
<h3>Program semantics</h3>
<p>Operate on real spreadsheets and SQLite while preserving formulas, schemas, and task-specific logic.</p>
<div class="mini-tags"><span>Spider2-Lite</span><span>PrepBench</span></div>
</article>
<article class="capability-card">
<span class="card-index">03</span>
<h3>Evidence binding</h3>
<p>Connect final claims to the correct source values across documents instead of returning plausible text.</p>
<div class="mini-tags"><span>DocFinQA</span><span>MultiHiertt</span></div>
</article>
<article class="capability-card">
<span class="card-index">04</span>
<h3>Delivery contracts</h3>
<p>Produce usable workbooks, decks, PDFs, reports, and exact output paths—not partial analysis.</p>
<div class="mini-tags"><span>BankerToolBench</span><span>DV-World</span></div>
</article>
</div>
</section>

<section id="results" class="section shell">
<div class="section-heading results-heading">
<div class="section-heading">
<div>
<p class="kicker">RESULTS</p>
<h2>Current Fixed-15 scoreboard</h2>
</div>
<p>One retained run per setting. Accuracy is official hard PASS, not partial rubric credit.</p>
</div>

<div class="scoreboard-layout">
<div class="ranking card-surface">
<div class="surface-head"><span>Accuracy</span><span>PASS / 15</span></div>
<div class="rank-row top">
<div class="rank-label"><strong>Penguin Manual</strong><span>v0.1.5 · Goal off</span></div>
<div class="rank-track"><i style="--score: 73.3%"></i></div><b>11</b>
</div>
<div class="rank-row top">
<div class="rank-label"><strong>Penguin Manual + Goal</strong><span>v0.1.5</span></div>
<div class="rank-track"><i style="--score: 73.3%"></i></div><b>11</b>
</div>
<div class="rank-row">
<div class="rank-label"><strong>Penguin Auto</strong><span>v0.1.5</span></div>
<div class="rank-track"><i style="--score: 66.7%"></i></div><b>10</b>
</div>
<div class="rank-row">
<div class="rank-label"><strong>Claude Code</strong><span>Opus 4.8 · max</span></div>
<div class="rank-track"><i style="--score: 66.7%"></i></div><b>10</b>
</div>
<div class="rank-row">
<div class="rank-label"><strong>Codex</strong><span>GPT-5.5 · xhigh</span></div>
<div class="rank-track"><i style="--score: 66.7%"></i></div><b>10</b>
</div>
</div>

<aside class="result-callout card-surface">
<p class="kicker">BEST CURRENT DEFAULT</p>
<strong class="big-score">11<span>/15</span></strong>
<h3>PenguinHarness v0.1.5 + Manual Skill</h3>
<p>Goal off · DeepSeek V4 Flash xhigh · full Skill injection</p>
<dl>
<div><dt>Tokens</dt><dd>12.38M</dd></div>
<div><dt>Cost</dt><dd>$0.1995</dd></div>
<div><dt>Wall total</dt><dd>97.8 min</dd></div>
</dl>
</aside>
</div>
<p class="lead-line">
Best current setting — <strong>PenguinHarness v0.1.5 + Manual Skill</strong>
<b>11/15</b> <span>73.3% · 12.38M tokens · $0.1995 · 97.8 min</span>
</p>

<div class="table-wrap card-surface" tabindex="0" aria-label="Complete benchmark result table">
<table>
<thead>
<tr><th>Setting</th><th>Version & configuration</th><th>Acc</th><th>Wall</th><th>Tokens</th><th>Cost</th></tr>
<tr>
<th class="col-rank">#</th>
<th>Setting</th>
<th>Version &amp; configuration</th>
<th class="col-acc">Acc</th>
<th>Wall</th>
<th>Tokens</th>
<th>Cost</th>
</tr>
</thead>
<tbody>
<tr><td>Penguin w/o Skill</td><td>v0.0.1 · DeepSeek V4 Flash · Goal off</td><td>8/15</td><td>136.5m</td><td>18.51M</td><td>$0.2407</td></tr>
<tr><td>Penguin w/o Skill</td><td>v0.1.5 · DeepSeek V4 Flash · Goal off</td><td>9/15</td><td>89.8m</td><td>17.78M</td><td>$0.2231</td></tr>
<tr><td>Penguin w/ Auto Skill</td><td>v0.1.5 · Auto Agent State · Goal off</td><td>10/15</td><td>94.4m</td><td>13.75M</td><td>$0.2158</td></tr>
<tr><td>Penguin w/ Manual Skill</td><td>v0.0.1 · Full injection · Goal off</td><td>9/15</td><td>97.7m</td><td>16.44M</td><td>$0.2250</td></tr>
<tr class="highlight"><td>Penguin w/ Manual Skill</td><td>v0.1.5 · Full injection · Goal off</td><td>11/15</td><td>97.8m</td><td>12.38M</td><td>$0.1995</td></tr>
<tr><td>Penguin Manual + Goal</td><td>v0.1.5 · Full injection · Goal on</td><td>11/15</td><td>87.6m</td><td>17.65M</td><td>$0.2267</td></tr>
<tr><td>Claude Code</td><td>CLI 2.1.191 · Claude Opus 4.8 · max</td><td>10/15</td><td>159.1m</td><td>16.93M</td><td>$34.27</td></tr>
<tr><td>Codex</td><td>CLI 0.146.0-alpha.9.2 · GPT-5.5 · xhigh</td><td>10/15</td><td>108.9m</td><td>12.22M</td><td>$18.94</td></tr>
<tr class="highlight">
<td class="col-rank">1</td>
<td>Penguin w/ Manual Skill</td>
<td>v0.1.5 · DeepSeek V4 Flash xhigh · Full injection · Goal off</td>
<td class="col-acc"><span class="acc"><i style="--score: 73.3%"></i>11/15</span></td>
<td>97.8m</td><td>12.38M</td><td>$0.1995</td>
</tr>
<tr>
<td class="col-rank">2</td>
<td>Penguin Manual + Goal</td>
<td>v0.1.5 · DeepSeek V4 Flash xhigh · Full injection · Goal on</td>
<td class="col-acc"><span class="acc"><i style="--score: 73.3%"></i>11/15</span></td>
<td>87.6m</td><td>17.65M</td><td>$0.2267</td>
</tr>
<tr>
<td class="col-rank">3</td>
<td>Penguin w/ Auto Skill</td>
<td>v0.1.5 · DeepSeek V4 Flash xhigh · Auto Agent State · Goal off</td>
<td class="col-acc"><span class="acc"><i style="--score: 66.7%"></i>10/15</span></td>
<td>94.4m</td><td>13.75M</td><td>$0.2158</td>
</tr>
<tr>
<td class="col-rank">4</td>
<td>Claude Code</td>
<td>CLI 2.1.191 · Claude Opus 4.8 · effort=max</td>
<td class="col-acc"><span class="acc"><i style="--score: 66.7%"></i>10/15</span></td>
<td>159.1m</td><td>16.93M</td><td>$34.27</td>
</tr>
<tr>
<td class="col-rank">5</td>
<td>Codex</td>
<td>CLI 0.146.0-alpha.9.2 · GPT-5.5 xhigh · no skill</td>
<td class="col-acc"><span class="acc"><i style="--score: 66.7%"></i>10/15</span></td>
<td>108.9m</td><td>12.22M</td><td>$18.94</td>
</tr>
<tr>
<td class="col-rank">6</td>
<td>Penguin w/o Skill</td>
<td>v0.1.5 · DeepSeek V4 Flash xhigh · no skill · Goal off</td>
<td class="col-acc"><span class="acc"><i style="--score: 60%"></i>9/15</span></td>
<td>89.8m</td><td>17.78M</td><td>$0.2231</td>
</tr>
<tr>
<td class="col-rank">7</td>
<td>Penguin w/ Manual Skill</td>
<td>v0.0.1 · DeepSeek V4 Flash xhigh · Full injection · Goal off</td>
<td class="col-acc"><span class="acc"><i style="--score: 60%"></i>9/15</span></td>
<td>97.7m</td><td>16.44M</td><td>$0.2250</td>
</tr>
<tr>
<td class="col-rank">8</td>
<td>Penguin w/o Skill</td>
<td>v0.0.1 · DeepSeek V4 Flash xhigh · no skill · Goal off</td>
<td class="col-acc"><span class="acc"><i style="--score: 53.3%"></i>8/15</span></td>
<td>136.5m</td><td>18.51M</td><td>$0.2407</td>
</tr>
</tbody>
</table>
</div>
<p class="result-note">
Evaluator note · Claude Code’s BankerToolBench artifact was regraded after the 2026-08-10 workbook-locator fix, moving from 9/15 to 10/15. The other seven Banker artifacts were unavailable and retain their historical result.
Evaluator note · Claude Code’s BankerToolBench artifact was regraded after the 2026-08-10
workbook-locator fix, moving from 9/15 to 10/15; the other seven Banker artifacts were unavailable
and retain their historical result. Penguin settings ran with Gemini visual judging enabled,
Claude Code and Codex did not. Cost reflects each harness on the model it is normally paired with,
not a shared one.
</p>
</section>

Expand All @@ -179,39 +166,13 @@ <h2>Public task in. External verdict out.</h2>
</div>
<div class="flow-grid">
<article><span>1</span><h3>Isolated workspace</h3><p>The agent sees only <code>task.md</code>, <code>data/</code>, and optional <code>env.md</code>.</p></article>
<article><span>2</span><h3>External truth</h3><p>Gold answers, validators, rubrics, and visual prompts remain outside the workspace.</p></article>
<article><span>2</span><h3>External validator</h3><p>Gold answers, validators, rubrics, and visual prompts remain outside the workspace.</p></article>
<article><span>3</span><h3>Hard-gate PASS</h3><p>Answer, evidence, intermediate semantics, and delivery must all satisfy the case contract.</p></article>
</div>

<div class="case-roster card-surface">
<div>
<p class="kicker">FIXED ROSTER</p>
<h3>15 representative cases</h3>
</div>
<ol>
<li>Spider2-Lite</li><li>DCI / BrowseComp+</li><li>DocFinQA</li><li>DocVQA</li><li>LongDA</li>
<li>MultiHiertt</li><li>WorkspaceBench</li><li>DV-World</li><li>BankerToolBench</li><li>FinLongDocQA</li>
<li>DABstep</li><li>PrepBench</li><li>SpreadsheetBench</li><li>HarveyLab</li><li>FDABench</li>
</ol>
</div>
</section>

<section class="cta shell">
<div>
<p class="kicker">RUN YOUR AGENT</p>
<h2>Reproduce the benchmark from one repository.</h2>
<p>Clone the payloads, stage an isolated case, then score the final workspace externally.</p>
</div>
<div class="terminal" aria-label="Quick start commands">
<div class="terminal-bar"><i></i><i></i><i></i><span>quick-start</span></div>
<pre><code><span>$</span> pip install -r requirements-eval.txt
<span>$</span> python scripts/stage_case.py \
--case-id prepbench_loyalty_tier_normalization_hard \
--workspace /tmp/rag-bench/prepbench
<span>$</span> python scripts/score_case.py \
--case-id prepbench_loyalty_tier_normalization_hard \
--workspace /tmp/rag-bench/prepbench</code></pre>
</div>
<p class="section-outro">
Payloads, evaluation packages, and the scoring scripts are in the repository —
<a href="https://github.com/Prism-Shadow/rag-bench-essential">Prism-Shadow/rag-bench-essential ↗</a>
</p>
</section>
</main>

Expand Down
Loading