diff --git a/site/index.html b/site/index.html index 80e3b1f..8fcb08a 100644 --- a/site/index.html +++ b/site/index.html @@ -31,7 +31,6 @@ RAG Bench Essential
- Benchmark Results Evaluation GitHub @@ -45,10 +44,10 @@
Open evaluation package · Fixed 15
-

Measure whether an agent can finish the whole job.

+

Fifteen real data tasks, judged by an external hard gate.

- Fifteen production-shaped cases across long documents, spreadsheets, SQLite, cross-source analysis, - evidence binding, and client-ready delivery. Public payloads stay isolated from external truth. + Long documents, spreadsheets, SQLite, cross-source analysis, evidence binding, and client-ready + delivery. The agent never sees the answer key — every verdict is produced outside its workspace.

Explore results → @@ -57,51 +56,12 @@

Measure whether an agent can finish the whole job.

15fixed cases
8published settings
-
4hard-gate layers
100%external scoring
-
-
-
-

THE BENCHMARK

-

Small roster. High diagnostic value.

-
-

- Each case is selected for a concrete capability boundary—not a formatting trick or a toy join. -

-
-
-
- 01 -

Agentic retrieval

-

Find the right evidence in long PDFs, OCR-heavy contracts, large corpora, and mixed business documents.

-
LongDADocVQADCI
-
-
- 02 -

Program semantics

-

Operate on real spreadsheets and SQLite while preserving formulas, schemas, and task-specific logic.

-
Spider2-LitePrepBench
-
-
- 03 -

Evidence binding

-

Connect final claims to the correct source values across documents instead of returning plausible text.

-
DocFinQAMultiHiertt
-
-
- 04 -

Delivery contracts

-

Produce usable workbooks, decks, PDFs, reports, and exact output paths—not partial analysis.

-
BankerToolBenchDV-World
-
-
-
-
-
+

RESULTS

Current Fixed-15 scoreboard

@@ -109,63 +69,90 @@

Current Fixed-15 scoreboard

One retained run per setting. Accuracy is official hard PASS, not partial rubric credit.

-
-
-
AccuracyPASS / 15
-
-
Penguin Manualv0.1.5 · Goal off
-
11 -
-
-
Penguin Manual + Goalv0.1.5
-
11 -
-
-
Penguin Autov0.1.5
-
10 -
-
-
Claude CodeOpus 4.8 · max
-
10 -
-
-
CodexGPT-5.5 · xhigh
-
10 -
-
- - -
+

+ Best current setting — PenguinHarness v0.1.5 + Manual Skill + 11/15 73.3% · 12.38M tokens · $0.1995 · 97.8 min +

- + + + + + + + + + - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SettingVersion & configurationAccWallTokensCost
#SettingVersion & configurationAccWallTokensCost
Penguin w/o Skillv0.0.1 · DeepSeek V4 Flash · Goal off8/15136.5m18.51M$0.2407
Penguin w/o Skillv0.1.5 · DeepSeek V4 Flash · Goal off9/1589.8m17.78M$0.2231
Penguin w/ Auto Skillv0.1.5 · Auto Agent State · Goal off10/1594.4m13.75M$0.2158
Penguin w/ Manual Skillv0.0.1 · Full injection · Goal off9/1597.7m16.44M$0.2250
Penguin w/ Manual Skillv0.1.5 · Full injection · Goal off11/1597.8m12.38M$0.1995
Penguin Manual + Goalv0.1.5 · Full injection · Goal on11/1587.6m17.65M$0.2267
Claude CodeCLI 2.1.191 · Claude Opus 4.8 · max10/15159.1m16.93M$34.27
CodexCLI 0.146.0-alpha.9.2 · GPT-5.5 · xhigh10/15108.9m12.22M$18.94
1Penguin w/ Manual Skillv0.1.5 · DeepSeek V4 Flash xhigh · Full injection · Goal off11/1597.8m12.38M$0.1995
2Penguin Manual + Goalv0.1.5 · DeepSeek V4 Flash xhigh · Full injection · Goal on11/1587.6m17.65M$0.2267
3Penguin w/ Auto Skillv0.1.5 · DeepSeek V4 Flash xhigh · Auto Agent State · Goal off10/1594.4m13.75M$0.2158
4Claude CodeCLI 2.1.191 · Claude Opus 4.8 · effort=max10/15159.1m16.93M$34.27
5CodexCLI 0.146.0-alpha.9.2 · GPT-5.5 xhigh · no skill10/15108.9m12.22M$18.94
6Penguin w/o Skillv0.1.5 · DeepSeek V4 Flash xhigh · no skill · Goal off9/1589.8m17.78M$0.2231
7Penguin w/ Manual Skillv0.0.1 · DeepSeek V4 Flash xhigh · Full injection · Goal off9/1597.7m16.44M$0.2250
8Penguin w/o Skillv0.0.1 · DeepSeek V4 Flash xhigh · no skill · Goal off8/15136.5m18.51M$0.2407

- Evaluator note · Claude Code’s BankerToolBench artifact was regraded after the 2026-08-10 workbook-locator fix, moving from 9/15 to 10/15. The other seven Banker artifacts were unavailable and retain their historical result. + Evaluator note · Claude Code’s BankerToolBench artifact was regraded after the 2026-08-10 + workbook-locator fix, moving from 9/15 to 10/15; the other seven Banker artifacts were unavailable + and retain their historical result. Penguin settings ran with Gemini visual judging enabled, + Claude Code and Codex did not. Cost reflects each harness on the model it is normally paired with, + not a shared one.

@@ -179,39 +166,13 @@

Public task in. External verdict out.

1

Isolated workspace

The agent sees only task.md, data/, and optional env.md.

-
2

External truth

Gold answers, validators, rubrics, and visual prompts remain outside the workspace.

+
2

External validator

Gold answers, validators, rubrics, and visual prompts remain outside the workspace.

3

Hard-gate PASS

Answer, evidence, intermediate semantics, and delivery must all satisfy the case contract.

- -
-
-

FIXED ROSTER

-

15 representative cases

-
-
    -
  1. Spider2-Lite
  2. DCI / BrowseComp+
  3. DocFinQA
  4. DocVQA
  5. LongDA
    6. -
  6. MultiHiertt
  7. WorkspaceBench
  8. DV-World
  9. BankerToolBench
  10. FinLongDocQA
    11. -
  11. DABstep
  12. PrepBench
  13. SpreadsheetBench
  14. HarveyLab
  15. FDABench
    16. -
-
- - -
-
-

RUN YOUR AGENT

-

Reproduce the benchmark from one repository.

-

Clone the payloads, stage an isolated case, then score the final workspace externally.

-
-
-
quick-start
- 
$ pip install -r requirements-eval.txt
-$ python scripts/stage_case.py \
-    --case-id prepbench_loyalty_tier_normalization_hard \
-    --workspace /tmp/rag-bench/prepbench
-$ python scripts/score_case.py \
-    --case-id prepbench_loyalty_tier_normalization_hard \
-    --workspace /tmp/rag-bench/prepbench
-
+

+ Payloads, evaluation packages, and the scoring scripts are in the repository — + Prism-Shadow/rag-bench-essential ↗ +

diff --git a/site/styles.css b/site/styles.css index f4a914f..70ac542 100644 --- a/site/styles.css +++ b/site/styles.css @@ -74,7 +74,7 @@ button { font: inherit; } .eyebrow, .kicker { font: 650 12px/1.2 ui-monospace, SFMono-Regular, Menlo, monospace; letter-spacing: .12em; text-transform: uppercase; color: var(--aqua); } .eyebrow { display: inline-flex; align-items: center; gap: 9px; padding: 8px 13px; border: 1px solid var(--line); border-radius: 999px; background: var(--surface); } .status-dot { width: 7px; height: 7px; border-radius: 50%; background: var(--aqua); box-shadow: 0 0 16px var(--aqua); } -h1 { max-width: 960px; margin: 28px auto 24px; font-size: clamp(48px, 7vw, 84px); line-height: .98; letter-spacing: -.062em; } +h1 { max-width: 960px; margin: 28px auto 24px; font-size: clamp(44px, 6.4vw, 76px); line-height: 1; letter-spacing: -.058em; text-wrap: balance; } h1 span { background: linear-gradient(100deg, var(--aqua), #74afff 58%, #c69af7); -webkit-background-clip: text; background-clip: text; color: transparent; } .hero-copy { max-width: 760px; margin: 0 auto; color: var(--muted); font-size: clamp(17px, 2vw, 20px); } .hero-actions { display: flex; justify-content: center; gap: 12px; margin-top: 34px; } @@ -82,7 +82,7 @@ h1 span { background: linear-gradient(100deg, var(--aqua), #74afff 58%, #c69af7) .button:hover { transform: translateY(-2px); } .button-primary { color: #06151a; background: var(--aqua); border-color: transparent; box-shadow: 0 10px 34px color-mix(in srgb, var(--aqua) 28%, transparent); } .button-secondary { background: var(--surface); } -.metric-strip { margin-top: 76px; display: grid; grid-template-columns: repeat(4, 1fr); border: 1px solid var(--line); border-radius: 18px; background: var(--surface); box-shadow: var(--shadow); overflow: hidden; } +.metric-strip { margin-top: 72px; display: grid; grid-template-columns: repeat(3, 1fr); border: 1px solid var(--line); border-radius: 18px; background: var(--surface); box-shadow: var(--shadow); overflow: hidden; } .metric-strip div { padding: 26px; border-right: 1px solid var(--line); } .metric-strip div:last-child { border-right: 0; } .metric-strip strong { display: block; font-size: 27px; letter-spacing: -.04em; } @@ -90,49 +90,43 @@ h1 span { background: linear-gradient(100deg, var(--aqua), #74afff 58%, #c69af7) .section { padding: 94px 0; } .section-heading { display: grid; grid-template-columns: 1.25fr .75fr; gap: 60px; align-items: end; margin-bottom: 42px; } -.section-heading h2, .cta h2 { margin: 9px 0 0; font-size: clamp(34px, 4.5vw, 52px); line-height: 1.04; letter-spacing: -.045em; } +.section-heading h2 { margin: 9px 0 0; font-size: clamp(34px, 4.5vw, 52px); line-height: 1.04; letter-spacing: -.045em; } .section-heading > p { color: var(--muted); margin: 0; } -.capability-grid { display: grid; grid-template-columns: repeat(2, 1fr); gap: 16px; } -.capability-card, .card-surface, .flow-grid article { border: 1px solid var(--line); background: var(--surface); border-radius: 18px; box-shadow: var(--shadow); } -.capability-card { min-height: 260px; padding: 28px; position: relative; overflow: hidden; } -.capability-card::after { content: ""; position: absolute; width: 180px; height: 180px; border-radius: 50%; background: var(--aqua); filter: blur(80px); opacity: .04; right: -50px; bottom: -70px; } -.capability-card h3 { margin: 28px 0 10px; font-size: 23px; } -.capability-card p { max-width: 520px; margin: 0; color: var(--muted); } -.card-index { font: 600 12px ui-monospace, monospace; color: var(--aqua); } -.mini-tags { display: flex; flex-wrap: wrap; gap: 7px; margin-top: 26px; } -.mini-tags span { padding: 5px 9px; border-radius: 7px; border: 1px solid var(--line); color: var(--muted); font: 11px ui-monospace, monospace; } - -.scoreboard-layout { display: grid; grid-template-columns: minmax(0, 1.45fr) minmax(300px, .55fr); gap: 16px; } -.ranking, .result-callout { padding: 26px; } -.surface-head { display: flex; justify-content: space-between; color: var(--muted); font: 11px ui-monospace, monospace; text-transform: uppercase; letter-spacing: .12em; margin-bottom: 20px; } -.rank-row { display: grid; grid-template-columns: 200px 1fr 28px; gap: 14px; align-items: center; padding: 12px 0; } -.rank-label strong, .rank-label span { display: block; } -.rank-label strong { font-size: 14px; } -.rank-label span { color: var(--muted); font-size: 11px; } -.rank-track { height: 8px; border-radius: 99px; background: var(--bg-soft); overflow: hidden; } -.rank-track i { display: block; width: var(--score); height: 100%; border-radius: inherit; background: linear-gradient(90deg, var(--blue), var(--aqua)); } -.rank-row.top .rank-track i { background: linear-gradient(90deg, var(--aqua), #8cf3dc); box-shadow: 0 0 18px color-mix(in srgb, var(--aqua) 35%, transparent); } -.rank-row b { font: 700 15px ui-monospace, monospace; } -.result-callout { background: linear-gradient(150deg, color-mix(in srgb, var(--aqua) 10%, var(--surface-solid)), var(--surface-solid)); } -.big-score { display: block; margin-top: 14px; font-size: 64px; line-height: 1; letter-spacing: -.07em; color: var(--aqua); } -.big-score span { font-size: 24px; color: var(--muted); } -.result-callout h3 { margin: 22px 0 5px; font-size: 20px; line-height: 1.25; } -.result-callout > p:not(.kicker) { color: var(--muted); font-size: 13px; } -.result-callout dl { margin: 24px 0 0; display: grid; grid-template-columns: repeat(3, 1fr); gap: 8px; } -.result-callout dl div { border-top: 1px solid var(--line); padding-top: 12px; } -.result-callout dt { color: var(--muted); font-size: 10px; text-transform: uppercase; } -.result-callout dd { margin: 2px 0 0; font: 650 13px ui-monospace, monospace; } - -.table-wrap { margin-top: 16px; overflow-x: auto; } -table { width: 100%; border-collapse: collapse; min-width: 900px; font-size: 13px; } +.card-surface { border: 1px solid var(--line); background: var(--surface); border-radius: 18px; box-shadow: var(--shadow); } +.flow-grid article { border: 1px solid var(--line); background: var(--surface); border-radius: 18px; box-shadow: var(--shadow); } + +/* Summary line above the scoreboard: the headline number, without a second card. */ +.lead-line { + margin: 0 0 16px; padding: 14px 18px; + border: 1px solid var(--line); border-left: 2px solid var(--aqua); border-radius: 12px; + background: color-mix(in srgb, var(--aqua) 6%, var(--surface)); + color: var(--muted); font-size: 14px; + display: flex; flex-wrap: wrap; align-items: baseline; gap: 6px 12px; +} +.lead-line strong { color: var(--text); } +.lead-line b { font: 700 19px ui-monospace, SFMono-Regular, Menlo, monospace; color: var(--aqua); letter-spacing: -.03em; } +.lead-line span { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; } + +.table-wrap { overflow-x: auto; } +.table-wrap:focus-visible { outline: 2px solid var(--aqua); outline-offset: 3px; } +table { width: 100%; border-collapse: collapse; min-width: 940px; font-size: 13px; } th, td { padding: 15px 18px; text-align: left; border-bottom: 1px solid var(--line); } th { color: var(--muted); font: 600 10px ui-monospace, monospace; letter-spacing: .1em; text-transform: uppercase; } -td:nth-child(n+3) { font-family: ui-monospace, monospace; } +td:nth-child(n+4) { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-variant-numeric: tabular-nums; } tbody tr:last-child td { border-bottom: 0; } tbody tr:hover { background: var(--surface-hover); } tbody tr.highlight { background: color-mix(in srgb, var(--aqua) 7%, transparent); } -tbody tr.highlight td:first-child, tbody tr.highlight td:nth-child(3) { color: var(--aqua); font-weight: 700; } -.result-note { margin: 16px 4px 0; color: var(--muted); font-size: 12px; } +tbody tr.highlight td:nth-child(2) { color: var(--aqua); font-weight: 700; } + +.col-rank { width: 46px; color: var(--muted); font: 600 12px ui-monospace, monospace; font-variant-numeric: tabular-nums; } +.col-acc { white-space: nowrap; } +.acc { display: inline-flex; align-items: center; gap: 11px; font-weight: 650; } +.acc i { position: relative; flex: none; width: 62px; height: 6px; border-radius: 99px; background: var(--bg-soft); overflow: hidden; } +.acc i::after { content: ""; position: absolute; inset: 0 auto 0 0; width: var(--score); border-radius: inherit; background: linear-gradient(90deg, var(--blue), var(--aqua)); } +tbody tr.highlight .acc { color: var(--aqua); } +tbody tr.highlight .acc i::after { background: linear-gradient(90deg, var(--aqua), #8cf3dc); box-shadow: 0 0 14px color-mix(in srgb, var(--aqua) 40%, transparent); } + +.result-note { margin: 16px 4px 0; color: var(--muted); font-size: 12px; max-width: 92ch; } .flow-grid { display: grid; grid-template-columns: repeat(3, 1fr); gap: 16px; } .flow-grid article { padding: 28px; } @@ -140,20 +134,10 @@ tbody tr.highlight td:first-child, tbody tr.highlight td:nth-child(3) { color: v .flow-grid h3 { margin: 44px 0 8px; font-size: 20px; } .flow-grid p { color: var(--muted); margin: 0; font-size: 14px; } code { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; } -.case-roster { margin-top: 16px; padding: 30px; display: grid; grid-template-columns: 260px 1fr; gap: 34px; } -.case-roster h3 { margin: 8px 0 0; font-size: 24px; } -.case-roster ol { margin: 0; padding: 0; list-style: none; display: grid; grid-template-columns: repeat(3, 1fr); gap: 9px; counter-reset: cases; } -.case-roster li { counter-increment: cases; padding: 10px 12px; border: 1px solid var(--line); border-radius: 9px; color: var(--muted); font-size: 12px; } -.case-roster li::before { content: counter(cases, decimal-leading-zero) " · "; color: var(--aqua); font-family: ui-monospace, monospace; } - -.cta { margin-top: 34px; margin-bottom: 100px; padding: 58px; display: grid; grid-template-columns: .85fr 1.15fr; gap: 64px; align-items: center; border: 1px solid var(--line); border-radius: 24px; background: linear-gradient(140deg, color-mix(in srgb, var(--blue) 8%, var(--surface-solid)), color-mix(in srgb, var(--aqua) 5%, var(--surface-solid))); box-shadow: var(--shadow); } -.cta > div > p:last-child { color: var(--muted); } -.terminal { border: 1px solid var(--line); border-radius: 14px; background: #06101c; overflow: hidden; box-shadow: 0 22px 50px rgba(0,0,0,.24); } -.terminal-bar { height: 38px; display: flex; align-items: center; gap: 7px; padding: 0 13px; border-bottom: 1px solid rgba(255,255,255,.08); color: #71869f; font: 10px ui-monospace, monospace; } -.terminal-bar i { width: 8px; height: 8px; border-radius: 50%; background: #405166; } -.terminal-bar span { margin-left: auto; } -pre { margin: 0; padding: 20px; overflow-x: auto; color: #bfd0e2; font: 11px/1.75 ui-monospace, SFMono-Regular, Menlo, monospace; } -pre span { color: #57e7c4; } + +.section-outro { margin: 26px 4px 0; color: var(--muted); font-size: 14px; } +.section-outro a { color: var(--aqua); border-bottom: 1px solid color-mix(in srgb, var(--aqua) 35%, transparent); } +.section-outro a:hover { border-bottom-color: var(--aqua); } .footer { min-height: 120px; display: flex; justify-content: space-between; align-items: center; border-top: 1px solid var(--line); color: var(--muted); font-size: 13px; } .footer .brand { color: var(--text); } @@ -161,28 +145,18 @@ pre span { color: #57e7c4; } @media (max-width: 900px) { .nav-links a { display: none; } - .metric-strip { grid-template-columns: repeat(2, 1fr); } - .metric-strip div:nth-child(2) { border-right: 0; } - .metric-strip div:nth-child(-n+2) { border-bottom: 1px solid var(--line); } - .section-heading, .scoreboard-layout, .cta { grid-template-columns: 1fr; } - .section-heading { gap: 18px; } - .capability-grid { grid-template-columns: 1fr; } + .section-heading { grid-template-columns: 1fr; gap: 18px; } .flow-grid { grid-template-columns: 1fr; } - .case-roster { grid-template-columns: 1fr; } - .case-roster ol { grid-template-columns: repeat(2, 1fr); } - .cta { padding: 34px; gap: 32px; } } @media (max-width: 620px) { .shell { width: min(100% - 24px, 1180px); } .hero { padding-top: 82px; } .hero-actions { flex-direction: column; } - .metric-strip div { padding: 20px 14px; } + .metric-strip { grid-template-columns: 1fr; } + .metric-strip div { padding: 20px 14px; border-right: 0; border-bottom: 1px solid var(--line); } + .metric-strip div:last-child { border-bottom: 0; } .section { padding: 68px 0; } - .rank-row { grid-template-columns: 145px 1fr 24px; gap: 9px; } - .result-callout dl { grid-template-columns: 1fr; } - .case-roster ol { grid-template-columns: 1fr; } - .cta { width: 100%; border-radius: 0; padding-inline: 20px; } .footer { padding: 30px 0; flex-direction: column; align-items: flex-start; gap: 12px; } }