From 5ce4fa87859ba7b8261da6f88715f4ad0960923c Mon Sep 17 00:00:00 2001 From: Alexander Putilin Date: Thu, 16 Jul 2026 22:05:47 +0100 Subject: [PATCH 1/7] Add triviaqa audit post draft --- blog/posts/triviaqa/_header.html | 12 + blog/posts/triviaqa/index.html | 685 +++++++++++++++++++++++++++++++ blog/posts/triviaqa/index.qmd | 142 +++++++ 3 files changed, 839 insertions(+) create mode 100644 blog/posts/triviaqa/_header.html create mode 100644 blog/posts/triviaqa/index.html create mode 100644 blog/posts/triviaqa/index.qmd diff --git a/blog/posts/triviaqa/_header.html b/blog/posts/triviaqa/_header.html new file mode 100644 index 0000000..184346f --- /dev/null +++ b/blog/posts/triviaqa/_header.html @@ -0,0 +1,12 @@ +
+

Draft · July 2026

+

TriviaQA Audit Report

+

+ TriviaQA is one of the classic QA benchmarks. We discuss its history, how it + is scored, and whether it has any signal left at the frontier. +

+
+ + +
diff --git a/blog/posts/triviaqa/index.html b/blog/posts/triviaqa/index.html new file mode 100644 index 0000000..d86fc8f --- /dev/null +++ b/blog/posts/triviaqa/index.html @@ -0,0 +1,685 @@ + + + + + + + + + +TriviaQA Audit Report - Generality Labs + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + + + + Back to blog + +
+

Draft · July 2026

+

TriviaQA Audit Report

+

+ TriviaQA is one of the classic QA benchmarks. We discuss its history, how it + is scored, and whether it has any signal left at the frontier. +

+
+ + +
+ + + + +
+
+
+
+
+ +
+
+
+
+
+
+ +
+
+
+
+
+
+ +
+
+
+
+

Summary of findings

+
    +
  • Saturated. The dashboard max is 87.6%. Opus 4.8 scores 89% (on a subset of 500 questions). We estimate the realistic ceiling of this dataset at ≈92%.
    • +
  • Contaminated by design. Questions are scraped from trivia websites.
    • +
  • Displays a mild inverted-U relationship with ECI and compute on the Epoch dashboard.
    • +
+

What is TriviaQA?

+

TriviaQA is introduced in the 2017 paper “TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension”:

+
+

TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average.

+
+

It was originally intended as a “reading comprehension dataset”. For each question–answer pair it provides evidence documents which are likely (but not guaranteed) to contain the answer to the question. The documents were gathered automatically from Wikipedia and web search results. The dataset was originally intended for ML models far weaker than current AIs — the original paper evaluates an RNN on evidence docs, which scores 40%, far below the 80% human baseline.

+

In recent years the dataset has been used in a different mode — closed book. The model isn’t provided with evidence docs, nor is it allowed to use web search. It answers using only internal knowledge. All of Epoch’s external runs for this dataset are done in this mode (manually verified by checking each paper).

+

That said, some papers still incorrectly describe this dataset as “reading comprehension” despite clearly using it in the closed-book mode, for example:

+
    +
  • Gemma: Open Models Based on Gemini Research and Technology
    • +
  • Claude 2 Model Card
    • +
  • Releasing Claude Instant 1.2
    • +
+

Splits: 76.5k train / 9.9k validation / 9.5k test. The answers for the test split are not provided.

+

Example items

+
+

In what year was the best foreign film category introduced to the Academy Awards?

+

1948 — alias: “one thousand, nine hundred and forty-eight”

+
+
+

What is the name of the strip of land between a golf tee and the green?

+

Fairway — aliases: “Fairway”, “Fair-way”, “Fair Way”, “Fairway (disambiguation)” [sic]

+
+
+

Which locks control entry to the Panama canal from the Atlantic end?

+

Gatun Lock — aliases: “Gatun lock”, “Gatun Locks”, “Panama Canal Locks”, “Pedro Miguel Locks”, “Pedro Miguel lock”, “Panama Canal locks”, “Gatún lock”

+
+

Construct validity

+

While the original paper describes the dataset as “reading comprehension”, in the closed-book mode it becomes a general QA dataset — there is nothing for the model to read, aside from the question text.

+

What is the dataset measuring then? Perhaps “breadth of factual trivia-like knowledge (prior to 2017) and the ability to retrieve it”. It’s hard to pinpoint the construct more exactly — the authors don’t balance the dataset in any way.

+

From looking at the individual items, the distribution is skewed toward US/UK pop culture, sports, and Western history, and it’s English-only — it’s unclear if the score would generalise well to multilingual or non-Western factual knowledge.

+

For humans this closed-book version would measure some forms of crystallised intelligence. Knowledge is hard-won for humans, so the ability to quickly retrieve factual knowledge can be used as a proxy for general intelligence. However, for an AI, knowledge arises as a byproduct of compressing the training data, and thus the case for it correlating with forms of crystallised intelligence is weaker.

+

Outcome validity (scoring)

+

The dataset provides a correct answer (with aliases) for the train/validation subsets. There is an official scoring script. Papers from which Epoch takes their data score via the “exact match” (“EM”) method. It compares model output to the gold truth (and its aliases) after applying normalisation. The score is accuracy: the percentage of matches, where a match for an item means a match with any of the aliases.

+

The normalisation applies four transformations to both prediction and reference: lowercase everything, remove punctuation, remove the English articles “a”, “an”, “the”, and collapse extra whitespace. Then it requires the resulting strings to be character-for-character identical.

+

There is no way to abstain from giving an answer — wrong answers are scored as 0, the same as no answer. The drawback of this dataset is that it doesn’t measure anything related to epistemic calibration.

+

Saturation

+

Saturated for several years:

+
    +
  • The Epoch dashboard page (with “Frontier trend only” enabled)
    • +
  • LLM stats lists some newer models
    • +
+

Scores for the current frontier models are not provided on either leaderboard.

+

The original paper lists 79.7% as the human performance level (with the evidence docs). There are multiple top scores in the 80+% range, with the max being 87.6%.

+

However, these do not come from the current frontier models. To evaluate whether there is any room for realistic improvement, we ran Opus 4.8 on 500 samples and then used an LLM judge (also Opus 4.8, but with web search) to classify the reasons for the model giving a wrong answer.

+

Opus 4.8 correctly answered 445 out of 500 questions (89%). Most of the wrong answers are inexact matches — the model giving a correct answer that doesn’t match any of the provided aliases for the “gold truth”, for example:

+
+

Which World War 2 American general was known as ‘Vinegar Joe’?

+

Model answer: Joseph Stilwell. Gold truth: “Stilwell” (aliases: “Stillwell”, “Stilwell (disambiguation)”)

+
+

There are occasional outdated questions, for example:

+
+

Who is currently the Supreme Governor of the Church of England?

+

Model answer: King Charles III. Gold truth: “Queen Elizabeth II” (correct at the time of collection, but she’s dead now)

+
+
+

In geology, Greywacke is classed as what form of sedimentary rock?

+

Model answer: Sandstone. Gold truth: “Deep Ocean” (incorrect — Wikipedia lists “Sandstone”)

+
+

Some questions had missing information, for example: “In which town in Greater Manchester is the TV series set?” (no TV series name — maybe the question was supposed to have a picture?) or “Measuring from the closest point of each of these countries to the equator, which is the furthest from the equator?” (no list of countries).

+

There are only 17 / 500 questions where Opus 4.8 was cleanly and clearly wrong — 3.4%. Verdict: the benchmark is basically saturated; the actual realistic ceiling is ≈92% (0.89 + 0.034).

+

Contamination

+

Contaminated by design. It originally contains scraped trivia questions: “First we gathered question-answer pairs from 14 trivia and quiz-league websites”. Even if the canonical version of the dataset is filtered out of the training data, these trivia question sets are unlikely to be removed.

+

From “A Comprehensive Survey of Contamination Detection Methods in Large Language Models”:

+
+

[…] the Llama pre-training corpus and The Pile (Gao et al., 2020) are found to be contaminated for many popular evaluation benchmarks, notably BigBench, HellaSwag, PiQA, MMLU and TriviaQA

+
+

From “Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data”:

+
+

Among the four tasks [TriviaQA, MMLU, WMT, and GSM8K], our analysis reveals that TriviaQA exhibits the strongest memorization effect, with task performance highly correlated to the n-gram distributions in the pretraining data.

+
+

Additionally, “Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets” highlights significant test–train overlap.

+

ECI vs TriviaQA and compute vs TriviaQA

+

TriviaQA seems to display a mild inverted-U relationship between compute (or ECI) and score on this benchmark. See the charts below — the shape of the data starts to bend downwards. The best models on this task aren’t the ones with the highest compute budget or the most capable.

+ +

Compare the TriviaQA dashboard to three randomly picked benchmarks: MATH Level 5, HellaSwag, BBH.

+ +

Hypotheses for the inverted U-shape

+
    +
  1. Saturation. Measurements near the ceiling are noisy — they don’t allow us to separate models by skill from each other.
    2. +
  2. Factual knowledge is not a good proxy for general capabilities in models. Models have a parameter budget, and you can “spend” this budget either to “buy” yourself better factual knowledge or general capabilities. Labs prefer the latter.
    3. +
  3. There aren’t a lot of measurements in general: only 29 if you don’t count duplicates (n-shot for different n).
    4. +
+ +
+
+
+
+ +
+
+ © 2026 Generality Labs +
+
+ + + + + + + + \ No newline at end of file diff --git a/blog/posts/triviaqa/index.qmd b/blog/posts/triviaqa/index.qmd new file mode 100644 index 0000000..62b0c9d --- /dev/null +++ b/blog/posts/triviaqa/index.qmd @@ -0,0 +1,142 @@ +--- +pagetitle: "TriviaQA Audit Report - Generality Labs" +format: + html: + page-layout: custom + theme: none + include-in-header: ../../../_includes/head.html + include-before-body: + - ../../../_includes/post-before.html + - _header.html + include-after-body: ../../../_includes/post-after.html + link-external-newwindow: true + embed-resources: false +--- + +```{ojs} +//| echo: false +glVersion = "dev" +gl = import(`/assets/gl-components/gl.js?v=${glVersion}`) +mc = import(`/assets/gl-components/model-colors.js?v=${glVersion}`) +``` + +## Summary of findings + +- **Saturated.** The dashboard max is 87.6%. Opus 4.8 scores 89% (on a subset of 500 questions). We estimate the realistic ceiling of this dataset at ≈92%. +- **Contaminated by design.** Questions are scraped from trivia websites. +- Displays a **mild inverted-U relationship** with ECI and compute on the Epoch dashboard. + +## What is TriviaQA? + +TriviaQA is introduced in the 2017 paper ["TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension"](https://arxiv.org/abs/1705.03551): + +> TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average. + +It was originally intended as a "reading comprehension dataset". For each question–answer pair it provides evidence documents which are likely (but not guaranteed) to contain the answer to the question. The documents were gathered automatically from Wikipedia and web search results. The dataset was originally intended for ML models far weaker than current AIs — the original paper evaluates an RNN on evidence docs, which scores 40%, far below the 80% human baseline. + +In recent years the dataset has been used in a different mode — **closed book**. The model isn't provided with evidence docs, nor is it allowed to use web search. It answers using only internal knowledge. All of Epoch's external runs for this dataset are done in this mode (manually verified by checking each paper). + +That said, some papers still incorrectly describe this dataset as "reading comprehension" despite clearly using it in the closed-book mode, for example: + +- Gemma: Open Models Based on Gemini Research and Technology +- Claude 2 Model Card +- Releasing Claude Instant 1.2 + +Splits: 76.5k train / 9.9k validation / 9.5k test. The answers for the test split are not provided. + +### Example items + +> In what year was the best foreign film category introduced to the Academy Awards? +> +> **1948** — alias: "one thousand, nine hundred and forty-eight" + +> What is the name of the strip of land between a golf tee and the green? +> +> **Fairway** — aliases: "Fairway", "Fair-way", "Fair Way", "Fairway (disambiguation)" [sic] + +> Which locks control entry to the Panama canal from the Atlantic end? +> +> **Gatun Lock** — aliases: "Gatun lock", "Gatun Locks", "Panama Canal Locks", "Pedro Miguel Locks", "Pedro Miguel lock", "Panama Canal locks", "Gatún lock" + +## Construct validity + +While the original paper describes the dataset as "reading comprehension", in the closed-book mode it becomes a general QA dataset — there is nothing for the model to read, aside from the question text. + +What is the dataset measuring then? Perhaps "breadth of factual trivia-like knowledge (prior to 2017) and the ability to retrieve it". It's hard to pinpoint the construct more exactly — the authors don't balance the dataset in any way. + +From looking at the individual items, the distribution is skewed toward US/UK pop culture, sports, and Western history, and it's English-only — it's unclear if the score would generalise well to multilingual or non-Western factual knowledge. + +For humans this closed-book version would measure some forms of crystallised intelligence. Knowledge is hard-won for humans, so the ability to quickly retrieve factual knowledge can be used as a proxy for general intelligence. However, for an AI, knowledge arises as a byproduct of compressing the training data, and thus the case for it correlating with forms of crystallised intelligence is weaker. + +## Outcome validity (scoring) + +The dataset provides a correct answer (with aliases) for the train/validation subsets. There is an official scoring script. Papers from which Epoch takes their data score via the "exact match" ("EM") method. It compares model output to the gold truth (and its aliases) after applying normalisation. The score is accuracy: the percentage of matches, where a match for an item means a match with any of the aliases. + +The normalisation applies four transformations to both prediction and reference: lowercase everything, remove punctuation, remove the English articles "a", "an", "the", and collapse extra whitespace. Then it requires the resulting strings to be character-for-character identical. + +There is no way to abstain from giving an answer — wrong answers are scored as 0, the same as no answer. The drawback of this dataset is that it doesn't measure anything related to epistemic calibration. + +## Saturation + +Saturated for several years: + +- The Epoch dashboard page (with "Frontier trend only" enabled) +- LLM stats lists some newer models + +Scores for the current frontier models are not provided on either leaderboard. + +The original paper lists 79.7% as the human performance level (with the evidence docs). There are multiple top scores in the 80+% range, with the max being 87.6%. + +However, these do not come from the current frontier models. To evaluate whether there is any room for realistic improvement, we ran Opus 4.8 on 500 samples and then used an LLM judge (also Opus 4.8, but with web search) to classify the reasons for the model giving a wrong answer. + +Opus 4.8 correctly answered 445 out of 500 questions (89%). Most of the wrong answers are inexact matches — the model giving a correct answer that doesn't match any of the provided aliases for the "gold truth", for example: + +> Which World War 2 American general was known as 'Vinegar Joe'? +> +> Model answer: **Joseph Stilwell**. Gold truth: "Stilwell" (aliases: "Stillwell", "Stilwell (disambiguation)") + +There are occasional outdated questions, for example: + +> Who is currently the Supreme Governor of the Church of England? +> +> Model answer: **King Charles III**. Gold truth: "Queen Elizabeth II" (correct at the time of collection, but she's dead now) + +> In geology, Greywacke is classed as what form of sedimentary rock? +> +> Model answer: **Sandstone**. Gold truth: "Deep Ocean" (incorrect — Wikipedia lists "Sandstone") + +Some questions had missing information, for example: "In which town in Greater Manchester is the TV series set?" (no TV series name — maybe the question was supposed to have a picture?) or "Measuring from the closest point of each of these countries to the equator, which is the furthest from the equator?" (no list of countries). + +There are only 17 / 500 questions where Opus 4.8 was cleanly and clearly wrong — 3.4%. Verdict: the benchmark is basically saturated; the actual realistic ceiling is ≈92% (0.89 + 0.034). + +## Contamination + +Contaminated by design. It originally contains scraped trivia questions: "First we gathered question-answer pairs from 14 trivia and quiz-league websites". Even if the canonical version of the dataset is filtered out of the training data, these trivia question sets are unlikely to be removed. + +From "A Comprehensive Survey of Contamination Detection Methods in Large Language Models": + +> [...] the Llama pre-training corpus and The Pile (Gao et al., 2020) are found to be contaminated for many popular evaluation benchmarks, notably BigBench, HellaSwag, PiQA, MMLU and TriviaQA + +From "Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data": + +> Among the four tasks [TriviaQA, MMLU, WMT, and GSM8K], our analysis reveals that TriviaQA exhibits the strongest memorization effect, with task performance highly correlated to the n-gram distributions in the pretraining data. + +Additionally, ["Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets"](https://arxiv.org/abs/2008.02637) highlights significant test–train overlap. + +## ECI vs TriviaQA and compute vs TriviaQA + +TriviaQA seems to display a mild inverted-U relationship between compute (or ECI) and score on this benchmark. See the charts below — the shape of the data starts to bend downwards. The best models on this task aren't the ones with the highest compute budget or the most capable. + + + +Compare the TriviaQA dashboard to three randomly picked benchmarks: MATH Level 5, HellaSwag, BBH. + + + +### Hypotheses for the inverted U-shape + +1. **Saturation.** Measurements near the ceiling are noisy — they don't allow us to separate models by skill from each other. +2. **Factual knowledge is not a good proxy for general capabilities in models.** Models have a parameter budget, and you can "spend" this budget either to "buy" yourself better factual knowledge or general capabilities. Labs prefer the latter. +3. **There aren't a lot of measurements in general:** only 29 if you don't count duplicates (n-shot for different n). From aa94a61288f46e11cdfc3f23bd4b543622f9cedf Mon Sep 17 00:00:00 2001 From: Alexander Putilin Date: Fri, 17 Jul 2026 17:11:14 +0100 Subject: [PATCH 2/7] Add .prettierignore --- .prettierignore | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 .prettierignore diff --git a/.prettierignore b/.prettierignore new file mode 100644 index 0000000..cb15471 --- /dev/null +++ b/.prettierignore @@ -0,0 +1,8 @@ +# Quarto include partials are intentionally unbalanced HTML fragments +# (post-before/_header open tags that post-after closes). Formatters +# "fix" the unclosed tags and break the page shell. +_includes/ +blog/posts/*/_header.html +blog/posts/*/_head-extra.html +# rendered output — not source +blog/posts/*/index.html From 52f61c176d5b364f8bf1a817117871dd0ea1740d Mon Sep 17 00:00:00 2001 From: Alexander Putilin Date: Sat, 18 Jul 2026 18:06:26 +0100 Subject: [PATCH 3/7] Slightly edit introduction + link to the pdfs describing triviaqa as a reading comprehension dataset --- blog/posts/triviaqa/index.qmd | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/blog/posts/triviaqa/index.qmd b/blog/posts/triviaqa/index.qmd index 62b0c9d..958a0a7 100644 --- a/blog/posts/triviaqa/index.qmd +++ b/blog/posts/triviaqa/index.qmd @@ -32,15 +32,15 @@ TriviaQA is introduced in the 2017 paper ["TriviaQA: A Large Scale Distantly Sup > TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average. -It was originally intended as a "reading comprehension dataset". For each question–answer pair it provides evidence documents which are likely (but not guaranteed) to contain the answer to the question. The documents were gathered automatically from Wikipedia and web search results. The dataset was originally intended for ML models far weaker than current AIs — the original paper evaluates an RNN on evidence docs, which scores 40%, far below the 80% human baseline. +It was originally intended as a "reading comprehension dataset". For each question–answer pair it provides evidence documents which are likely (but not guaranteed) to contain the answer to the question. The documents were gathered automatically from Wikipedia and web search results. The dataset was intended for ML models far weaker than the current AIs. For example, the paper evaluates an RNN, which scores 40%, far below the 80% human baseline. -In recent years the dataset has been used in a different mode — **closed book**. The model isn't provided with evidence docs, nor is it allowed to use web search. It answers using only internal knowledge. All of Epoch's external runs for this dataset are done in this mode (manually verified by checking each paper). +In recent years the dataset has been used in a different mode — _closed book_. The model isn't provided with evidence docs, nor is it allowed to use web search. It has to answer using only internal knowledge. All of Epoch's external runs for this dataset are done in this mode (manually verified by checking each paper). -That said, some papers still incorrectly describe this dataset as "reading comprehension" despite clearly using it in the closed-book mode, for example: +Despite this change, some papers still incorrectly describe this dataset as "reading comprehension" despite clearly using it in the closed-book mode, for example: -- Gemma: Open Models Based on Gemini Research and Technology -- Claude 2 Model Card -- Releasing Claude Instant 1.2 +- [Gemma: Open Models Based on Gemini Research and Technology](https://arxiv.org/pdf/2403.08295) +- [Claude 2 Model Card](https://www-cdn.anthropic.com/5c49cc247484cecf107c699baf29250302e5da70/claude-2-model-card.pdf) +- [Releasing Claude Instant 1.2](https://www.anthropic.com/news/releasing-claude-instant-1-2) Splits: 76.5k train / 9.9k validation / 9.5k test. The answers for the test split are not provided. From 465de040a1618f4e89c8c7c4713b24c6f40f7c56 Mon Sep 17 00:00:00 2001 From: Alexander Putilin Date: Sat, 18 Jul 2026 18:20:37 +0100 Subject: [PATCH 4/7] Add triviqa card to the blog --- blog/index.html | 24 ++++++++++++++++++++++++ blog/posts/triviaqa/index.html | 12 ++++++------ 2 files changed, 30 insertions(+), 6 deletions(-) diff --git a/blog/index.html b/blog/index.html index 0705518..2c786d2 100644 --- a/blog/index.html +++ b/blog/index.html @@ -294,6 +294,30 @@

Blog

+ + +

July 2026

+

TriviaQA Audit Report

+

+ TriviaQA is one of the classic QA benchmarks. We discuss its history, how it is scored, and whether it has any signal left at the frontier. +

+ + Read more + + + + + +

July 2026 · By James Mann

diff --git a/blog/posts/triviaqa/index.html b/blog/posts/triviaqa/index.html index d86fc8f..82f08a1 100644 --- a/blog/posts/triviaqa/index.html +++ b/blog/posts/triviaqa/index.html @@ -167,13 +167,13 @@

What is TriviaQA?

TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average.

-

It was originally intended as a “reading comprehension dataset”. For each question–answer pair it provides evidence documents which are likely (but not guaranteed) to contain the answer to the question. The documents were gathered automatically from Wikipedia and web search results. The dataset was originally intended for ML models far weaker than current AIs — the original paper evaluates an RNN on evidence docs, which scores 40%, far below the 80% human baseline.

-

In recent years the dataset has been used in a different mode — closed book. The model isn’t provided with evidence docs, nor is it allowed to use web search. It answers using only internal knowledge. All of Epoch’s external runs for this dataset are done in this mode (manually verified by checking each paper).

-

That said, some papers still incorrectly describe this dataset as “reading comprehension” despite clearly using it in the closed-book mode, for example:

+

It was originally intended as a “reading comprehension dataset”. For each question–answer pair it provides evidence documents which are likely (but not guaranteed) to contain the answer to the question. The documents were gathered automatically from Wikipedia and web search results. The dataset was intended for ML models far weaker than the current AIs. For example, the paper evaluates an RNN, which scores 40%, far below the 80% human baseline.

+

In recent years the dataset has been used in a different mode — closed book. The model isn’t provided with evidence docs, nor is it allowed to use web search. It has to answer using only internal knowledge. All of Epoch’s external runs for this dataset are done in this mode (manually verified by checking each paper).

+

Despite this change, some papers still incorrectly describe this dataset as “reading comprehension” despite clearly using it in the closed-book mode, for example:

Splits: 76.5k train / 9.9k validation / 9.5k test. The answers for the test split are not provided.

Example items

From 34b3aeea159af0d7b67c6d35ac7ff35bc93664c1 Mon Sep 17 00:00:00 2001 From: Alexander Putilin Date: Mon, 20 Jul 2026 20:16:14 +0100 Subject: [PATCH 5/7] Add component for displaying example items --- assets/gl-components/gl.js | 1309 ++++++++++++++++++++++++++------ blog/posts/triviaqa/index.html | 36 +- blog/posts/triviaqa/index.qmd | 29 +- 3 files changed, 1114 insertions(+), 260 deletions(-) diff --git a/assets/gl-components/gl.js b/assets/gl-components/gl.js index f6fa085..2035425 100644 --- a/assets/gl-components/gl.js +++ b/assets/gl-components/gl.js @@ -15,13 +15,25 @@ // the helpers below so each component states only what is unique to it. export const palette = { - ink: "#1a1919", muted: "#5b5858", soft: "#837878", hairline: "#e6e6e6", - bg: "#fafafa", paper: "#f2f2f2", - mint: "#17cfb9", mintSoft: "#ecf8f5", mintEdge: "#c5e8e1", mintDeep: "#24857a", + ink: "#1a1919", + muted: "#5b5858", + soft: "#837878", + hairline: "#e6e6e6", + bg: "#fafafa", + paper: "#f2f2f2", + mint: "#17cfb9", + mintSoft: "#ecf8f5", + mintEdge: "#c5e8e1", + mintDeep: "#24857a", series: ["#24857a", "#d97757", "#7c4dbe", "#3a5fa8", "#eda100", "#d1495b"], - good: "#1baf7a", warn: "#eda100", warnInk: "#b07d00", - bad: "#d1495b", badSoft: "#faeceb", - neutral: "#7a93ad", neutralInk: "#5a748c", neutralSoft: "#edf1f5", + good: "#1baf7a", + warn: "#eda100", + warnInk: "#b07d00", + bad: "#d1495b", + badSoft: "#faeceb", + neutral: "#7a93ad", + neutralInk: "#5a748c", + neutralSoft: "#edf1f5", cleanFill: "#9fc3b4", }; @@ -30,7 +42,10 @@ const SVG_NS = "http://www.w3.org/2000/svg"; /* ---------------------------------------------------------------- helpers */ -const esc = (s) => String(s ?? "").replace(/&/g, "&").replace(/ + String(s ?? "") + .replace(/&/g, "&") + .replace(/ - ({ background: "transparent", fontFamily: FONT, fontSize }); +const plotStyle = (fontSize = "13px") => ({ + background: "transparent", + fontFamily: FONT, + fontSize, +}); /** A centred x-axis config with enough offset to clear two-line tick labels. */ -const centredAxis = (label, labelOffset, extra = {}) => - ({ label, labelAnchor: "center", labelOffset, ...extra }); +const centredAxis = (label, labelOffset, extra = {}) => ({ + label, + labelAnchor: "center", + labelOffset, + ...extra, +}); /** An HTML x-axis label rendered below the plot (used where Plot's own label would collide with tick labels). */ -const labelBelow = (text, css) => el("div", - `text-align:center;font-family:${FONT};margin-top:.2rem;${css}`, esc(text)); +const labelBelow = (text, css) => + el( + "div", + `text-align:center;font-family:${FONT};margin-top:.2rem;${css}`, + esc(text), + ); /** Colour-swatch legend row rendered below a plot (house style: the model family legend sits under the graph, never above it). */ function legendRow(domain, range) { - return el("div", + return el( + "div", `display:flex;flex-wrap:wrap;justify-content:center;gap:.35rem 1.1rem;` + - `font-family:${FONT};font-size:12px;color:${palette.muted};margin-top:.4rem;`, - domain.map((name, i) => `` + - `${esc(name)}`).join("")); + `font-family:${FONT};font-size:12px;color:${palette.muted};margin-top:.4rem;`, + domain + .map( + (name, i) => + `` + + `${esc(name)}`, + ) + .join(""), + ); } /* -------------------------------------------------------------- glStatBar */ @@ -105,7 +141,9 @@ function legendRow(domain, range) { // items: [{ label, value, detail? }] export function glStatBar(_, items, spec = {}) { const { caption, provenance } = spec; - injectOnce("gl-statbar-css", ` + injectOnce( + "gl-statbar-css", + ` .gl-statbar { display:flex; flex-wrap:wrap; border-bottom:2px solid ${palette.hairline}; font-family:${FONT}; } @@ -121,15 +159,22 @@ export function glStatBar(_, items, spec = {}) { margin-top:.05rem; } .gl-statbar .detail { font-size:10.5px; color:${palette.muted}; margin-top:.05rem; line-height:1.3; } - `); + `, + ); const bar = el("div", null); bar.className = "gl-statbar"; for (const it of items) { - bar.appendChild(el("div", null, ` + bar.appendChild( + el( + "div", + null, + `
${esc(it.label)}
${esc(it.value)}
${it.detail ? `
${esc(it.detail)}
` : ""} - `)).className = "gl-stat"; + `, + ), + ).className = "gl-stat"; } return frame(bar, { caption, provenance }); } @@ -144,10 +189,22 @@ export function glStatBar(_, items, spec = {}) { // out to the latest x to make stagnation legible. export function glScatter({ Plot }, rows, spec = {}) { const { - x = "x", y = "y", label = "label", group = "group", - xLabel = "", yLabel = "", caption, provenance, - width = 700, height = 460, xFmt = (v) => v, yFmt = (v) => v, - vintage, dateKey = "date", lo, hi, + x = "x", + y = "y", + label = "label", + group = "group", + xLabel = "", + yLabel = "", + caption, + provenance, + width = 700, + height = 460, + xFmt = (v) => v, + yFmt = (v) => v, + vintage, + dateKey = "date", + lo, + hi, } = spec; const cut = vintage ? new Date(vintage) : null; const isNew = (d) => cut && new Date(d[dateKey]) > cut; @@ -161,49 +218,119 @@ export function glScatter({ Plot }, rows, spec = {}) { const steps = []; let best = -Infinity; for (const d of sorted) { - if (d[y] > best) { best = d[y]; onFrontier.add(d); steps.push({ _fx: d[x], _fy: best }); } + if (d[y] > best) { + best = d[y]; + onFrontier.add(d); + steps.push({ _fx: d[x], _fy: best }); + } } steps.push({ _fx: sorted[sorted.length - 1][x], _fy: best }); // hold flat to latest release - marks.push(Plot.line(steps, { x: "_fx", y: "_fy", curve: "step-after", - stroke: palette.soft, strokeWidth: 1.3, strokeDasharray: "2,4" })); + marks.push( + Plot.line(steps, { + x: "_fx", + y: "_fy", + curve: "step-after", + stroke: palette.soft, + strokeWidth: 1.3, + strokeDasharray: "2,4", + }), + ); } // with a frontier drawn, the field fades slightly so the frontier carries the story const dotAlpha = (d) => (!spec.frontier || onFrontier.has(d) ? 0.9 : 0.45); const ciAlpha = (d) => (!spec.frontier || onFrontier.has(d) ? 0.5 : 0.28); if (lo && hi && rows.some((d) => d[lo] != null)) { - marks.push(Plot.ruleX(rows, { x, y1: lo, y2: hi, stroke: group, strokeWidth: 1.4, strokeOpacity: ciAlpha })); + marks.push( + Plot.ruleX(rows, { + x, + y1: lo, + y2: hi, + stroke: group, + strokeWidth: 1.4, + strokeOpacity: ciAlpha, + }), + ); // CI end caps: short horizontal segments at lo and hi. const xs = rows.map((d) => +d[x]); const capW = (Math.max(...xs) - Math.min(...xs)) * 0.006 || 1; - const caps = rows.flatMap((d) => [ - { ...d, _cy: d[lo], _x1: +d[x] - capW, _x2: +d[x] + capW, _a: ciAlpha(d) }, - { ...d, _cy: d[hi], _x1: +d[x] - capW, _x2: +d[x] + capW, _a: ciAlpha(d) }, - ]).filter((d) => d._cy != null); - marks.push(Plot.ruleY(caps, { y: "_cy", x1: (d) => new Date(d._x1), x2: (d) => new Date(d._x2), - stroke: group, strokeWidth: 1.4, strokeOpacity: (d) => d._a })); + const caps = rows + .flatMap((d) => [ + { + ...d, + _cy: d[lo], + _x1: +d[x] - capW, + _x2: +d[x] + capW, + _a: ciAlpha(d), + }, + { + ...d, + _cy: d[hi], + _x1: +d[x] - capW, + _x2: +d[x] + capW, + _a: ciAlpha(d), + }, + ]) + .filter((d) => d._cy != null); + marks.push( + Plot.ruleY(caps, { + y: "_cy", + x1: (d) => new Date(d._x1), + x2: (d) => new Date(d._x2), + stroke: group, + strokeWidth: 1.4, + strokeOpacity: (d) => d._a, + }), + ); } - marks.push(Plot.dot(before, { x, y, fill: group, r: 5.5, fillOpacity: dotAlpha })); + marks.push( + Plot.dot(before, { x, y, fill: group, r: 5.5, fillOpacity: dotAlpha }), + ); if (after.length) marks.push( - Plot.dot(after, { x, y, stroke: palette.mint, strokeWidth: 2.5, r: 6.5, fill: palette.bg }), + Plot.dot(after, { + x, + y, + stroke: palette.mint, + strokeWidth: 2.5, + r: 6.5, + fill: palette.bg, + }), Plot.dot(after, { x, y, fill: palette.mint, r: 3 }), ); - marks.push(Plot.tip(rows, Plot.pointer({ x, y, maxRadius: 24, title: (d) => - `${d[label]}${isNew(d) ? " (added since publication)" : ""}\n${xFmt(d[x])} · ${yFmt(d[y])}` }))); + marks.push( + Plot.tip( + rows, + Plot.pointer({ + x, + y, + maxRadius: 24, + title: (d) => + `${d[label]}${isNew(d) ? " (added since publication)" : ""}\n${xFmt(d[x])} · ${yFmt(d[y])}`, + }), + ), + ); const node = Plot.plot({ - width, height, marginBottom: 58, + width, + height, + marginBottom: 58, style: plotStyle(), x: centredAxis(xLabel || null, 52, { grid: false, domain: spec.xDomain }), y: { label: yLabel || null, grid: true, domain: spec.yDomain }, - color: { legend: !spec.legendBelow, domain: spec.colorDomain, range: spec.colorRange ?? palette.series }, + color: { + legend: !spec.legendBelow, + domain: spec.colorDomain, + range: spec.colorRange ?? palette.series, + }, marks, }); let out = node; if (spec.legendBelow && spec.colorDomain) { out = el("div"); out.appendChild(node); - out.appendChild(legendRow(spec.colorDomain, spec.colorRange ?? palette.series)); + out.appendChild( + legendRow(spec.colorDomain, spec.colorRange ?? palette.series), + ); } const prov = cut ? `${provenance || ""} · ● as published ${vintage} · ◉ added since (live-refreshing)` @@ -216,30 +343,56 @@ export function glScatter({ Plot }, rows, spec = {}) { // rows: {value, series}; spec.thresholds may be a count or bin-edge array. export function glHistogram({ Plot }, rows, spec = {}) { const { - value = "value", series = "series", - xLabel = "", caption, provenance, - width = 688, height = 470, thresholds = 18, + value = "value", + series = "series", + xLabel = "", + caption, + provenance, + width = 688, + height = 470, + thresholds = 18, } = spec; const byValues = {}; - for (const r of rows) (byValues[r[series]] = byValues[r[series]] || []).push(r[value]); - const meanRows = Object.entries(byValues).map(([k, vs]) => - ({ [series]: k, m: vs.reduce((a, b) => a + b, 0) / vs.length })); + for (const r of rows) + (byValues[r[series]] = byValues[r[series]] || []).push(r[value]); + const meanRows = Object.entries(byValues).map(([k, vs]) => ({ + [series]: k, + m: vs.reduce((a, b) => a + b, 0) / vs.length, + })); const node = Plot.plot({ - width, height, + width, + height, style: plotStyle(), x: { label: null, grid: false }, y: { label: null, grid: true, insetTop: 16 }, - color: { legend: true, domain: spec.colorDomain, range: spec.colorRange ?? [palette.series[0], palette.series[1]] }, + color: { + legend: true, + domain: spec.colorDomain, + range: spec.colorRange ?? [palette.series[0], palette.series[1]], + }, marks: [ // y2 (not y) so the two distributions overlay instead of stacking. - Plot.rectY(rows, Plot.binX({ y2: "count" }, { x: value, fill: series, thresholds, fillOpacity: 0.5, inset: 0.5 })), - Plot.ruleX(meanRows, { x: "m", stroke: series, strokeWidth: 2.6, strokeDasharray: "5,3" }), + Plot.rectY( + rows, + Plot.binX( + { y2: "count" }, + { x: value, fill: series, thresholds, fillOpacity: 0.5, inset: 0.5 }, + ), + ), + Plot.ruleX(meanRows, { + x: "m", + stroke: series, + strokeWidth: 2.6, + strokeDasharray: "5,3", + }), ], }); const wrap = el("div"); wrap.appendChild(node); - wrap.appendChild(labelBelow(xLabel, `font-size:.95rem;color:${palette.muted};`)); + wrap.appendChild( + labelBelow(xLabel, `font-size:.95rem;color:${palette.muted};`), + ); return frame(wrap, { caption, provenance }); } @@ -250,29 +403,59 @@ export function glHistogram({ Plot }, rows, spec = {}) { // tooltip's value line. export function glBars({ Plot }, rows, spec = {}) { const { - value = "value", label = "label", group = "group", - xLabel = "", caption, provenance, width = 688, - valueFmt = (v) => v, sort = true, + value = "value", + label = "label", + group = "group", + xLabel = "", + caption, + provenance, + width = 688, + valueFmt = (v) => v, + sort = true, } = spec; const data = sort ? rows.slice().sort((a, b) => b[value] - a[value]) : rows; const height = Math.max(160, data.length * 16 + 80); const node = Plot.plot({ - width, height, marginLeft: 210, marginBottom: 46, + width, + height, + marginLeft: 210, + marginBottom: 46, style: plotStyle("12px"), x: centredAxis(xLabel, 38, { grid: true }), y: { label: null, domain: data.map((d) => d[label]) }, - color: { legend: false, domain: spec.colorDomain, range: spec.colorRange ?? palette.series }, + color: { + legend: false, + domain: spec.colorDomain, + range: spec.colorRange ?? palette.series, + }, marks: [ - Plot.barX(data, { y: label, x: value, fill: group, rx: 2, insetTop: 1.5, insetBottom: 1.5 }), + Plot.barX(data, { + y: label, + x: value, + fill: group, + rx: 2, + insetTop: 1.5, + insetBottom: 1.5, + }), // pointerY: hovering anywhere along a bar's row triggers that bar's tip. - Plot.tip(data, Plot.pointerY({ y: label, x: value, title: (d) => - `${d[label]}\n${spec.tipName || xLabel || value}: ${valueFmt(d[value])}` })), + Plot.tip( + data, + Plot.pointerY({ + y: label, + x: value, + title: (d) => + `${d[label]}\n${spec.tipName || xLabel || value}: ${valueFmt(d[value])}`, + }), + ), ], }); const wrap = el("div"); wrap.appendChild(node); - if (spec.colorDomain) wrap.appendChild(legendRow(spec.colorDomain, spec.colorRange ?? palette.series)); + if (spec.colorDomain) + wrap.appendChild( + legendRow(spec.colorDomain, spec.colorRange ?? palette.series), + ); return frame(wrap, { caption, provenance }); } @@ -284,19 +467,38 @@ export function glSolveDist({ Plot }, rows, spec = {}) { const { caption, provenance, width = 688, height = 340 } = spec; const nModels = rows[0]?.n_models ?? 49; const counts = new Map(); - for (const r of rows) counts.set(r.n_solved, (counts.get(r.n_solved) || 0) + 1); - const bins = Array.from({ length: nModels + 1 }, (_, k) => ({ k, n: counts.get(k) || 0 })); + for (const r of rows) + counts.set(r.n_solved, (counts.get(r.n_solved) || 0) + 1); + const bins = Array.from({ length: nModels + 1 }, (_, k) => ({ + k, + n: counts.get(k) || 0, + })); const node = Plot.plot({ - width, height, marginBottom: 48, + width, + height, + marginBottom: 48, style: plotStyle(), x: centredAxis(`models solving the question (of ${nModels})`, 40), y: { label: "questions", grid: true, insetTop: 14 }, marks: [ - Plot.rectY(bins, { x1: (d) => d.k - 0.45, x2: (d) => d.k + 0.45, y: "n", - fill: (d) => (d.k === 0 ? palette.bad : palette.mintDeep), fillOpacity: 0.85, rx: 1.5 }), - Plot.tip(bins, Plot.pointerX({ x: "k", y: "n", title: (d) => - `${d.n} questions solved by exactly ${d.k} model${d.k === 1 ? "" : "s"}` })), + Plot.rectY(bins, { + x1: (d) => d.k - 0.45, + x2: (d) => d.k + 0.45, + y: "n", + fill: (d) => (d.k === 0 ? palette.bad : palette.mintDeep), + fillOpacity: 0.85, + rx: 1.5, + }), + Plot.tip( + bins, + Plot.pointerX({ + x: "k", + y: "n", + title: (d) => + `${d.n} questions solved by exactly ${d.k} model${d.k === 1 ? "" : "s"}`, + }), + ), ], }); return frame(node, { caption, provenance }); @@ -311,44 +513,112 @@ export function glSolveDist({ Plot }, rows, spec = {}) { // rows: {label, hex, nat_score, for_score, nat_se, for_se, nat_abst, for_abst} export function glForcingPairs({ Plot }, rows, spec = {}) { const { caption, provenance, width = 688 } = spec; - const order = rows.slice() - .sort((a, b) => (b.for_score - b.nat_score) - (a.for_score - a.nat_score)) + const order = rows + .slice() + .sort((a, b) => b.for_score - b.nat_score - (a.for_score - a.nat_score)) .map((d) => d.label); const panel = (aKey, bKey, seA, seB, yLabel, withCI) => { const flat = rows.flatMap((d) => [ - { label: d.label, arm: "natural", v: d[aKey], se: seA ? d[seA] : null, hex: d.hex }, - { label: d.label, arm: "forced", v: d[bKey], se: seB ? d[seB] : null, hex: d.hex }, + { + label: d.label, + arm: "natural", + v: d[aKey], + se: seA ? d[seA] : null, + hex: d.hex, + }, + { + label: d.label, + arm: "forced", + v: d[bKey], + se: seB ? d[seB] : null, + hex: d.hex, + }, ]); - const tips = rows.map((d) => ({ label: d.label, a: d[aKey], b: d[bKey], - top: Math.max(d[aKey], d[bKey]) })); + const tips = rows.map((d) => ({ + label: d.label, + a: d[aKey], + b: d[bKey], + top: Math.max(d[aKey], d[bKey]), + })); return Plot.plot({ - width, height: 330, marginBottom: 64, marginLeft: 56, + width, + height: 330, + marginBottom: 64, + marginLeft: 56, style: plotStyle("12.5px"), - fx: { label: null, domain: order, tickRotate: -28, padding: 0.25, paddingOuter: 0.5, align: 0.6 }, + fx: { + label: null, + domain: order, + tickRotate: -28, + padding: 0.25, + paddingOuter: 0.5, + align: 0.6, + }, x: { axis: null, domain: ["natural", "forced"], padding: 0.12 }, y: { label: yLabel, grid: true, insetTop: 14 }, marks: [ - Plot.barY(flat, { fx: "label", x: "arm", y: "v", rx: 2, - fill: (d) => d.hex, fillOpacity: (d) => (d.arm === "natural" ? 0.32 : 0.92) }), - ...(withCI ? [ - Plot.ruleX(flat.filter((d) => d.se), { fx: "label", x: "arm", - y1: (d) => d.v - 1.96 * d.se, y2: (d) => d.v + 1.96 * d.se, - stroke: palette.ink, strokeWidth: 1.3 }), - ] : []), + Plot.barY(flat, { + fx: "label", + x: "arm", + y: "v", + rx: 2, + fill: (d) => d.hex, + fillOpacity: (d) => (d.arm === "natural" ? 0.32 : 0.92), + }), + ...(withCI + ? [ + Plot.ruleX( + flat.filter((d) => d.se), + { + fx: "label", + x: "arm", + y1: (d) => d.v - 1.96 * d.se, + y2: (d) => d.v + 1.96 * d.se, + stroke: palette.ink, + strokeWidth: 1.3, + }, + ), + ] + : []), // one tip per model, anchored between the two bars (dx = half a band) - Plot.tip(tips, Plot.pointerX({ fx: "label", x: () => "natural", y: "top", dx: 15, title: (d) => - `${d.label}\nnatural: ${(100 * d.a).toFixed(1)}%\nforced: ${(100 * d.b).toFixed(1)}%` })), + Plot.tip( + tips, + Plot.pointerX({ + fx: "label", + x: () => "natural", + y: "top", + dx: 15, + title: (d) => + `${d.label}\nnatural: ${(100 * d.a).toFixed(1)}%\nforced: ${(100 * d.b).toFixed(1)}%`, + }), + ), ], }); }; const wrap = el("div"); - wrap.appendChild(el("div", `font-size:.8rem;color:${palette.muted};margin-bottom:.25rem;font-family:${FONT};`, - "pale = natural  ·  solid = forced answer")); - wrap.appendChild(panel("nat_score", "for_score", "nat_se", "for_se", "headline accuracy", true)); + wrap.appendChild( + el( + "div", + `font-size:.8rem;color:${palette.muted};margin-bottom:.25rem;font-family:${FONT};`, + "pale = natural  ·  solid = forced answer", + ), + ); + wrap.appendChild( + panel( + "nat_score", + "for_score", + "nat_se", + "for_se", + "headline accuracy", + true, + ), + ); wrap.appendChild(el("div", "height:14px;")); - wrap.appendChild(panel("nat_abst", "for_abst", null, null, "abstention rate", false)); + wrap.appendChild( + panel("nat_abst", "for_abst", null, null, "abstention rate", false), + ); return frame(wrap, { caption, provenance }); } @@ -360,55 +630,120 @@ export function glForcingPairs({ Plot }, rows, spec = {}) { // rows: {x: Date, label, group, } export function glTripletScatter({ Plot }, rows, spec = {}) { const { - x = "x", label = "label", group = "group", metrics = [], - xLabel = "", caption, provenance, - width = 688, height = 480, valueFmt = (v) => v.toFixed(3), + x = "x", + label = "label", + group = "group", + metrics = [], + xLabel = "", + caption, + provenance, + width = 688, + height = 480, + valueFmt = (v) => v.toFixed(3), } = spec; - const JITTER_DAYS = 1.3, DAY_MS = 86400000; - const jmap = new Map(rows.map((r) => { - const h = [...String(r[label])].reduce((a, c) => a + c.charCodeAt(0), 0); - return [r[label], new Date(+r[x] + ((h % 13) - 6) * JITTER_DAYS * DAY_MS)]; - })); + const JITTER_DAYS = 1.3, + DAY_MS = 86400000; + const jmap = new Map( + rows.map((r) => { + const h = [...String(r[label])].reduce((a, c) => a + c.charCodeAt(0), 0); + return [ + r[label], + new Date(+r[x] + ((h % 13) - 6) * JITTER_DAYS * DAY_MS), + ]; + }), + ); const jx = (d) => jmap.get(d[label]); const flat = []; - for (const r of rows) for (const m of metrics) - if (r[m] != null) flat.push({ ...r, metric: m, value: r[m] }); + for (const r of rows) + for (const m of metrics) + if (r[m] != null) flat.push({ ...r, metric: m, value: r[m] }); const heads = flat.filter((d) => d.metric === metrics[0]); const others = flat.filter((d) => d.metric !== metrics[0]); - const extent = (d, fn) => fn(...metrics.map((m) => d[m]).filter((v) => v != null)); + const extent = (d, fn) => + fn(...metrics.map((m) => d[m]).filter((v) => v != null)); const node = Plot.plot({ - width, height, marginBottom: 46, + width, + height, + marginBottom: 46, style: plotStyle(), x: { label: null, grid: false, domain: spec.xDomain }, y: { label: null, grid: true, domain: spec.yDomain }, - color: { legend: false, domain: spec.colorDomain, range: spec.colorRange ?? palette.series }, - symbol: { legend: false, domain: metrics, range: ["circle", "diamond2", "times"] }, + color: { + legend: false, + domain: spec.colorDomain, + range: spec.colorRange ?? palette.series, + }, + symbol: { + legend: false, + domain: metrics, + range: ["circle", "diamond2", "times"], + }, marks: [ - Plot.ruleX(rows, { x: jx, y1: (d) => extent(d, Math.min), y2: (d) => extent(d, Math.max), - stroke: group, strokeWidth: 1.1, strokeOpacity: 0.35 }), - Plot.dot(heads, { x: jx, y: "value", fill: group, symbol: "circle", r: 4.5 }), - Plot.dot(others, { x: jx, y: "value", stroke: group, symbol: "metric", r: 4.5, strokeWidth: 1.8 }), - Plot.tip(flat, Plot.pointer({ x: jx, y: "value", title: (d) => - `${d[label]}\n${d.metric}: ${valueFmt(d.value)}` })), + Plot.ruleX(rows, { + x: jx, + y1: (d) => extent(d, Math.min), + y2: (d) => extent(d, Math.max), + stroke: group, + strokeWidth: 1.1, + strokeOpacity: 0.35, + }), + Plot.dot(heads, { + x: jx, + y: "value", + fill: group, + symbol: "circle", + r: 4.5, + }), + Plot.dot(others, { + x: jx, + y: "value", + stroke: group, + symbol: "metric", + r: 4.5, + strokeWidth: 1.8, + }), + Plot.tip( + flat, + Plot.pointer({ + x: jx, + y: "value", + title: (d) => `${d[label]}\n${d.metric}: ${valueFmt(d.value)}`, + }), + ), ], }); // metric key drawn inside the plot area (colour carries provider, shape carries metric) const GLYPH = ["●", "◇", "✕"]; - const key = el("div", + const key = el( + "div", `position:absolute;top:10px;left:52px;font-family:${FONT};font-size:12px;` + - `color:${palette.muted};display:flex;flex-direction:column;gap:.15rem;pointer-events:none;`, - metrics.map((m, i) => `${GLYPH[i] ?? "•"}${esc(m)}`).join("")); + `color:${palette.muted};display:flex;flex-direction:column;gap:.15rem;pointer-events:none;`, + metrics + .map( + (m, i) => + `${GLYPH[i] ?? "•"}${esc(m)}`, + ) + .join(""), + ); const plotBox = el("div", "position:relative;"); plotBox.appendChild(node); plotBox.appendChild(key); const wrap = el("div"); wrap.appendChild(plotBox); - wrap.appendChild(labelBelow(xLabel, `font-size:.92rem;font-weight:600;color:${palette.ink};margin-top:.15rem;`)); - if (spec.colorDomain) wrap.appendChild(legendRow(spec.colorDomain, spec.colorRange ?? palette.series)); + wrap.appendChild( + labelBelow( + xLabel, + `font-size:.92rem;font-weight:600;color:${palette.ink};margin-top:.15rem;`, + ), + ); + if (spec.colorDomain) + wrap.appendChild( + legendRow(spec.colorDomain, spec.colorRange ?? palette.series), + ); return frame(wrap, { caption, provenance }); } @@ -418,50 +753,90 @@ export function glTripletScatter({ Plot }, rows, spec = {}) { // spec: colors {model: hex}, rMin/rMax (radial domain), legend: false to // suppress the built-in legend (e.g. when two radars share one). export function glRadar(_, data, spec = {}) { - const { caption, provenance, width = 560, rMin = 0.4, rMax = 0.85, colors = {} } = spec; + const { + caption, + provenance, + width = 560, + rMin = 0.4, + rMax = 0.85, + colors = {}, + } = spec; const RING_VALUES = [0.5, 0.6, 0.7, 0.8]; - const LABEL_OVERHANG = 0.11; // axis labels sit just past rMax - const R_MARGIN = 58; // svg edge to polygon edge + const LABEL_OVERHANG = 0.11; // axis labels sit just past rMax + const R_MARGIN = 58; // svg edge to polygon edge const axes = data.axes; const models = [...new Set(data.rows.map((r) => r.model))]; - const size = width, cx = size / 2, cy = size / 2 + 6, R = size / 2 - R_MARGIN; + const size = width, + cx = size / 2, + cy = size / 2 + 6, + R = size / 2 - R_MARGIN; const ang = (i) => (Math.PI * 2 * i) / axes.length - Math.PI / 2; const rr = (v) => R * Math.max(0, Math.min(1, (v - rMin) / (rMax - rMin))); - const pt = (i, v) => [cx + rr(v) * Math.cos(ang(i)), cy + rr(v) * Math.sin(ang(i))]; + const pt = (i, v) => [ + cx + rr(v) * Math.cos(ang(i)), + cy + rr(v) * Math.sin(ang(i)), + ]; const svg = svgEl("svg", { viewBox: `0 0 ${size} ${size + 10}` }); svg.style.cssText = `width:100%;max-width:${size}px;height:auto;font-family:${FONT};display:block;margin:0 auto;`; for (const g of RING_VALUES) { - svg.appendChild(svgEl("polygon", { - points: axes.map((_, i) => pt(i, g).join(",")).join(" "), - fill: "none", stroke: palette.hairline })); + svg.appendChild( + svgEl("polygon", { + points: axes.map((_, i) => pt(i, g).join(",")).join(" "), + fill: "none", + stroke: palette.hairline, + }), + ); const [x, y] = pt(0, g); - const t = svgEl("text", { x: x + 4, y: y + 3, style: `font-size:9.5px;fill:${palette.soft};` }); + const t = svgEl("text", { + x: x + 4, + y: y + 3, + style: `font-size:9.5px;fill:${palette.soft};`, + }); t.textContent = g.toFixed(1); svg.appendChild(t); } axes.forEach((a, i) => { - const [x1, y1] = pt(i, rMin), [x2, y2] = pt(i, rMax); - svg.appendChild(svgEl("line", { x1, y1, x2, y2, stroke: palette.hairline })); + const [x1, y1] = pt(i, rMin), + [x2, y2] = pt(i, rMax); + svg.appendChild( + svgEl("line", { x1, y1, x2, y2, stroke: palette.hairline }), + ); const [lx, ly] = pt(i, rMax + (rMax - rMin) * LABEL_OVERHANG); - const anchor = Math.abs(Math.cos(ang(i))) < 0.35 ? "middle" : (Math.cos(ang(i)) > 0 ? "start" : "end"); - const t = svgEl("text", { x: lx, y: ly + 4, "text-anchor": anchor, - style: `font-size:11px;fill:${palette.muted};` }); + const anchor = + Math.abs(Math.cos(ang(i))) < 0.35 + ? "middle" + : Math.cos(ang(i)) > 0 + ? "start" + : "end"; + const t = svgEl("text", { + x: lx, + y: ly + 4, + "text-anchor": anchor, + style: `font-size:11px;fill:${palette.muted};`, + }); t.textContent = a; svg.appendChild(t); }); const byModel = {}; - for (const r of data.rows) (byModel[r.model] = byModel[r.model] || {})[r.axis] = r; + for (const r of data.rows) + (byModel[r.model] = byModel[r.model] || {})[r.axis] = r; models.forEach((mName, mi) => { const col = colors[mName] || palette.series[mi]; const pts = axes.map((a, i) => pt(i, byModel[mName][a]?.acc ?? rMin)); - svg.appendChild(svgEl("polygon", { - points: pts.map((p) => p.join(",")).join(" "), - fill: col, "fill-opacity": "0.13", - stroke: col, "stroke-width": "2.2", "stroke-linejoin": "round" })); + svg.appendChild( + svgEl("polygon", { + points: pts.map((p) => p.join(",")).join(" "), + fill: col, + "fill-opacity": "0.13", + stroke: col, + "stroke-width": "2.2", + "stroke-linejoin": "round", + }), + ); pts.forEach(([x, y], i) => { const c = svgEl("circle", { cx: x, cy: y, r: 3.6, fill: col }); const r = byModel[mName][axes[i]]; @@ -475,10 +850,18 @@ export function glRadar(_, data, spec = {}) { const wrap = el("div"); wrap.appendChild(svg); if (spec.legend !== false) { - wrap.appendChild(el("div", - `display:flex;gap:1.2rem;justify-content:center;font-size:.85rem;color:${palette.muted};font-family:${FONT};margin-top:.2rem;`, - models.map((mName, mi) => - `● ${mName}`).join(""))); + wrap.appendChild( + el( + "div", + `display:flex;gap:1.2rem;justify-content:center;font-size:.85rem;color:${palette.muted};font-family:${FONT};margin-top:.2rem;`, + models + .map( + (mName, mi) => + `● ${mName}`, + ) + .join(""), + ), + ); } return frame(wrap, { caption, provenance }); } @@ -493,29 +876,73 @@ export function glInfScale({ Plot }, data, spec = {}) { const flat = []; for (const s of data.series) { const model = s.name.replace(/ (serial|parallel).*$/, ""); - for (const p of s.points) flat.push({ ...p, model, arm: s.arm, series: s.name }); + for (const p of s.points) + flat.push({ ...p, model, arm: s.arm, series: s.name }); } const tipTitle = (d) => { - const detail = d.k ? ` (k=${d.k})` - : d.budget != null && d.arm === "serial" ? ` (budget ${d.budget || "off"})` : ""; + const detail = d.k + ? ` (k=${d.k})` + : d.budget != null && d.arm === "serial" + ? ` (budget ${d.budget || "off"})` + : ""; return `${d.model} · ${d.arm}${detail}\n~${d.x} tokens · ${(100 * d.y).toFixed(1)}%`; }; const node = Plot.plot({ - width, height, marginBottom: 48, + width, + height, + marginBottom: 48, style: plotStyle(), - x: centredAxis("median generated tokens per question", 40, - { type: spec.xType ?? "linear", grid: false, domain: spec.xDomain }), + x: centredAxis("median generated tokens per question", 40, { + type: spec.xType ?? "linear", + grid: false, + domain: spec.xDomain, + }), y: { label: "accuracy", grid: true, domain: [0, 0.85] }, - color: { legend: true, domain: Object.keys(colors), range: Object.values(colors) }, - symbol: { legend: true, domain: ["serial", "parallel"], range: ["times", "circle"] }, + color: { + legend: true, + domain: Object.keys(colors), + range: Object.values(colors), + }, + symbol: { + legend: true, + domain: ["serial", "parallel"], + range: ["times", "circle"], + }, marks: [ - Plot.line(flat, { x: "x", y: "y", z: "series", stroke: "model", strokeWidth: 2, - strokeDasharray: (d) => (d.arm === "parallel" ? "5,4" : null) }), - Plot.dot(flat.filter((d) => d.arm === "serial"), - { x: "x", y: "y", stroke: "model", symbol: "times", r: 5.5, strokeWidth: 2.2 }), - Plot.dot(flat.filter((d) => d.arm === "parallel"), - { x: "x", y: "y", stroke: "model", symbol: "circle", r: 5, strokeWidth: 2 }), - Plot.tip(flat, Plot.pointer({ x: "x", y: "y", maxRadius: 26, title: tipTitle })), + Plot.line(flat, { + x: "x", + y: "y", + z: "series", + stroke: "model", + strokeWidth: 2, + strokeDasharray: (d) => (d.arm === "parallel" ? "5,4" : null), + }), + Plot.dot( + flat.filter((d) => d.arm === "serial"), + { + x: "x", + y: "y", + stroke: "model", + symbol: "times", + r: 5.5, + strokeWidth: 2.2, + }, + ), + Plot.dot( + flat.filter((d) => d.arm === "parallel"), + { + x: "x", + y: "y", + stroke: "model", + symbol: "circle", + r: 5, + strokeWidth: 2, + }, + ), + Plot.tip( + flat, + Plot.pointer({ x: "x", y: "y", maxRadius: 26, title: tipTitle }), + ), ], }); return frame(node, { caption }); @@ -526,45 +953,79 @@ export function glInfScale({ Plot }, data, spec = {}) { // in mint, off-diagonal heat in red scaled by row share. // data: {flash: {labels, counts, n_runs}, mini: {...}} export function glConfusion(_, data, spec = {}) { - const { caption, titles = { - flash: "Flash-era runs, regraded by mini", - mini: "mini-era runs, regraded by mini (noise floor)", - } } = spec; + const { + caption, + titles = { + flash: "Flash-era runs, regraded by mini", + mini: "mini-era runs, regraded by mini (noise floor)", + }, + } = spec; const CELL = { w: 74, h: 52 }; - const HEAT_GAIN = 14; // off-diagonal share → red opacity + const HEAT_GAIN = 14; // off-diagonal share → red opacity - const wrap = el("div", `display:flex;gap:2rem;flex-wrap:wrap;justify-content:center;font-family:${FONT};`); + const wrap = el( + "div", + `display:flex;gap:2rem;flex-wrap:wrap;justify-content:center;font-family:${FONT};`, + ); for (const key of ["flash", "mini"]) { const d = data[key]; const G = d.labels; const rowTot = d.counts.map((r) => r.reduce((a, b) => a + b, 0)); const box = el("div"); - box.appendChild(el("div", `font-size:.85rem;color:${palette.muted};margin-bottom:.45rem;text-align:center;`, - esc(titles[key]))); + box.appendChild( + el( + "div", + `font-size:.85rem;color:${palette.muted};margin-bottom:.45rem;text-align:center;`, + esc(titles[key]), + ), + ); const tbl = el("table", "border-collapse:separate;border-spacing:3px;"); - tbl.appendChild(el("tr", null, - `` + G.map((g) => - `→ ${g}`).join(""))); + tbl.appendChild( + el( + "tr", + null, + `` + + G.map( + (g) => + `→ ${g}`, + ).join(""), + ), + ); d.counts.forEach((row, i) => { - const cells = row.map((v, j) => { - const share = rowTot[i] ? v / rowTot[i] : 0; - const diag = i === j; - const bg = diag - ? `rgba(23,207,185,${0.10 + 0.5 * share})` - : `rgba(209,73,91,${Math.min(0.85, share * HEAT_GAIN)})`; - const ink = (!diag && share * HEAT_GAIN > 0.45) ? "#fff" : palette.ink; - return `` + - `
${v.toLocaleString()}
` + - `
${(100 * share).toFixed(1)}%
`; - }).join(""); - tbl.appendChild(el("tr", null, - `${G[i]}` + cells)); + const cells = row + .map((v, j) => { + const share = rowTot[i] ? v / rowTot[i] : 0; + const diag = i === j; + const bg = diag + ? `rgba(23,207,185,${0.1 + 0.5 * share})` + : `rgba(209,73,91,${Math.min(0.85, share * HEAT_GAIN)})`; + const ink = !diag && share * HEAT_GAIN > 0.45 ? "#fff" : palette.ink; + return ( + `` + + `
${v.toLocaleString()}
` + + `
${(100 * share).toFixed(1)}%
` + ); + }) + .join(""); + tbl.appendChild( + el( + "tr", + null, + `${G[i]}` + + cells, + ), + ); }); box.appendChild(tbl); - box.appendChild(el("div", `font-size:.7rem;color:${palette.soft};margin-top:.3rem;text-align:center;`, - `rows: original grade · ${d.n_runs} runs`)); + box.appendChild( + el( + "div", + `font-size:.7rem;color:${palette.soft};margin-top:.3rem;text-align:center;`, + `rows: original grade · ${d.n_runs} runs`, + ), + ); wrap.appendChild(box); } return frame(wrap, { caption }); @@ -580,29 +1041,63 @@ export function glSankey(_, verdicts, spec = {}) { const { caption, provenance, width = 688, height = 500, total = 1000 } = spec; // Layout constants. - const BAR = 13; // band thickness - const GAP = 15; // vertical gap between bands - const MIN_BAND = 2.5; // slivers stay visible - const HIT_MIN = 30, HIT_PAD = 10; // invisible hover target size - const COL = { c1: 0.38, c2: 0.70 }; // column positions as width fractions - const M = { t: 18, b: 18, l: 142, r: 138 }; // l fits "1000 questions" even at hover font size + const BAR = 13; // band thickness + const GAP = 15; // vertical gap between bands + const MIN_BAND = 2.5; // slivers stay visible + const HIT_MIN = 30, + HIT_PAD = 10; // invisible hover target size + const COL = { c1: 0.38, c2: 0.7 }; // column positions as width fractions + const M = { t: 18, b: 18, l: 142, r: 138 }; // l fits "1000 questions" even at hover font size const count = (k) => verdicts.filter((v) => v.final === k).length; - const bitesOf = (k) => verdicts.filter((v) => v.final === k && v.bites === true).length; - const FLAG = verdicts.length, NON = total - FLAG; + const bitesOf = (k) => + verdicts.filter((v) => v.final === k && v.bites === true).length; + const FLAG = verdicts.length, + NON = total - FLAG; const terminals = [ - { id: "fine", name: "fine", n: count("ACTUALLY_FINE"), color: palette.good, side: "right" }, - { id: "unv", name: "unverifiable", n: count("UNVERIFIABLE"), color: palette.neutral, side: "right" }, - { id: "amb", name: "ambiguous", n: count("CONFIRMED_AMBIGUOUS"), color: palette.warn, side: "left" }, - { id: "wrong", name: "wrong", n: count("CONFIRMED_BAD"), color: palette.bad, side: "left" }, + { + id: "fine", + name: "fine", + n: count("ACTUALLY_FINE"), + color: palette.good, + side: "right", + }, + { + id: "unv", + name: "unverifiable", + n: count("UNVERIFIABLE"), + color: palette.neutral, + side: "right", + }, + { + id: "amb", + name: "ambiguous", + n: count("CONFIRMED_AMBIGUOUS"), + color: palette.warn, + side: "left", + }, + { + id: "wrong", + name: "wrong", + n: count("CONFIRMED_BAD"), + color: palette.bad, + side: "left", + }, ]; - const ambB = bitesOf("CONFIRMED_AMBIGUOUS"), ambH = count("CONFIRMED_AMBIGUOUS") - ambB; - const wrB = bitesOf("CONFIRMED_BAD"), wrH = count("CONFIRMED_BAD") - wrB; - const nCosts = ambB + wrB, nHarm = ambH + wrH; + const ambB = bitesOf("CONFIRMED_AMBIGUOUS"), + ambH = count("CONFIRMED_AMBIGUOUS") - ambB; + const wrB = bitesOf("CONFIRMED_BAD"), + wrH = count("CONFIRMED_BAD") - wrB; + const nCosts = ambB + wrB, + nHarm = ambH + wrH; - const W = width - M.l - M.r, H = height - M.t - M.b; + const W = width - M.l - M.r, + H = height - M.t - M.b; const sc = (v) => (v / total) * (H - 6 * GAP); - const x0 = M.l, x1 = M.l + W * COL.c1, x2 = M.l + W * COL.c2, x3 = M.l + W; + const x0 = M.l, + x1 = M.l + W * COL.c1, + x2 = M.l + W * COL.c2, + x3 = M.l + W; const svg = svgEl("svg", { viewBox: `0 0 ${width} ${height}` }); svg.style.cssText = `width:100%;height:auto;font-family:${FONT};`; @@ -624,59 +1119,149 @@ export function glSankey(_, verdicts, spec = {}) { return `M${xa},${a0} C${mx},${a0} ${mx},${b0} ${xb},${b0} L${xb},${b1} C${mx},${b1} ${mx},${a1} ${xa},${a1} Z`; }; - const groups = {}, ribbons = {}; + const groups = {}, + ribbons = {}; function band(id, x, y, h, color, name, n, side) { const g = svgEl("g", { class: "gl-band" }); g.dataset.id = id; const bandH = Math.max(h, MIN_BAND); const hitH = Math.max(bandH + HIT_PAD, HIT_MIN); - g.appendChild(svgEl("rect", { class: "hit", - x: x - 6, y: y + bandH / 2 - hitH / 2, width: BAR + 12, height: hitH })); - g.appendChild(svgEl("rect", { class: "viz", x, y, width: BAR, height: bandH, rx: 2, fill: color })); + g.appendChild( + svgEl("rect", { + class: "hit", + x: x - 6, + y: y + bandH / 2 - hitH / 2, + width: BAR + 12, + height: hitH, + }), + ); + g.appendChild( + svgEl("rect", { + class: "viz", + x, + y, + width: BAR, + height: bandH, + rx: 2, + fill: color, + }), + ); const t = svgEl("text", { y: y + bandH / 2 + 4.5, x: side === "left" ? x - 10 : x + BAR + 10, - "text-anchor": side === "left" ? "end" : "start" }); + "text-anchor": side === "left" ? "end" : "start", + }); t.innerHTML = n === "" ? name : `${name} ${n}`; g.appendChild(t); svg.appendChild(g); groups[id] = g; } function ribbon(id, xa, xb, a0, a1, b0, b1, color) { - const p = svgEl("path", { class: "gl-ribbon", d: ribbonPath(xa, xb, a0, a1, b0, b1), fill: color }); + const p = svgEl("path", { + class: "gl-ribbon", + d: ribbonPath(xa, xb, a0, a1, b0, b1), + fill: color, + }); p.dataset.id = id; - svg.insertBefore(p, svg.firstChild.nextSibling); // ribbons under bands + svg.insertBefore(p, svg.firstChild.nextSibling); // ribbons under bands (ribbons[id] = ribbons[id] || []).push(p); } // Column 0: source. - band("src", x0 - BAR, M.t, sc(total) + GAP, palette.soft, "1000 questions", "", "left"); + band( + "src", + x0 - BAR, + M.t, + sc(total) + GAP, + palette.soft, + "1000 questions", + "", + "left", + ); // Column 1: not flagged on top, flagged below. - const nY = M.t, nH = sc(NON); - const fY = nY + nH + GAP, fH = sc(FLAG); + const nY = M.t, + nH = sc(NON); + const fY = nY + nH + GAP, + fH = sc(FLAG); band("non", x1, nY, nH, palette.good, "not flagged", NON, "right"); band("flag", x1, fY, fH, palette.soft, "flagged", FLAG, "left"); ribbon("non", x0, x1, M.t, M.t + nH, nY, nY + nH, palette.good); ribbon("flag", x0, x1, M.t + nH, M.t + nH + fH, fY, fY + fH, palette.soft); // Column 2: verdicts from flagged; ambiguous + wrong continue to column 3. - let ySrc = fY, yDst = fY; + let ySrc = fY, + yDst = fY; const pos = {}; for (const t of terminals) { const h = Math.max(sc(t.n), MIN_BAND); band(t.id, x2, yDst, sc(t.n), t.color, t.name, t.n, t.side); ribbon(t.id, x1 + BAR, x2, ySrc, ySrc + sc(t.n), yDst, yDst + h, t.color); pos[t.id] = { y: yDst, h }; - ySrc += sc(t.n); yDst += h + GAP; + ySrc += sc(t.n); + yDst += h + GAP; } // Column 3: harmless above, costs-points below. const hY = pos.amb.y - 2; const cY = hY + Math.max(sc(nHarm), MIN_BAND) + GAP * 1.6; - band("harmless", x3, hY, sc(nHarm), palette.cleanFill, "harmless", nHarm, "right"); - band("costs", x3, cY, sc(nCosts), palette.bad, "costs points", nCosts, "right"); - ribbon("harmless", x2 + BAR, x3, pos.amb.y, pos.amb.y + sc(ambH), hY, hY + sc(ambH), palette.warn); - ribbon("harmless", x2 + BAR, x3, pos.wrong.y, pos.wrong.y + sc(wrH), hY + sc(ambH), hY + sc(ambH) + sc(wrH), palette.bad); - ribbon("costs", x2 + BAR, x3, pos.amb.y + sc(ambH), pos.amb.y + sc(ambH) + sc(ambB), cY, cY + sc(ambB), palette.warn); - ribbon("costs", x2 + BAR, x3, pos.wrong.y + sc(wrH), pos.wrong.y + sc(wrH) + sc(wrB), cY + sc(ambB), cY + sc(ambB) + sc(wrB), palette.bad); + band( + "harmless", + x3, + hY, + sc(nHarm), + palette.cleanFill, + "harmless", + nHarm, + "right", + ); + band( + "costs", + x3, + cY, + sc(nCosts), + palette.bad, + "costs points", + nCosts, + "right", + ); + ribbon( + "harmless", + x2 + BAR, + x3, + pos.amb.y, + pos.amb.y + sc(ambH), + hY, + hY + sc(ambH), + palette.warn, + ); + ribbon( + "harmless", + x2 + BAR, + x3, + pos.wrong.y, + pos.wrong.y + sc(wrH), + hY + sc(ambH), + hY + sc(ambH) + sc(wrH), + palette.bad, + ); + ribbon( + "costs", + x2 + BAR, + x3, + pos.amb.y + sc(ambH), + pos.amb.y + sc(ambH) + sc(ambB), + cY, + cY + sc(ambB), + palette.warn, + ); + ribbon( + "costs", + x2 + BAR, + x3, + pos.wrong.y + sc(wrH), + pos.wrong.y + sc(wrH) + sc(wrB), + cY + sc(ambB), + cY + sc(ambB) + sc(wrB), + palette.bad, + ); // Hover wiring: a band and its ribbons heat together, from either side. const setHot = (id, hot) => { @@ -706,14 +1291,25 @@ export function glAuditCard(_, verdicts, spec = {}) { const { caption } = spec; const CLASSES = { CONFIRMED_BAD: { name: "wrong", color: palette.bad, bg: palette.badSoft }, - CONFIRMED_AMBIGUOUS: { name: "ambiguous", color: palette.warnInk, bg: "#faf3e0" }, - UNVERIFIABLE: { name: "unverifiable", color: palette.neutralInk, bg: palette.neutralSoft }, + CONFIRMED_AMBIGUOUS: { + name: "ambiguous", + color: palette.warnInk, + bg: "#faf3e0", + }, + UNVERIFIABLE: { + name: "unverifiable", + color: palette.neutralInk, + bg: palette.neutralSoft, + }, }; const pool = verdicts.filter((v) => CLASSES[v.final]); - const card = el("div", `border-radius:6px;padding:.85rem 1.1rem;font-family:${FONT};` + - `font-size:.93rem;line-height:1.55;transition:opacity .25s ease, background .25s ease, border-color .25s ease;` + - `border-left:3px solid;min-height:11em;`); + const card = el( + "div", + `border-radius:6px;padding:.85rem 1.1rem;font-family:${FONT};` + + `font-size:.93rem;line-height:1.55;transition:opacity .25s ease, background .25s ease, border-color .25s ease;` + + `border-left:3px solid;min-height:11em;`, + ); const render = () => { const v = pool[Math.floor(Math.random() * pool.length)]; const c = CLASSES[v.final]; @@ -723,21 +1319,34 @@ export function glAuditCard(_, verdicts, spec = {}) { `${c.name}` + `
“${esc(v.question)}” — gold: ${esc(v.gold)}
` + - (v.correct_answer ? `
verifier's answer: ${esc(v.correct_answer)}
` : "") + + (v.correct_answer + ? `
verifier's answer: ${esc(v.correct_answer)}
` + : "") + `
${esc(v.reason)}
` + - (v.source_url ? `` : ""); + (v.source_url + ? `` + : ""); }; render(); - const btn = el("button", `background:transparent;color:${palette.mintDeep};border:1px solid ${palette.mintEdge};` + - `border-radius:999px;padding:.45rem 1.05rem;font-family:${FONT};font-size:.85rem;cursor:pointer;`); + const btn = el( + "button", + `background:transparent;color:${palette.mintDeep};border:1px solid ${palette.mintEdge};` + + `border-radius:999px;padding:.45rem 1.05rem;font-family:${FONT};font-size:.85rem;cursor:pointer;`, + ); btn.textContent = "↻ show another flagged question"; btn.onclick = () => { card.style.opacity = 0; - setTimeout(() => { render(); card.style.opacity = 1; }, 180); + setTimeout(() => { + render(); + card.style.opacity = 1; + }, 180); }; - const btnRow = el("div", "display:flex;justify-content:center;margin-top:.8rem;"); + const btnRow = el( + "div", + "display:flex;justify-content:center;margin-top:.8rem;", + ); btnRow.appendChild(btn); const wrap = el("div"); @@ -781,8 +1390,12 @@ export function glQuestionTriplet(_, rows, spec = {}) { injectOnce("gl-question-triplet-css", TRIPLET_CSS); const byItem = new Map(rows.map((r) => [r.item, r])); - const current = seedItems.map((i) => byItem.get(i)).filter(Boolean).slice(0, count); - while (current.length < count) current.push(rows[Math.floor(Math.random() * rows.length)]); + const current = seedItems + .map((i) => byItem.get(i)) + .filter(Boolean) + .slice(0, count); + while (current.length < count) + current.push(rows[Math.floor(Math.random() * rows.length)]); const wrap = el("div"); const render = (card, q) => { @@ -805,16 +1418,240 @@ export function glQuestionTriplet(_, rows, spec = {}) { for (const card of cards) { const inner = card.querySelector(".inner"); inner.classList.add("leaving"); - inner.addEventListener("animationend", () => { - inner.classList.remove("leaving"); - render(card, rows[Math.floor(Math.random() * rows.length)]); - inner.classList.add("entering"); - inner.addEventListener("animationend", () => inner.classList.remove("entering"), { once: true }); - }, { once: true }); + inner.addEventListener( + "animationend", + () => { + inner.classList.remove("leaving"); + render(card, rows[Math.floor(Math.random() * rows.length)]); + inner.classList.add("entering"); + inner.addEventListener( + "animationend", + () => inner.classList.remove("entering"), + { once: true }, + ); + }, + { once: true }, + ); } }; - const btnRow = el("div", "display:flex;justify-content:center;margin-top:.9rem;"); + const btnRow = el( + "div", + "display:flex;justify-content:center;margin-top:.9rem;", + ); btnRow.appendChild(btn); wrap.appendChild(btnRow); return frame(wrap, { caption }); } + +/* ------------------------------------------------------------ glHfSamples */ +// Hugging-Face-dataset-viewer-style table of sample rows with a refresh +// button that draws a fresh random window live from the datasets-server +// /rows API (CORS-open; serves the CURRENT revision only, no pinning — +// anything that must stay true to the audited snapshot belongs in seedRows). +// Seeds render immediately and deterministically (offline-safe); the live +// fetch happens only on refresh, or on load when no seeds are given. The +// footer reports the served slice and the revision sha from X-Revision. +// seedRows: rows shown on load and used as fallback when the API fails. +// spec: { dataset, config, split, — HF ids, e.g. "mandarjoshi/trivia_qa" +// count = 3, — rows per draw +// columns, — subset/order of columns (default: all) +// format = {}, — per-column value -> display string +// caption } +const HFS_API = "https://datasets-server.huggingface.co"; +const HFS_SIZES = new Map(); // "ds|cfg|split" -> promise of split row count + +const HFS_CSS = ` + .gl-hfs { border: 1px solid ${palette.hairline}; border-radius: 12px; background: #fff; + font-family: ${FONT}; overflow: hidden; } + .gl-hfs-bar { display: flex; justify-content: space-between; align-items: baseline; gap: .8rem; + padding: .6rem .95rem; border-bottom: 1px solid ${palette.hairline}; background: ${palette.paper}; } + .gl-hfs-id { font-family: "Geist Mono", monospace; font-size: .74rem; color: ${palette.muted}; } + .gl-hfs-id a { color: ${palette.mintDeep}; text-decoration: none; + border-bottom: 1px solid ${palette.mintEdge}; } + .gl-hfs-badge { font-family: "Geist Mono", monospace; font-size: .66rem; letter-spacing: .07em; + text-transform: uppercase; color: ${palette.soft}; white-space: nowrap; } + .gl-hfs-scroll { overflow-x: auto; } + .gl-hfs table { width: 100%; border-collapse: collapse; table-layout: fixed; font-size: .86rem; } + .gl-hfs th { text-align: left; font-weight: 600; font-size: .74rem; color: ${palette.ink}; + padding: .5rem .75rem; border-bottom: 1px solid ${palette.hairline}; } + .gl-hfs th .t { display: block; font-family: "Geist Mono", monospace; font-weight: 400; + font-size: .64rem; color: ${palette.soft}; } + .gl-hfs td { padding: .55rem .75rem; border-bottom: 1px solid #f0f0f0; vertical-align: top; + color: ${palette.ink}; line-height: 1.45; overflow-wrap: break-word; } + .gl-hfs td > div { display: -webkit-box; -webkit-line-clamp: 3; -webkit-box-orient: vertical; + overflow: hidden; } + .gl-hfs .idx { width: 5.6em; padding-left: .4rem; padding-right: .6rem; text-align: right; + font-family: "Geist Mono", monospace; font-size: .68rem; color: ${palette.soft}; + white-space: nowrap; overflow-wrap: normal; } + .gl-hfs tbody { transition: opacity .18s ease; } + .gl-hfs tbody tr:hover { background: ${palette.bg}; } + .gl-hfs tbody tr:last-child td { border-bottom: none; } + .gl-hfs-foot { display: flex; justify-content: space-between; align-items: center; gap: .8rem; + padding: .5rem .95rem; border-top: 1px solid ${palette.hairline}; } + .gl-hfs-status { font-family: "Geist Mono", monospace; font-size: .68rem; color: ${palette.soft}; } + .gl-hfs-status.err { color: #b4443c; } + .gl-hfs-btn { background: transparent; color: ${palette.mintDeep}; + border: 1px solid ${palette.mintEdge}; border-radius: 999px; padding: .4rem .95rem; + font-family: ${FONT}; font-size: .8rem; cursor: pointer; white-space: nowrap; + transition: background .15s, border-color .15s; } + .gl-hfs-btn:hover { background: ${palette.mintSoft}; border-color: ${palette.mintDeep}; } + .gl-hfs-btn:disabled { opacity: .45; cursor: default; } +`; + +export function glHfSamples(_, seedRows, spec = {}) { + const { + dataset, + config, + split, + count = 3, + columns, + format = {}, + caption, + } = spec; + injectOnce("gl-hfs-css", HFS_CSS); + const seeds = (seedRows ?? []).slice(); + let cols = + columns ?? + (seeds.length ? Object.keys(seeds[0]).filter((k) => k !== "row_idx") : []); + let types = {}; // column -> dtype label, learned from the API's features + + const card = el("div"); + card.className = "gl-hfs"; + const hfUrl = `https://huggingface.co/datasets/${dataset}`; + card.innerHTML = ` +
+ ${ + dataset + ? `${esc(dataset)} + · ${esc(config)} · ${esc(split)}` + : "committed sample" + } + +
+
+ +
+
+ + +
`; + const [badge, thead, tbody, status, btn] = [ + card.querySelector('[data-role="badge"]'), + card.querySelector("thead tr"), + card.querySelector("tbody"), + card.querySelector('[data-role="status"]'), + card.querySelector(".gl-hfs-btn"), + ]; + + const fmt = (c, v) => + format[c] + ? format[c](v) + : typeof v === "string" + ? v + : v == null + ? "" + : JSON.stringify(v); + const head = () => { + thead.innerHTML = + `#` + + cols + .map( + (c) => + `${esc(c)}${esc(types[c] ?? "")}`, + ) + .join(""); + }; + const body = (entries) => { + tbody.innerHTML = entries + .map( + ({ idx, row }) => ` + ${idx} + ${cols.map((c) => `
${esc(String(fmt(c, row[c])).slice(0, 600))}
`).join("")} + `, + ) + .join(""); + }; + const set = (b, s, err = false) => { + badge.textContent = b; + status.textContent = s; + status.classList.toggle("err", err); + }; + + const showSeeds = (label) => { + const rows = + seeds.length > count + ? seeds + .slice() + .sort(() => Math.random() - 0.5) + .slice(0, count) + : seeds; + head(); + body(rows.map((row) => ({ idx: row.row_idx ?? "—", row }))); + console.log("[glHfSamples] seed rows:", rows); + set("audit sample", label); + }; + + const sizeOf = () => { + const key = `${dataset}|${config}|${split}`; + if (!HFS_SIZES.has(key)) + HFS_SIZES.set( + key, + fetch( + `${HFS_API}/size?dataset=${encodeURIComponent(dataset)}&config=${config}&split=${split}`, + ) + .then((r) => r.json()) + .then((d) => d.size.splits.find((s) => s.split === split).num_rows), + ); + return HFS_SIZES.get(key); + }; + + async function draw() { + if (!dataset) return showSeeds("committed sample · shuffled"); + btn.disabled = true; + tbody.style.opacity = ".35"; + set(badge.textContent, "fetching…"); + try { + const total = await sizeOf(); + const offset = Math.floor(Math.random() * Math.max(1, total - count)); + const res = await fetch( + `${HFS_API}/rows?dataset=${encodeURIComponent(dataset)}` + + `&config=${config}&split=${split}&offset=${offset}&length=${count}`, + ); + if (!res.ok) throw new Error(`HTTP ${res.status}`); + const rev = (res.headers.get("x-revision") || "").slice(0, 7); + const data = await res.json(); + console.log( + `[glHfSamples] ${dataset} ${config}/${split} ` + + `offset=${offset} of ${total} rev=${rev || "?"} — raw response:`, + data, + ); + if (!cols.length) cols = data.features.map((f) => f.name); + types = Object.fromEntries( + data.features.map((f) => [ + f.name, + f.type?.dtype ?? f.type?._type ?? (f.type ? "struct" : ""), + ]), + ); + head(); + body(data.rows.map((r) => ({ idx: r.row_idx, row: r.row }))); + set( + "live · unpinned", + `rows ${offset.toLocaleString()}–${(offset + count - 1).toLocaleString()}` + + ` of ${total.toLocaleString()}${rev ? ` · rev ${rev}` : ""}`, + ); + } catch (e) { + console.warn("[glHfSamples] live fetch failed:", e); + if (seeds.length) showSeeds("live fetch failed — committed sample shown"); + else set("error", `live fetch failed (${e.message}) — try again`, true); + } finally { + btn.disabled = false; + tbody.style.opacity = "1"; + } + } + + btn.onclick = draw; + if (seeds.length) + showSeeds("the three items examined above — refresh for live rows"); + else draw(); + return frame(card, { caption }); +} diff --git a/blog/posts/triviaqa/index.html b/blog/posts/triviaqa/index.html index 82f08a1..10bdce7 100644 --- a/blog/posts/triviaqa/index.html +++ b/blog/posts/triviaqa/index.html @@ -177,18 +177,28 @@

What is TriviaQA?

Splits: 76.5k train / 9.9k validation / 9.5k test. The answers for the test split are not provided.

Example items

-
-

In what year was the best foreign film category introduced to the Academy Awards?

-

1948 — alias: “one thousand, nine hundred and forty-eight”

-
-
-

What is the name of the strip of land between a golf tee and the green?

-

Fairway — aliases: “Fairway”, “Fair-way”, “Fair Way”, “Fairway (disambiguation)” [sic]

-
-
-

Which locks control entry to the Panama canal from the Atlantic end?

-

Gatun Lock — aliases: “Gatun lock”, “Gatun Locks”, “Panama Canal Locks”, “Pedro Miguel Locks”, “Pedro Miguel lock”, “Panama Canal locks”, “Gatún lock”

-
+
+
+
+
+ +
+
+

Construct validity

While the original paper describes the dataset as “reading comprehension”, in the closed-book mode it becomes a general QA dataset — there is nothing for the model to read, aside from the question text.

What is the dataset measuring then? Perhaps “breadth of factual trivia-like knowledge (prior to 2017) and the ability to retrieve it”. It’s hard to pinpoint the construct more exactly — the authors don’t balance the dataset in any way.

@@ -259,7 +269,7 @@

Hypotheses for the