From 744f4e4fbfbe4013a22d5b91235a75639c11cf0a Mon Sep 17 00:00:00 2001 From: Dan Reynolds Date: Tue, 23 Jun 2026 15:23:35 -0400 Subject: [PATCH] experiment pipeline: generate README tables from per-experiment fragments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes the conflict-reduction work (the deferred README half of phase 2). README.md was the last shared file experiments hand-edited (and a *source* for generate_history, which is why it was deferred). Now: - experiments/index/NNN.json — one fragment per experiment holding its README row {file, title, impact, status, link}. A "split" experiment (014: an accepted finding + a rejected one) stores an array of rows. - experiments/README.template.md — prose + table headers with {{ROWS:status}} placeholders. - generate_readme.dart assembles README.md from template + fragments, newest first (id descending). README.md becomes a generated, bot-owned aggregate. - generate_history.dart now sources experiments from the fragments (it builds the equivalent table markdown in memory and feeds the UNCHANGED, golden- tested parser), with a README.md fallback for test fixtures. Its sort is made stable (insertion-order tiebreak) so split/suffixed ids (008/008b, the two 014 rows) are deterministic regardless of source. - check_generated_data builds README too; check_experiment_signals and finalize_experiment read the index fragments instead of README rows. - The bot regenerates README.md; instructions updated to "add index/NNN.json, never edit the README table". Verified: README round-trips content-exactly (same row set per section, prose byte-identical) — only the row ORDER normalizes to id-descending (the prior order was an irregular historical artifact). history.json is byte-identical except the two duplicate-014 entries normalize to accepted-first via the new stable sort. Golden-fixture + pipeline tests pass; analyze clean. Two follow-ups, mirroring the signals.json rollout: - add experiments/README.md to guard-generated-docs once this baseline lands (left out so this bootstrap PR can commit the reordered README). - README-table generation does not touch the chart pipeline beyond the stable sort. Co-Authored-By: Claude Opus 4.8 --- .claude/skills/resqlite-experiment/SKILL.md | 56 ++--- .github/workflows/update-experiments.yml | 12 +- benchmark/check_experiment_signals.dart | 55 +++-- benchmark/check_generated_data.dart | 3 + benchmark/finalize_experiment.dart | 19 +- benchmark/generate_history.dart | 99 +++++++-- benchmark/generate_readme.dart | 110 ++++++++++ experiments/README.md | 230 ++++++++++---------- experiments/README.template.md | 135 ++++++++++++ experiments/RUNNER_INSTRUCTIONS.md | 30 ++- experiments/index/001.json | 7 + experiments/index/002.json | 7 + experiments/index/003.json | 7 + experiments/index/004.json | 7 + experiments/index/005.json | 7 + experiments/index/006.json | 7 + experiments/index/007.json | 7 + experiments/index/008.json | 7 + experiments/index/008b.json | 7 + experiments/index/009.json | 7 + experiments/index/010.json | 7 + experiments/index/011.json | 7 + experiments/index/012.json | 7 + experiments/index/013.json | 7 + experiments/index/014.json | 16 ++ experiments/index/015.json | 7 + experiments/index/016.json | 7 + experiments/index/017.json | 7 + experiments/index/018.json | 7 + experiments/index/019.json | 7 + experiments/index/020.json | 7 + experiments/index/021.json | 7 + experiments/index/022.json | 7 + experiments/index/023.json | 7 + experiments/index/024.json | 7 + experiments/index/025.json | 7 + experiments/index/026.json | 7 + experiments/index/027.json | 7 + experiments/index/028.json | 7 + experiments/index/029.json | 7 + experiments/index/030.json | 7 + experiments/index/031.json | 7 + experiments/index/032.json | 7 + experiments/index/033.json | 7 + experiments/index/034.json | 7 + experiments/index/035.json | 7 + experiments/index/036.json | 7 + experiments/index/037.json | 7 + experiments/index/038.json | 7 + experiments/index/039.json | 7 + experiments/index/040.json | 7 + experiments/index/041.json | 7 + experiments/index/042.json | 7 + experiments/index/043.json | 7 + experiments/index/044.json | 7 + experiments/index/045.json | 7 + experiments/index/046.json | 7 + experiments/index/047.json | 7 + experiments/index/051.json | 7 + experiments/index/052.json | 7 + experiments/index/053.json | 7 + experiments/index/054.json | 7 + experiments/index/055.json | 7 + experiments/index/057.json | 7 + experiments/index/058.json | 7 + experiments/index/059.json | 7 + experiments/index/060.json | 7 + experiments/index/063.json | 7 + experiments/index/064.json | 7 + experiments/index/065.json | 7 + experiments/index/066.json | 7 + experiments/index/067.json | 7 + experiments/index/068.json | 7 + experiments/index/069.json | 7 + experiments/index/070.json | 7 + experiments/index/071.json | 7 + experiments/index/072.json | 7 + experiments/index/073.json | 7 + experiments/index/074.json | 7 + experiments/index/075.json | 7 + experiments/index/076.json | 7 + experiments/index/077.json | 7 + experiments/index/081.json | 7 + experiments/index/082.json | 7 + experiments/index/083.json | 7 + experiments/index/084.json | 7 + experiments/index/085.json | 7 + experiments/index/088.json | 7 + experiments/index/089.json | 7 + experiments/index/090.json | 7 + experiments/index/092.json | 7 + experiments/index/093.json | 7 + experiments/index/094.json | 7 + experiments/index/095.json | 7 + experiments/index/096.json | 7 + experiments/index/097.json | 7 + experiments/index/099.json | 7 + experiments/index/100.json | 7 + experiments/index/101.json | 7 + experiments/index/102.json | 7 + experiments/index/103.json | 7 + experiments/index/104.json | 7 + experiments/index/105.json | 7 + experiments/index/106.json | 7 + experiments/index/108.json | 7 + experiments/index/109.json | 7 + experiments/index/110.json | 7 + experiments/index/111.json | 7 + experiments/index/112.json | 7 + experiments/index/113.json | 7 + experiments/index/114.json | 7 + experiments/index/115.json | 7 + experiments/index/116.json | 7 + experiments/index/117.json | 7 + experiments/index/118.json | 7 + experiments/index/119.json | 7 + experiments/index/120.json | 7 + experiments/index/121.json | 7 + experiments/index/122.json | 7 + experiments/index/125.json | 7 + experiments/index/126.json | 7 + experiments/index/134.json | 7 + experiments/index/136.json | 7 + experiments/index/142.json | 7 + experiments/index/143.json | 7 + experiments/index/144.json | 7 + experiments/index/145.json | 7 + experiments/index/146.json | 7 + experiments/index/147.json | 7 + experiments/index/148.json | 7 + experiments/index/149.json | 7 + experiments/index/150.json | 7 + experiments/index/151.json | 7 + experiments/index/158.json | 7 + experiments/index/159.json | 7 + experiments/index/161.json | 7 + experiments/index/164.json | 7 + experiments/index/167.json | 7 + experiments/index/169.json | 7 + experiments/index/170.json | 7 + experiments/index/171.json | 7 + experiments/index/172.json | 7 + experiments/index/173.json | 7 + experiments/index/174.json | 7 + experiments/index/175.json | 7 + experiments/index/176.json | 7 + experiments/index/177.json | 7 + experiments/index/178.json | 7 + experiments/index/179.json | 7 + experiments/index/180.json | 7 + experiments/index/181.json | 7 + experiments/index/182.json | 7 + experiments/index/183.json | 7 + experiments/index/184.json | 7 + experiments/index/185.json | 7 + experiments/index/186.json | 7 + experiments/index/187.json | 7 + experiments/index/188.json | 7 + experiments/index/189.json | 7 + experiments/index/190.json | 7 + experiments/index/191.json | 7 + experiments/index/192.json | 7 + experiments/index/193.json | 7 + experiments/index/194.json | 7 + experiments/index/195.json | 7 + 165 files changed, 1639 insertions(+), 204 deletions(-) create mode 100644 benchmark/generate_readme.dart create mode 100644 experiments/README.template.md create mode 100644 experiments/index/001.json create mode 100644 experiments/index/002.json create mode 100644 experiments/index/003.json create mode 100644 experiments/index/004.json create mode 100644 experiments/index/005.json create mode 100644 experiments/index/006.json create mode 100644 experiments/index/007.json create mode 100644 experiments/index/008.json create mode 100644 experiments/index/008b.json create mode 100644 experiments/index/009.json create mode 100644 experiments/index/010.json create mode 100644 experiments/index/011.json create mode 100644 experiments/index/012.json create mode 100644 experiments/index/013.json create mode 100644 experiments/index/014.json create mode 100644 experiments/index/015.json create mode 100644 experiments/index/016.json create mode 100644 experiments/index/017.json create mode 100644 experiments/index/018.json create mode 100644 experiments/index/019.json create mode 100644 experiments/index/020.json create mode 100644 experiments/index/021.json create mode 100644 experiments/index/022.json create mode 100644 experiments/index/023.json create mode 100644 experiments/index/024.json create mode 100644 experiments/index/025.json create mode 100644 experiments/index/026.json create mode 100644 experiments/index/027.json create mode 100644 experiments/index/028.json create mode 100644 experiments/index/029.json create mode 100644 experiments/index/030.json create mode 100644 experiments/index/031.json create mode 100644 experiments/index/032.json create mode 100644 experiments/index/033.json create mode 100644 experiments/index/034.json create mode 100644 experiments/index/035.json create mode 100644 experiments/index/036.json create mode 100644 experiments/index/037.json create mode 100644 experiments/index/038.json create mode 100644 experiments/index/039.json create mode 100644 experiments/index/040.json create mode 100644 experiments/index/041.json create mode 100644 experiments/index/042.json create mode 100644 experiments/index/043.json create mode 100644 experiments/index/044.json create mode 100644 experiments/index/045.json create mode 100644 experiments/index/046.json create mode 100644 experiments/index/047.json create mode 100644 experiments/index/051.json create mode 100644 experiments/index/052.json create mode 100644 experiments/index/053.json create mode 100644 experiments/index/054.json create mode 100644 experiments/index/055.json create mode 100644 experiments/index/057.json create mode 100644 experiments/index/058.json create mode 100644 experiments/index/059.json create mode 100644 experiments/index/060.json create mode 100644 experiments/index/063.json create mode 100644 experiments/index/064.json create mode 100644 experiments/index/065.json create mode 100644 experiments/index/066.json create mode 100644 experiments/index/067.json create mode 100644 experiments/index/068.json create mode 100644 experiments/index/069.json create mode 100644 experiments/index/070.json create mode 100644 experiments/index/071.json create mode 100644 experiments/index/072.json create mode 100644 experiments/index/073.json create mode 100644 experiments/index/074.json create mode 100644 experiments/index/075.json create mode 100644 experiments/index/076.json create mode 100644 experiments/index/077.json create mode 100644 experiments/index/081.json create mode 100644 experiments/index/082.json create mode 100644 experiments/index/083.json create mode 100644 experiments/index/084.json create mode 100644 experiments/index/085.json create mode 100644 experiments/index/088.json create mode 100644 experiments/index/089.json create mode 100644 experiments/index/090.json create mode 100644 experiments/index/092.json create mode 100644 experiments/index/093.json create mode 100644 experiments/index/094.json create mode 100644 experiments/index/095.json create mode 100644 experiments/index/096.json create mode 100644 experiments/index/097.json create mode 100644 experiments/index/099.json create mode 100644 experiments/index/100.json create mode 100644 experiments/index/101.json create mode 100644 experiments/index/102.json create mode 100644 experiments/index/103.json create mode 100644 experiments/index/104.json create mode 100644 experiments/index/105.json create mode 100644 experiments/index/106.json create mode 100644 experiments/index/108.json create mode 100644 experiments/index/109.json create mode 100644 experiments/index/110.json create mode 100644 experiments/index/111.json create mode 100644 experiments/index/112.json create mode 100644 experiments/index/113.json create mode 100644 experiments/index/114.json create mode 100644 experiments/index/115.json create mode 100644 experiments/index/116.json create mode 100644 experiments/index/117.json create mode 100644 experiments/index/118.json create mode 100644 experiments/index/119.json create mode 100644 experiments/index/120.json create mode 100644 experiments/index/121.json create mode 100644 experiments/index/122.json create mode 100644 experiments/index/125.json create mode 100644 experiments/index/126.json create mode 100644 experiments/index/134.json create mode 100644 experiments/index/136.json create mode 100644 experiments/index/142.json create mode 100644 experiments/index/143.json create mode 100644 experiments/index/144.json create mode 100644 experiments/index/145.json create mode 100644 experiments/index/146.json create mode 100644 experiments/index/147.json create mode 100644 experiments/index/148.json create mode 100644 experiments/index/149.json create mode 100644 experiments/index/150.json create mode 100644 experiments/index/151.json create mode 100644 experiments/index/158.json create mode 100644 experiments/index/159.json create mode 100644 experiments/index/161.json create mode 100644 experiments/index/164.json create mode 100644 experiments/index/167.json create mode 100644 experiments/index/169.json create mode 100644 experiments/index/170.json create mode 100644 experiments/index/171.json create mode 100644 experiments/index/172.json create mode 100644 experiments/index/173.json create mode 100644 experiments/index/174.json create mode 100644 experiments/index/175.json create mode 100644 experiments/index/176.json create mode 100644 experiments/index/177.json create mode 100644 experiments/index/178.json create mode 100644 experiments/index/179.json create mode 100644 experiments/index/180.json create mode 100644 experiments/index/181.json create mode 100644 experiments/index/182.json create mode 100644 experiments/index/183.json create mode 100644 experiments/index/184.json create mode 100644 experiments/index/185.json create mode 100644 experiments/index/186.json create mode 100644 experiments/index/187.json create mode 100644 experiments/index/188.json create mode 100644 experiments/index/189.json create mode 100644 experiments/index/190.json create mode 100644 experiments/index/191.json create mode 100644 experiments/index/192.json create mode 100644 experiments/index/193.json create mode 100644 experiments/index/194.json create mode 100644 experiments/index/195.json diff --git a/.claude/skills/resqlite-experiment/SKILL.md b/.claude/skills/resqlite-experiment/SKILL.md index 6efe18c8..333e7db2 100644 --- a/.claude/skills/resqlite-experiment/SKILL.md +++ b/.claude/skills/resqlite-experiment/SKILL.md @@ -21,8 +21,9 @@ and those files feed the live charts on the GitHub Pages site 1. **The code change** — `native/`, `lib/`, `hook/`, or `test/` as appropriate 2. **The experiment doc** — `experiments/NNN-short-name.md` with - `**Date:** YYYY-MM-DD` frontmatter and a row in `experiments/README.md` - (under Accepted or Rejected) + `**Date:** YYYY-MM-DD` frontmatter and an `experiments/index/NNN.json` row + fragment (`{file, title, impact, status, link}`). `experiments/README.md` is + *generated* from those fragments — never hand-edit the README table. 3. **At least one benchmark result file** — `benchmark/results/-