experiment pipeline: generate README tables from per-experiment fragments

.claude/skills/resqlite-experiment/SKILL.md

@@ -21,8 +21,9 @@ and those files feed the live charts on the GitHub Pages site
 
 1. **The code change** — `native/`, `lib/`, `hook/`, or `test/` as appropriate
 2. **The experiment doc** — `experiments/NNN-short-name.md` with
-   `**Date:** YYYY-MM-DD` frontmatter and a row in `experiments/README.md`
-   (under Accepted or Rejected)
+   `**Date:** YYYY-MM-DD` frontmatter and an `experiments/index/NNN.json` row
+   fragment (`{file, title, impact, status, link}`). `experiments/README.md` is
+   *generated* from those fragments — never hand-edit the README table.
 3. **At least one benchmark result file** — `benchmark/results/<ISO-timestamp>-<label>.md`
    whose **filename timestamp's date** matches the experiment doc's `**Date:**`
 
@@ -112,16 +113,19 @@ Run this mental (or literal) checklist:
 
 - [ ] `git status --short benchmark/results/` — is there an untracked result file?
 - [ ] Does that file's filename timestamp match `grep "^**Date:**" experiments/NNN-*.md`?
-- [ ] Is the experiment listed in `experiments/README.md` (Accepted or Rejected section)?
+- [ ] Is the experiment's README row in its own file,
+      `experiments/index/NNN.json` (`{file, title, impact, status, link}`)?
+      `README.md` is generated from these — don't hand-edit the table.
 - [ ] Is the signal entry in its own file, `experiments/signals/entries/NNN.json`
       (not hand-edited into the generated `signals.json`)?
 - [ ] Does the experiment doc have the headings the parser expects?
       (`Problem`, `Hypothesis`, `Approach` or `What We Built`, `Results`,
       `Decision` or `Why Accepted` / `Why Rejected`)
 - [ ] Did `finalize_experiment.dart` pass? It only *validates* sources — it does
       not write the generated aggregates, and you must not commit
-      `docs/experiments/history.json`, `docs/benchmarks/devices.json`, or
-      `experiments/signals.json` (the bot regenerates them on `main`).
+      `docs/experiments/history.json`, `docs/benchmarks/devices.json`,
+      `experiments/signals.json`, or `experiments/README.md` (the bot
+      regenerates them on `main`).
 
 The generator's section extraction tolerates a few heading variants; see
 `_extractSection` in `generate_history.dart` for the full list.
@@ -309,16 +313,17 @@ git fetch origin
 
 Experiment PRs used to collide on shared generated files; that is now designed
 out. The generated aggregates — `docs/experiments/history.json`,
-`docs/benchmarks/devices.json`, and `experiments/signals.json` — are
-**bot-owned on `main` and never committed on a branch** (CI's
-`guard-generated-docs` job blocks them, and `check_generated_data.dart` only
-checks that the *sources* build). Each experiment's signal data lives in its
-own file, `experiments/signals/entries/NNN.json`, so two experiments never
-touch the same one. The only files a normal run still shares are
-`experiments/README.md` (rows append) and `experiments/signals/base.json` (the
-per-direction synthesis). You must still claim your number — concurrent runs
-that grab the same number, or ship the same follow-up, still collide (exp 168
-was claimed by three PRs; exp 175 by two runs shipping the *same* follow-up).
+`docs/benchmarks/devices.json`, `experiments/signals.json`, and
+`experiments/README.md` — are **bot-owned on `main` and never committed on a
+branch** (CI's `guard-generated-docs` job blocks them, and
+`check_generated_data.dart` only checks that the *sources* build). Each
+experiment's signal entry and README row each live in their own file —
+`experiments/signals/entries/NNN.json` and `experiments/index/NNN.json` — so
+two experiments never touch the same one. The only file a normal run still
+shares is `experiments/signals/base.json` (the per-direction synthesis). You
+must still claim your number — concurrent runs that grab the same number, or
+ship the same follow-up, still collide (exp 168 was claimed by three PRs; exp
+175 by two runs shipping the *same* follow-up).
 
 **1. Claim the number atomically.** A plain "check open PRs, then pick the next
 free" *races*: two runs check, both see N free, both take N. That is exactly
@@ -371,16 +376,17 @@ which case expect to regenerate `history.json` on the later one.
 ## Resolving a stale derived-file conflict
 
 Generated aggregates no longer conflict: `docs/experiments/history.json`,
-`docs/benchmarks/devices.json`, and `experiments/signals.json` are bot-owned
-and never committed on a branch, so a stale branch just takes `main`'s copy
-with a one-sided auto-merge. If a PR falls behind `main`, the only files that
-can really conflict are hand-edited *sources*:
-
-- `experiments/README.md` — rows append; usually auto-merges. If two
-  experiments inserted at the same spot, fix the row order by hand.
-- `experiments/signals/entries/NNN.json` — one file per experiment, so a true
-  collision only happens when two runs claimed the same number `N`. That's a
-  numbering bug: renumber, never overwrite a prior experiment's entry.
+`docs/benchmarks/devices.json`, `experiments/signals.json`, and
+`experiments/README.md` are bot-owned and never committed on a branch, so a
+stale branch just takes `main`'s copy with a one-sided auto-merge. If a PR
+falls behind `main`, the only files that can really conflict are hand-edited
+*sources*, and each is per-experiment:
+
+- `experiments/index/NNN.json` (the README row) and
+  `experiments/signals/entries/NNN.json` (the signal entry) — one file per
+  experiment, so a true collision only happens when two runs claimed the same
+  number `N`. That's a numbering bug: renumber, never overwrite a prior
+  experiment's file.
 - `experiments/signals/base.json` — two experiments editing the same
   direction's `currentRead` / `notesForExperimenters` narrative is a real
   weave; keep both contributions.

.github/workflows/update-experiments.yml

@@ -6,7 +6,10 @@ on:
     paths:
       - 'experiments/*.md'
       - 'experiments/signals/**'
+      - 'experiments/index/**'
+      - 'experiments/README.template.md'
       - 'benchmark/generate_signals.dart'
+      - 'benchmark/generate_readme.dart'
       - 'benchmark/results/*.md'
       - 'benchmark/results/*.json'
       - 'benchmark/HARDWARE_RESULTS.md'
@@ -44,6 +47,9 @@ jobs:
       - name: Generate signals JSON
         run: dart run benchmark/generate_signals.dart
 
+      - name: Generate experiments README
+        run: dart run benchmark/generate_readme.dart
+
       - name: Generate devices JSON
         run: dart run benchmark/generate_devices.dart
 
@@ -56,13 +62,13 @@ jobs:
       - name: Check for changes
         id: diff
         run: |
-          git diff --quiet docs/ experiments/signals.json && echo "changed=false" >> "$GITHUB_OUTPUT" || echo "changed=true" >> "$GITHUB_OUTPUT"
+          git diff --quiet docs/ experiments/signals.json experiments/README.md && echo "changed=false" >> "$GITHUB_OUTPUT" || echo "changed=true" >> "$GITHUB_OUTPUT"
 
       - name: Commit and push
         if: steps.diff.outputs.changed == 'true'
         run: |
           git config user.name "github-actions[bot]"
           git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
-          git add docs/ experiments/signals.json
-          git commit -m "Auto-update docs (experiments + signals + devices + blog + releases)"
+          git add docs/ experiments/signals.json experiments/README.md
+          git commit -m "Auto-update docs (experiments + signals + README + devices + blog + releases)"
           git push

benchmark/check_experiment_signals.dart

@@ -1,3 +1,4 @@
+import 'dart:convert';
 import 'dart:io';
 
 import 'generate_signals.dart' as generate_signals;
@@ -31,40 +32,42 @@ void main() {
   print('Experiment signal map is valid.');
 }
 
+/// Builds the experiment index from the per-experiment fragments
+/// (`experiments/index/NNN.json`). README.md is generated from these, so the
+/// fragments are the fresh source on a branch that hasn't regenerated it. A
+/// fragment is one row object, or a list of rows for a split experiment.
 Map<String, _ExperimentEntry> _readExperimentIndex(
   List<_ValidationError> errors,
 ) {
-  final file = File(_readmePath);
-  if (!file.existsSync()) {
-    _readmeError(errors, 'Missing $_readmePath.');
+  final indexDir = Directory('$_experimentsDir/index');
+  if (!indexDir.existsSync()) {
+    _signalError(errors, 'Missing ${indexDir.path}/ (experiment index fragments).');
     return const {};
   }
 
   final entries = <String, _ExperimentEntry>{};
-  final rowPattern = RegExp(r'^\|\s*\[(\d+\w?)\]\(([^)]+)\)\s*\|');
-  var currentStatus = 'accepted';
-  for (final line in file.readAsStringSync().split('\n')) {
-    if (line.startsWith('## Accepted')) {
-      currentStatus = 'accepted';
-    } else if (line.startsWith('## In Review')) {
-      currentStatus = 'in_review';
-    } else if (line.startsWith('## Rejected')) {
-      currentStatus = 'rejected';
+  for (final file in indexDir.listSync().whereType<File>()) {
+    if (!file.path.endsWith('.json')) continue;
+    final id = file.uri.pathSegments.last.replaceFirst(RegExp(r'\.json$'), '');
+    Object? decoded;
+    try {
+      decoded = json.decode(file.readAsStringSync());
+    } on FormatException catch (error) {
+      _signalError(errors, '${file.path} is not valid JSON: ${error.message}');
+      continue;
+    }
+    final rows = decoded is List ? decoded : [decoded];
+    for (final row in rows) {
+      if (row is! Map) continue;
+      entries[id] = _ExperimentEntry(
+        id: id,
+        filename: row['file']?.toString() ?? '$id.md',
+        status: row['status']?.toString() ?? 'accepted',
+      );
     }
-
-    final match = rowPattern.firstMatch(line);
-    if (match == null) continue;
-
-    final id = match.group(1)!;
-    final filename = match.group(2)!;
-    entries[id] = _ExperimentEntry(
-      id: id,
-      filename: filename,
-      status: currentStatus,
-    );
   }
   if (entries.isEmpty) {
-    _readmeError(errors, 'No experiment rows found in $_readmePath.');
+    _signalError(errors, 'No experiment fragments found in ${indexDir.path}/.');
   }
   return entries;
 }
@@ -631,10 +634,6 @@ void _checkOpenCandidates(
   }
 }
 
-void _readmeError(List<_ValidationError> errors, String message) {
-  errors.add(_ValidationError(_readmePath, message));
-}
-
 void _signalError(List<_ValidationError> errors, String message) {
   errors.add(_ValidationError(_signalsPath, message));
 }

benchmark/check_generated_data.dart

@@ -2,6 +2,7 @@ import 'dart:io';
 
 import 'generate_devices.dart' as generate_devices;
 import 'generate_history.dart' as generate_history;
+import 'generate_readme.dart' as generate_readme;
 import 'generate_signals.dart' as generate_signals;
 
 /// Verifies that the generated-docs sources are well-formed enough that the
@@ -35,6 +36,8 @@ Future<void> main() async {
       signalsSourceDir: Directory('experiments/signals'),
       generatedAt: null,
     ),
+    'README.md': () =>
+        generate_readme.buildReadme(experimentsDir: Directory('experiments')),
   };
 
   final failures = <String>[];

benchmark/finalize_experiment.dart

@@ -117,14 +117,19 @@ String? _validateExperimentFile(File experimentFile) {
         'experiments/147-short-slug.md: ${p.relative(experimentPath)}';
   }
 
-  final readme = File(p.join(experimentsDir, 'README.md'));
-  if (!readme.existsSync()) {
-    return 'Missing experiments/README.md.';
+  // README.md is generated from the per-experiment index fragments, so a new
+  // experiment is registered by adding its fragment, not by editing README.md
+  // (which the bot regenerates on main). Validate the fragment, not the row.
+  final id = idMatch.group(1)!;
+  final indexFragment = File(p.join(experimentsDir, 'index', '$id.json'));
+  if (!indexFragment.existsSync()) {
+    return 'Experiment $id has no index fragment '
+        'experiments/index/$id.json (its README row: file, title, impact, '
+        'status, link). README.md is generated from these — do not edit it.';
   }
-  final readmeText = readme.readAsStringSync();
-  if (!RegExp(r'\]\(' + RegExp.escape(filename) + r'\)').hasMatch(readmeText)) {
-    return 'Experiment ${idMatch.group(1)} is not linked from '
-        'experiments/README.md.';
+  if (!indexFragment.readAsStringSync().contains('"$filename"')) {
+    return 'experiments/index/$id.json does not reference "$filename" in a '
+        '"file" field.';
   }
 
   final content = experimentFile.readAsStringSync();

benchmark/generate_history.dart

@@ -151,9 +151,16 @@ Map<String, Object?> buildHistoryData({
   final experiments = <Map<String, Object?>>[];
 
   if (experimentsDir.existsSync()) {
-    final readmeFile = File('${experimentsDir.path}/README.md');
-    if (readmeFile.existsSync()) {
-      final readme = readmeFile.readAsStringSync();
+    // The per-experiment index fragments (experiments/index/NNN.json) are the
+    // source of truth. README.md is a generated, bot-owned aggregate that a
+    // branch never keeps fresh, so we synthesize the equivalent table markdown
+    // from the fragments and reuse the existing README parser unchanged. Fall
+    // back to a committed README.md when there is no index dir (test fixtures,
+    // legacy trees).
+    final readme =
+        _readmeFromIndex(Directory('${experimentsDir.path}/index')) ??
+        _maybeReadFile('${experimentsDir.path}/README.md');
+    if (readme != null) {
       experiments.addAll(
         _parseExperimentsReadme(readme, experimentsDir, allKeys),
       );
@@ -562,6 +569,57 @@ int? _pickClosestUnclaimedRun(
   return null;
 }
 
+String? _maybeReadFile(String path) {
+  final file = File(path);
+  return file.existsSync() ? file.readAsStringSync() : null;
+}
+
+/// Synthesizes README-equivalent table markdown from the per-experiment index
+/// fragments (`experiments/index/NNN.json`), so [_parseExperimentsReadme] can
+/// consume the fragments without any change to its (golden-fixture-tested)
+/// logic. A fragment is one row object, or a list of rows for a "split"
+/// experiment (e.g. 014 has both an accepted and a rejected finding). Returns
+/// null when there is no index dir or it holds no fragments.
+String? _readmeFromIndex(Directory indexDir) {
+  if (!indexDir.existsSync()) return null;
+  final files = indexDir
+      .listSync()
+      .whereType<File>()
+      .where((f) => f.path.endsWith('.json'))
+      .toList();
+  if (files.isEmpty) return null;
+
+  final rows = <String, List<String>>{
+    'accepted': [],
+    'in_review': [],
+    'rejected': [],
+  };
+  for (final file in files) {
+    final id = file.uri.pathSegments.last.replaceFirst(RegExp(r'\.json$'), '');
+    final decoded = json.decode(file.readAsStringSync());
+    final entries = decoded is List ? decoded : [decoded];
+    for (final entry in entries) {
+      if (entry is! Map) continue;
+      final bucket = rows[entry['status']?.toString()];
+      if (bucket == null) continue;
+      final fileName = entry['file']?.toString() ?? '$id.md';
+      final title = entry['title']?.toString() ?? '';
+      final impact = entry['impact']?.toString() ?? '';
+      final link = entry['link']?.toString() ?? '';
+      bucket.add('| [$id]($fileName) | $title | $impact | $link |');
+    }
+  }
+
+  final buffer = StringBuffer();
+  buffer.writeln('## Accepted');
+  rows['accepted']!.forEach(buffer.writeln);
+  buffer.writeln('## In Review');
+  rows['in_review']!.forEach(buffer.writeln);
+  buffer.writeln('## Rejected');
+  rows['rejected']!.forEach(buffer.writeln);
+  return buffer.toString();
+}
+
 /// Parse experiment entries from the README.md table rows and individual files.
 List<Map<String, Object?>> _parseExperimentsReadme(
   String readme,
@@ -705,18 +763,29 @@ List<Map<String, Object?>> _parseExperimentsReadme(
     }
   }
 
-  // Sort by experiment number.
-  experiments.sort((a, b) {
-    final aNum =
-        int.tryParse((a['id'] as String).replaceAll(RegExp(r'[^0-9]'), '')) ??
-        0;
-    final bNum =
-        int.tryParse((b['id'] as String).replaceAll(RegExp(r'[^0-9]'), '')) ??
-        0;
-    return aNum.compareTo(bNum);
-  });
-
-  return experiments;
+  // Sort by experiment number, kept stable on insertion order so that ids
+  // sharing a number — a suffixed sibling (008 vs 008b) or a split experiment's
+  // two rows (014's accepted + rejected findings) — preserve their source
+  // order. (Dart's List.sort is not stable, so an explicit index tiebreak is
+  // required; without it the tie order depends on sort internals and on whether
+  // the rows came from README.md or the index fragments.)
+  final ordered = experiments.asMap().entries.toList()
+    ..sort((a, b) {
+      final aNum =
+          int.tryParse(
+            (a.value['id'] as String).replaceAll(RegExp(r'[^0-9]'), ''),
+          ) ??
+          0;
+      final bNum =
+          int.tryParse(
+            (b.value['id'] as String).replaceAll(RegExp(r'[^0-9]'), ''),
+          ) ??
+          0;
+      if (aNum != bNum) return aNum.compareTo(bNum);
+      return a.key.compareTo(b.key);
+    });
+
+  return ordered.map((entry) => entry.value).toList();
 }
 
 String? _experimentHeaderValue(String content, String label) {