optimize: fix corpus corruption, make CI link check real, remove dead code

[aaronjmars]

Ran the 8-agent /optimize sweep over the repo. Five agents applied (dedup, dead-code, defensive-code, legacy-paths, ai-slop); type-consolidation, circular-deps and weak-types were skipped as inapplicable — no typed language, no module graph, no dependency manifest.

One commit per domain so any dimension can be reverted independently.

The headline: three committed data files were corrupt

karpathy/scripts/fetch-data.sh used curl -sL without -f, so HTTP errors were written to disk as the response body and curl still exited 0 — the author's own || echo "[WARN]" could never fire, and the [ -f ] cache guard meant the poisoned file was never retried.

Verified against the live URL: curl -sL exits 0 and writes a 9379-byte 404 page; curl -fsSL exits 56 and writes nothing.

File	Was	Now
data/writing/software-2.0.html	9379 B GitHub Pages 404	241 KB real essay
data/github/karpathy_char-rnn_README.md	14 B 404: Not Found	12.8 KB real README
data/x/steipete.replies.json	[] (2 B)	still [] — needs a SURF_API_KEY re-pull

The first one mattered most: SOUL.md:20 calls Software 2.0 "my core thesis" and weak-model-test.mjs:239 scores model output for that literal string — so the grounding for the file's central claim was a 404 page. The URL was simply wrong (the essay is on Medium). char-rnn ships Readme.md, not README.md.

Sources were fixed before re-fetching, so a pipeline re-run reproduces the good files. Swept all of examples/*/data for other empty or error-body files: none.

The CI job named "Check relative links" checked nothing

scripts/check-links.mjs scanned only root-level *.md — five files containing zero relative links. It printed ✓ 0 relative link(s) across 5 doc(s) and passed unconditionally. Cause: 9edda35 moved README/CONTRIBUTING/SECURITY into .github/ and the scope was never updated, so its own comment named the exact failure mode it could no longer see.

Now scans root + .github/ + folder guides + per-example README/QUICKSTART, still excluding examples/*/data/ and persona files. 22 links across 20 docs. Verified it fails correctly by injecting a broken link (exits 1, names the target).

Also fixed

• SURF_API_KEY vs SURF_AI_API_KEY — pull-x.mjs reads the former, fetch-data.sh and the README passed the latter, so following the documented instructions hit exit(1). That pipeline had never run. Unified across 11 sites.
• Stale star counts — all 9 rows in the frameworks table were understated. Verified against the GitHub API: OpenClaw 322k→383k, Hermes Agent 8.7k→218k (checked for a redirect; none), and 6 others. Re-sorted.
• Dead code — unused dirname import, HANDLE const, _slug() plus the re import it was the sole consumer of.
• Silent failures — bare || true, discarded yt-dlp stderr, a swallowed MCP protocol violation, errors on stdout instead of stderr, and run_weak_model.py writing [ERROR 401: ...] into a results file as if it were the model's answer, then exiting 0.
• Broken idempotency — the transcript cache checked for .vtt files when zero are committed, so every transcript re-downloaded on every run.
• Stale docs — soul-builder pointed at SKILL.md (it's BUILD.md); a documented test command that could not run; a "live run pending" status for a run that shipped 2026-04-14; README trees documenting directories that don't exist; a 40-line embedded "reference implementation" contradicting the real script.
• −48 lines collapsing byte-identical openai/openrouter fetch branches.

Deliberately not done

• No cross-persona consolidation despite a 24.76% JS clone rate. Each examples/<persona>/ folder is meant to be copied out whole, and what would need parameterizing (12 prompts, system prompt, scoring regexes) is the persona. The two fetch-data.sh files scored 0 clones — independently written, not copy-paste.
• No persona prose touched. SOUL.md, STYLE.md, good-outputs.md are the product; bad-outputs.md contains deliberately bad writing as negative examples.
• examples/*/data/** untouched apart from the two repairs. Its 164 "broken" links are third-party source material pointing at original repo layouts — correct by design.

Verification

Gate after every domain: node --check ×4, bash -n ×2 (+ shellcheck), python3 -m py_compile ×4, node scripts/check-links.mjs, markdownlint-cli2. All green, no new failures vs. the 5cc4a00 baseline.

Scripts were not executed end-to-end — they hit paid APIs and need credentials. The riskiest edit is the fetch-branch collapse; the two branches were confirmed byte-identical apart from the URL and output stays in scope, but it hasn't run against a live API.