Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
6708f33
task #603: port #551/#541/#518 rig onto issue-603 + response_sink cap…
superkaiba Jun 11, 2026
e7777b3
task #603: write_decomposition.py + V1 gate vs #551 published control…
superkaiba Jun 11, 2026
315bfe7
task #603: pod-side extraction dispatcher + worker, source priors, Ph…
superkaiba Jun 11, 2026
6c91331
issue-603: sync workflow-surface specs from main (spec-freshness)
superkaiba Jun 11, 2026
98659a9
task #603 round 2: manifest-validated resume, --from-hub priors stagi…
superkaiba Jun 11, 2026
4120770
task #603 r3: step-12 refusal implant check + decompose gate wiring +…
superkaiba Jun 11, 2026
8dbacac
task #603 r3: step-12 production output — 6/6 refusal sources pass th…
superkaiba Jun 11, 2026
e57e21e
task #603 r4: pin refusal-implant judge to temperature=0.0 (#518 pari…
superkaiba Jun 11, 2026
231b265
methodology: issue #603 findings-blind reference
superkaiba Jun 11, 2026
4a0d4ff
task #603 Phase-2: decomposition results (lattice indeterminate_on_h1…
superkaiba Jun 11, 2026
cee4c82
task #603: analyzer-pass figures (fact/extensions hero split, guard A…
superkaiba Jun 11, 2026
64de93e
task #603: INDEX.md row
superkaiba Jun 11, 2026
4429225
task #603 round-2: fact-only disattenuated hero (panel-matched to raw…
superkaiba Jun 11, 2026
91cc7c5
task #603 round-2: INDEX row corrected (CMF-cosine read, clean-u flip…
superkaiba Jun 11, 2026
10846e4
task #603: fix refusal/EM binary-verdict parsing + Messages-Batch re-…
superkaiba Jun 11, 2026
3292c68
task #603 round-5 micro-fix: meta.family_caveats + realized-pattern d…
superkaiba Jun 11, 2026
4c3df33
task #603 round-5 micro-fix: regenerate expression_strata.json with f…
superkaiba Jun 11, 2026
291ec10
task #603: guard-B v3 analyzer figure (post-re-judge: raw vs clean-te…
superkaiba Jun 11, 2026
22d8786
task #603 follow-up base-text-extraction: thread --variant base throu…
superkaiba Jun 11, 2026
d7d2ddf
task #603: bind expression-strata label caches to the extraction vari…
superkaiba Jun 11, 2026
de78e5f
issue-603: sync workflow-surface specs from main (spec-freshness)
superkaiba Jun 12, 2026
a24c4fa
task #603 follow-up base-text-extraction Phase 2: decomposition (21 c…
superkaiba Jun 12, 2026
e6a4c09
task #603 base-text re-fold: same-vs-base rank-correlation comparison…
superkaiba Jun 12, 2026
365868d
methodology: issue #603 EXTEND — base-text-extraction arm section
superkaiba Jun 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .claude/agents/analyzer.md
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,7 @@ Write first to a local file `.claude/cache/experiment-<N>-clean-result.md` (thro
- **`**Artifacts:**`** — links to training data, model checkpoints, eval JSONs, figure source, raw completions. The training-data dropdown lives under `### What I ran`; eval examples live near the finding that consumed them (NOT here); this Artifacts block just lists the full artifact links. **GROUND every path-specific artifact claim in a live Hub listing — never type it from the plan's intent.** When you write a bullet that names specific subfolders, checkpoint directories, intermediate-fraction adapters, file counts, or HF Hub paths (e.g. "per-cell LoRA adapters at intermediate fractions {0.25, 0.50, 0.75, 1.00} uploaded to `adapters/issue_<N>/<cell>/`", "520 files at `<path>`"), run `huggingface_hub.list_repo_files` on the relevant repo + revision at write time and copy what the listing actually shows. The `hf` CLI has no `api` subcommand and false-reports "0 files" on a path that exists, so use the Python Hub API (see `.claude/rules/upload-policy.md` for the canonical snippet). If a planned subfolder is missing — e.g. a band-stop callback halted training before the planned intermediate-fraction checkpoint was saved — the body says what is ACTUALLY on the Hub, not what the plan intended; the missing piece becomes a methodology-correction beat inside the relevant `#### <finding>` H4 (the silent-fail rule in CLAUDE.md § "After Every Experiment" #8). A plan-intent claim that doesn't survive the listing is a data-integrity bug that propagates: it gets carried into follow-up-proposer's reuse premises (incident #530→#534, 2026-06-09) and into any future task whose planner mines this body for prior-art artifacts. **Reuse provenance — when ANY reader-facing claim in this body rests on a trained artifact REUSED from a prior issue** (a LoRA adapter, merged checkpoint, training-mix dataset, raw-completion bucket, or `eval_results/` JSON produced by a previous `/issue` run rather than freshly produced by THIS task), record one bullet per reused artifact under this block stating: (a) the producing issue number `#M` as a markdown link to `https://eps.superkaiba.com/tasks/M`; (b) the permanent HF Hub path (pinned to `/tree/<sha>` or `@<sha>`) or repo-relative `eval_results/issue_M/...` path the artifact was pulled from; and (c) a one-line fitness rationale stating WHY this artifact was the right one to reuse — recipe match (same base model + training-recipe / hyperparameters the new question demands), measurement-regime fit (the artifact's eval surface contains the conditions THIS result reads off; for marker work specifically, the artifact is NOT saturated where this read needs headroom — source `log P − base ∈ [5,12]` nat per `.claude/rules/marker-training-recipe.md`), and required conditions present. Mirror the positive fitness check the planner ran at plan §5 / §10 (CLAUDE.md § "Reuse existing trained artifacts when fit-for-purpose — never reuse a wrong one") so the clean-result carries the same justification forward. Format: `- Reused <kind> from [#M](...): <hf path or local path> — fit: <one line: recipe + regime + conditions>`. Source the reuse list from the plan body (§5 reusable + §10/§11 artifact citations) and from any explicit `Source: #M` / `from-issue` references in the training-script SHA at the `**Code:**` link; never invent reuse the plan didn't approve. When THIS task produced every artifact it stands on, omit the reuse-provenance bullets entirely (most fresh-train experiments). The clean-result-critic Lens 5 audits this.
- **`**Compute:**`** — wall time, GPU type/count, pod label.
- **`**Code:**`** — dataset-build script, pipeline driver, Hydra config, git commit hash, one-block reproduce snippet.
- **`**Context:**`** — run-context provenance (REQUIRED for v2 bodies; SPEC.md § `**Context:**` row; verifier check 17). Three bullets: **Created / run** (frontmatter `created_at` + the date/window results landed), **Follow-up to** (`[#K](https://eps.superkaiba.com/tasks/K) — <one line>` from frontmatter `parent_id` / the lineage that seeded the task, or `fresh direction (no parent)`; for same-issue follow-up rounds also name the round's `followup_label`), and **Originating prompt(s), verbatim** (blockquoted). Source the prompt from, in priority order: frontmatter `origin_prompt`; the ORIGINAL task body's `## Provenance` section — read it BEFORE Step 6's `set-body --snapshot` replaces the live body (post-promotion it lives only in `original-body.md`, so on re-drafts read it from there); and `epm:followup-scope v1` markers with `source: user-chat` (via `task.py latest-marker` / `view --json`, never a hand-built `tasks/...` path). VERBATIM means verbatim — never paraphrase, trim, or fix typos. When no prompt was recorded, write the literal `origin prompt not recorded` — never omit the row, never fabricate. Provenance lives ONLY here: the "state facts, not sources" rule still bans prompt/person attributions in `## TL;DR` and finding prose.

**Confidence lives in the H1 title tag only** for v2 nested-design bodies. Do NOT emit a `Confidence: LOW|MODERATE|HIGH — <rationale>` sentence in `## Reproducibility`. There is NO "Why confidence is where it is" section. If you need to convey the binding constraint that drove the title's confidence level, weave it into the relevant `#### <finding>` read paragraph.

Expand Down Expand Up @@ -434,7 +435,7 @@ uv run python "$REPO_ROOT"/scripts/verify_task_body.py --file .claude/cache/expe

Every FAIL must be fixed before posting. WARNs may ship when explicitly acknowledged in the body (e.g. the qualitative-data-link WARN for runs whose raw completions weren't uploaded — pair with a "re-run with raw-completion upload" bullet in Next steps). Do NOT proceed to Step 6 until the verifier is FAIL-free.

The verifier enforces the mechanical checks for the 2-content-section nested-design (v2) spec (see `scripts/verify_task_body.py` docstring for the canonical enumeration): body-nonstub (check 0, defense against the cache → body.md silent-handoff failure); no-duplicate-frontmatter (check 0b); title confidence tag (`(LOW|MODERATE|HIGH confidence)`); three required H2s in order (`## Human TL;DR`, `## TL;DR`, `## Reproducibility`) — a stray `## Details` or `## Figure` H2 is a hard FAIL (forces clean migration to the 2-content-section spec); `## TL;DR` opens with the Motivation block, either as `### Motivation` H3 (preferred) or `**Motivation:**` boldface bullet (legacy form); v2 nested-shape check (sentinel-gated) — `### Motivation` / `### What I ran` / `### Findings` H3s in order with ≥1 `#### <finding>` H4 child under `### Findings`; at least one `![alt](url)` image exists inline under `## TL;DR` (every finding H4 carries its own figure); every image URL is absolute + commit-pinned; Confidence — for v2 bodies the H1 title tag is the source of truth (PASSes with NO body sentence); for legacy (pre-sentinel) bodies `Confidence: …` line scanned whole-body, matches the title's level + ≥20 chars of rationale (lives in `## Reproducibility` by convention); `## Reproducibility` carries all three boldface subgroups (`**Artifacts:**`, `**Compute:**`, `**Code:**`); URL permanence in Reproducibility (HF Hub `/tree/<ref>`, WandB `/runs/<id>`, GitHub `/blob/<sha>`; no `main`/`master`/`HEAD`); no `{{` / `TBD` / `see config` / `default` sentinels in Reproducibility (write `n/a` explicitly); cherry-picked label preceding every sample-output block in `## TL;DR` (fenced OR `<details>`-wrapped table/long-text); qualitative-data link in the same prelude or inside the `<details>` block (raw text-level artifact, not aggregate); Reproducibility lr matches plan (check 16, v2-only) — the learning rate in the Parameters table must appear in the approved `plans/plan.md` (FAIL unless a documented run-vs-plan deviation downgrades it to WARN; NO-OP PASS when it cannot reconcile). Soft WARN: `check_details_narrative_flow` flags outline-label H3s + figure-dump runs (>2 consecutive figures without prose between) inside `## TL;DR` (but does NOT flag `### Findings` or `### What I ran` — those are required structural H3s under v2). See `CLAUDE.md § Experiment Report Structure` for the canonical body shape this verifier checks.
The verifier enforces the mechanical checks for the 2-content-section nested-design (v2) spec (see `scripts/verify_task_body.py` docstring for the canonical enumeration): body-nonstub (check 0, defense against the cache → body.md silent-handoff failure); no-duplicate-frontmatter (check 0b); title confidence tag (`(LOW|MODERATE|HIGH confidence)`); three required H2s in order (`## Human TL;DR`, `## TL;DR`, `## Reproducibility`) — a stray `## Details` or `## Figure` H2 is a hard FAIL (forces clean migration to the 2-content-section spec); `## TL;DR` opens with the Motivation block, either as `### Motivation` H3 (preferred) or `**Motivation:**` boldface bullet (legacy form); v2 nested-shape check (sentinel-gated) — `### Motivation` / `### What I ran` / `### Findings` H3s in order with ≥1 `#### <finding>` H4 child under `### Findings`; at least one `![alt](url)` image exists inline under `## TL;DR` (every finding H4 carries its own figure); every image URL is absolute + commit-pinned; Confidence — for v2 bodies the H1 title tag is the source of truth (PASSes with NO body sentence); for legacy (pre-sentinel) bodies `Confidence: …` line scanned whole-body, matches the title's level + ≥20 chars of rationale (lives in `## Reproducibility` by convention); `## Reproducibility` carries all three boldface subgroups (`**Artifacts:**`, `**Compute:**`, `**Code:**`); URL permanence in Reproducibility (HF Hub `/tree/<ref>`, WandB `/runs/<id>`, GitHub `/blob/<sha>`; no `main`/`master`/`HEAD`); no `{{` / `TBD` / `see config` / `default` sentinels in Reproducibility (write `n/a` explicitly); cherry-picked label preceding every sample-output block in `## TL;DR` (fenced OR `<details>`-wrapped table/long-text); qualitative-data link in the same prelude or inside the `<details>` block (raw text-level artifact, not aggregate); Reproducibility lr matches plan (check 16, v2-only) — the learning rate in the Parameters table must appear in the approved `plans/plan.md` (FAIL unless a documented run-vs-plan deviation downgrades it to WARN; NO-OP PASS when it cannot reconcile); Reproducibility Context provenance row (check 17, v2-only) — the `**Context:**` row (created/run dates, follow-up lineage, verbatim originating prompt) must be present (FAIL only when recorded origin data — frontmatter `origin_prompt` or a `## Provenance` section in `original-body.md` — exists but the body dropped it; WARN otherwise). Soft WARN: `check_details_narrative_flow` flags outline-label H3s + figure-dump runs (>2 consecutive figures without prose between) inside `## TL;DR` (but does NOT flag `### Findings` or `### What I ran` — those are required structural H3s under v2). See `CLAUDE.md § Experiment Report Structure` for the canonical body shape this verifier checks.

### Step 6: Promote the source experiment to a clean-result (inline)

Expand Down
33 changes: 31 additions & 2 deletions .claude/agents/clean-result-critic.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,12 @@ anti-pattern audit:
# in the Parameters table must appear in `plans/plan.md`
# (FAIL unless a documented run-vs-plan deviation → WARN; NO-OP
# PASS when it cannot reconcile). Task #489's 1e-4-vs-2e-6 typo.
# 17. Reproducibility Context provenance row (v2-only) — the
# `**Context:**` row ships created/run dates, follow-up lineage,
# and the verbatim originating prompt (FAIL only when recorded
# origin data — frontmatter `origin_prompt` or original-body.md
# `## Provenance` — exists but the body dropped it; WARN
# otherwise; legacy bodies skip).
# 13. (WARN) TL;DR narrative flow — outline-label H3s + figure-dumps
uv run python scripts/verify_task_body.py --issue <N>

Expand All @@ -119,9 +125,13 @@ the verifier's FAILs into two classes before deciding the verdict:
`![alt](url)` figure exists anywhere under `## TL;DR` (check 4), a
Reproducibility boldface subgroup is absent (check 7), a retired
`## Details` / `## Figure` H2 is present (check 2 clean-migration),
the body is a stub (nonstub check), or the Reproducibility learning
the body is a stub (nonstub check), the Reproducibility learning
rate does not match the plan (check 16) — a wrong load-bearing
hyperparameter is a data-integrity defect, never cosmetic. These are
hyperparameter is a data-integrity defect, never cosmetic — or
recorded origin provenance was dropped (check 17 FAIL: frontmatter
`origin_prompt` / an original-body `## Provenance` section exists but
the body carries no `**Context:**` row; the check's WARN form — no
recorded origin data — is not a FAIL and never blocks). These are
like a missing/wrong report section: record the failed check as a
blocking finding, but STILL read all fifteen lenses in the same
pass and report every substantive finding you see. **Beyond the
Expand Down Expand Up @@ -453,6 +463,25 @@ in Lens 12. Lens number kept stable so downstream tooling that reads
been linked.
- No `{{`, `TBD`, `default`, `see config` sentinels — write `n/a`
explicitly when truly non-applicable (verifier check #9).
- **Context-row audit (run-context provenance; v2 bodies).** The
`**Context:**` row in `## Reproducibility` (SPEC.md
§ `**Context:**` row; verifier check 17 covers presence — this
bullet adds the substantive read) must carry: (a) **real dates** —
the created date matches frontmatter `created_at`, the run
date/window is plausible against the events.jsonl timeline; (b)
**correct lineage** — the `Follow-up to` line matches frontmatter
`parent_id` / the Motivation's actual prior-task citation (a
fabricated or wrong parent is a FAIL), or says `fresh direction
(no parent)`; (c) **verbatim prompts** — cross-check the quoted
originating prompt against frontmatter `origin_prompt` and/or the
`## Provenance` section in `original-body.md`; a paraphrased,
trimmed, or typo-corrected prompt is a FAIL (verbatim means
verbatim), and the literal `origin prompt not recorded` is
accepted only when no origin data actually exists. Also confirm
provenance stays CONFINED to this row — prompt/person attributions
woven into `## TL;DR` or finding prose violate the "state facts,
not sources" rule. Forward-only: legacy (pre-sentinel) bodies are
never failed for lacking the row.
- **Reuse-provenance audit (semantic, not mechanical).** When any
reader-facing claim in `## TL;DR` rests on a trained artifact
REUSED from a prior issue — a LoRA adapter, merged checkpoint,
Expand Down
Loading