Skip to content

Documentation sweep: reconcile every doc claim with the code - #20

Merged
logannye merged 1 commit into
mainfrom
docs/sweep-corrections
Aug 19, 2026
Merged

Documentation sweep: reconcile every doc claim with the code#20
logannye merged 1 commit into
mainfrom
docs/sweep-corrections

Conversation

@logannye

Copy link
Copy Markdown
Owner

A five-reader audit of every .md against the code, verifying by execution rather than by reading.

The statistical claims all held — the conformance figures (5,504 / 4,765 / 4,606 / 159), the 2,596 covariate verdicts, the 94.3% coverage, all nine GRN cost-table rows and every IntractableQuery message reproduce exactly. What had rotted was everything around them.

Self-contradictions — and the wrong half was the one a skimmer reads first

Stale output blocks

block was is
E[Y | do] sum_{TF,genotype,stim} … * P(genotype) * P(stim), footprint of 3 sum_{TF} E[target | TF] * P0_reg(TF), footprint {'TF'} — PR #12's post-intervention closure severs those. The block also never showed its graph, so it raised when run
empty-stratum 3 lines 5 — PR #16 inserted Policy support between them
suite count 317 passed 352 passed
whitepaper ×3 60 tests 62
FOUNDATIONS.md:189 12 tests 14

★ The empty-stratum gate asserted three substrings, so it stayed green while the document under-reported. It now compares the block contiguously against both the tool's output and the README's text — mutation-tested: deleting the two lines from the README turns it red.

Dangling citations — 7 cited test IDs pytest does not collect

MINIMAL_EXAMPLE.md cited six test_example.py:: tests under names none of them have, plus test_irreducibility_witness, which exists nowhere — its coverage lives in test_observational_correlation_one, now said plainly. THEOREM_T4_T5.md cited test_T4_under_latent_confounding; the test is test_T4_identifiable_for_observed_mechanism.

Undocumented shipped surface

  • PolicySupport prints from Estimate.summary() by default and was documented nowhere in the README. A user saw a Policy support: line in their own output with nothing to read. Now a fourth paragraph in Estimating from data, stating why it is deliberately not a positivity certificate.
  • demo/ had zero mentions in the README — the one artifact that runs the library on real published data. Now in Install (as the first thing to run), the Documentation table, and Repository layout. tests/idcorpus/ was missing from the layout too.

Wrong dependency

README said "the only runtime dependency is NumPy" and pyproject declared it. grep -rn numpy src/ returns nothing — verified by importing all six subpackages with a meta_path blocker that raises on any numpy import: all succeed. NumPy is used by minimal_model/ (not shipped) and parts of the test suite, so it moves to the dev extra, which CI installs and therefore validates. The shipped wheel is pure stdlib.

My own errors in demo/, from yesterday

The cycle-survival claim was evidence about the wrong object. It cited the strongly connected component of the whole ~64k-edge network (253 under TRRUST, ~103 under DoRothEA-A) to support a statement about the demo's twenty-gene pathway. At pathway scope the count is 12 → 6 → 4. The cycles do survive independent curation, but the surviving core is CEBPA/MYB/SPI1/TAL1 — and KLF1, whose deletion verdict 3 reports, is cyclic only in the aggregated graph. The demo now computes the pathway figures instead of quoting a number about something else, and they are pinned.

  • "a third of the scope undefined" understated it: all 3 points of the scope are undefined.
  • The module docstring said 106 single-gene arms; it is 105 plus the control.
  • The README pointed at estimator.py for the CEBPA diagnosis, which lives in no source file.
  • test_demo_smoke.py pinned the refusals but not the two disclosures — the verdicts that return numbers. Verdicts 2, 3 and 4 are now pinned; a rework flipping GATA1 to refused would previously have left the suite green.

Left open deliberately

  • pyright is documented, pinned and configured — but no CI job runs it. Either add a step or drop the claim; that's a CI decision.
  • whitepaper.md:345 (§8.3) claims do(¬m_W) is identifiable where the repo's own T7 solver appears to disagree. A substantive maths claim, not a doc typo — worth its own look.

352 passed, 1 xfailed. ruff check . clean. Every corrected figure re-derived by running the code.

🤖 Generated with Claude Code

A five-reader audit of every .md against the code, verifying by execution rather than by
reading. The statistical claims all held -- the conformance figures (5,504/4,765/4,606/159),
the 2,596 covariate verdicts, the 94.3% coverage, all nine GRN cost-table rows and every
`IntractableQuery` message reproduce exactly. What had rotted was everything around them.

SELF-CONTRADICTIONS -- the docs disagreed with themselves, and the wrong half was the one
a skimmer reads first:
- README:495 and SPEC.md:42 both said C1 is checked at CONSTRUCTION. It is not, and has not
  been since PR #9 -- `graph/model.py:175` says "C1 is deliberately NOT enforced here" --
  and README:525 already said the opposite three sections later. ⚠️ I reported this fixed
  last cycle after finding the CORRECT statement and missing the incorrect one.
- README:592 said a cycle strictly upstream of a knockdown "is refused". PR #12 made it
  ANSWERABLE, README:117 already showed it answering, and `test_cyclic_graphs.py:620` pins
  it. Verbatim survivor of pre-#12 text, listed under "Not yet supported".

STALE OUTPUT -- blocks that no longer match what the code prints:
- README's `E[Y | do]` block showed `sum_{TF,genotype,stim} ... * P(genotype) * P(stim)`
  and a 3-element footprint. PR #12's post-intervention closure severs those; it is
  `sum_{TF} E[target | TF] * P0_reg(TF)`, footprint `{'TF'}`. The block also never showed
  the graph it needs -- run as written it raised. Both fixed.
- README's empty-stratum block lost two lines when PR #16 inserted `Policy support`.
  ★ Its gate asserted three substrings, so it stayed green while the document
  under-reported. Now compared as a CONTIGUOUS BLOCK against both the tool's output and the
  README's text, and mutation-tested: deleting the two lines from the README turns it red.
- README:609 said `317 passed`; it is 352.
- whitepaper x3 said `60 tests`; `minimal_model` has 62. FOUNDATIONS said 12 for
  `test_dseparation.py`; it has 14.

DANGLING CITATIONS -- 7 cited test node IDs that pytest does not collect:
MINIMAL_EXAMPLE.md cited six `test_example.py::` tests under names none of them have, and a
seventh, `test_irreducibility_witness`, that exists nowhere -- its coverage lives in
`test_observational_correlation_one`, now said plainly. THEOREM_T4_T5 cited
`test_T4_under_latent_confounding`; the test is `test_T4_identifiable_for_observed_mechanism`.

UNDOCUMENTED SHIPPED SURFACE:
- `PolicySupport` prints from `Estimate.summary()` by DEFAULT and was documented nowhere in
  the README. A user saw a `Policy support:` line in their own output with nothing to read.
  Now a fourth paragraph in "Estimating from data", stating why it is deliberately not a
  positivity certificate, and citing the run that produced it.
- `demo/` had ZERO mentions in the README -- the one artifact that runs the library on real
  published data. Now in Install (as the first thing to run), the Documentation table, and
  Repository layout. `tests/idcorpus/` was missing from the layout too.

WRONG DEPENDENCY: README said "the only runtime dependency is NumPy" and pyproject declared
it. `grep -rn numpy src/` returns NOTHING -- verified by importing all six subpackages with
a `meta_path` blocker that raises on any numpy import: all succeed. NumPy is used by
`minimal_model/` (not shipped) and parts of the test suite, so it moves to the dev extra,
which CI installs and therefore validates. The shipped wheel is pure stdlib.

MY OWN ERRORS IN demo/, from yesterday:
★ The cycle-survival claim was evidence about the WRONG OBJECT. It cited the strongly
  connected component of the whole ~64k-edge network (253 under TRRUST, ~103 under
  DoRothEA-A) to support a statement about the demo's TWENTY-GENE pathway. At pathway scope
  the count is 12 -> 6 -> 4. The cycles do survive independent curation, but the surviving
  core is CEBPA/MYB/SPI1/TAL1, not all twelve -- and KLF1, whose deletion verdict 3 reports,
  is cyclic only in the aggregated graph. The demo now COMPUTES the pathway figures instead
  of quoting a number about something else, and they are pinned.
- "a third of the scope undefined" understated it: all 3 points of the scope are undefined.
- The module docstring said 106 single-gene arms; it is 105 plus the control.
- The README pointed at `estimator.py` for the CEBPA diagnosis, which lives in no source
  file. The finding is now stated where it is made.
- `test_demo_smoke.py` pinned the refusals but not the two DISCLOSURES -- the verdicts that
  return numbers. Verdicts 2, 3 and 4 are now pinned; a rework flipping GATA1 to refused
  would previously have left the suite green.

`.ruff_cache/` added to `.gitignore` beside `.pytest_cache/`.

352 passed, 1 xfailed. ruff clean. Every corrected figure re-derived by running the code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@logannye
logannye merged commit db56c05 into main Aug 19, 2026
4 checks passed
@logannye
logannye deleted the docs/sweep-corrections branch August 19, 2026 00:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant