Documentation sweep: reconcile every doc claim with the code - #20
Merged
Conversation
A five-reader audit of every .md against the code, verifying by execution rather than by reading. The statistical claims all held -- the conformance figures (5,504/4,765/4,606/159), the 2,596 covariate verdicts, the 94.3% coverage, all nine GRN cost-table rows and every `IntractableQuery` message reproduce exactly. What had rotted was everything around them. SELF-CONTRADICTIONS -- the docs disagreed with themselves, and the wrong half was the one a skimmer reads first: - README:495 and SPEC.md:42 both said C1 is checked at CONSTRUCTION. It is not, and has not been since PR #9 -- `graph/model.py:175` says "C1 is deliberately NOT enforced here" -- and README:525 already said the opposite three sections later.⚠️ I reported this fixed last cycle after finding the CORRECT statement and missing the incorrect one. - README:592 said a cycle strictly upstream of a knockdown "is refused". PR #12 made it ANSWERABLE, README:117 already showed it answering, and `test_cyclic_graphs.py:620` pins it. Verbatim survivor of pre-#12 text, listed under "Not yet supported". STALE OUTPUT -- blocks that no longer match what the code prints: - README's `E[Y | do]` block showed `sum_{TF,genotype,stim} ... * P(genotype) * P(stim)` and a 3-element footprint. PR #12's post-intervention closure severs those; it is `sum_{TF} E[target | TF] * P0_reg(TF)`, footprint `{'TF'}`. The block also never showed the graph it needs -- run as written it raised. Both fixed. - README's empty-stratum block lost two lines when PR #16 inserted `Policy support`. ★ Its gate asserted three substrings, so it stayed green while the document under-reported. Now compared as a CONTIGUOUS BLOCK against both the tool's output and the README's text, and mutation-tested: deleting the two lines from the README turns it red. - README:609 said `317 passed`; it is 352. - whitepaper x3 said `60 tests`; `minimal_model` has 62. FOUNDATIONS said 12 for `test_dseparation.py`; it has 14. DANGLING CITATIONS -- 7 cited test node IDs that pytest does not collect: MINIMAL_EXAMPLE.md cited six `test_example.py::` tests under names none of them have, and a seventh, `test_irreducibility_witness`, that exists nowhere -- its coverage lives in `test_observational_correlation_one`, now said plainly. THEOREM_T4_T5 cited `test_T4_under_latent_confounding`; the test is `test_T4_identifiable_for_observed_mechanism`. UNDOCUMENTED SHIPPED SURFACE: - `PolicySupport` prints from `Estimate.summary()` by DEFAULT and was documented nowhere in the README. A user saw a `Policy support:` line in their own output with nothing to read. Now a fourth paragraph in "Estimating from data", stating why it is deliberately not a positivity certificate, and citing the run that produced it. - `demo/` had ZERO mentions in the README -- the one artifact that runs the library on real published data. Now in Install (as the first thing to run), the Documentation table, and Repository layout. `tests/idcorpus/` was missing from the layout too. WRONG DEPENDENCY: README said "the only runtime dependency is NumPy" and pyproject declared it. `grep -rn numpy src/` returns NOTHING -- verified by importing all six subpackages with a `meta_path` blocker that raises on any numpy import: all succeed. NumPy is used by `minimal_model/` (not shipped) and parts of the test suite, so it moves to the dev extra, which CI installs and therefore validates. The shipped wheel is pure stdlib. MY OWN ERRORS IN demo/, from yesterday: ★ The cycle-survival claim was evidence about the WRONG OBJECT. It cited the strongly connected component of the whole ~64k-edge network (253 under TRRUST, ~103 under DoRothEA-A) to support a statement about the demo's TWENTY-GENE pathway. At pathway scope the count is 12 -> 6 -> 4. The cycles do survive independent curation, but the surviving core is CEBPA/MYB/SPI1/TAL1, not all twelve -- and KLF1, whose deletion verdict 3 reports, is cyclic only in the aggregated graph. The demo now COMPUTES the pathway figures instead of quoting a number about something else, and they are pinned. - "a third of the scope undefined" understated it: all 3 points of the scope are undefined. - The module docstring said 106 single-gene arms; it is 105 plus the control. - The README pointed at `estimator.py` for the CEBPA diagnosis, which lives in no source file. The finding is now stated where it is made. - `test_demo_smoke.py` pinned the refusals but not the two DISCLOSURES -- the verdicts that return numbers. Verdicts 2, 3 and 4 are now pinned; a rework flipping GATA1 to refused would previously have left the suite green. `.ruff_cache/` added to `.gitignore` beside `.pytest_cache/`. 352 passed, 1 xfailed. ruff clean. Every corrected figure re-derived by running the code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A five-reader audit of every
.mdagainst the code, verifying by execution rather than by reading.The statistical claims all held — the conformance figures (5,504 / 4,765 / 4,606 / 159), the 2,596 covariate verdicts, the 94.3% coverage, all nine GRN cost-table rows and every
IntractableQuerymessage reproduce exactly. What had rotted was everything around them.Self-contradictions — and the wrong half was the one a skimmer reads first
README.md:495andSPEC.md:42both said C1 is checked at construction. It is not, and hasn't been since PR Lift C1: feedback is a fact about biology, not a construction error #9 (graph/model.py:175: "C1 is deliberately NOT enforced here") — whileREADME.md:525said the opposite three sections later.README.md:592said a cycle strictly upstream of a knockdown "is refused". PR Close the ancestry over the post-intervention graph, not the observational one #12 made it answerable,README.md:117already showed it answering, andtest_cyclic_graphs.py:620pins it. A verbatim survivor of pre-Close the ancestry over the post-intervention graph, not the observational one #12 text, sitting under "Not yet supported".Stale output blocks
E[Y | do]sum_{TF,genotype,stim} … * P(genotype) * P(stim), footprint of 3sum_{TF} E[target | TF] * P0_reg(TF), footprint{'TF'}— PR #12's post-intervention closure severs those. The block also never showed its graph, so it raised when runPolicy supportbetween them317 passed352 passed60 testsFOUNDATIONS.md:18912 tests★ The empty-stratum gate asserted three substrings, so it stayed green while the document under-reported. It now compares the block contiguously against both the tool's output and the README's text — mutation-tested: deleting the two lines from the README turns it red.
Dangling citations — 7 cited test IDs pytest does not collect
MINIMAL_EXAMPLE.mdcited sixtest_example.py::tests under names none of them have, plustest_irreducibility_witness, which exists nowhere — its coverage lives intest_observational_correlation_one, now said plainly.THEOREM_T4_T5.mdcitedtest_T4_under_latent_confounding; the test istest_T4_identifiable_for_observed_mechanism.Undocumented shipped surface
PolicySupportprints fromEstimate.summary()by default and was documented nowhere in the README. A user saw aPolicy support:line in their own output with nothing to read. Now a fourth paragraph in Estimating from data, stating why it is deliberately not a positivity certificate.demo/had zero mentions in the README — the one artifact that runs the library on real published data. Now in Install (as the first thing to run), the Documentation table, and Repository layout.tests/idcorpus/was missing from the layout too.Wrong dependency
README said "the only runtime dependency is NumPy" and
pyprojectdeclared it.grep -rn numpy src/returns nothing — verified by importing all six subpackages with ameta_pathblocker that raises on any numpy import: all succeed. NumPy is used byminimal_model/(not shipped) and parts of the test suite, so it moves to the dev extra, which CI installs and therefore validates. The shipped wheel is pure stdlib.My own errors in
demo/, from yesterday★ The cycle-survival claim was evidence about the wrong object. It cited the strongly connected component of the whole ~64k-edge network (253 under TRRUST, ~103 under DoRothEA-A) to support a statement about the demo's twenty-gene pathway. At pathway scope the count is 12 → 6 → 4. The cycles do survive independent curation, but the surviving core is CEBPA/MYB/SPI1/TAL1 — and KLF1, whose deletion verdict 3 reports, is cyclic only in the aggregated graph. The demo now computes the pathway figures instead of quoting a number about something else, and they are pinned.
estimator.pyfor the CEBPA diagnosis, which lives in no source file.test_demo_smoke.pypinned the refusals but not the two disclosures — the verdicts that return numbers. Verdicts 2, 3 and 4 are now pinned; a rework flipping GATA1 to refused would previously have left the suite green.Left open deliberately
pyrightis documented, pinned and configured — but no CI job runs it. Either add a step or drop the claim; that's a CI decision.whitepaper.md:345(§8.3) claimsdo(¬m_W)is identifiable where the repo's own T7 solver appears to disagree. A substantive maths claim, not a doc typo — worth its own look.352 passed, 1 xfailed.ruff check .clean. Every corrected figure re-derived by running the code.🤖 Generated with Claude Code