Add a pre-flight demo: which causal questions can this screen answer? - #17
Merged
Conversation
A perturbation screen is expensive and its analysis is cheap, so the analysis is where the
mistakes go unnoticed. This points the compiler at a real published screen -- Norman et al.
2019, GSE133344, 65,359 K562 cells, 105 single-gene arms -- and asks a grid of
mechanism-level causal questions: which of these can the data answer?
Three are refused outright and two are answered with a disclosure that changes what the
answer means. EVERY VERDICT IS PRINTED NEXT TO EVIDENCE ASSEMBLED INDEPENDENTLY OF THE
COMPILER, because a tool that says "no" is only worth having if the "no" is right.
cycles 89 of 95 refused, 6 identified, 0 of them non-trivial
post-treatment total UMI and cell-cycle score refused; structural, not an assumption
dead readout answered, resting on 19 of 11,183 cells -- positivity PASSES here,
so no refusal catches it and only the disclosure does
degenerate binning refused; the median non-zero UMI count is 1, so the middle level of
the obvious three-level binning is unreachable while the domain
still claims it
policy leverage 3,734 effective rows of 11,183, a 3x overstatement
"Non-trivial" is measured, not asserted: a deletion estimand with ONE estimated conditional
factor and a point-mass policy is `P(Y | X=x)`, agreeing with a group-by bit for bit at
every sample size tested. Two or more factors and it is a genuinely different estimator.
The demo counts factors, which is why it reports 0 non-trivial answers on the curated graph.
The constructive half, because a refusal that does not say what would work is a complaint
rather than an instrument. A cycle is a statement about variables measured at one instant;
resolved in time it is a DAG, and NO EDGE IS REMOVED -- pruning until acyclic would
manufacture the property the refusal is about.
timepoints 1 (as curated) 2 3 4
non-trivial 0 0 35 69
max width - - 8 12
Time-resolution, not more cells, and every answer inside the elimination budget.
What it does NOT claim: that the compiler predicts a held-out interventional arm. That was
tried on the CEBPA arm and lost to "nothing changed". CRISPRa drives its target 24.8x past
anything the control cells show, so a kernel fitted on control data answers with the
observational noise-slope. Off-support extrapolation, which no identification machinery
repairs -- and the reason the policy certificate in the parent commit exists.
Data vendored under `demo/data/` (843 KB): derived counts for 23 named genes only, neither
source redistributed. `demo/build_cache.py` regenerates it by streaming the 362M-entry
matrix once. Runs in 2.3s from a clean checkout with no third-party packages.
`tests/test_demo_smoke.py` pins every figure quoted in `demo/README.md` against a real
subprocess run, for the reason `test_readme_smoke.py` exists: three README claims here were
once wrong rather than stale, and a demo is the artefact people run to decide whether to
believe the library. 329 passed, 1 xfailed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
logannye
force-pushed
the
feat/preflight-demo
branch
from
August 18, 2026 21:29
d17fb3f to
2325c28
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Points the compiler at a real published screen — Norman et al. 2019 (GSE133344), 65,359 K562 cells, 105 single-gene arms, 8 gemgroups — and asks a grid of mechanism-level causal questions: which of these can the data answer?
Three are refused outright; two are answered with a disclosure that changes what the answer means. Every verdict is printed next to evidence assembled independently of the compiler, because a tool that says "no" is only worth having if the "no" is right.
Verdicts 3 and 5 both return a number. Both would read as clean results without the certificate beside them — which is the case for shipping the disclosure half, not just the refusal half.
"Non-trivial" is measured, not asserted
A deletion estimand with one estimated conditional factor and a point-mass policy is
P(Y | X=x)— agreeing with a group-by bit for bit at every sample size tested. At two or more it is a genuinely different, Markov-restricted estimator. The demo counts factors, which is why it reports 0 non-trivial answers on the curated graph rather than 6.The constructive half
A refusal that does not say what would work is a complaint, not an instrument. A cycle is a statement about variables measured at one instant; resolved in time it is a DAG — and no edge is removed, because pruning a network until it is acyclic would manufacture the very property the refusal is about.
Time-resolution, not more cells — and every answer inside the elimination budget, which is the half of the claim that makes the other half useful.
What this does not claim
That the compiler predicts a held-out interventional arm. It was tried on the CEBPA arm and lost to "nothing changed" (TV 0.2061 vs 0.2054). CRISPRa drives its target 24.8× past anything the control cells show, so a kernel fitted on control data answers with the observational noise-slope. That is off-support extrapolation, which no identification machinery repairs — and it is why #16 exists. Stated plainly in
demo/README.mdrather than omitted.Data
demo/data/is 843 KB of derived counts for 23 named genes only; neither source is redistributed.demo/build_cache.pyregenerates it by streaming the 362M-entry matrix once. Cycles are shown to survive restriction to TRRUST alone (253-node SCC) and DoRothEA-A alone (103-node SCC), so they are not an artefact of aggregating databases — which is why the resource tags are kept.Verification
tests/test_demo_smoke.pyruns the demo as a subprocess and pins every figure quoted indemo/README.md— cell/arm/gemgroup counts, the 89/6/0 split, the 0→35→69 unrolling table with its widths, and the policy certificate's effective count. Same reasontest_readme_smoke.pyexists: three README claims in this repo were once wrong rather than stale, and a demo is the artefact people run to decide whether to believe the library.329 passed, 1 xfailed.ruff check .clean.🤖 Generated with Claude Code