Skip to content

Add a pre-flight demo: which causal questions can this screen answer? - #17

Merged
logannye merged 1 commit into
mainfrom
feat/preflight-demo
Aug 18, 2026
Merged

Add a pre-flight demo: which causal questions can this screen answer?#17
logannye merged 1 commit into
mainfrom
feat/preflight-demo

Conversation

@logannye

Copy link
Copy Markdown
Owner

Stacked on #16. Merge that first.

python3 demo/preflight.py     # 2.3s, clean checkout, no third-party packages

What

Points the compiler at a real published screen — Norman et al. 2019 (GSE133344), 65,359 K562 cells, 105 single-gene arms, 8 gemgroups — and asks a grid of mechanism-level causal questions: which of these can the data answer?

Three are refused outright; two are answered with a disclosure that changes what the answer means. Every verdict is printed next to evidence assembled independently of the compiler, because a tool that says "no" is only worth having if the "no" is right.

# question verdict why it is right
1 95 mechanism-deletion queries over the curated pathway 89 refused, 6 identified, 0 non-trivial 12 of 20 mechanisms lie on a cycle
2 adjust KLF1 → HBG1 for total UMI and cell-cycle score refused, both post-treatment measured on the same cell after perturbation. Structural, not an assumption
3 what does activating KLF1 do to SLC4A1? answered, resting on 19 of 11,183 cells SLC4A1 is detected in 0.8% of cells. Positivity passes — scarce, not absent — so no refusal catches this and only the disclosure does
4 three expression levels per gene, declared the obvious way refused, a third of the scope undefined the median non-zero UMI count is 1, so the middle level is unreachable while the domain still claims it
5 is the estimate really backed by 11,183 cells? 3× overstatement, 3,734 effective rows uses the certificate from #16

Verdicts 3 and 5 both return a number. Both would read as clean results without the certificate beside them — which is the case for shipping the disclosure half, not just the refusal half.

"Non-trivial" is measured, not asserted

A deletion estimand with one estimated conditional factor and a point-mass policy is P(Y | X=x) — agreeing with a group-by bit for bit at every sample size tested. At two or more it is a genuinely different, Markov-restricted estimator. The demo counts factors, which is why it reports 0 non-trivial answers on the curated graph rather than 6.

The constructive half

A refusal that does not say what would work is a complaint, not an instrument. A cycle is a statement about variables measured at one instant; resolved in time it is a DAG — and no edge is removed, because pruning a network until it is acyclic would manufacture the very property the refusal is about.

timepoints 1 (as curated) 2 3 4
non-trivial answers 0 0 35 69
max induced width 8 12

Time-resolution, not more cells — and every answer inside the elimination budget, which is the half of the claim that makes the other half useful.

What this does not claim

That the compiler predicts a held-out interventional arm. It was tried on the CEBPA arm and lost to "nothing changed" (TV 0.2061 vs 0.2054). CRISPRa drives its target 24.8× past anything the control cells show, so a kernel fitted on control data answers with the observational noise-slope. That is off-support extrapolation, which no identification machinery repairs — and it is why #16 exists. Stated plainly in demo/README.md rather than omitted.

Data

demo/data/ is 843 KB of derived counts for 23 named genes only; neither source is redistributed. demo/build_cache.py regenerates it by streaming the 362M-entry matrix once. Cycles are shown to survive restriction to TRRUST alone (253-node SCC) and DoRothEA-A alone (103-node SCC), so they are not an artefact of aggregating databases — which is why the resource tags are kept.

Verification

tests/test_demo_smoke.py runs the demo as a subprocess and pins every figure quoted in demo/README.md — cell/arm/gemgroup counts, the 89/6/0 split, the 0→35→69 unrolling table with its widths, and the policy certificate's effective count. Same reason test_readme_smoke.py exists: three README claims in this repo were once wrong rather than stale, and a demo is the artefact people run to decide whether to believe the library.

329 passed, 1 xfailed. ruff check . clean.

🤖 Generated with Claude Code

@logannye
logannye changed the base branch from feat/policy-support-certificate to main August 18, 2026 21:28
A perturbation screen is expensive and its analysis is cheap, so the analysis is where the
mistakes go unnoticed. This points the compiler at a real published screen -- Norman et al.
2019, GSE133344, 65,359 K562 cells, 105 single-gene arms -- and asks a grid of
mechanism-level causal questions: which of these can the data answer?

Three are refused outright and two are answered with a disclosure that changes what the
answer means. EVERY VERDICT IS PRINTED NEXT TO EVIDENCE ASSEMBLED INDEPENDENTLY OF THE
COMPILER, because a tool that says "no" is only worth having if the "no" is right.

  cycles              89 of 95 refused, 6 identified, 0 of them non-trivial
  post-treatment      total UMI and cell-cycle score refused; structural, not an assumption
  dead readout        answered, resting on 19 of 11,183 cells -- positivity PASSES here,
                      so no refusal catches it and only the disclosure does
  degenerate binning  refused; the median non-zero UMI count is 1, so the middle level of
                      the obvious three-level binning is unreachable while the domain
                      still claims it
  policy leverage     3,734 effective rows of 11,183, a 3x overstatement

"Non-trivial" is measured, not asserted: a deletion estimand with ONE estimated conditional
factor and a point-mass policy is `P(Y | X=x)`, agreeing with a group-by bit for bit at
every sample size tested. Two or more factors and it is a genuinely different estimator.
The demo counts factors, which is why it reports 0 non-trivial answers on the curated graph.

The constructive half, because a refusal that does not say what would work is a complaint
rather than an instrument. A cycle is a statement about variables measured at one instant;
resolved in time it is a DAG, and NO EDGE IS REMOVED -- pruning until acyclic would
manufacture the property the refusal is about.

  timepoints   1 (as curated)     2       3       4
  non-trivial               0     0      35      69
  max width                 -     -       8      12

Time-resolution, not more cells, and every answer inside the elimination budget.

What it does NOT claim: that the compiler predicts a held-out interventional arm. That was
tried on the CEBPA arm and lost to "nothing changed". CRISPRa drives its target 24.8x past
anything the control cells show, so a kernel fitted on control data answers with the
observational noise-slope. Off-support extrapolation, which no identification machinery
repairs -- and the reason the policy certificate in the parent commit exists.

Data vendored under `demo/data/` (843 KB): derived counts for 23 named genes only, neither
source redistributed. `demo/build_cache.py` regenerates it by streaming the 362M-entry
matrix once. Runs in 2.3s from a clean checkout with no third-party packages.

`tests/test_demo_smoke.py` pins every figure quoted in `demo/README.md` against a real
subprocess run, for the reason `test_readme_smoke.py` exists: three README claims here were
once wrong rather than stale, and a demo is the artefact people run to decide whether to
believe the library. 329 passed, 1 xfailed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@logannye
logannye force-pushed the feat/preflight-demo branch from d17fb3f to 2325c28 Compare August 18, 2026 21:29
@logannye
logannye merged commit fcaf750 into main Aug 18, 2026
4 checks passed
@logannye
logannye deleted the feat/preflight-demo branch August 18, 2026 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant