From 749ffbec2eee23e2577e9e6987c7f5b1270fbd20 Mon Sep 17 00:00:00 2001 From: Logan Nye <87274608+logannye@users.noreply.github.com> Date: Tue, 18 Aug 2026 16:14:41 -0700 Subject: [PATCH] A covariate check that cannot report is worse than no covariate check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `check_covariates` decided path-opening as separated_alone = d_separated(back_door, target, outcome) # computed ONCE opens = separated_alone and not d_separated(back_door, target, outcome, (name,)) so whenever the target and outcome were ALREADY d-connected in the back-door graph -- which is what an unmeasured common cause produces, and therefore the normal case in any real system -- `opens` was False for every covariate and the report read Admissible: ['A', 'B', 'C', 'W'] with both finding sections empty. Three defects at once, and the second is what makes this more than cosmetic: 1. Nothing distinguished "tested and clean" from "the test could not fire". 2. A collider the module flags correctly on a clean graph goes UNFLAGGED the moment a confounder exists anywhere. The finding is not merely unstated, it is LOST. 3. `W` is unobserved. Covariates were validated against `variable_set`, never `observed_set`, so the report advised conditioning on a latent variable. The module's own docstring warned about exactly this shape for the name-clash case -- "a check that runs, passes, and cannot report -- every covariate would come back admissible for the wrong reason" -- and the guard was installed there and not here. ★ The repair is not a warning label. Path opening asks whether conditioning BREAKS a separation; when nothing was separated there is nothing to break, and the question with an answer is whether the covariate CLOSES the open path -- which is the entire purpose of an adjustment set, and which a check that only ever asks "does this open a path?" cannot express. So the previously-vacuous branch now answers rather than merely refusing. back_door_open graph-level finding, reported once, leading the summary blocks_path names the covariates that close the open path undecided no path-level verdict was reachable; never admissible observed unobserved is a STRUCTURAL refusal, not an undecided one, so the report does not file a hard fact under an "unanswerable" heading ⚠️ FOUND WHILE MUTATION-TESTING, and unrelated to the above: the O(n^2) gate on `mechanism_dependencies` had stopped gating anything. It timed CONSTRUCTION, on the stated grounds that "every MechanismGraph validates acyclicity at construction" -- true when written, and PR #9 made it false by lifting C1 to a query-time condition. Restoring the pairwise implementation left construction at 0.0038s/0.0154s, unchanged, while `mechanism_dependencies` itself went 0.0006s -> 0.2764s at n=1500. The gate is now timed on the routine (mutation-tested: fires, 28s vs 0.08s), and a companion test COUNTS the calls to pin the premise, because a timing proxy there would be the same mistake one layer down. That gate also took min-of-5 after a warm-up: a single 4ms sample was dominated by whatever ran before it, and it fired when this branch merely ADDED a test file. Five mutations on the covariate repair, all dead, restored and verified by content: path_test_applicable := True (the bug reinstated), admissible := True in the open branch, drop the observed check, back_door_open := False, closes := False. README gains the third finding with its real output, gated by a new `test_readme_smoke` case including `Closes it: ['donor']`. 340 passed, 1 xfailed. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 22 +++ .../identification/covariates.py | 137 ++++++++++++- tests/test_covariate_vacuity.py | 185 ++++++++++++++++++ tests/test_mechanism_graph.py | 94 +++++++-- tests/test_readme_smoke.py | 37 ++++ 5 files changed, 453 insertions(+), 22 deletions(-) create mode 100644 tests/test_covariate_vacuity.py diff --git a/README.md b/README.md index a917c6c..5363528 100644 --- a/README.md +++ b/README.md @@ -370,6 +370,28 @@ sound but complete only under faithfulness, it is a warning rather than a proof `admissible` means only that neither failure mode was detected. It is not a certificate that adjusting for the covariate yields an unbiased estimate. +There is a third case, and getting it wrong is how a check becomes decoration. Path opening +asks whether conditioning *breaks* a separation that held — so when the target and the +outcome are already d-connected in the back-door graph, which is what any unmeasured common +cause produces, there is no separation to break and the question has no answer. The report +says so rather than reporting silence as a clean bill of health: + +```text +Conditioning around do(knockdown) with outcome 'IFNG': + + 'knockdown' and 'IFNG' are d-connected in the back-door graph given nothing: + there is an open back-door path, so the effect is confounded before any covariate is chosen. + Closes it: ['donor'] + For the rest, no path-level verdict is reachable -- reported as undecided, not as clean. +``` + +`report.back_door_open` carries the graph-level finding, `report.blocks_path` names the +covariates that *close* the open path — the reason adjustment sets exist, and something a +check that only ever asks "does this open a path?" cannot express — and `report.undecided` +names the rest. A covariate whose verdict was not reachable is never reported as admissible. +A covariate that is not in `observed_set` is never admissible either, however well it does +in the graph: the graph does not know what the assay measured. + Both findings are checked against exact ground truth over 120 generated models — 2,596 verdicts. Post-treatment is structural, so it is verified in *both* directions against a reachability walk written independently of the library's. Path-opening is verified in the diff --git a/src/causal_hypergraphs/identification/covariates.py b/src/causal_hypergraphs/identification/covariates.py index 554ca43..cfa5ede 100644 --- a/src/causal_hypergraphs/identification/covariates.py +++ b/src/causal_hypergraphs/identification/covariates.py @@ -42,6 +42,30 @@ class CovariateVerdict: post_treatment: bool opens_path: bool reason: str + blocks_path: bool = False + """Conditioning on it *closes* a back-door path that was open. + + The positive finding, and the reason adjustment sets exist. Only reachable when the + back-door path is open to begin with, which is why the old code -- which asked solely + whether a covariate opens a path -- could not express it. + """ + observed: bool = True + """Whether the covariate is in the model's observed set. + + Unobserved is a *structural* refusal, not an undecided one -- you cannot condition on + what was not measured, and no further evidence would change that. It shares + `path_test_applicable=False` with the undecided case because neither reached a path + verdict, which is what that flag is for; the two are separated here so the report does + not file a hard fact under a heading that says the question was unanswerable. + """ + path_test_applicable: bool = True + """Whether a path-level verdict was actually reached for this covariate. + + False means the question could not be decided here, not that it was decided + favourably, and `admissible` is never True alongside it. Distinguishing the two is the + whole repair: a check that runs, passes, and cannot report is worse than no check, + because it is read as a clean bill of health. + """ def __str__(self) -> str: mark = "ok " if self.admissible else "!! " @@ -55,11 +79,32 @@ class CovariateReport: target: str outcome: str verdicts: tuple[CovariateVerdict, ...] + back_door_open: bool = False + """Whether target and outcome are d-connected in the back-door graph given nothing. + + A fact about the *query*, not about any covariate, and the dominant one when true: + some non-causal path is open, so the effect is confounded until something closes it. + It is reported on the report rather than repeated per covariate because that is where + it belongs, and because leaving it unsaid is what made the old output read as clean. + """ @property def admissible(self) -> tuple[str, ...]: return tuple(v.covariate for v in self.verdicts if v.admissible) + @property + def blocks_path(self) -> tuple[str, ...]: + return tuple(v.covariate for v in self.verdicts if v.blocks_path) + + @property + def undecided(self) -> tuple[str, ...]: + """Covariates for which no path-level verdict was reached.""" + return tuple( + v.covariate + for v in self.verdicts + if not v.path_test_applicable and not v.post_treatment and v.observed + ) + @property def post_treatment(self) -> tuple[str, ...]: return tuple(v.covariate for v in self.verdicts if v.post_treatment) @@ -71,11 +116,31 @@ def opens_path(self) -> tuple[str, ...]: def summary(self) -> str: lines = [ f"Conditioning around do({self.target}) with outcome {self.outcome!r}:", + ] + if self.back_door_open: + closers = list(self.blocks_path) + lines.extend([ + "", + f" {self.target!r} and {self.outcome!r} are d-connected in the back-door " + "graph given nothing:", + " there is an open back-door path, so the effect is confounded before any " + "covariate is chosen.", + f" Closes it: {closers}" if closers + else " No candidate offered closes it.", + " For the rest, no path-level verdict is reachable -- reported as " + "undecided, not as clean.", + ]) + lines.extend([ "", " Structural -- post-treatment, no distributional assumption involved:", - ] + ]) blocked = [v for v in self.verdicts if v.post_treatment or not v.admissible] - structural = [v for v in blocked if v.post_treatment or not v.opens_path] + undecided = set(self.undecided) + structural = [ + v + for v in blocked + if (v.post_treatment or not v.opens_path) and v.covariate not in undecided + ] lines.extend(f" {v}" for v in structural) if structural else lines.append( " (none)" ) @@ -86,6 +151,13 @@ def summary(self) -> str: ) warnings = [v for v in self.verdicts if v.opens_path and not v.post_treatment] lines.extend(f" {v}" for v in warnings) if warnings else lines.append(" (none)") + if self.undecided: + lines.append("") + lines.append( + " Undecided -- the path test could not reach a verdict for these; this is " + "not a clean result:" + ) + lines.extend(f" {v}" for v in self.verdicts if v.covariate in self.undecided) lines.append("") lines.append(f" Admissible: {list(self.admissible) or '(none)'}") return "\n".join(lines) @@ -207,7 +279,59 @@ def check_covariates( ) continue - opens = separated_alone and not d_separated(back_door, target, outcome, (name,)) + if name not in graph.observed_set: + # A latent variable can be the graphically ideal covariate -- W below closes + # the back-door path perfectly -- and you still cannot condition on it. The + # graph does not know what the assay measured, so this is checked separately + # rather than inferred. + verdicts.append( + CovariateVerdict( + covariate=name, + admissible=False, + post_treatment=False, + opens_path=False, + observed=False, + path_test_applicable=False, + reason=( + "not observed in this model, so it cannot be conditioned on " + "whatever the graph says about it." + ), + ) + ) + continue + + given = d_separated(back_door, target, outcome, (name,)) + if not separated_alone: + # The back-door path is already open, so "does conditioning OPEN one" has no + # verdict to give -- the old code silently answered "no" here and called the + # covariate admissible. The answerable question is whether it CLOSES the path. + closes = given + verdicts.append( + CovariateVerdict( + covariate=name, + admissible=closes, + post_treatment=False, + opens_path=False, + blocks_path=closes, + path_test_applicable=closes, + reason=( + f"closes an open back-door path: conditioning on it d-separates " + f"{target!r} from {outcome!r} in the graph with {target!r}'s " + "outgoing edges severed. Rests on faithfulness in the same way the " + "opening warning does." + ) + if closes + else ( + f"undecided: {target!r} and {outcome!r} are already d-connected in " + "the back-door graph, so no covariate can be cleared of opening a " + "path here, and this one does not close the open path either. Not " + "a clean result -- the question was not answerable, not answered." + ), + ) + ) + continue + + opens = not given verdicts.append( CovariateVerdict( covariate=name, @@ -228,4 +352,9 @@ def check_covariates( ) ) - return CovariateReport(target=target, outcome=outcome, verdicts=tuple(verdicts)) + return CovariateReport( + target=target, + outcome=outcome, + verdicts=tuple(verdicts), + back_door_open=not separated_alone, + ) diff --git a/tests/test_covariate_vacuity.py b/tests/test_covariate_vacuity.py new file mode 100644 index 0000000..af02614 --- /dev/null +++ b/tests/test_covariate_vacuity.py @@ -0,0 +1,185 @@ +"""A check that runs, passes, and cannot report is the failure this module names. + +`check_covariates` decided path-opening as + + separated_alone = d_separated(back_door, target, outcome) # computed once + opens = separated_alone and not d_separated(back_door, target, outcome, (name,)) + +so whenever the target and outcome were *already* d-connected in the back-door graph -- +which is what an unmeasured common cause produces, and therefore the normal case in any +real system -- `opens` was False for every covariate and the report read + + Admissible: ['A', 'B', 'C', 'W'] + +with both finding sections empty. Three things were wrong at once, and the second is the +one that makes it more than cosmetic: + +1. Nothing distinguished "tested and clean" from "the test could not fire". +2. A collider the module flags correctly on a clean graph goes **unflagged** the moment a + confounder exists. The finding is not merely unstated, it is lost. +3. `W` is *unobserved*. Covariates were validated against `graph.variable_set`, never + `graph.observed_set`, so the report advised conditioning on a latent variable. + +The module's own docstring warned about exactly this shape for the name-clash case -- +"skipping the path-opening test would leave a check that runs, passes, and cannot report +-- every covariate would come back admissible for the wrong reason" -- and the guard was +installed there and not here. +""" + +from __future__ import annotations + +import pytest + +from causal_hypergraphs import DeleteMechanism, MechanismGraph, check_covariates + +QUERY = DeleteMechanism("m_T", outcomes=("Y",)) +BASE = { + "m_A": {"inputs": (), "outputs": ("A",)}, + "m_B": {"inputs": (), "outputs": ("B",)}, + "m_C": {"inputs": ("A", "B"), "outputs": ("C",)}, +} + + +def clean_graph() -> MechanismGraph: + """A -> m_T, and a collider C with parents A and B; no open back-door path.""" + return MechanismGraph( + variables={"A", "B", "C", "T", "Y"}, + mechanisms={ + **BASE, + "m_T": {"inputs": ("A",), "outputs": ("T",)}, + "m_Y": {"inputs": ("T", "B"), "outputs": ("Y",)}, + }, + ) + + +def confounded_graph(*, w_observed: bool) -> MechanismGraph: + """The same graph plus W, a common cause of the target and the outcome.""" + observed = {"A", "B", "C", "T", "Y"} | ({"W"} if w_observed else set()) + return MechanismGraph( + variables={"A", "B", "C", "T", "Y", "W"}, + mechanisms={ + **BASE, + "m_W": {"inputs": (), "outputs": ("W",)}, + "m_T": {"inputs": ("A", "W"), "outputs": ("T",)}, + "m_Y": {"inputs": ("T", "B", "W"), "outputs": ("Y",)}, + }, + observed_variables=observed, + ) + + +def verdict(report, name: str): + (found,) = [v for v in report.verdicts if v.covariate == name] + return found + + +def test_the_collider_is_still_flagged_when_the_back_door_is_clean() -> None: + """The behaviour that was already right, pinned so the repair cannot cost it.""" + report = check_covariates(clean_graph(), QUERY, "Y", ("A", "B", "C")) + + assert not report.back_door_open + assert report.opens_path == ("C",) + assert report.admissible == ("A", "B") + + +def test_a_confounder_no_longer_hides_the_collider() -> None: + """Defect 2, and the reason this is not cosmetic: the finding was LOST, not unstated. + + `C` is the same collider as above. Adding an unmeasured common cause elsewhere in the + graph must not silence a verdict about `C`. + """ + report = check_covariates(confounded_graph(w_observed=False), QUERY, "Y", ("A", "B", "C")) + + assert report.back_door_open + assert "C" not in report.admissible + + +def test_an_open_back_door_is_reported_and_not_read_as_clean() -> None: + """Defect 1. `admissible` must not be asserted on a test that could not run.""" + report = check_covariates(confounded_graph(w_observed=False), QUERY, "Y", ("A", "B", "C")) + + assert report.back_door_open + for name in ("A", "B", "C"): + found = verdict(report, name) + assert not found.path_test_applicable + assert not found.admissible + assert report.admissible == () + text = report.summary() + assert "open back-door path" in text + assert "Admissible: (none)" in text or "Admissible: []" in text + + +def test_a_covariate_that_closes_the_back_door_is_named() -> None: + """The previously-vacuous branch now answers the question a user actually has. + + `W` is the confounder. Conditioning on it *closes* the open back-door path, which is + the entire purpose of covariate adjustment -- and the old code, which only ever asked + whether a covariate OPENS a path, could not express it. + """ + report = check_covariates(confounded_graph(w_observed=True), QUERY, "Y", ("A", "B", "C", "W")) + + assert report.back_door_open + assert report.blocks_path == ("W",) + assert verdict(report, "W").admissible + assert not verdict(report, "A").admissible + assert "closes" in verdict(report, "W").reason + + +def test_an_unobserved_covariate_is_never_admissible() -> None: + """Defect 3. `W` is latent here; you cannot condition on what you did not measure. + + It closes the back-door path in the graph, so a purely graphical check calls it the + ideal covariate. That is precisely why observation has to be checked separately: the + graph cannot tell you what the assay measured. + """ + report = check_covariates(confounded_graph(w_observed=False), QUERY, "Y", ("W",)) + + found = verdict(report, "W") + assert not found.admissible + assert not found.path_test_applicable + assert "not observed" in found.reason + assert report.blocks_path == () + + +def test_post_treatment_still_outranks_everything() -> None: + """A descendant of the intervention is refused whatever the back-door graph says.""" + graph = MechanismGraph( + variables={"A", "B", "C", "T", "Y", "W", "M"}, + mechanisms={ + **BASE, + "m_W": {"inputs": (), "outputs": ("W",)}, + "m_T": {"inputs": ("A", "W"), "outputs": ("T",)}, + "m_M": {"inputs": ("T",), "outputs": ("M",)}, + "m_Y": {"inputs": ("T", "B", "W"), "outputs": ("Y",)}, + }, + observed_variables={"A", "B", "C", "T", "Y", "M"}, + ) + report = check_covariates(graph, DeleteMechanism("m_T", outcomes=("Y",)), "Y", ("M",)) + + found = verdict(report, "M") + assert found.post_treatment + assert not found.admissible + assert "Structural" in found.reason + + +def test_the_report_leads_with_the_graph_level_finding() -> None: + """An open back-door is a fact about the query, not about any one covariate.""" + confounded = check_covariates(confounded_graph(w_observed=True), QUERY, "Y", ("A", "W")) + clean = check_covariates(clean_graph(), QUERY, "Y", ("A", "B")) + + assert "open back-door path" in confounded.summary() + assert "open back-door path" not in clean.summary() + # And the closing covariate is surfaced, not buried in the per-covariate reasons. + assert "W" in confounded.summary() + + +@pytest.mark.parametrize("w_observed", [True, False]) +def test_admissible_never_includes_a_covariate_whose_test_could_not_run( + w_observed: bool, +) -> None: + """The invariant the whole repair exists to establish, stated once, directly.""" + report = check_covariates( + confounded_graph(w_observed=w_observed), QUERY, "Y", ("A", "B", "C", "W") + ) + for found in report.verdicts: + if not found.path_test_applicable: + assert not found.admissible, found.covariate diff --git a/tests/test_mechanism_graph.py b/tests/test_mechanism_graph.py index 86f8586..2deb725 100644 --- a/tests/test_mechanism_graph.py +++ b/tests/test_mechanism_graph.py @@ -70,34 +70,92 @@ def test_hidden_and_fallback_partitions_are_explicit() -> None: def test_the_dependency_graph_is_built_by_index_not_by_comparing_every_pair() -> None: - """Construction must scale with edges, not with mechanisms squared. - - Every `MechanismGraph` validates acyclicity at construction, so the cost of building - the dependency graph is the cost of *loading* a network. Comparing all pairs makes a - genome-scale graph take tens of seconds before a single query is asked -- which would - leave the whole affordable-query story true of one module and false of the tool. - - Measured as a growth ratio rather than a wall-clock budget: quadratic construction - costs about 16x for a 4x larger network, linear about 4x. The gate sits between them - with room for a noisy machine on either side, and it fires against the pairwise - implementation this replaced. + """`mechanism_dependencies` must scale with incidences, not with mechanisms squared. + + Comparing all pairs makes a genome-scale graph take tens of seconds before a single + query is asked, which would leave the whole affordable-query story true of one module + and false of the tool. + + **Timed on the routine, not on construction, and that is the repair.** This gate used + to build a `MechanismGraph` and time *that*, on the stated grounds that "every + MechanismGraph validates acyclicity at construction, so the cost of building the + dependency graph is the cost of loading a network". That was true when it was written + and PR #9 made it false: lifting C1 to a query-time condition means construction no + longer walks the dependency graph at all. The gate went on passing and stopped + measuring anything -- restoring the pairwise implementation left construction at + 0.0038s/0.0154s, unchanged, while `mechanism_dependencies` itself went from 0.0006s to + 0.2764s at n=1500. A gate whose premise a later change invalidated is worse than no + gate, because the green tick is read as coverage. + + Measured as a growth ratio rather than a wall-clock budget: quadratic costs about 16x + for a 4x larger network, linear about 4x, and the gate sits between them with room for + a noisy machine on either side. """ import time - def build(count: int) -> float: + def cost(count: int) -> float: + """The best of several runs, after a warm-up. + + The minimum, not the mean: scheduling noise, allocator state and a cold cache only + ever *add* time, so the fastest run is the closest estimate of the cost being + measured. A single sample at this size is dominated by whatever ran before it in + the session, which made an earlier form of this gate fire when an unrelated test + file was added -- reporting a scaling regression that had not happened. + """ specs = { f"m{i}": {"inputs": (f"g{i}", f"g{i + 1}"), "outputs": (f"g{i + 2}",)} for i in range(count) } variables = {f"g{i}" for i in range(count + 3)} - start = time.perf_counter() - MechanismGraph(variables=variables, mechanisms=specs) - return time.perf_counter() - start + graph = MechanismGraph(variables=variables, mechanisms=specs) + graph.mechanism_dependencies() # warm-up, not timed + best = float("inf") + for _ in range(5): + start = time.perf_counter() + graph.mechanism_dependencies() + best = min(best, time.perf_counter() - start) + return best + + small = cost(1_500) + large = cost(6_000) + + assert large / small < 8.0, f"{small:.4f}s -> {large:.4f}s is {large / small:.1f}x for 4x" + - small = build(1_500) - large = build(6_000) +def test_construction_no_longer_walks_the_dependency_graph() -> None: + """Pins the fact that invalidated the gate above, so it cannot silently come back. + + Since PR #9 acyclicity is a *query* condition, not a construction veto. If some later + change reinstates a construction-time walk, the cost story changes and the gate above + is measuring the wrong thing again -- so the premise is asserted rather than assumed. + + Counted, not timed. A timing proxy here would be the same mistake one layer down: it + would pass whenever the machine happened to be fast, and it is the *call* that matters. + """ + original = MechanismGraph.mechanism_dependencies + calls = 0 + + def counting(self: MechanismGraph) -> dict[str, set[str]]: + nonlocal calls + calls += 1 + return original(self) + + MechanismGraph.mechanism_dependencies = counting # type: ignore[method-assign] + try: + graph = MechanismGraph( + variables={"a", "b", "c"}, + mechanisms={ + "m1": {"inputs": ("a",), "outputs": ("b",)}, + "m2": {"inputs": ("b",), "outputs": ("c",)}, + }, + ) + assert calls == 0, f"construction walked the dependency graph {calls} time(s)" - assert large / small < 8.0, f"{small:.3f}s -> {large:.3f}s is {large / small:.1f}x for 4x" + # And it is still reachable on demand -- the walk moved, it did not disappear. + graph.is_mechanism_acyclic() + assert calls == 1 + finally: + MechanismGraph.mechanism_dependencies = original # type: ignore[method-assign] def test_the_dependency_graph_still_names_the_right_successors() -> None: diff --git a/tests/test_readme_smoke.py b/tests/test_readme_smoke.py index 4c6ba9a..abd66ab 100644 --- a/tests/test_readme_smoke.py +++ b/tests/test_readme_smoke.py @@ -210,3 +210,40 @@ def test_readme_empty_stratum_block_reports_the_counts_it_prints() -> None: assert "! P(F | C,E) undefined at C=1, E=1 (16 point(s) unreachable)" in summary # Absent, never nan: the affected points are not in `values` at all. assert (0, 1, 1, 0, 1, 1) not in est.values + + +def test_readme_open_back_door_block_prints_what_the_readme_shows() -> None: + """The third covariate finding, gated like the two the README already showed. + + The README block is a claim about output, and the last three claims it made about this + module's output were wrong rather than stale. This one is checked line for line -- + including `Closes it: ['donor']`, which is the whole point of the section: a check that + only asks "does conditioning OPEN a path?" cannot name the covariate that closes one. + """ + graph = MechanismGraph( + variables={"donor", "stim", "batch", "TF", "exhaustion_marker", "IFNG"}, + mechanisms={ + # `donor` now feeds the outcome as well as the target: an open back-door path. + "knockdown": {"inputs": {"donor", "stim"}, "outputs": {"TF"}}, + "m_marker": {"inputs": {"TF"}, "outputs": {"exhaustion_marker"}}, + "m_ifng": {"inputs": {"TF", "batch", "donor"}, "outputs": {"IFNG"}}, + }, + ) + + report = check_covariates( + graph, + DeleteMechanism("knockdown"), + "IFNG", + ["donor", "stim", "exhaustion_marker", "batch"], + ) + summary = report.summary() + + assert report.back_door_open + assert report.blocks_path == ("donor",) + assert "there is an open back-door path, so the effect is confounded before any" in summary + assert "Closes it: ['donor']" in summary + assert "reported as undecided, not as clean" in summary + # And the invariant the whole repair exists for, asserted on the README's own example. + for verdict in report.verdicts: + if not verdict.path_test_applicable: + assert not verdict.admissible, verdict.covariate