RL recentering, part 2: causalrl.ope, the real-data suite, and v3.0.0

.github/workflows/ci.yml

@@ -56,7 +56,7 @@ jobs:
         run: uv run pytest --nbmake --no-cov examples/
       - name: Smoke-test CGFA-PPO example
         run: uv run python examples/cgfa_ppo_example.py
-      - name: Run the five 2.0 task guides (§10 acceptance)
+      - name: Run the task guides (five that certify a policy, one that trains one)
         run: |
           for guide in examples/guides/*.py; do
             echo "== $guide =="

CITATION.cff

@@ -1,12 +1,12 @@
 cff-version: 1.2.0
 message: "If you use causalrl in research, please cite this software."
-title: "causalrl: Causal intervention-selection and causal-RL research tools"
+title: "causalrl: Causal reinforcement learning -- the 9-task causal RL taxonomy, made runnable"
 type: software
 authors:
   - family-names: "Coelho"
     given-names: "Raphael"
 repository-code: "https://github.com/raphaelrrcoelho/causalrl"
 url: "https://github.com/raphaelrrcoelho/causalrl"
 license: MIT
-version: "1.0.0"
-date-released: "2026-06-08"
+version: "3.0.0"
+date-released: "2026-08-07"

README.md

@@ -7,15 +7,25 @@
 [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
 [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 
-Causal intervention-selection and causal-RL research tools.
+Causal reinforcement learning: the 9-task causal RL taxonomy, made runnable.
 
-`causalrl` provides graph algorithms for causal bandits, demonstration environments and agents,
-and explicit-latent structural causal models with `see` (L1), `do` (L2), and `counterfactual`
-(L3) queries, organised around the [9-task taxonomy of causal RL](https://crl.causalai.net/).
+`causalrl` supplies **the causal layer for sequential decisions, not a new trainer**: agents that
+plan inside a given or learned structural causal model, graph algorithms for causal bandits,
+demonstration environments, and explicit-latent SCMs with `see` (L1), `do` (L2), and
+`counterfactual` (L3) queries, organised around the
+[9-task taxonomy of causal RL](https://crl.causalai.net/). Train a policy however you like, then
+hand its actions to `certify_policy`: it bounds whether the value improvement over the logging
+policy survives hidden confounding, and abstains when it cannot.
 
-Scope is explicit and enforced in code: out-of-class identification queries raise
+Scope is explicit and enforced in code. Out-of-class identification queries raise
 `NotIdentifiableError` with the witnessing hedge (or return `None` for the conservative helpers)
-rather than guessing a formula, and learning agents are tabular/demo-scale, not production RL. See
+rather than guessing a formula. The two halves also sit at deliberately different maturities: the
+**planners and environments are demo-scale** — tabular to modest function approximation, on
+synthetic worlds built to isolate one failure mode each — while the **decision, certificate and
+off-policy-evaluation layers run on real data**. The `examples/causal_mbrl_*.py` scripts are the
+evidence: on NHEFS, LaLonde and Twins they fit an agent, call `.act()` per unit and certify the
+resulting policy; on the Open Bandit Dataset and Coat they run the off-policy-evaluation and
+sensitivity kernels (`certify_policy`, `msm_policy_value_bounds`) against measured ground truth. See
 [Guarantees & Scope](https://raphaelrrcoelho.github.io/causalrl/guarantees/).
 
 ## Install
@@ -89,6 +99,7 @@ see the [causal-MBRL results note](docs/causal_mbrl_agent/RESULTS.md).
 | Decision under confounding | Counterfactual Thompson sampling on the MABUC | `CausalThompsonSampling` |
 | Confounded offline agent | One front-door → back-door / discovery / transport / function-approx / sequential | `CausalMBRLAgent` |
 | Learned world model | Fit an SCM from logs, then act in it as a Gymnasium env | `fit_scm`, `orient`, `counterfactual_interval` |
+| Learn the model while acting | Refit the I-MEC from the agent's own experiments; Thompson-sample over structure | `OnlineCausalMBRL` |
 | 1 — Offline→online | Learn from confounded logs via causal bounds | `UCDTR`, `DOVI`, `DeepDeconfoundedQ` |
 | 2 — Where to intervene | POMIS / MIS, incl. non-manipulable variables | `pomis`, `minimal_intervention_sets` |
 | 3 — Counterfactual policy | Act on `E[Y_do(a) \| intent]` | `CounterfactualOptimalPolicy` |
@@ -103,6 +114,29 @@ see the [causal-MBRL results note](docs/causal_mbrl_agent/RESULTS.md).
 A runnable example for every row is in the
 [**Tour by Task**](https://raphaelrrcoelho.github.io/causalrl/tour/); end-to-end notebooks are in
 [`examples/`](examples) and the [Tutorials](https://raphaelrrcoelho.github.io/causalrl/tutorials/).
+Six [task guides](https://raphaelrrcoelho.github.io/causalrl/guides/) — five that certify a policy
+and one that trains an agent online — are scripts in [`examples/guides/`](examples/guides) executed
+end to end in CI.
+
+## What the numbers say — a deliberate negative
+
+Scored the way an RL practitioner scores things — `.act()` per unit, then **regret** against ground
+truth — the causal *point estimates* do not win on real data, and the shipped examples print that
+themselves rather than hiding it:
+
+- **Twins** (`examples/causal_mbrl_twins.py`, 11,984 pairs with both potential outcomes): our
+  policy reaches 0.8316 survival against the per-pair oracle's 0.8747 — **regret 0.0431**, the
+  worst of the learned policies, and behind the trivial constant "always the heavier twin"
+  (0.8358).
+- **LaLonde** (`examples/causal_mbrl_lalonde.py`, priced by the NSW randomized experiment): the
+  contextual policy enrols 73.5% of the population for **$476/person of regret**, where the
+  marginal rule it is built from ("enrol everyone") leaves $0 — and its off-policy value from the
+  observational logs, −$453/person, has the wrong sign outright.
+
+Both examples then **abstain**: `certify_policy` refuses Twins at Γ≈1.07 and LaLonde at Γ≈1.10, and
+on LaLonde the randomized experiment vindicates the refusal. That is the recorded finding, and it
+is the positioning — **the defensible edge is the decision and certificate layer, not the number.**
+Full write-up: [real-data results](docs/causal_mbrl_agent/REAL_DATA.md).
 
 ## How it compares
 
@@ -113,12 +147,28 @@ A runnable example for every row is in the
   `causalrl` instead targets *sequential decision-making*: intervention-set selection (POMIS),
   confounded offline-to-online RL, counterfactual policies, and causal curricula / shaping /
   games. Those are the parts of the Bareinboim taxonomy these libraries do not cover.
-- For pure graph identification it overlaps with **Ananke / pgmpy / Y0**. It deliberately does
-  **not** reimplement offline RL at scale; pair it with a dedicated library such as
-  [`d3rlpy`](https://github.com/takuseno/d3rlpy) for that.
+- For pure graph identification it overlaps with **Ananke / pgmpy / Y0**.
+
+On the RL side it is a layer, not a competitor:
+
+- **`d3rlpy` (and offline-RL libraries generally) train the policy**; `causalrl` does not
+  reimplement any of that, and pairs with them instead. `src/causalrl/scale/d3rlpy.py` is the
+  bridge in both directions — `to_mdp_dataset` hands a `ConfoundedTrajectoryDataset` to a d3rlpy
+  algorithm, `policy_actions` reads the trained policy's greedy actions back, and `certify_fqe`
+  wraps a fitted-Q evaluation as a certificate. `pip install causalrl[scale]`; see the
+  [Scale guide](https://raphaelrrcoelho.github.io/causalrl/scale/).
+- **What it adds on top of a trained policy** is the assumption those libraries' evaluators take
+  for granted. Off-policy evaluation — importance sampling, doubly-robust, FQE — is valid only when
+  the logged actions are unconfounded given the recorded state, and logs written by a human, a
+  clinician or a legacy heuristic often are not. `certify_policy` bounds the value improvement over
+  the logging policy under Tan's marginal sensitivity model, reports the **tipping Γ** at which the
+  ship/keep decision flips, and with `alpha=…` gates it on a finite-sample conformal lower bound
+  (`conformal_action_value`). When the bound will not carry the decision, its `recommendation` is
+  `abstain` rather than a green light.
 
 Use `causalrl` when your problem is a causal *decision* over time; use DoWhy/EconML when it is a
-treatment-effect *estimate*.
+treatment-effect *estimate*; use d3rlpy when you need the policy *trained*, and `causalrl` to
+decide whether to ship it.
 
 ## Stability
 

benchmarks/bench_causal_core.py

@@ -18,7 +18,7 @@
 import numpy as np
 from torch.distributions import Normal
 
-from causalrl.identification.bounds import ipw_sensitivity_bounds
+from causalrl.ope.bounds import ipw_sensitivity_bounds
 from causalrl.scm.graph import CausalGraph
 from causalrl.scm.mechanisms import LinearGaussianMechanism
 from causalrl.scm.scm import StructuralCausalModel

benchmarks/bench_streaming.py

@@ -6,9 +6,7 @@
   (1) The streaming accumulators (``StreamingMoments`` / ``WeightedStreamingRatio``) reproduce the
       one-shot NumPy statistics EXACTLY, and run within a conservative constant factor of a single
       vectorised pass (hard-fail on a >2x slowdown vs the recorded relative floor).
-  (2) The Greenwald-Khanna sketch answers quantiles within its ε rank-error bound in sub-linear
-      space over a large stream.
-  (3) The end-to-end streamed OPE certificate (``stream_policy_value`` over a columnar log) equals
+  (2) The end-to-end streamed OPE certificate (``stream_policy_value`` over a columnar log) equals
       the materialised Hájek estimate — correctness never regresses with scale.
 
 The shipped closed-form guards (the 874x MSM speedup, exact known-noise counterfactual) live in
@@ -21,10 +19,9 @@
 
 import numpy as np
 
-from causalrl.backends.quantile_sketch import GKQuantileSketch
 from causalrl.backends.streaming import StreamingMoments, WeightedStreamingRatio
 from causalrl.data.trajectory import TrajectoryLog
-from causalrl.estimate.streaming import stream_policy_value
+from causalrl.ope.ipw import stream_policy_value
 
 # Conservative relative floor: streaming may be at most this many times a single vectorised pass.
 # Set well above the observed ~1-2x so only a genuine >2x regression trips it (not machine noise).
@@ -98,28 +95,6 @@ def one_shot() -> tuple[float, float]:
     assert ratio < _MAX_SLOWDOWN, f"ratio streaming regressed: {ratio:.1f}x > {_MAX_SLOWDOWN}x"
 
 
-def bench_quantile_sketch() -> None:
-    """GK sketch over a large stream: quantiles within ε rank error, sub-linear space."""
-    rng = np.random.default_rng(2)
-    x = rng.standard_normal(2_000_000)
-    eps = 0.01
-    t_build, sketch = _timed(lambda: GKQuantileSketch(eps).update(x), repeat=1)
-    sk: GKQuantileSketch = sketch  # type: ignore[assignment]
-    srt = np.sort(x)
-    n = x.shape[0]
-    worst = 0.0
-    for q in (0.01, 0.1, 0.5, 0.9, 0.99):
-        rank = int(np.searchsorted(srt, sk.quantile(q), side="right"))
-        worst = max(worst, abs(rank - q * n) / n)
-    thru = n / t_build / 1e6
-    print(
-        f"[sketch]  {n:,} in {t_build * 1e3:.0f} ms ({thru:.2f}M/s); {len(sk._entries)} entries "
-        f"(vs {n:,}); worst rank-error {worst:.4f} (ε={eps})"
-    )
-    assert worst <= eps + 1e-3, f"sketch rank error {worst} exceeds ε={eps}"
-    assert len(sk._entries) < n / 100, "sketch space is not sub-linear"
-
-
 def bench_stream_policy_value_end_to_end() -> None:
     """End-to-end streamed OPE certificate over a columnar log == the materialised Hájek value."""
     rng = np.random.default_rng(3)
@@ -150,7 +125,6 @@ def main() -> None:
     print("causal-core streaming micro-benchmarks (Phase 3 §9 scale path)")
     bench_streaming_moments()
     bench_weighted_ratio()
-    bench_quantile_sketch()
     bench_stream_policy_value_end_to_end()
     print("OK — all streaming bench assertions passed")