Causal MBRL agent: confounded/offline/transfer agent (M0–M3) + from_causal_gym seam - #28
Merged
Merged
Conversation
triton 3.7.0's wheel is broken against torch 2.12+cu130 (AttributeError: module 'triton' has no attribute 'language' at torch._dynamo import time), and nothing in causalrl or its tests uses torch.compile. The impossible marker drops it from resolution, so uv sync / uv run never reinstall it (previously required a manual 'uv pip uninstall triton' after every sync). pip end users are unaffected: overrides are uv-only and never reach the published wheel metadata.
…, progress map THEORY.md states and proves the trichotomy for the shipped classes: T1 (E-stability <=> learning do() equals equilibrium do(), linear class, both directions, with the CCM/selection caveat), T2 (finite-time measured-regret CCE containment + degeneracy and asymptotic corollaries), T3 (certified-failure characterization + the certificate ladder table across both instruments). Experiments E1-E5 live in experiments/eqcf/ (untracked by repo convention) with results recorded in experiments/eqcf/RESULTS.md. equilibrium_counterfactuals/ is excluded from the published docs site like the other research-notes directories.
…ntation plan Local research branch only (experiments/ is gitignored on main by convention). E1 cobweb ladder (incl. Hommes adaptive-gain chaos, Lyapunov +0.22), E2 SAF chaos (Nash misses by 0.037, measured-eps BOUNDED interval catches), E3 Q-learners violate the degenerate exact-CCE bounds under intervention (eps_T 0.31-0.76 vs Hedge 0.034) with the finite-time interval absorbing it, E4 certified policy sign-flip at the Bullard-Mitra threshold, E5 jax garnish (env-gated). Numbers recorded in RESULTS.md from the actual runs.
…E6/E7/E3b
THEORY.md gains the unified identification trichotomy (Main Theorem: POINT under stability or
CCE degeneracy / SET at measured regret / NONE with diagnostics, all decidable by shipped
instruments) and the hyperbolic local T1'. Probes: E6 memory-1 Q-learners collude (welfare 35.9
vs the game's DEGENERATE exact-CCE welfare point {32}; measured eps_T ~ 1.94 = temptation x
collusive mass — a collusion meter); E7 both basins locally certified (+0.9 margins) yet
do(u=0.2) leaves the population mixture mean at +0.26 vs equilibrium-tracked +1.20 (selection
gap 0.94, the CCM caveat measured); E3b (GPU, 512 replicate populations) policy-gradient
learners stay exactly no-regret/Nash even with memory — the collusion driver is value
bootstrapping, not memory. Results tables updated with real run numbers.
…ected claims, venue verdict
ADVERSARIAL_REVIEW.md records the red-team pass. Confirmed by machine-checked weak-duality
certificates (slack ~1e-14): E6's welfare CCE {32} and profit CCE [12,20] are exact, explained by
three grid-induced pure Nash on the Q=8 anti-diagonal (disclosure added: degeneracy partly a
discretization artifact; the 35.9 escape is not). Fixed: Main Theorem exclusivity (bistable
systems satisfied two cases — now classifies the global query, case 1 requires unique stable
sigma-solution); E3b reframed (myopic-by-construction PG isolates state from farsightedness;
horizon confound and the open farsighted-PG cell named); E4's threshold relabeled Taylor-type
(not Bullard-Mitra; recovers phi>1 only as beta->1); E2's chaos label softened to non-convergent
learning; eps_T-as-collusion-meter qualified (signature = eps + concentrated supra-competitive
play). Venue verdict: CLeaR nearest (2-4 weeks), EC contingent on the matched-horizon 2x2 +
seed/grid robustness, JEDC months out.
…rapping; E7 dispersion curve E8 (matched 2M-step horizon, GPU, R=64/cell): farsighted (gamma=0.95, truncated MC returns, no critic) memory-1 policy-gradient populations do NOT collude — 64/64 land on the point-identified welfare 32 with median eps_T 0.005 — while the bootstrapped memory-1 Q anchor colludes 3/3 (welfare 34.8-35.9, eps_T ~1.9). The folk-theorem prediction for the open cell is falsified in the interesting direction: state + motive is not sufficient; the driver narrows to value bootstrapping, matching the Q-vs-PG asymmetry in the algorithmic-collusion literature. E7 gains the dispersion-robustness curve (crossing mass 16.4%->1.4% over spreads 0.25-3.0, gap to the tracked root persistent throughout). Adversarial-review item B1 resolved; results and progress updated.
… 19pp compiled When Can You Trust an Equilibrium Counterfactual? A certified identification trichotomy for interventions on learning systems. - Main Theorem restated with explicit regularity Assumption (R) for the nonlinear class (sharper than THEORY.md 0.5, which it supersedes); decidability scope stated honestly (guarded, not decided, for (R2)-(R3)). - T1 + equations-vs-maps corollary; T1' with full noise-condition proofs (appendices A/B: martingale assumptions, Kushner-Yin limit-set, Pemantle/Brandiere-Duflo converse, excitation condition A4). - T2 kept to the set-identification rung with the provenance paragraph crediting the four adjacent literatures; demarcation/collusion material deferred to the EC-shaped companion per the paper-split decision. - Experiments E1/E2/E4/E7 with all ADVERSARIAL_REVIEW.md honesty items baked in; every LP value duality-verified; figs regenerated deterministically (paper_figs.py, E7 numbers reproduced exactly). - Compiles clean: latexmk, 0 errors, 0 undefined refs, 19 pages. - Pre-submission TODOs in the main.tex header (CLeaR style swap, anonymization, owner's residual reading checks).
…endix Why the library appears at all: the operational claim (every trichotomy condition is decidable by released instruments) is a core contribution and CLeaR values the artifact. But naming causalrl and its API throughout the body was (a) a double-blind leak (API names are googleable), (b) an invitation to read a theory paper as a software paper, and (c) against the standing strategy of keeping the library's identity out of the paper wave (the MLOSS/JOSS library paper is its own later artifact). Body now says 'instruments we release as an open-source artifact' once; Table 1 uses generic instrument names (comparator / game certifier); all concrete API names live in Appendix E's reproducibility note, to be swapped for an anonymized artifact link at submission. Also untracks the accidentally-committed experiments/eqcf/__pycache__ files. Rebuilt clean: 0 errors, 0 undefined refs, 19 pages.
Appendix E's real API function names (googleable straight to the public package) replaced by generic instrument descriptions plus an anonymized code-artifact statement; author footnote standardized. Zero library names, zero API identifiers anywhere in the document (PDF Author metadata already empty). Remaining step at submission is attaching the artifact itself. Rebuilt clean: 0 errors, 0 undefined refs, 19 pages.
…ifact, reading checks folded in Style: typeset under the jmlr class [pmlr] mode (CLeaR proceedings are PMLR volumes; per-year CLeaR wrapper swaps in when the 2027 kit is released). jmlr.cls/jmlrutils.sty/algorithm2e.sty vendored. 22pp, 0 errors, 0 undefined refs. Page budget flagged: main text ends p.15 vs the historical 12pp limit (trim candidates in the header). Artifact: paper/artifact/ + artifact.zip — eqcert/ is the minimal 24-file closure of the four experiments (package renamed, CausalRLError->EqcertError, identification engine reduced to a Domain stub), identity-scan clean; all four experiments reproduce the paper's numbers standalone. Reading checks (web research, three agents; results in prose+bib): - 'Mishra-Fox 2024' corrected to Mishra, Fox & Wooldridge, 'Characterising Interventions in Causal Games', UAI 2024, PMLR 244:2560-2572. - Dogra = FRBNY Staff Report 1093 (2024); Magnolfi-Roncoroni assumption is BCE, not no-regret (prose fixed); STZ still a working paper; P-R JACM 55(3) optimize-over-CE NP-hardness confirmed in-paper; all bib TODOs cleared. - Hammond et al. read in full: Def. 21 already gives model-theoretic set-valued interventional queries over equilibrium sets; Section 4 delta re-scoped to the identification layer; taxonomy mapping added. - EC 2024-26 scan surfaced the nearest neighbor: Lomys-Magnolfi (EC 2025), 'Estimation of Games under No Regret'. T2 delta re-anchored on measured realized regret / intervened-game containment / abstention / dual prices. New citations: Lomys-Magnolfi, Kline-Tamer, Hartline survey + audits, Weinberger. Deltas B (trichotomy) and C (E-stability bridge) scanned clear.
Five trim rounds bring the main text (abstract through Conclusion) from p.14.7 to exactly p.12 in PMLR format (References start p.13; 19pp total with appendices), matching CLeaR's historical 12-page main-text limit. How: prose tightening throughout; E1 regime-D time-average honesty note moved to Appendix E; Section 4's Provenance paragraph merged with the Related-Work learning-in-games paragraph (they duplicated each other; the four-choices delta vs Lomys-Magnolfi now lives in Related Work with a pointer from Section 4); figure heights reduced in paper_figs.py (both the experiments copy and the artifact copy, artifact re-zipped); \looseness=-1 on four late paragraphs. No content dropped: all honesty items, theorem statements, proofs, and novelty positioning intact. Build clean: 0 errors, 0 undefined refs. Remaining pre-submission step is external: swap in the CLeaR 2027 wrapper style and re-fit when the author kit is released (CFP expected ~Oct 2026), and attach artifact.zip at submission.
Discover -> robustly plan -> certify -> transport loop over an SCM belief, reusing existing causalrl primitives (discover_interventional, causal_q_bounds, certify_policy, identify_transport). Falsifiable M0-M3 probe with an early kill gate; general-use, domain-neutral (DTR/medicine, finance, control as testbeds).
…g = behavior policy)
…ntifiable causal win)
…or agent 0.50 vs naive 0.40)
…ack-door adjust) + do-sampling + M1 gate
… (0.46 vs 0.40, discovery imperfect)
…10/10, hits optimum 0.50)
…n the confounded medicine DTR (1.04 vs 0.85)
…ats correlational TransportableConfoundedBandit puts the two failure modes on orthogonal variables (confounder Z drives the Γ axis; shift variable W drives the covariate-shift axis, additive on the safe arm), so a 2-D phase diagram does not entangle them. TransportBackdoorAgent deconfounds Z and transports W by the target P(W), dogfooding backdoor_adjustment_set + is_transportable_effect. run_m2_phase_diagram returns a PhaseDiagram (per-cell gap BenchmarkEstimates + monotone-in-both- axes flags) — the 'confounding bites where theory predicts' signature. The design was verified end-to-end at finite sample, including the tight-overlap γ=1 edge where a 0.4 propensity slope preserves positivity (a deterministic A=Z would make the effect unidentifiable). TDD + exports.
…uous confounder ContinuousConfoundedBandit keeps a continuous observed confounder Z~Uniform and a nonlinear arm-1 reward bump, with overlap-preserving confounded behavior that over-samples the bump — so recovering the interventional value needs a learned continuous outcome model, not a lookup table. FunctionApproxBackdoorAgent fits qhat(a,z) by per-action ridge on RBF features and back-door-adjusts by MC-integrating it over the observed Z (single continuous confounder read from the graph via backdoor_adjustment_set). At gamma=1 it recovers E[Y|do(1)]≈0.38 and keeps the safe optimal arm 0 across seeds, while the confounded marginal is fooled into the harmful arm 1 — the discover→adjust recipe carried past the tabular regime. run_m3_function_approx_gate reads the verdict. The external d3rlpy/CausalGym tier stays an out-of-CI deferral (numpy-2 risk, no interop seam) per DESIGN §6. TDD; pyright/ruff clean; public-API exports.
M2 (2-D phase diagram γ×shift, 10 seeds): gap monotone nondecreasing in both axes, 0.000 on the no-confounding edge, 0.200 at the high corner — the 'confounding bites where theory predicts' signature (CI green f9e6de6). M3 (function-approx tier, 10 seeds): ridge-RBF back-door recovers E[Y|do(1)]≈0.38 and keeps the optimal arm 0 (0.500) while the confounded marginal is fooled to 0.381 (CI green 85f967d). Records the honest d3rlpy/CausalGym out-of-CI deferral. Status table now spans M0–M3 (all GO); Resume-here updated to the completed spine.
from_causal_gym rolls out any Gymnasium-API env under a supplied behavior policy into a ConfoundedTrajectoryDataset — the offline structure the causal-MBRL probes and certificate layer consume. Duck-typed (the external causal_gym package is never imported, matching the PettingZoo/DoWhy/EconML house style), so causalrl agents can consume external Causal-Gymnasium rollouts; unit-tested with fake Gymnasium envs. This builds the DESIGN §7 interop seam that PROGRESS previously flagged missing. The heavier d3rlpy CQL / CartPoleWind comparison stays an honest out-of-CI deferral (numpy-2 risk), not a fabricated result. RESULTS.md pulls the five verdicts (M0 back-door, M1a discovery, M1b sequential medicine DTR, M2 γ×shift phase diagram, M3 function-approx) into one note, with the dividing line and honest boundaries stated up front. TDD; ruff/pyright clean; from_causal_gym exported.
…_causal_gym Strict pyright on src flags the intentionally Any-typed Gymnasium env handling (reportUnknownArgumentType/VariableType), exactly as the sibling pettingzoo adapter — add the same per-file pyright suppression header. Verified with 'uv run pyright src' (0 errors), the CI-equivalent check.
raphaelrrcoelho
changed the title
raphaelrrcoelho
marked this pull request as ready for review
July 19, 2026 17:47
raphaelrrcoelho
added a commit
that referenced
this pull request
Jul 19, 2026
PR #28 merged (M0–M3 + from_causal_gym + RESULTS.md); the unified CausalMBRLAgent front-door is this PR. Resume-here now lists the real open directions: real-data demo, adoption push, and the deferred d3rlpy tier.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A general, domain-neutral causal model-based RL agent — discover structure, plan under
confounding, transport across a shift — that beats correlational offline RL exactly in the
confounded/offline/transfer regime the theory targets (and makes no clean-benchmark claim, by
design). Five milestones, every verdict CI-green.
Verdicts
{Z}on 10/10, 0.50M2 is the headline: the causal-minus-naive gap is zero on the no-confounding edge and grows
monotonically in both confounding strength and shift — the "confounding bites where theory predicts"
signature (two failure modes on orthogonal variables; positivity preserved through γ=1).
Also in this PR
from_causal_gyminterop seam (DESIGN §7): rolls any Gymnasium-API env under a behavior policyinto a
ConfoundedTrajectoryDataset, duck-typed (external package never imported), unit-tested.(0.45) — which is why the program uses an active deconfounding optimizer, not a safety gate.
docs/causal_mbrl_agent/{DESIGN,PROGRESS,RESULTS}.md.Honest boundaries
No clean-benchmark SOTA claim (wins are confined to the confounded/offline/transfer regime).
Perception→variables is assumed away. At zero overlap the effect is unidentifiable and the causal
agent cannot help either (M2 stays inside the identifiable regime). The heavier d3rlpy CQL /
CartPoleWind comparison is deliberately out of CI (numpy-2 risk) and left as a user-runnable,
unverified example rather than a fabricated result.
New agents, envs, and harnesses are all TDD'd, ruff/pyright-clean, and exported from
causalrl.