mlr3autoiml is an auditing and automation layer for interpretable
machine learning (IML) on top of the mlr3 ecosystem. It implements a
gate-based workflow (G0A/G0B → G1 → G2 → G3 → G4 → G5 → G6 →
G7A/G7B) and assigns claim-scoped Interpretation Evidence Levels
(IEL-0…3) that scope the strength of permissible interpretive claims.
Core dependencies: mlr3, mlr3measures, mlr3misc, data.table,
checkmate, R6. Optional integrations (pipelines, SHAP, iml,
plotting) activate when the corresponding packages are available.
# install.packages("remotes")
remotes::install_github("coorsaa/mlr3autoiml")Task: german_credit (1 000 rows, 20 mixed-type features, binary
credit_risk). Full gate path including G3 calibration + decision
utility and G7A subgroup audit.
library(mlr3)
library(mlr3autoiml)
library(mlr3learners)
library(mlr3pipelines)
task = tsk("german_credit")
task$set_col_roles("personal_status_sex", add_to = "group")
learner = lrn("classif.rpart", predict_type = "prob", maxdepth = 6L)The ctx$claim block is a pre-analysis declaration, not a model output.
The operative part is the claim scope, semantics, stakes, and any
decision thresholds. AutoIML can derive conservative defaults for
non-use, prohibited interpretations, and decision-policy wording from
those declarations, so you only need to spell them out when the analysis
warrants narrower boundaries.
auto = AutoIML$new(
task = task,
learner = learner,
resampling = rsmp("cv", folds = 3),
purpose = "decision_support",
seed = 42
)
auto$ctx$claim = make_claim_card(
purpose = "decision_support",
semantics = "within_support",
stakes = "medium",
claims = list(global = TRUE, local = FALSE, decision = TRUE),
decision_spec = list(
thresholds = seq(0.20, 0.60, by = 0.05),
utility = list(tp = 1, fp = -0.5, fn = -1, tn = 0)
)
)
auto$ctx$measurement = make_measurement_card(
level = "item"
)
auto$ctx$sensitive_features = "personal_status_sex"
auto$ctx$alt_learners = list(
featureless = lrn("classif.featureless", predict_type = "prob")
)
auto$ctx$multiplicity$transport_mode = "group_performance"The measurement card now only needs the part the analysis cannot infer reliably. Here that is just the measurement level. Gate 0B derives task- and pipeline-level notes for missingness and scoring automatically, and only asks for comparability or reliability evidence when the measurement type makes those claims material.
res = auto$run()knitr::kable(auto$report_card()[, .(gate_id, gate_name, status, summary)])| gate_id | gate_name | status | summary |
|---|---|---|---|
| G0A | Scope claim & use | pass | Claim scope set (global=TRUE, local=FALSE, decision=TRUE) with semantics=within_support. |
| G0B | Measurement readiness | pass | Measurement readiness screened (psychometric evidence is user-supplied when needed, and pipeline notes may be analysis-derived). |
| G1 | Modeling and data validity (preflight) | pass | Predictive adequacy established with honest resampling. |
| G2 | What is being summarized? (dependence & interactions) | warn | Claim semantics=“within_support”: dependence and/or heterogeneity detected; prefer dependence- and interaction-aware summaries (ALE/ICE + regionalization) and avoid overinterpreting simple global narratives. |
| G3 | Calibration and decision utility | warn | Binary calibration/utility checks computed (intercept/slope, ECE, reliability, net benefit, cost-/utility sweep). |
| G5 | Stability and robustness | pass | Permutation-based stability check suggests robust importance ordering under bootstrap perturbations. |
| G6 | Multiplicity and transport | pass | Multiplicity and transport checks did not raise major concerns (given available evidence). |
| G7A | Subgroups / measurement audit | warn | Subgroup audit computed (binary classification performance + calibration; utility if specified). |
IEL (overall): IEL-1
With the operative claim card, analysis-derived measurement notes, and subgroup audit in place, the remaining warnings are substantive rather than setup-related: calibration, net benefit in the declared threshold range, and the interaction pattern that makes a simple one-feature story too thin.
auto$plot("g3_calibration")
auto$plot("g3_dca")
Gate 2 then shows why the package warns instead of presenting a single
global effect curve as the main story: amount and duration dominate,
and they interact strongly.
auto$plot("g2_hstats", top_n = 3L)
auto$plot("g2_ale_2d", feature1 = "amount", feature2 = "duration", class_label = task$positive)
When Gate 2 regionalization is enabled
(auto$ctx$structure$regionalize = TRUE) and the interaction screen
finds materially heterogeneous effects, the GADGET-style views decompose
the global effect into interpretable regions and show the split rules
that produced those regions. If no such regions are found, the
paper-aligned Gate 2 evidence remains the global effect plot, the
interaction screen, and the top 2D ALE surface. Optional
pint_enabled = TRUE adds a permutation-based interaction screen before
regionalization.
if (!is.null(auto$result$gate_results$G2$artifacts$gadget_regions) &&
nrow(auto$result$gate_results$G2$artifacts$gadget_regions) > 0L) {
auto$plot("g2_gadget")
auto$plot("g2_gadget_tree")
}
auto$plot("g2_pint")export_analysis_bundle(auto, dir = "bundle_german_credit", prefix = "german_credit")Task: california_housing (20 433 complete cases from the 20
640-row benchmark, 8 numeric + 1 factor feature, continuous
median_house_value). We use a complete-case subset so ranger, lasso,
and xgboost benchmark the same design matrix. Correlated spatial
predictors still trigger ALE selection in G2, and the mixed learner
family makes Gate 6 Rashomon diagnostics realistic.
task = tsk("california_housing")
task$filter(task$row_ids[stats::complete.cases(task$data())])
learner = lrn(
"regr.ranger",
num.trees = 200L,
mtry.ratio = 0.5,
min.node.size = 5L
)The claim card here is not extra decoration; it fixes the semantics and scope before Gate 6 compares models. Analysis-specific transport and preprocessing notes can remain implicit unless you need to declare a narrower boundary than the task- and pipeline-level defaults.
auto = AutoIML$new(
task = task,
learner = learner,
resampling = rsmp("cv", folds = 3),
purpose = "global_insight",
seed = 42
)
auto$ctx$claim = make_claim_card(
purpose = "global_insight",
semantics = "within_support",
stakes = "low",
claims = list(global = TRUE, local = FALSE, decision = FALSE)
)
auto$ctx$measurement = make_measurement_card(
level = "item"
)
auto$ctx$sensitive_features = "ocean_proximity"
auto$ctx$alt_learners = list(
lasso = as_learner(po("encode") %>>% lrn("regr.glmnet", alpha = 1)),
xgboost = as_learner(
po("encode") %>>%
lrn(
"regr.xgboost",
nrounds = 150L,
eta = 0.05,
max_depth = 6L,
subsample = 0.8,
colsample_bytree = 0.8,
verbose = 0
)
),
tree = lrn("regr.rpart", maxdepth = 12L, cp = 1e-03),
featureless = lrn("regr.featureless")
)
auto$ctx$structure$max_features = 8Lres = auto$run()knitr::kable(auto$report_card()[, .(gate_id, gate_name, status, summary)])| gate_id | gate_name | status | summary |
|---|---|---|---|
| G0A | Scope claim & use | pass | Claim scope set (global=TRUE, local=FALSE, decision=FALSE) with semantics=within_support. |
| G0B | Measurement readiness | pass | Measurement readiness screened (psychometric evidence is user-supplied when needed, and pipeline notes may be analysis-derived). |
| G1 | Modeling and data validity (preflight) | pass | Predictive adequacy established with honest resampling. |
| G2 | What is being summarized? (dependence & interactions) | warn | Claim semantics=“within_support”: dependence and/or heterogeneity detected; prefer dependence- and interaction-aware summaries (ALE/ICE + regionalization) and avoid overinterpreting simple global narratives. |
| G5 | Stability and robustness | pass | Permutation-based stability check suggests robust importance ordering under bootstrap perturbations. |
| G6 | Multiplicity and transport | warn | Evidence of multiplicity (Rashomon set contains multiple near-tie models) and/or limited transportability across groups; scope interpretive claims accordingly. |
| G7A | Subgroups / measurement audit | pass | Subgroup audit computed (regression RMSE, R², mean_y by group). |
IEL (overall): IEL-1
The effect plots focus on the housing variables that dominate the model
story: income and coastal location. ranger keeps the global effects
readable, while Gate 6 benchmarks it against lasso, xgboost, and a
single tree baseline.
auto$plot("g2_effect", feature = c("median_income", "latitude", "longitude"))
The supporting views below add two complementary angles: conditional SHAP importance for the fitted tree, and a Rashomon rank heatmap for the near-tie models in Gate 6.
auto$plot("shap_importance", n_rows = 40L, sample_size = 15L, background_n = 40L, top_n = 8L)
auto$plot("g6_rank_heatmap", top_n = 8L)
export_analysis_bundle(auto, dir = "bundle_california", prefix = "california")| Gate | Triggered when | Evidence checks |
|---|---|---|
| G0A | always | Claim and semantics declaration; hard-stop on causal_recourse without identification |
| G0B | always | Measurement readiness; reliability / invariance evidence for high-stakes |
| G1 | always | CV performance + calibration snapshot |
| G2 | always | Feature dependence · ALE vs PDP selection · pairwise interaction screening |
| G3 | claims$decision = TRUE |
Calibration curve + Decision Curve Analysis |
| G4 | claims$local = TRUE |
SHAP faithfulness + perturbation sensitivity checks |
| G5 | always | Stability of narrated patterns under bootstrap resampling |
| G6 | high-resolution profile | Rashomon multiplicity + transport probes |
| G7A | sensitive_features set |
Subgroup performance + calibration audit |
| G7B | user-facing + high-stakes | Human-factors (task-based evaluation) evidence |
IELs (0–3) are assigned per claim scope (global / local / decision). The overall IEL is the minimum across all requested scopes.
The package follows a claim-first, gate-based architecture with explicit
semantics, claim-scoped IELs, and evidence-driven gate planning. Default
runs use the high_resolution profile; quick_start = TRUE is
available for rapid prototyping. IEL-3 requires G6 evidence across all
scopes and, for decision claims, a passed G7A subgroup audit.
LGPL-3.