Support XGBoost categorical splits in CPU TreeExplainer with background

[04pallav]

Status: Work in progress — I am still finalizing this PR; it is not fully ready for review yet.

Related issues: shap#2662, shap#5068
Upstream draft PR: shap#5107

Summary

TreeExplainer on an XGBoost model with native categoricals (enable_categorical=True) works today without a background dataset: SHAP delegates to XGBoost’s own pred_contribs path. It breaks when you pass background data with feature_perturbation="tree_path_dependent" — that configuration forces SHAP’s CPU _cext implementation, which did not support XGB categorical trees end-to-end.

On master, that _cext path either raises NotImplementedError for categorical XGB models, or (if reached) cannot traverse the trees correctly: the loader never marks categorical nodes or builds their bitmask thresholds, the C++ core has no XGB split type (only numeric and LightGBM), and pandas category columns are passed through to_numpy() as labels instead of the integer codes the booster trained on.

This PR connects the full pipeline for that path:

• Loader (XGBTreeModelLoader): mark categorical split nodes as threshold_types=2 and set threshold bitmasks from category codes.
• C++ (tree_shap.h): add XGB routing (in-set → right, 2^code bitmask) via a shared split dispatcher used in predict, weight update, and SHAP recursion.
• Python (_tree.py): convert categorical columns to codes through transform_input / _to_input_array; allow _cext when type-2 nodes are present; keep background and explain rows in sync (TreeExplainer.data ← encoded TreeEnsemble.data).

Testing

Success criterion: with background + tree_path_dependent, row-level SHAP values satisfy additivity — sum(shap_values) + expected_value ≈ predict(..., output_margin=True).

Automated (TestExplainerXGBoostCategorical):

• Fixture: adult dataset (300 rows), Workclass as pandas category, XGBClassifier(enable_categorical=True, tree_method="hist"), 100-row background + 20 explain rows.
• test_tree_path_dependent_background_additivity: background forces _cext (not pred_contribs); asserts threshold_types == 2 nodes exist; runs shap_values(..., check_additivity=True); explicitly checks shap.sum(1) + expected_value vs output_margin (rtol=atol=1e-5).
• test_interventional_raises: same model/background with feature_perturbation="interventional" raises NotImplementedError (expected gap).

CI: full upstream tests workflow green on this branch (all OS/Python matrix jobs, including oldest supported numpy).

Not tested: per-feature agreement with XGB pred_contribs on the no-background shortcut path.

Not in this PR

Item	Status
Interventional + categoricals	Still NotImplementedError
GPU (_cext_gpu)	Deferred
Per-feature vs pred_contribs	Not tested — row-level additivity only

[04pallav]

Superseded by upstream draft PR: shap#5107