Structure of multi-output trees

[david-cortes]

In multi-output trees, leafs should have one value for each output, but if inspecting the trees, they appear to have more - guess multiplied by the depth or something like that. Example:

import numpy as np
rng = np.random.default_rng(seed=123)
X = rng.standard_normal(size=(100,10))
y = rng.integers(5, size=X.shape[0])

import xgboost as xgb
model = xgb.train(
    dtrain=xgb.DMatrix(X, y),
    num_boost_round=1,
    params={
        "objective": "multi:softprob",
        "max_depth": 2,
        "multi_strategy": "multi_output_tree",
        "num_class": 5,
    },
)

import json
json.loads(
    model.save_raw(raw_format="json")
)["learner"]["gradient_booster"]["model"]["trees"][-1]["leaf_weights"]

[0.13317914,
 -0.12695906,
 0.12011204,
 -0.16318427,
 0.10858103,
 -0.0880465,
 -0.083756156,
 -0.13272737,
 0.26806664,
 0.04055968,
 -0.13248305,
 0.22173962,
 -0.13717051,
 -0.017899837,
 -0.046972796,
 0.01588887,
 -0.03370758,
 0.033632353,
 0.0655839,
 -0.057034157]

Has 20 values, yet the raw predictions output something with 5 values.

How do those numbers relate to the raw/margin predictions?

[trivialfis]

Those are the raw values; all leaves are stored in a single buffer. How to get leaf value with vector leaf:

xgboost/include/xgboost/multi_target_tree_model.h

Line 137 in cfa442b

[[nodiscard]] linalg::VectorView<float const> LeafValue(bst_node_t nidx) const {

[david-cortes]

Those are the raw values; all leaves are stored in a single buffer. How to get leaf value with vector leaf:

xgboost/include/xgboost/multi_target_tree_model.h

Line 137 in cfa442b

[[nodiscard]] linalg::VectorView LeafValue(bst_node_t nidx) const {

Shouldn't it have some other name in the json then? Would btw be helpful for external usage of these jsons to have the leaf values of each terminal leaf alone in that leaf's json.

[trivialfis]

We can provide a helper script to parse the model format into Python in memory representation. But the model format itself is designed to be as compact as possible.

[wgu9]

Would a small Python helper for this be useful if it stays outside the model format itself?

I am thinking of a minimal utility/example that takes the JSON model representation, uses the vector-leaf indexing layout from MultiTargetTree::LeafValue, and exposes each terminal node’s vector leaf as a per-node Python object. If that direction sounds right, I can put together a focused PR with a small test/example rather than changing the serialized JSON schema.

[trivialfis]

We can provide a helper script to parse the model format into Python in memory representation.

Does this work for you #12293 ? @david-cortes

[david-cortes]

Yes, thanks, that helps.

But would still be ideal if the JSON dump of leafs could have the values specifically for each leaf, since in that format they wouldn't share a memory buffer either way.