[RFC] Introduce a Single "dropout" Parameter for DART

[RAMitchell]

Summary

I propose introducing a new DART parameter:

dropout ∈ [0, 1]

and deprecating the existing DART-specific parameters:

• "rate_drop"
• "sample_type"
• "normalize_type"
• "one_drop"
• "skip_drop"

The new "dropout" parameter would implement the canonical DART algorithm described in the original paper with the following fixed behavior:

• "sample_type = uniform"
• "normalize_type = tree"
• "one_drop = false"
• "skip_drop = 0"

The existing parameters can remain supported during a deprecation period for backwards compatibility.

Motivation

The original DART paper defines a remarkably simple algorithm:

1. Drop each previous tree independently with probability "dropout".
2. If no tree is selected, drop one uniformly chosen tree ("Binomial-plus-one").
3. Train the new tree.
4. Apply a fixed normalization scheme.

The paper does not experimentally justify alternative sampling schemes, normalization methods, or skipping dropout iterations. These options were introduced later as implementation extensions rather than core components of DART.

Today XGBoost exposes five DART-specific parameters, making DART considerably more complicated than both the original algorithm and the analogous dropout mechanism used in neural networks, where users typically tune a single probability.

Rationale

"dropout"

This is the fundamental DART hyperparameter and should be the only user-facing regularization parameter.

"one_drop"

The original paper always ensures that at least one tree is dropped. This is a boundary condition rather than an independent modeling choice.

Without this rule, for small dropout probabilities,

$P(K=0)=(1-p)^T$,

meaning many boosting rounds perform no dropout at all and degenerate to ordinary GBDT.

I would propose that the user should increase the dropout probability instead of using one drop.

"sample_type"

The paper uses uniform sampling and does not provide empirical evidence comparing it with weighted sampling.

Uniform sampling should therefore become the canonical behavior.

"normalize_type"

The paper defines a single normalization rule.

While XGBoost currently supports multiple normalization strategies, there is little published evidence that exposing this choice as a user-facing hyperparameter provides consistent benefit.

"skip_drop"

The original algorithm performs dropout every boosting iteration.

"skip_drop" effectively introduces a second regularization parameter controlling dropout frequency in addition to dropout probability. This increases tuning complexity without clear evidence that users benefit from controlling these effects independently.

Benefits

• A significantly simpler and more intuitive API.
• Closer alignment with the original DART algorithm.
• Easier parameter tuning.
• Consistency with dropout interfaces in deep learning, where a single probability controls regularization.
• Existing models remain compatible during a deprecation period while new users are presented with a much simpler interface.

[trivialfis]

I looked into dart again, and am running into a couple of issues:

• The eta is wrong. It should be eta / n_parallel_trees if we want to split up the dropout for num_parallel_tree, and eta if we want it to be global. But currently it's eta / (n_parallel_trees * n_outputs).
• The dropout is performed on individual trees, which doesn't make sense with softmax.
• Also, the dropout should be aware of the num_parallel_tree.
• With dart merged into gbtree, we need to have a defined behaviour for what happens to training continuation, either starting from a gbtree model or from a dart model.

All solvable; perhaps we don't need a breaking change in model format. But we might need to provide tree weights and dropout an explicit concept of output groups.

[jfpuget]

How would DART be disabled with this proposal? Is it when dropout is 0?

There must be a way to use XGBoost without DART, i.e. we can't say that at least one tree is dropped in general.

[RAMitchell]

Yes dropout 0 would mean no dropped trees.

[jfpuget]

Then it is not true that it is equivalent to:

• "one_drop = true"

[RAMitchell]

I've changed the description I think it should be "one_drop = false" and the user should increase the dropout probability if too few trees are being dropped.