Skip to content

warnings about probabilities from weighted classifiers and calibration for simulation #56

Description

@jason-bentley

Is your feature request related to a problem? Please describe.
When sample weights are applied to a learner classifier the up weighting on one class will be reflected in a higher predicted probability than what was observed in the unweighted data.

Describe the solution you'd like
There are two aspects to an ideal solution:

  1. when ever using weights with a classifier and performing simulation a warning should be displayed noting as such and that uncalibrated probabilities may not align with observed rates for the positive class.
  2. post training calibration could be added as a toggled option for the ClassifierPipelineDF class, where the default is set to true but can be turned off if desired. This could help ensure that naively even with weights applied to learning the probabilities shown in the simulation outputs align reasonably well with those observed in the data.

Describe alternatives you've considered
None.

Additional context
None.

Metadata

Metadata

Assignees

Labels

APINew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions