Class FeatureImp$new() in multiclass problem computes metrics on probabilities instead of class labels

[gi-marengo]

When computing Permutation Feature Importance (PFI) using FeatureImp$new() per class in a multiclass classification task, the loss function appears to be evaluated directly on predicted probabilities instead of class labels, producing incorrect values for metrics that expect lables rather than probabilities (e.g. Metrics::ce).

In contrast, when computing the global multiclass PFI, the probabilities are correctly converted to the label with maximum probability using iml:::probs.to.labels() before the loss function is applied. However, conversions are not applied when Predictor$predict() returns a single column of probabilities (as in class PFI).

library(iml)
library(mlr3)
library(mlr3learners)
library(Metrics)

ce_fun <- function(actual, predicted) { # to explor the metric computation
  cat("Actual: ", actual, "\n\n")
  cat("Predicted: ", predicted, "\n\n")
  cat("METRIC: ", Metrics::ce(actual, predicted), "\n\n")

  return(Metrics::ce(actual, predicted))
}

tsk = as_task_classif(iris, target = "Species")
lrn = lrn("classif.ranger", predict_type = "prob")

lrn$train(tsk)

# Class Feature Importance ----

set.seed(123)
rows = sample(nrow(iris), 10)

mod_class <- Predictor$new(lrn,
                     data = iris[rows,], y = iris$Species[rows] == "setosa",
                     type = "prob", class = "setosa")

mod_class$predict(tsk$data())
iml:::probs.to.labels(mod_class$predict(tsk$data()))

imp <- FeatureImp$new(mod_class, loss = ce_fun, compare = "ratio", n.repetitions = 1)

Example output:

Actual:  TRUE FALSE TRUE FALSE FALSE TRUE FALSE FALSE FALSE FALSE 

Predicted:  1 0 0.9995 0 0 1 0 0 0 0 
 
METRIC:  0.1

A probability of 0.9995 to be TRUE is treated as a misclassification because the loss function receives a numeric probability and not a class label. Shouldn't it be converted to 1 (i.e., TRUE) using a threshold of 0.5? Or am I wrong?

# For a more clear explanation (predicting on the first 10 rows, i.e, always setosa)

set.seed(123)
mod <- Predictor$new(lrn,
                     data = iris[1:10,], y = iris$Species[1:10] == "setosa",
                     type = "prob", class = "setosa")

mod$predict(tsk$data())
iml:::probs.to.labels(mod$predict(tsk$data()))

imp <- FeatureImp$new(mod, loss = ce_fun, compare = "ratio", n.repetitions = 1)

# Example output:
# Actual:  TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE 
#
# Predicted:  1 1 1 1 1 1 1 0.9993333 0.9993333 1 
#
# METRIC:  0.2 
#
# The metric should not be 0?

> sessionInfo()
R version 4.5.1 (2025-06-13 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 11 x64 (build 26100)

Matrix products: default
  LAPACK version 3.12.1

locale:
[1] LC_COLLATE=Italian_Italy.utf8  LC_CTYPE=Italian_Italy.utf8    LC_MONETARY=Italian_Italy.utf8 LC_NUMERIC=C                  
[5] LC_TIME=Italian_Italy.utf8    

time zone: Europe/Rome
tzcode source: internal

attached base packages:
 [1] tools     grid      parallel  compiler  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] Formula_1.2-5        labeling_0.4.3       withr_3.0.2          lattice_0.22-7       Rcpp_1.1.0           ranger_0.17.0       
 [7] Matrix_1.7-3         pkgconfig_2.0.3      parallelly_1.45.1    codetools_0.2-20     listenv_0.9.1        farver_2.1.2        
[13] data.table_1.17.8    glue_1.8.0           vctrs_0.6.5          lifecycle_1.0.4      rstudioapi_0.17.1    digest_0.6.37       
[19] magrittr_2.0.4       cli_3.6.5            mlr3misc_0.19.0      lgr_0.5.0            rlang_1.1.6          RColorBrewer_1.1-3  
[25] pillar_1.11.1        paradox_1.0.1        Metrics_0.1.4        tibble_3.3.0         checkmate_2.3.3      backports_1.5.0     
[31] palmerpenguins_0.1.1 generics_0.1.4       R6_2.6.1             ggplot2_3.5.2        uuid_1.2-1           scales_1.4.0        
[37] globals_0.18.0       dichromat_2.0-0.1    tidyselect_1.2.1     rpart_4.1.24         crayon_1.5.3         dplyr_1.1.4         
[43] gtable_0.3.6         future.apply_1.20.0  future_1.67.0        mlr3learners_0.13.0  mlr3_1.2.0           iml_0.11.4