pppms/README.Rmd at main · pascalrink/pppms · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116

---
title: "pppms: Confidence Limits for Prediction Performance"
output: github_document
---

# pppms

`pppms` is an R package for post-selection inference in predictive modeling. It implements multiplicity-adjusted bootstrap tilting methods for lower confidence limits for prediction performance after selecting the empirically best candidate model.

The methods implemented in this package originate from the dissertation

> Rink, P. (2025). *Confidence Limits for Prediction Performance*. PhD thesis, University of Bremen, https://doi.org/10.26092/elib/3822

The package is intended as a **methods package for post-selection inference
in predictive modeling**.

---

## Motivation

In many predictive modeling workflows several candidate models are trained
and compared using the same evaluation data.

Typical workflow:

1. Fit multiple candidate models
2. Estimate their prediction performance
3. Select the empirically best model
4. Report its estimated performance

However, this procedure ignores the uncertainty introduced by the **model
selection step**. Selecting the best model among several candidates inflates
the observed performance and can lead to overly optimistic conclusions.

`pppms` provides **statistically valid lower confidence limits for prediction
performance that explicitly account for model selection**.

---

## Installation

```r
# install.packages("remotes")
remotes::install_github("pascalrink/pppms")
```

---

## Example

```r
library(pppms)

true_labels <- c(0,0,1,1,0,1)

pred_labels <- cbind(
  model1 = c(0,0,1,1,1,1),
  model2 = c(0,1,1,0,0,1)
)

res <- MabtCI(
  true_labels,
  pred_labels,
  B = 200,
  seed = 1
)

res
```

Returned values:

- **bound** – lower confidence limit for prediction performance
- **tau** – estimated tilting parameter
- **t0** – empirical performance of the selected model
- **selected_idx** – index of the selected model

---

## Methodological idea

The procedure combines two ideas:

**Multiplicity adjustment**
Model selection creates a multiple comparison problem. The procedure therefore
uses a **max-type calibration** across candidate models.

**Bootstrap tilting**
Bootstrap resampling is modified using weights

```
w_i(tau) ∝ exp(tau * psi_i)
```

where `psi_i` is an empirical influence quantity and `tau` is a tilting
parameter chosen so that the bootstrap distribution matches the target
significance level.

---

## Further details

For methodological background see

```r
vignette("methodological-background", package = "pppms")
```

---

## Reference

Rink, P. (2025).
*Confidence Limits for Prediction Performance*.
Doctoral thesis, University of Bremen.