multiple-testing-toolkit

This repo is simple implementation of Bonferroni and Benjamini-Hochberg for Family-wise error rate (FWER) and False discovery rate (FDR).

A Python toolkit for correcting multiple hypothesis testing problems using statistically rigorous methods like Benjamini–Hochberg (FDR) and Bonferroni correction.

📌 The Problem Statement

When testing multiple hypotheses, the chance of false positives increases rapidly:

• Testing 100 hypotheses at α = 0.05

• 👉 Expected false positives = 5

Even worse, the probability of at least one false positive becomes very high.

This leads to misleading conclusions in:

• A/B testing
• Feature selection
• Experimentation platforms and so on

💡 The Solution

This toolkit implements:

✅ Benjamini–Hochberg (FDR)

• Controls the False Discovery Rate
• More powerful (finds more true signals)
• Ideal for data science & experimentation

✅ Bonferroni Correction

• Controls Family-Wise Error Rate (FWER)
• Very strict (minimises false positives)
• Best for high-risk decisions

Methods Included

⚡ Quick Start

from fdr_toolkit.fdr import benjamini_hochberg

p_vals = [0.12, 0.03, 0.001, 0.20, 0.04, 0.002, 0.15, 0.05, 0.30, 0.01]

results = benjamini_hochberg(p_vals, alpha=0.05)

print(results)

📊 Example Output

📈 Visualisation

from fdr_toolkit.visualization import plot_bh

plot_bh(p_vals)

This produces a plot where:

• Points = sorted p-values
• Line = BH threshold
• ✅ Values below the line → rejected

🚀 Future Improvements

• Holm–Bonferroni method
• Storey q-values
• Integration with scikit-learn pipelines
• CLI tool for quick analysis

⭐Rationale

This project demonstrates:

• Translating statistical theory → real-world code
• Understanding of multiple testing problems
• Opportunity to build reusable data science tools

📬 Feedback & Contributions

Feel free to open issues or submit pull requests!