Feature/bias benchmark

README.md

@@ -8,6 +8,8 @@
 [![PyPI - License](https://img.shields.io/github/license/holistic-ai/holisticai)](https://img.shields.io/github/license/holistic-ai/holisticai)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/holisticai)](https://img.shields.io/pypi/dm/holisticai)
 [![Slack](https://img.shields.io/badge/Slack-Join-blue)](https://join.slack.com/t/holisticaicommunity/shared_invite/zt-2jamouyrn-BrMfeoBZIHT8HbLzB3P9QQ)
+[![jax_badge][jax_badge_link]](https://github.com/google/jax)
+
 
 ---
 
@@ -107,3 +109,6 @@ brew install cmake
 ### Contributing
 
 We welcome contributions from the community To learn more about contributing to Holistic AI, please refer to our [Contributing Guide](CONTRIBUTING.md).
+
+
+[jax_badge_link]: https://img.shields.io/badge/JAX-Accelerated-9cf.svg?style=flat-square&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAC0AAAAaCAYAAAAjZdWPAAAIx0lEQVR42rWWBVQbWxOAkefur%2B7u3les7u7F3ZIQ3N2tbng8aXFC0uAuKf2hmlJ3AapIgobMv7t0w%2Ba50JzzJdlhlvNldubeq%2FY%2BXrTS1z%2B6sttrKfQOOY4ns13ecFImb47pVvIkukNe4y3Junr1kSZ%2Bb3Na248tx7rKiHlPo6Ryse%2F11NKQuk%2FV3tfL52yHtXm8TGYS1wk4J093wrPQPngRJH9HH1x2fAjMhcIeIaXKQCmd2Gn7IqSvG83BueT0CMkTyESUqm3vRRggTdOBIb1HFDaNl8Gdg91AFGkO7QXe8gJInpoDjEXC9gbhtWH3rjZ%2F9yK6t42Y9zyiC1iLhZA8JQe4eqKXklrJF0MqfPv2bc2wzPZjpnEyMEVlEZCKQzYCJhE8QEtIL1RaXEVFEGmEaTn96VuLDzWflLFbgvqUec3BPVBmeBnNwUiakq1I31UcPaTSR8%2B1LnditsscaB2A48K6D9SoZDD2O6bELvA0JGhl4zIYZzcWtD%2BMfdvdHNsDOHciXwBPN18lj7sy79qQCTNK3nxBZXakqbZFO2jHskA7zBs%2BJhmDmr0RhoadIZjYxKIVHpCZngPMZUKoQKrfEoz1PfZZdKAe2CvP4XnYE8k2LLMdMumwrLaNlomyVqK0UdwN%2BD7AAz73dYBpPg6gPiCN8TXFHCI2s7AWYesJgTabD%2FS5uXDTuwVaAvvghncTdk1DYGkL0daAs%2BsLiutLrn0%2BRMNXpunC7mgkCpshfbw4OhrUvMkYo%2F0c4XtHS1waY4mlG6To8oG1TKjs78xV5fAkSgqcZSL0GoszfxEAW0fUludRNWlIhGsljzVjctr8rJOkCpskKaDYIlgkVoCmF0kp%2FbW%2FU%2F%2B8QNdXPztbAc4kFxIEmNGwKuI9y5gnBMH%2BakiZxlfGaLP48kyj4qPFkeIPh0Q6lt861zZF%2BgBpDcAxT3gEOjGxMDLQRSn9XaDzPWdOstkEN7uez6jmgLOYilR7NkFwLh%2B4G0SQMnMwRp8jaCrwEs8eEmFW2VsNd07HQdP4TgWxNTYcFcKHPhRYFOWLfJJBE5FefTQsWiKRaOw6FBr6ob1RP3EoqdbHsWFDwAYvaVI28DaK8AHs51tU%2BA3Z8CUXvZ1jnSR7SRS2SnwKw4O8B1rCjwrjgt1gSrjXnWhBxjD0Hidm4vfj3e3riUP5PcUCYlZxsYFDK41XnLlUANwVeeILFde%2BGKLhk3zgyZNeQjcSHPMEKSyPPQKfIcKfIqCf8yN95MGZZ1bj98WJ%2BOorQzxsPqcYdX9orw8420jBQNfJVVmTOStEUqFz5dq%2F2tHUY3LbjMh0qYxCwCGxRep8%2FK4ZnldzuUkjJLPDhkzrUFBoHYBjk3odtNMYoJVGx9BG2JTNVehksmRaGUwMbYQITk3Xw9gOxbNoGaA8RWjwuQdsXdGvpdty7Su2%2Fqn0qbzWsXYp0nqVpet0O6zzugva1MZHUdwHk9G8aH7raHua9AIxzzjxDaw4w4cpvEQlM84kwdI0hkpsPpcOtUeaVM8hQT2Qtb4ckUbaYw4fXzGAqSVEd8CGpqamj%2F9Q2pPX7miW0NlHlDE81AxLSI2wyK6xf6vfrcgEwb0PAtPaHM1%2BNXzGXAlMRcUIrMpiE6%2Bxv0cyxSrC6FmjzvkWJE3OxpY%2BzmpsANFBxK6RuIJvXe7bUHNd4zfCwvPPh9unSO%2BbIL2JY53QDqvdbsEi2%2BuwEEHPsfFRdOqjHcjTaCLmWdBewtKzHEwKZynSGgtTaSqx7dwMeBLRhR1LETDhu76vgTFfMLi8zc8F7hoRPpAYjAWCp0Jy5dzfSEfltGU6M9oVCIATnPoGKImDUJNfK0JS37QTc9yY7eDKzIX5wR4wN8RTya4jETAvZDCmFeEPwhNXoOlQt5JnRzqhxLZBpY%2BT5mZD3M4MfLnDW6U%2Fy6jkaDXtysDm8vjxY%2FXYnLebkelXaQtSSge2IhBj9kjMLF41duDUNRiDLHEzfaigsoxRzWG6B0kZ2%2BoRA3dD2lRa44ZrM%2FBW5ANziVApGLaKCYucXOCEdhoew5Y%2Btu65VwJqxUC1j4lav6UwpIJfnRswQUIMawPSr2LGp6WwLDYJ2TwoMNbf6Tdni%2FEuNvAdEvuUZAwFERLVXg7pg9xt1djZgqV7DmuHFGQI9Sje2A9dR%2FFDd0osztIRYnln1hdW1dff%2B1gtNLN1u0ViZy9BBlu%2BzBNUK%2BrIaP9Nla2TG%2BETHwq2kXzmS4XxXmSVan9KMYUprrbgFJqCndyIw9fgdh8dMvzIiW0sngbxoGlniN6LffruTEIGE9khBw5T2FDmWlTYqrnEPa7aF%2FYYcPYiUE48Ul5jhP82tj%2FiESyJilCeLdQRpod6No3xJNNHeZBpOBsiAzm5rg2dBZYSyH9Hob0EOFqqh3vWOuHbFR5eXcORp4OzwTUA4rUzVfJ4q%2FIa1GzCrzjOMxQr5uqLAWUOwgaHOphrgF0r2epYh%2FytdjBmUAurfM6CxruT3Ee%2BDv2%2FHAwK4RUIPskqK%2Fw4%2FR1F1bWfHjbNiXcYl6RwGJcMOMdXZaEVxCutSN1SGLMx3JfzCdlU8THZFFC%2BJJuB2964wSGdmq3I2FEcpWYVfHm4jmXd%2BRn7agFn9oFaWGYhBmJs5v5a0LZUjc3Sr4Ep%2FmFYlX8OdLlFYidM%2B731v7Ly4lfu85l3SSMTAcd5Bg2Sl%2FIHBm3RuacVx%2BrHpFcWjxztavOcOBcTnUhwekkGlsfWEt2%2FkHflB7WqKomGvs9F62l7a%2BRKQQQtRBD9VIlZiLEfRBRfQEmDb32cFQcSjznUP3um%2FkcbV%2BjmNEvqhOQuonjoQh7QF%2BbK811rduN5G6ICLD%2BnmPbi0ur2hrDLKhQYiwRdQrvKjcp%2F%2BL%2BnTz%2Fa4FgvmakvluPMMxbL15Dq5MTYAhOxXM%2FmvEpsoWmtfP9RxnkAIAr%2F5pVxqPxH93msKodRSXIct2l0OU0%2FL4eY506L%2B3GyJ6UMEZfjjCDbysNcWWmFweJP0Jz%2FA0g2gk80pGkYAAAAAElFTkSuQmCC

docs/source/bmbench/index.rst

@@ -0,0 +1,6 @@
+
+BMBENCH
+=========
+
+The bias mitigation benchmark (BMBENCH) is a framework to simple evaluate the performance of bias mitigation algorithms. The benchmark is based on four empirical machine learning tasks: binary classification, regression, multi-class classification, and clustering. The benchmark is designed to be simple to use and easy to extend at the top of `holisticai` library. 
+

docs/source/getting_started/datasets.csv

@@ -1,9 +1,9 @@
 adult,Adult,Binary Classification, race / sex
 clinical_records,Heart Clinical Records,Binary Classification, sex
-law_school,Law School,Binary Classification, race / gender
+law_school,Law School,Regression, race / gender
 lastfm,Last FM,Recommender, 
-student,Student,Binary Classification, sex / address
-student_multiclass,Student (Multiclass),Multiclassification, sex / address
+student,Student, Regression, sex / address
+student_multiclass,Student (Multiclass), Multiclassification, sex / address
 us_crime,US Crime,Regression, race
 us_crime_multiclass,US Crime (Multiclass),Regression, race
 german_credit, German Credit, Binary Classification, sex

docs/source/index.rst

@@ -65,6 +65,15 @@ Welcome to the Holistic AI library! This is an open-source tool to assess and im
 
         Learn how to contribute to the library
 
+    .. grid-item-card:: Bias Mitigation Benchmark (BMBENCH)
+        :img-top: _static/contributor.svg
+        :class-card: intro-card
+        :link: bmbench/index.html
+        :link-alt: bmbench
+        :shadow: md
+
+        Bias mitigation benchmark for AI systems
+
 .. admonition:: Learn more about the Holistic AI company
         :class: dropdown admonition-youtube
 

src/holisticai/benchmark/BMBENCH.md

@@ -0,0 +1,43 @@
+# Empirical Benchmarking of Algorithmic Fairness in Machine Learning Models
+
+## Motivation
+
+The development and assessment of bias mitigation methods require rigorous benchmarks. This paper introduces BMBench, a comprehensive benchmarking framework to evaluate bias mitigation strategies across multitask machine learning predictions (binary classification, multiclass classification, regression, and clustering). Our benchmark leverages state-of-the-art and proposed datasets to improve fairness research, offering a broad spectrum of fairness metrics for a robust evaluation of bias mitigation methods. We provide an open-source repository to allow researchers to test and refine their bias mitigation approaches easily, promoting advancements in creating fair machine learning models.
+
+## Get started
+
+### Installation and usage
+
+```bash
+pip install holisticai
+```
+
+```python
+from holisticai.benchmark import BiasMitigationBenchmark
+
+stage = # preprocessing, inprocessing, postprocessing
+task = # binary_classification, multiclass, regression, clustering
+
+benchmark = BiasMitigationBenchmark(task, stage)
+
+my_mitigator = MyMitigator()
+
+my_results = benchmark.run(custom_mitigator=my_mitigator)
+benchmark.submit()
+```
+
+There are many other functionalities available in the `BiasMitigationBenchmark` class. Please refer to the [API documentation](https://holisticai.github.io/holistic-ai/).
+
+
+## How to cite
+
+If you use BMBench in your research, please cite the following paper:
+
+```bibtex
+@article{dacosta2025bmbench,
+  title={BMBENCH: Empirical Benchmarking of Algorithmic Fairness in Machine Learning Models},
+  author={da Costa, Kleyton and Munoz, Cristian and Fernandez, Franklin and Modenesi, Bernardo and Kazim, Emre and Koshiyama, Adriano},
+  url={},
+  year={2025}
+}
+```

src/holisticai/benchmark/__init__.py

@@ -0,0 +1,5 @@
+from holisticai.benchmark._bias import BiasMitigationBenchmark
+
+__all__ = [
+    "BiasMitigationBenchmark",
+]