Open-source framework for stress-testing AI systems, bringing together benchmarks to evaluate bias, toxicity, truthfulness, robustness, and adversarial risk in modern AI and LLM systems. Built for reproducibility, grounded in academic research, and designed for real-world governance, risk, and safety use cases.
Related project: AI-regulatory-monitor tracks real-time regulatory developments (EU AI Act, NIST AI RMF, and more) that contextualize the governance dimensions evaluated here.
Three commands to clone, install, and run your first evaluation:
git clone https://github.com/felipelago17/Responsible-AI-evaluation.git && cd Responsible-AI-evaluation
pip install -r requirements.txt
python -c "from evaluation.runner import EvaluationRunner; from benchmarks.membench_rai import MemBenchRAIAdapter; from benchmarks.cybergym_glasswing import CyberGymGlasswingAdapter; r = EvaluationRunner([MemBenchRAIAdapter(), CyberGymGlasswingAdapter()]); print(r.run_all(model=lambda p: ['safe response'] * len(p)))"To evaluate your own model, replace the lambda with your inference function:
from evaluation.runner import EvaluationRunner
from benchmarks.membench_rai import MemBenchRAIAdapter
def my_model(prompts: list[str]) -> list[str]:
# Replace with your model's inference call
return ["<placeholder response>" for _ in prompts]
runner = EvaluationRunner(benchmarks=[
MemBenchRAIAdapter(),
CyberGymGlasswingAdapter(),
])
results = runner.run_all(model=my_model)See results/v1.0.0/example-model-v1/summary.json for an example of the output schema.
pip install pyyaml
# Optional — required only when running the TruthfulQA benchmark:
pip install datasets
# Optional — required for MemPalace knowledge-graph queries:
pip install networkxOr install all dependencies at once:
pip install -r requirements.txtbenchmarks/ # Dataset adapters and benchmark implementations
├── base.py # BenchmarkAdapter ABC and result dataclasses
├── truthfulness/ # TruthfulQA (MC1 / MC2)
├── truthfulqa/ # TruthfulQA re-export (BenchmarkAdapter-conformant)
├── membench_rai.py # MemBench-RAI (Safety Consistency Under Memory)
└── cybergym_glasswing.py # CyberGym-Glasswing (Zero-Day Autonomous Risk)
evaluation/ # Evaluation harness and auxiliary modules
├── runner.py # EvaluationRunner — orchestrates benchmark execution
├── session_memory.py # MemPalace longitudinal result store
├── disclosure_compliance.py # Coordinated disclosure compliance checker
└── agentic_autonomy.py # Agentic autonomy risk evaluator
docs/ # MkDocs documentation source
└── literature/ # Annotated bibliographies by evaluation dimension
eia/ # UNESCO EIA templates and machine-readable schemas
├── templates/eia-template.md # Fillable Markdown assessment template
├── templates/impact-mapping.md # Impact tables (one per principle)
├── templates/eia.schema.json # JSON Schema draft-07 for machine-readable EIAs
└── templates/eia.example.yaml # Example: LLM-based HR screening tool
results/ # Versioned evaluation outputs
tests/ # Unit tests
Full documentation is available at https://felipelago17.github.io/Responsible-AI-evaluation/.
Framework documentation:
| Section | Description |
|---|---|
| Methodology | Evaluation workflow and academic foundations |
| Bias | Stereotype, representation, and allocation bias |
| Toxicity | Hate speech, harassment, and harmful content |
| Truthfulness | Factual accuracy and hallucination rates |
| Robustness | Adversarial inputs and prompt injection |
| Red Teaming | Structured adversarial probing |
| Scoring | Metrics, aggregation, and reporting |
| Governance | Data handling, versioning, and responsible use |
Annotated literature by evaluation dimension:
| Dimension | Reference Collection |
|---|---|
| Bias & Fairness | Bias benchmarks, fairness metrics, debiasing methods |
| Toxicity | Toxicity detection, hate speech datasets, evaluation tools |
| Truthfulness | Hallucination benchmarks, factuality evaluation |
| Red Teaming | Red teaming methodologies, safety evaluation frameworks |
| Adversarial Attacks | Data poisoning, model extraction, robustness defences |
| Governance Frameworks | AI governance standards by jurisdiction |
This framework is grounded in peer-reviewed research and regulatory standards:
| Reference | Relevance |
|---|---|
| TruthfulQA (Lin et al., 2022) | Truthfulness benchmark methodology |
| EU AI Act — Regulation (EU) 2024/1689 | Governance and risk classification framework |
| NIST AI Risk Management Framework 1.0 | Risk assessment methodology |
| Measuring Massive Multitask Language Understanding (Hendrycks et al., 2020) | Robustness evaluation benchmark |
| BBQ: A Hand-Built Bias Benchmark (Parrish et al., 2022) | Bias evaluation methodology |
For the full methodology, see docs/methodology.md.
- AI-regulatory-monitor — Real-time tracking of AI governance developments (EU AI Act, NIST AI RMF, global regulatory signals). Use alongside this framework to connect evaluation scores to the regulatory requirements they address.
This framework ships a self-contained UNESCO EIA module aligned with the UNESCO Recommendation on the Ethics of AI (2021).
- Run a full EIA as GitHub Issues (one per stage) using the issue templates
- Machine-readable assessments:
eia/templates/eia.schema.json - Regulatory crosswalk: UNESCO principles → NIST AI RMF · EU AI Act · OECD · ISO 42001 · CoE HUDERIA
Attribution: The UNESCO Ethical Impact Assessment is a tool developed by UNESCO. Official resources, the EIA workbook, and the Recommendation on the Ethics of Artificial Intelligence (2021) are available at https://www.unesco.org/ethics-ai/en/eia.
Contributions are welcome! See CONTRIBUTING.md for guidelines on adding benchmark adapters, reporting issues, and submitting pull requests.
If you use this framework in your research, please cite:
@software{lago2026responsible,
author = {Lago, Felipe},
title = {Responsible-AI-evaluation},
year = {2026},
url = {https://github.com/felipelago17/Responsible-AI-evaluation},
license = {MIT},
note = {Open-source framework for stress-testing AI systems across bias, toxicity, truthfulness, robustness, and adversarial risk}
}See CITATION.cff for the machine-readable citation.
pip install pytest
pytest tests/