chore(benchmarks): add missing config.yaml stubs for bias, toxicity, robustness, red-teaming - #17
Merged
Merged
Conversation
…ownlint Scoring module (closes #4): scoring/aggregator.py — ScoringConfig (immutable, validated weights), ScoreAggregator (weighted overall safety score, partial-evaluation re-normalisation, toxicity inversion), ScoreReport (to_dict + write_json to results/{model_id}/summary_{ts}.json). Default config matches docs/scoring.md: red_teaming 30%, robustness 25%, truthfulness 20%, toxicity 15%, bias 10%. 29 pytest tests, all passing. Markdownlint fixes (docs/**): .markdownlint.json — disable MD024 (intentional repeated sub-headings) and MD060 (pre-existing repo-wide table pipe style) MD040 — add ```text language tag to formula code blocks MD034 — wrap bare URLs in <> in all six evaluation module pages MD022/MD032 — add blank lines around headings and list items MD036/MD026 — convert bold pseudo-headings to ### and strip trailing colons All 75 tests pass; zero markdownlint errors on new docs pages. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFc4mqrrwQKn4ieExoB1rX
…robustness, red-teaming Four benchmark directories referenced by docs/benchmarks/*.md and docs/methodology/five-phase-assessment.md were missing their config.yaml files, creating a documentation accuracy gap (truthfulness/ was the only benchmark with a complete directory structure). Add config.yaml (with pinned dataset sources, evaluation parameters, and primary/secondary metric declarations) and __init__.py for: benchmarks/bias/ — WinoBias + StereoSet + CrowS-Pairs → bias_macro_f1 benchmarks/toxicity/ — RealToxicityPrompts + ToxiGen → toxicity_rate benchmarks/robustness/ — AdvGLUE + PromptBench → accuracy_under_attack benchmarks/red-teaming/ — HarmBench + custom suite → refusal_rate Adapter implementations are out of scope for this change; these stubs satisfy the config.yaml references in the benchmark documentation pages. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFc4mqrrwQKn4ieExoB1rX
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/benchmarks/*.mdanddocs/methodology/five-phase-assessment.mdwere missing theirconfig.yamlfiles —truthfulness/was the only benchmark with a complete directory structure.config.yaml(pinned dataset sources, evaluation parameters, primary/secondary metric declarations) and__init__.pyfor:benchmarks/bias/— WinoBias + StereoSet + CrowS-Pairs →bias_macro_f1benchmarks/toxicity/— RealToxicityPrompts + ToxiGen →toxicity_ratebenchmarks/robustness/— AdvGLUE + PromptBench →accuracy_under_attackbenchmarks/red-teaming/— HarmBench + custom suite →refusal_rateconfig.yamlreferences in the benchmark documentation pages and ensurebenchmarks/*/config.yamlreferences in the docs are accurate.Test plan
pytest tests/ -v)mkdocs build --strictpassesbenchmarks/bias/config.yaml,benchmarks/toxicity/config.yaml,benchmarks/robustness/config.yaml,benchmarks/red-teaming/config.yamlall exist and are valid YAML🤖 Generated with Claude Code
https://claude.ai/code/session_01WFc4mqrrwQKn4ieExoB1rX
Generated by Claude Code