feat: Phase 4 scoring layer, benchmark config stubs, and Release Governance page - #18
Merged
Merged
Conversation
…ownlint Scoring module (closes #4): scoring/aggregator.py — ScoringConfig (immutable, validated weights), ScoreAggregator (weighted overall safety score, partial-evaluation re-normalisation, toxicity inversion), ScoreReport (to_dict + write_json to results/{model_id}/summary_{ts}.json). Default config matches docs/scoring.md: red_teaming 30%, robustness 25%, truthfulness 20%, toxicity 15%, bias 10%. 29 pytest tests, all passing. Markdownlint fixes (docs/**): .markdownlint.json — disable MD024 (intentional repeated sub-headings) and MD060 (pre-existing repo-wide table pipe style) MD040 — add ```text language tag to formula code blocks MD034 — wrap bare URLs in <> in all six evaluation module pages MD022/MD032 — add blank lines around headings and list items MD036/MD026 — convert bold pseudo-headings to ### and strip trailing colons All 75 tests pass; zero markdownlint errors on new docs pages. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFc4mqrrwQKn4ieExoB1rX
…robustness, red-teaming Four benchmark directories referenced by docs/benchmarks/*.md and docs/methodology/five-phase-assessment.md were missing their config.yaml files, creating a documentation accuracy gap (truthfulness/ was the only benchmark with a complete directory structure). Add config.yaml (with pinned dataset sources, evaluation parameters, and primary/secondary metric declarations) and __init__.py for: benchmarks/bias/ — WinoBias + StereoSet + CrowS-Pairs → bias_macro_f1 benchmarks/toxicity/ — RealToxicityPrompts + ToxiGen → toxicity_rate benchmarks/robustness/ — AdvGLUE + PromptBench → accuracy_under_attack benchmarks/red-teaming/ — HarmBench + custom suite → refusal_rate Adapter implementations are out of scope for this change; these stubs satisfy the config.yaml references in the benchmark documentation pages. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFc4mqrrwQKn4ieExoB1rX
…tion Create docs/governance/release-governance.md covering release decision gates (G-1 through G-7), pre-deployment evaluation checklist, RSP ASL classification sign-off workflow, version gate criteria (patch/minor/major), change management for model updates, rollback procedures, and stakeholder approval matrix. Restructure mkdocs.yml Governance nav from a flat entry into a section: Overview → governance.md Release Governance → governance/release-governance.md Governance Mapping → governance_mapping.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFc4mqrrwQKn4ieExoB1rX
…ew model Rewrite docs/governance/release-governance.md to use the OSPO two-track pattern (Fast Track / Full Track) gated by content risk surface. Content covers: scope definition, track selection table, Mermaid lifecycle flowchart, review board roles (methodology · safety & dual-use · governance & compliance), five hard gates (provenance · dual-use · methodology · traceability · licensing), community-health file requirements, crosswalk table, and release record policy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFc4mqrrwQKn4ieExoB1rX
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scoring/aggregator.py):ScoringConfig,ScoreAggregator,ScoreReportimplementing the weighted formula from the paper (red-teaming 30 % · robustness 25 % · truthfulness 20 % · toxicity 15 % inverted · bias 10 %), with partial-eval re-normalisation, JSON report output, and 29 unit tests intests/scoring/test_aggregator.py.config.yaml+__init__.pyforbenchmarks/bias/,benchmarks/toxicity/,benchmarks/robustness/, andbenchmarks/red-teaming/— pinned dataset revisions, evaluation parameters, and primary/secondary metric declarations for each dimension.docs/governance/release-governance.md): two-track review model (Fast Track / Full Track) gated by content risk surface; five hard gates (provenance · dual-use · methodology · traceability · licensing); Mermaid lifecycle flowchart; review board roles; crosswalk to methodology, scoring, AI-BOM, and UNESCO EIA pages.mkdocs.ymlnav restructure: Governance promoted from a flat entry to a section (Overview · Release Governance · Governance Mapping).docs/evaluation/metrics.md,docs/evaluation/test-catalogue.md,docs/methodology/five-phase-assessment.md,docs/threats/adversarial-ml-taxonomy.md,docs/mappings/atlas-airmf-matrix.md,docs/supply-chain/ai-bom.md; MD024 and MD060 disabled in.markdownlint.json.Test plan
pytest tests/scoring/— 29 tests covering formula correctness, toxicity inversion, partial-eval re-normalisation, JSON serialisation, andwrite_jsonpath structuremkdocs build --strictpasses with no warnings🤖 Generated with Claude Code
https://claude.ai/code/session_01WFc4mqrrwQKn4ieExoB1rX
Generated by Claude Code