Skip to content

fix(metrics): metric calibration — fidelity improvements across all dimensions - #17

Merged
srbouffard merged 16 commits into
mainfrom
feat/metric-calibration-task1
Aug 11, 2026
Merged

fix(metrics): metric calibration — fidelity improvements across all dimensions#17
srbouffard merged 16 commits into
mainfrom
feat/metric-calibration-task1

Conversation

@srbouffard

Copy link
Copy Markdown
Collaborator

Summary

Replaces semi-fake POC metric logic with deterministic, well-tested, production-grade detectors across all five dimensions. All scorers are now measurability-aware: unmeasurable signals return None and propagate to UNRATED at the engine level rather than forcing false zeros/bronze.

Changes by dimension

test_verification

  • latest_build_passing now falls back to GitHub default-branch check-runs when no Allure URL is set — fixes the saml-integrator false negative
  • coverage_pct / stability_pct return None (unmeasurable) instead of 0 when Allure is absent

documentation

  • Adds accepted workflow family name aliases (docs-checks / vale, docs-checks / linkcheck, docs-checks / docs build)
  • Adds equivalent README heading variants (Quickstart, Getting help)

substrate_compat

  • Full YAML-aware workflow parser: handles matrix.include list-items, inline comments, literal/folded/quoted block scalars, heredoc skip, quoted run: scalars
  • Exact juju channel matching (24/stable no longer matches juju4)
  • Canonical-k8s detection scoped to actual execution contexts
  • Fixes crash on defaults.run: mapping blocks

security_ssdlc

  • canonical_repo_automation_registered now checks git tree of canonical/canonical-repo-automation for a per-repo config file instead of code search — scoped to canonical/* owner only

support_engagement

  • avg_triage_days, avg_pr_review_days, response_coverage_rate return None when total_items < MIN_SAMPLE (5) — separates insufficient data from measured zero

Engine changes

  • compute_leaf_applicability and compute_product now check required_metrics_for_scoring from dimensions.yaml; any None value returns INSUFFICIENT_DATA / UNRATED
  • UNRATED dimensions excluded from overall medal min (only SCORED dimensions count)

Config/docs

  • config/dimensions.yaml: adds required_metrics_for_scoring lists for all dimensions
  • config/schemas/dimensions.schema.json: schema updated for new field
  • docs/local-scoring.md: documents measurability vs unrated semantics
  • docs/superpowers/artifacts/2026-07-23-metric-calibration-audit-postfix.json: before/after snapshot

Test coverage

  • 239 tests passing (up from 209)
  • Regression tests for every new detection case
  • Integration test guards required_metrics_for_scoring entries are valid declared outputs

srbouffard and others added 16 commits July 23, 2026 18:38
When Allure summary is unavailable, latest_build_passing now falls back to the default branch's latest terminal check-run.

Coverage and stability are now marked unmeasurable (None) instead of forcing zero when no Allure source is present.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ants

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ck-scalar commands

Handle list-item key syntax in matrix.include entries, strip inline YAML comments before scalar checks, and recognize block-scalar run commands so substrate compatibility signals are preserved across common workflow styles.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nt heredoc tokens

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ource

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ree check

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…efaults.run

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@srbouffard
srbouffard merged commit ab9b49c into main Aug 11, 2026
41 checks passed
@srbouffard
srbouffard deleted the feat/metric-calibration-task1 branch August 11, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant