Fix and measure the risk ranking (tracescope eval) - #4
Merged
Conversation
computeReviewScore re-derived signals the risk tier already encodes: the 80/50/20 base comes from the tier (itself a function of production callers + export status), and the function then re-added a linear caller bonus and a +10 for exported. Two consequences: exportedness and caller count each moved the score twice, and the linear caller bonus saturated at 24 so a 6-caller and a 600-caller HIGH function tied — the most impactful cases compressed into a single value. - Tier base carries the category (unchanged). - Replace the two saturating caps with one uncapped, sub-linear log1p term on production callers — within-tier granularity, no single hub dominating. - Drop the redundant +10 exported (double-count with the tier). - Keep only signals orthogonal to the tier: depth proximity, is-test, and the heuristic-confidence penalty. - Drop the unused callerCount param from RiskScorer.Score. Two new tests lock the properties: a 60-caller function now outranks a 10-caller one in the same tier (was equal), and export status no longer adds a bonus on top of its tier. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BcVscS2c2WCee2ZcU353bF
Makes the hand-tuned risk score falsifiable. For each recent integration commit the harness indexes the tree at that commit in an isolated git worktree (in-process parser backend — no SCIP, so hundreds of commits are tractable), analyzes the commit's own diff, and records a risk feature. It labels each commit by whether it was later reverted (mined from revert-commit bodies) or hot-fixed (a fix-shaped commit within a window touching overlapping files), then reports whether the risk ranking sorts the bad commits to the top. Every metric ships with a random baseline (sanity: AUC ~0.5) and a changed-lines/churn baseline (the surprisingly hard baseline the JIT literature warns about), so "the ladder beats them" is visible, not asserted. Metrics: AUC (Mann-Whitney, ties at 0.5), Precision@k, and IFA. All pure-stdlib, no new dependency. Metric math is unit-tested. Deferred (documented as next steps, gated on a larger corpus): a logistic- regression ranker and function-level Precision@5 — the per-commit features are already recorded so both are a small bolt-on. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BcVscS2c2WCee2ZcU353bF
Ran `tracescope eval` over 300 commits of gin-gonic/gin (59 labeled risky, 19.7% base rate, 30-day fix window): ranker AUC P@5 P@10 IFA ladder 0.613 0.00 0.20 6.0 churn 0.595 0.00 0.00 13.0 random 0.494 0.22 0.20 5.7 The full-corpus AUC (0.61) looks like signal, but it is almost entirely the coarse bit "does this change touch real functions at all": two-thirds of commits score zero, and restricted to the 99 the tool actually scores, AUC collapses to 0.516 — below raw fan-in (0.541). The hand-tuned weighting adds no measurable value on this corpus. That is the category error made concrete: the score models Impact, and Impact barely predicts defects. P@5 = 0 (the biggest-blast-radius commits were refactors, not the hot-fixed ones) is a real finding, not a bug. - docs/EVALUATION.md: full methodology, results, the scored-subset collapse, limitations, and prioritized next steps (add the P(defect) axis; use SCIP in the eval; then a learned ranker on a larger corpus). - README: document `tracescope eval` and link the writeup. - Architecture explorer: risk-model panel now shows the fixed scoring and cites the measured AUC instead of "unvalidated magic numbers". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BcVscS2c2WCee2ZcU353bF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the risk ranking falsifiable: fixes two structural bugs in the review score, adds an evaluation harness that replays the tool over a repo's history, and writes up the honest — and sobering — result.
Scoring fixes (
24f1b74)computeReviewScoredouble-counted (export + caller count fed both the risk tier and an additive bonus) and saturated (a 6-caller and a 600-caller HIGH function tied). Now the tier carries the category, alog1pterm adds uncapped within-tier granularity, and only tier-orthogonal signals adjust further. Dropped the deadcallerCountparam fromRiskScorer.Score. Two property tests lock the fixes (one written test-first, red → green).tracescope eval(c56d25b)Replays the analysis over a repo's history: worktree-isolated in-process indexing (parser backend, no SCIP, so hundreds of commits are tractable), revert + hot-fix labeling, and ladder-vs-churn-vs-random ranking reporting AUC (Mann–Whitney), Precision@k, and IFA. Pure stdlib, metric math unit-tested. No new dependencies.
The measured result (
7619a09)On 300 commits of gin (59 risky, 19.7% base rate):
The full-corpus AUC looks like signal, but restricted to the 99 commits the tool actually scores it collapses to 0.516 — below raw fan-in (0.541). Two-thirds of commits score zero, so the 0.61 is mostly the coarse "touches functions" bit, not the weighting. The hand-tuned weights add no measurable value on this corpus — the category error (modeling Impact, which barely predicts defects) made concrete.
docs/EVALUATION.mdhas the full methodology, the scored-subset collapse, limitations, and prioritized next steps (add the P(defect) axis; use SCIP in the eval; then a learned ranker on a larger corpus).README documents
tracescope eval; the architecture explorer's risk panel now cites the measured number instead of "unvalidated magic numbers".Verification
go build ./...,go vet ./..., andgo test ./... -raceall green, including the newinternal/evalpackage.🤖 Generated with Claude Code