Skip to content

Commit b454129

Browse files
haasonsaasclaude
andcommitted
Fix PR lens review: inline comments + split confidence thresholds
The "EvalOps PR lens review" system published 0 findings across the last 12 scheduled runs (~70 lens reviews) yet reported success every time. Three bugs combined to make it silently green: 1. One blunt threshold (DEFAULT_MIN_CONFIDENCE = 0.82) decided both what to show a human and what fails the status, so real medium-confidence findings (0.6-0.8) were silently discarded. 2. Findings posted as one issue comment at the bottom of the PR with `path:line` as plain text, despite every finding carrying an exact code_location. 3. normalize_finding silently returned nil for malformed findings. Changes: - Publish as a single PR review (POST pulls/{pr}/reviews, event COMMENT) with inline comments anchored to each finding's code_location. meta-review parses each file's patch hunks to compute the set of addable right-side line numbers; findings whose path:line is in that set are inlined, the rest are folded into the review summary body (avoids GitHub's 422 on off-diff lines). - Split the threshold in two: comment_min_confidence (default 0.55, env PR_LENS_COMMENT_MIN_CONFIDENCE) for surfacing; block_min_confidence (default 0.80, env PR_LENS_BLOCK_MIN_CONFIDENCE) for failing the status, and only on P0/P1. PR_LENS_MIN_CONFIDENCE / --min-confidence kept as a back-compat alias mapping to the block threshold. - Honest green status: "N lenses · 0 findings >= 0.55" instead of implying nothing was found. - normalize_finding now raises DroppedFinding; the caller counts and warns per dropped finding and records dropped_findings in the ledger. - Idempotency preserved: prior marker issue-comment and prior marker inline review comments are deleted before posting, so re-running on the same head replaces rather than duplicates. - Workflow gains comment_min_confidence / block_min_confidence inputs and keeps min_confidence for back-compat. Test Plan: - ruby -Itest -e 'ARGV.each { |path| require "./#{path}" }' test/*_test.rb => 125 runs, 1016 assertions, 0 failures, 0 errors, 0 skips - New coverage: diff-hunk line parsing, inline-vs-summary split, the two thresholds, idempotent replacement, dropped-finding logging. Rollback: revert this commit; behavior returns to the single-threshold issue-comment publisher. No state migration involved. Signal: 12 consecutive green scheduled runs with 0 published findings (gh run list evalops-pr-lens-review.yml, 2026-06-08..09). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent bedf594 commit b454129

3 files changed

Lines changed: 662 additions & 76 deletions

File tree

0 commit comments

Comments
 (0)