Skip to content

Commit 03c96d9

Browse files
haasonsaasclaudecursoragent
authored
Fix PR lens review: inline comments + split confidence thresholds (#147)
* Fix PR lens review: inline comments + split confidence thresholds The "EvalOps PR lens review" system published 0 findings across the last 12 scheduled runs (~70 lens reviews) yet reported success every time. Three bugs combined to make it silently green: 1. One blunt threshold (DEFAULT_MIN_CONFIDENCE = 0.82) decided both what to show a human and what fails the status, so real medium-confidence findings (0.6-0.8) were silently discarded. 2. Findings posted as one issue comment at the bottom of the PR with `path:line` as plain text, despite every finding carrying an exact code_location. 3. normalize_finding silently returned nil for malformed findings. Changes: - Publish as a single PR review (POST pulls/{pr}/reviews, event COMMENT) with inline comments anchored to each finding's code_location. meta-review parses each file's patch hunks to compute the set of addable right-side line numbers; findings whose path:line is in that set are inlined, the rest are folded into the review summary body (avoids GitHub's 422 on off-diff lines). - Split the threshold in two: comment_min_confidence (default 0.55, env PR_LENS_COMMENT_MIN_CONFIDENCE) for surfacing; block_min_confidence (default 0.80, env PR_LENS_BLOCK_MIN_CONFIDENCE) for failing the status, and only on P0/P1. PR_LENS_MIN_CONFIDENCE / --min-confidence kept as a back-compat alias mapping to the block threshold. - Honest green status: "N lenses · 0 findings >= 0.55" instead of implying nothing was found. - normalize_finding now raises DroppedFinding; the caller counts and warns per dropped finding and records dropped_findings in the ledger. - Idempotency preserved: prior marker issue-comment and prior marker inline review comments are deleted before posting, so re-running on the same head replaces rather than duplicates. - Workflow gains comment_min_confidence / block_min_confidence inputs and keeps min_confidence for back-compat. Test Plan: - ruby -Itest -e 'ARGV.each { |path| require "./#{path}" }' test/*_test.rb => 125 runs, 1016 assertions, 0 failures, 0 errors, 0 skips - New coverage: diff-hunk line parsing, inline-vs-summary split, the two thresholds, idempotent replacement, dropped-finding logging. Rollback: revert this commit; behavior returns to the single-threshold issue-comment publisher. No state migration involved. Signal: 12 consecutive green scheduled runs with 0 published findings (gh run list evalops-pr-lens-review.yml, 2026-06-08..09). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix PR lens review publication idempotency * Fix PR lens publication edge cases * Fix PR lens inline publication fallback * test: stub gh_api_paginated_json so discover_open_prs test makes no live API call #147 changed pr_files_metadata to the paginated helper (gh api --paginate --slurp), which the existing gh_api_json stub no longer intercepts; the test then hit the live API and failed in CI (no GH_TOKEN). Stub gh_api_paginated_json so the test is hermetic. Verified passing under a no-auth gh environment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>
1 parent bedf594 commit 03c96d9

3 files changed

Lines changed: 1037 additions & 94 deletions

File tree

0 commit comments

Comments
 (0)