Commit 03c96d9
Fix PR lens review: inline comments + split confidence thresholds (#147)
* Fix PR lens review: inline comments + split confidence thresholds
The "EvalOps PR lens review" system published 0 findings across the last
12 scheduled runs (~70 lens reviews) yet reported success every time.
Three bugs combined to make it silently green:
1. One blunt threshold (DEFAULT_MIN_CONFIDENCE = 0.82) decided both what
to show a human and what fails the status, so real medium-confidence
findings (0.6-0.8) were silently discarded.
2. Findings posted as one issue comment at the bottom of the PR with
`path:line` as plain text, despite every finding carrying an exact
code_location.
3. normalize_finding silently returned nil for malformed findings.
Changes:
- Publish as a single PR review (POST pulls/{pr}/reviews, event COMMENT)
with inline comments anchored to each finding's code_location.
meta-review parses each file's patch hunks to compute the set of
addable right-side line numbers; findings whose path:line is in that
set are inlined, the rest are folded into the review summary body
(avoids GitHub's 422 on off-diff lines).
- Split the threshold in two: comment_min_confidence (default 0.55,
env PR_LENS_COMMENT_MIN_CONFIDENCE) for surfacing; block_min_confidence
(default 0.80, env PR_LENS_BLOCK_MIN_CONFIDENCE) for failing the
status, and only on P0/P1. PR_LENS_MIN_CONFIDENCE / --min-confidence
kept as a back-compat alias mapping to the block threshold.
- Honest green status: "N lenses · 0 findings >= 0.55" instead of
implying nothing was found.
- normalize_finding now raises DroppedFinding; the caller counts and
warns per dropped finding and records dropped_findings in the ledger.
- Idempotency preserved: prior marker issue-comment and prior marker
inline review comments are deleted before posting, so re-running on
the same head replaces rather than duplicates.
- Workflow gains comment_min_confidence / block_min_confidence inputs
and keeps min_confidence for back-compat.
Test Plan:
- ruby -Itest -e 'ARGV.each { |path| require "./#{path}" }' test/*_test.rb
=> 125 runs, 1016 assertions, 0 failures, 0 errors, 0 skips
- New coverage: diff-hunk line parsing, inline-vs-summary split, the two
thresholds, idempotent replacement, dropped-finding logging.
Rollback: revert this commit; behavior returns to the single-threshold
issue-comment publisher. No state migration involved.
Signal: 12 consecutive green scheduled runs with 0 published findings
(gh run list evalops-pr-lens-review.yml, 2026-06-08..09).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix PR lens review publication idempotency
* Fix PR lens publication edge cases
* Fix PR lens inline publication fallback
* test: stub gh_api_paginated_json so discover_open_prs test makes no live API call
#147 changed pr_files_metadata to the paginated helper (gh api --paginate --slurp),
which the existing gh_api_json stub no longer intercepts; the test then hit the live
API and failed in CI (no GH_TOKEN). Stub gh_api_paginated_json so the test is hermetic.
Verified passing under a no-auth gh environment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>1 parent bedf594 commit 03c96d9
3 files changed
Lines changed: 1037 additions & 94 deletions
File tree
- .github
- scripts
- workflows
- test
0 commit comments