Repository navigation
feat(qa-review): advisory review job that complements existing PR checks - #116
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Step 2 of the QA-review build. Adds the
reviewjob toqa-review.yml(runs after the gate), still manual-dispatch only.Re-scoped after inspecting the pilot repo:
user-registration-proalready has PHPCS-on-PR, an AI security scan, and claudegrill's E2E suite +.themegrill-qa/data (and claudegrill deliberately keeps AI off the PR path). So this does NOT run its own static tools or a sandbox. It reads the diff + the base branch's.themegrill-qa/(area map, test-case titles, knowledge) + other checks' results, makes ONE model call, and posts one advisory sticky comment.suite.json), other checks' status.file:lineis an added line in the diff; a suggested test only if the title exists verbatim in the index. Others are dropped and counted. No confidence number.post_comment=false(default for manual runs) writes the result to the run summary only.Testing
npm test: 50/50 (12 new: glob/area mapping, line-number annotation, diff budget, sanitizer, citation verification, prompt-injection escaping, end-to-end with a fake model incl. stale head / empty summary / repo without QA data).pricing.mjshas no row for the default model) and the comment posting path in a live run.Notes
Uses the shared
OPENAI_API_KEY. A dedicated spend-capped key is advisable before n8n makes this fire automatically. Model override:QA_REVIEW_MODELrepo variable.