Skip to content

feat(qa-review): advisory review job that complements existing PR checks - #116

Merged
lihsaa591 merged 1 commit into
masterfrom
feat/qa-review-job
Oct 9, 2026
Merged

lihsaa591 merged 1 commit into
masterfrom
feat/qa-review-job

Conversation

@lihsaa591

Copy link
Copy Markdown
Contributor

Summary

Step 2 of the QA-review build. Adds the review job to qa-review.yml (runs after the gate), still manual-dispatch only.

Re-scoped after inspecting the pilot repo: user-registration-pro already has PHPCS-on-PR, an AI security scan, and claudegrill's E2E suite + .themegrill-qa/ data (and claudegrill deliberately keeps AI off the PR path). So this does NOT run its own static tools or a sandbox. It reads the diff + the base branch's .themegrill-qa/ (area map, test-case titles, knowledge) + other checks' results, makes ONE model call, and posts one advisory sticky comment.

  • Never checks out or runs PR code (API reads only).
  • Deterministic: areas touched (from suite.json), other checks' status.
  • Verified, not trusted: a risk is shown only if its file:line is an added line in the diff; a suggested test only if the title exists verbatim in the index. Others are dropped and counted. No confidence number.
  • Model output is stripped of URLs, @mentions, HTML, backticks; untrusted text can't close the prompt's data blocks; unparseable/empty model output fails the job instead of posting a clean-looking review.
  • post_comment=false (default for manual runs) writes the result to the run summary only.

Testing

  • npm test: 50/50 (12 new: glob/area mapping, line-number annotation, diff budget, sanitizer, citation verification, prompt-injection escaping, end-to-end with a fake model incl. stale head / empty summary / repo without QA data).
  • Gather half run against the real PR (themegrill/user-registration-pro#1765) with a stubbed model: prompt about 94k chars (~24k tokens).
  • NOT verified: a real model call (quality, cost; pricing.mjs has no row for the default model) and the comment posting path in a live run.

Notes

Uses the shared OPENAI_API_KEY. A dedicated spend-capped key is advisable before n8n makes this fire automatically. Model override: QA_REVIEW_MODEL repo variable.

@lihsaa591
lihsaa591 merged commit e0430ea into master Oct 9, 2026
2 checks passed
@lihsaa591
lihsaa591 deleted the feat/qa-review-job branch October 9, 2026 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant