[ziadantar90]

Requested by Ziad · project thread

What

PullPicket now ships production-grade rules files instead of three short examples. Each file has sample pull requests that test it.

• Standard and quality bar. docs/rules-files.md defines the file layout and how a good rule is written. A file ships only when, on a large model at default settings and in each of three runs, every planted CRITICAL is caught at CRITICAL, at least 80% of planted WARNINGs are caught, and no clean sample gets a CRITICAL or WARNING comment. Small-model scores are published per stack instead of held to the same bar (Ziad's call).
• Test harness. tools/Invoke-RulesEval.ps1 (functions in tools/RulesEval.ps1) uses the engine's own functions to build the exact prompt the pipeline sends. It calls Anthropic, OpenAI or Azure OpenAI, or scores saved answers (-Provider manual, -Rescore), and reports caught, underrated, missed and noise per stack and run. It exits 1 on a failure.
• Samples. 54 samples under evals/<stack>/<sample>/, each a before//after/ tree plus an expect.json. Every stack has a clean sample. ai-code-review/tests/RulesEval.Tests.ps1 checks every sample and the scoring logic. No model is called in CI.
• Rules files. Next.js, NestJS and Terraform are rewritten. New: React/Vite, Angular, Vue, Node/Express, ASP.NET Core, Spring Boot, Go, Django, FastAPI, C++, Rust, Bicep, Kubernetes/Helm, Dockerfiles/Compose and SQL migrations.
• Base prompt. Three noise and severity rules: drop comments that conclude the code is fine, cap comments that depend on code outside the diff at SUGGESTION, and use the severity a repository rule names.
• Docs. docs/rules-results.md publishes the per-stack scores. README, CHANGELOG and CONTRIBUTING are updated.

Results (3 runs per sample, 2 Nov 2026)

Model	Stack runs passing the bar
Claude Opus 5.5 (large)	53 of 54
Claude Haiku 5.5 (small)	23 of 54

The one large-model miss was a latest image tag rated SUGGESTION instead of WARNING in one run. The small model catches most bugs but adds noise on clean Terraform, Bicep, Django, Next.js and C++ samples, and rates SQL migration and Terraform criticals lower. The docs recommend a mid-sized or large model. Full table: docs/rules-results.md.

Checks run locally

• Pester: 239 passed, 0 failed.
• PSScriptAnalyzer with the CI settings: no findings.
• gitleaks over full history: no leaks.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01D8A4vxmHJ5PCcJknu3yJHn