PR Integrity Review

Design specifications and reproducible build process for an Archestra application that reviews GitHub pull requests for scope creep, weakened tests, dependency risk and incomplete validation

Status

• The Archestra application was completed and stored inside Archestra
• The hackathon submission was merged
• The public demo pull request remains available for reproducible evaluation

What this repository is

This is the preparation workspace, written before the application existed. It holds the material that made a single authoring session possible:

• MVP and architecture specifications, including inputs, interface states, interaction flow, size caps and acceptance criteria
• Six deterministic review signals with their detection rules, thresholds and false positive boundaries
• Evidence and confidence semantics, covering how partial or missing input changes a result
• GitHub MCP tool constraints, expressed as capabilities rather than invented tool names
• Synthetic demo planning, including the fixture codebase, the demo pull request and the expected per signal outcome
• The final Archestra build prompt, pasted once as the first authoring message
• Submission and validation notes, including a privacy checklist and stop conditions

Nothing here is executed: no build step, no package manifest, no test suite

What this repository is not

• Not the source repository of the generated application. It was created and stored inside Archestra as a single sandboxed HTML document, not a package based project checked in here
• Not a production GitHub application, and not an installable GitHub App or Action
• Not a security scanner. It performs no static analysis, no dependency vulnerability lookup and no licence checking
• Not an AI-authorship detector. It never attempts to decide who or what wrote a change
• Not a replacement for human code review. It orders evidence for a reviewer who then reads the diff
• Not a repository containing client or private code. Every example in it is synthetic

Product overview

The application takes one pull request at a time and turns it into an ordered review checklist:

GitHub pull request
-> metadata
-> changed files and patches
-> checks and commit statuses
-> deterministic integrity signals
-> ordered evidence
-> reviewer decision

The reviewer ends at one of four recorded decisions, with an optional note.

The runtime does not use an LLM. Every signal is computed from string and numeric comparisons over the GitHub response, so two reviewers see the same result and any card can be explained by pointing at the rule that produced it. The Archestra platform does expose a runtime completion call, and it is deliberately unused

The application performs no GitHub write actions. It does not approve, merge, comment, or create review state. It requests read capabilities only, and the reviewer decision is written to private user storage rather than back to GitHub

Deterministic review model

Six signals, each rendered as a card with its own evidence:

• task-context asks whether the description gives a reviewer enough to review against, looking at body length, issue links and acceptance criteria, escalating on large changes
• scope-spread measures how broadly the change is distributed across reviewable files, top level paths and changed lines
• test-coverage-balance asks whether production behavior changed without any corresponding test change
• test-weakening looks for tests being removed, skipped, or relaxed, and for coverage thresholds being lowered
• dependency-surface reports manifest and lockfile changes, and escalates when a new dependency is never mentioned in the description
• validation-evidence reports what actually ran on the head commit, and whether anything failed or is still pending

Each signal resolves to one of three states:

• attention means the rule matched on evidence actually seen. It marks something worth a reviewer's eyes, and is explicitly not a defect claim
• clear means the rule ran on sufficient data and did not match. The measured values are still shown, so a green card is still readable evidence
• inconclusive means the evidence in hand supports neither of the above, because required input was missing, denied, capped, or truncated

Two rules govern incomplete input. Positive evidence survives it, so a condition already demonstrated stays at attention and the card discloses that coverage was partial. Missing input prevents a clear result, because an unread file could have changed the answer

Page confidence is a separate axis from signal state. It measures changed file and patch coverage only, that is, how much of the diff the analysis actually read. It does not measure whether CI data is complete, so a denied or incomplete CI stage affects validation-evidence without changing the page badge

The full rules, including every threshold, pattern and false positive boundary, are in docs/REVIEW_SIGNALS.md. They are deliberately not reproduced here

Evidence stages

The application requests exactly three read-only capabilities for one pull request:

• E1: pull request metadata, including title, body, author, state, draft flag, head SHA, changed file count, additions and deletions
• E2: the changed files with per file status, additions, deletions and unified diff patch text where GitHub provides it
• E3: CI results for the head commit, meaning check runs and commit statuses with name, status and conclusion

The application uses the smallest available read-only GitHub MCP tool set covering those three stages. Issues, commits, repository trees, code search and existing review state are not fetched, because no signal consumes them. Linked issue context is read from the body text alone

Why a separate demo repository exists

The fixture lives at olgaiv39/pr-integrity-demo and exists to provide:

• Public synthetic data that anyone can inspect
• A stable pull request that does not change under evaluation
• Reproducible patches and CI outcomes on a known head commit
• No dependency on client, employer or private repositories
• A controlled combination of positive and negative review evidence, so both attention and clear cards appear in one view

It is a small TypeScript checkout pricing library with Jest tests and a GitHub Actions workflow. It is not a production library and it is not the application source

The recorded demo pull request is pr-integrity-demo#2. It combines an ordinary looking bug fix with a widened blast radius, a skipped test, a lowered coverage threshold, an undeclared runtime dependency and a failing check

The application is not hard-coded to that repository, its filenames, or its expected result. It accepts any pull request the authenticated GitHub connection can read, classifies files by extension and path convention rather than by name, and computes each signal from the response it actually receives. The fixture is a stable demonstration target, not a special case in the logic

The one-prompt build

The final application was generated in one authoring session, from one build prompt.

That result was possible because the work in this repository came first. Product behavior, signal semantics, size caps, the demo fixture and the authoring tool workflow had already been specified and validated, so the authoring session had to make no product decisions. The complete project did not take one prompt. One prompt was the final step of a longer preparation sequence

The application was built with Claude Sonnet 5 inside Archestra. During authoring, the process inspected the real GitHub MCP tool schemas and previewed real tool responses against the public demo pull request before the final interface was generated, so the bound tool names and response shapes came from observed behavior rather than assumption

The runtime application itself does not call Sonnet or any other model. The model was used to write the application, not to run it

Build process

1. Define product boundaries, including explicit non-goals
2. Specify deterministic evidence rules for every signal
3. Create a clean synthetic TypeScript baseline that genuinely builds and tests
4. Construct a controlled demo pull request against that baseline
5. Verify the real GitHub CI and patch evidence the pull request produces
6. Update the authoring prompt with the public demo URL
7. Inspect the available GitHub MCP schemas and select the smallest covering tool set
8. Scaffold the app in Archestra and persist the specification
9. Preview real tool responses for each bound capability
10. Validate the generated app
11. Record and submit the application

Repository structure

CLAUDE.md
docs/MVP_SPEC.md
docs/REVIEW_SIGNALS.md
docs/ARCHESTRA_BUILD_PROMPT.md
docs/DEMO_AND_SUBMISSION.md

• CLAUDE.md sets the working rules for this preparation repository, including its scope boundaries, source of truth table and stop conditions
• docs/MVP_SPEC.md is the product specification: problem, users, inputs, required GitHub evidence, interface states, size limits, decision workflow, non-goals and acceptance criteria
• docs/REVIEW_SIGNALS.md is the deterministic logic: file classification, patch parsing, and the complete rule for each of the six signals with its thresholds and false positive boundary
• docs/ARCHESTRA_BUILD_PROMPT.md is the authoring prompt that was pasted into Archestra, plus the short procedure for using it
• docs/DEMO_AND_SUBMISSION.md plans the fixture repository and demo pull request, states the exact evidence each signal should surface, and holds the recording, privacy and submission notes

Public and legal boundary

This project uses only independently designed logic, public GitHub data, synthetic demo code, and synthetic issues and pull requests written for the demonstration.

It contains no client code, no internal benchmarks, no private datasets, no private communications and no credentials. It makes no claim to have been built for, or to evaluate, any specific company

Limitations

• The signals are review heuristics, not proofs. An attention card means look here, not defect found
• Renamed test helpers or unusual test syntax may affect text based evidence detection, which reads patch text rather than executing the suite
• Unavailable, capped, or truncated patches can produce inconclusive rather than a definite answer
• GitHub response schemas and the available MCP tools may differ between installations, so a capability present in one environment may be missing in another
• The application does not execute code, run tests, or reproduce a build
• It does not determine whether code was generated by an AI system
• It does not establish that a pull request is safe to merge

Related project

claude-oss-skills controls coding agent behavior before and during implementation, shaping how the work is carried out. PR Integrity Review examines the resulting pull request afterwards, before human approval or merge

The two are independent and address different points in the same lifecycle. Neither requires the other, and this application is not an interface to that repository

Hackathon submission

The merged submission is archestra-ai/apps-gallery#77. The application was submitted in the Workflow Tools category

Reviewing the project

For a technical reviewer, the shortest useful path is:

1. Read docs/MVP_SPEC.md for scope, inputs and non-goals
2. Inspect the complete signal definitions in docs/REVIEW_SIGNALS.md
3. Open the synthetic demo pull request and form your own view of what a reviewer should notice there
4. Review the final build prompt in docs/ARCHESTRA_BUILD_PROMPT.md
5. Compare the expected evidence in docs/DEMO_AND_SUBMISSION.md with the recorded submission