diff --git a/.github/actions/verify/README.md b/.github/actions/verify/README.md index 75787d3..203b656 100644 --- a/.github/actions/verify/README.md +++ b/.github/actions/verify/README.md @@ -40,7 +40,7 @@ Reproduce a report from its evidence: on_evidence: artifacts/personalization-on.json ``` -After a release tag contains the Action, replace `@main` with that immutable tag. +The examples pin `@submission-rc-03`, the immutable release that contains this Action. Use `@main` only if you intentionally want to track later changes on the default branch. ## Inputs diff --git a/ADOPTION.md b/ADOPTION.md index c424a11..5bb0d22 100644 --- a/ADOPTION.md +++ b/ADOPTION.md @@ -124,6 +124,37 @@ npm run promiseproof -- check \ `check` also accepts a single report with `--report ... --evidence ...`. +## Committed examples + +The repository ships a ready evidence set under `artifacts/verify/`, so you can reproduce a result without scaffolding anything first: + +- `passing-off.example.json`, `passing-on.example.json`: a passing OFF and ON pair. +- `passing-gate.report.json`, `passing-gate.report.md`: the sealed gate report for that pair. +- `broken-off.example.json`: an OFF bundle that breaks the promise. + +```bash +# Regenerate the sealed report byte for byte: +npm run promiseproof -- check \ + --report artifacts/verify/passing-gate.report.json \ + --off artifacts/verify/passing-off.example.json \ + --on artifacts/verify/passing-on.example.json # BOUND_AND_REPRODUCED (exit 0) + +# Watch a broken bundle fail the gate: +npm run promiseproof -- gate \ + --off artifacts/verify/broken-off.example.json \ + --on artifacts/verify/passing-on.example.json # BROKEN_PROMISE (exit 2) +``` + +## Programmatic use + +PromiseProof does not publish an npm package, and the verdict is not hidden behind a service. Inside a checkout, the entire verdict comes from one pure function, `evaluatePromise(evidence)` in `src/shared/evaluator.ts`: no network, no state, no model. This deterministic function is the authority that owns the verdict. The CLI, the GitHub Action, and the hosted browser verifier all call it, which is why their results are identical, and the models never call it or produce its output. + +```ts +import { evaluatePromise } from "./src/shared/evaluator"; + +const evaluation = evaluatePromise(evidence); // deterministic verdict, clause results, and violations +``` + ## Reports `verify` and `gate` write two files to `--out`: diff --git a/BUILD_WEEK.md b/BUILD_WEEK.md index ab5f5ac..d2852a0 100644 --- a/BUILD_WEEK.md +++ b/BUILD_WEEK.md @@ -1,5 +1,47 @@ # PromiseProof Build Week Log +## What PromiseProof shipped by 2026-07-21 + +PromiseProof is a new OpenAI Build Week project, built July 14 to July 21, 2026. This section summarizes the finished release. The dated milestone log that follows is the original historical record. + +The finished release carries one broken product promise from symptom to proof: + +- a synthetic reference product with a real browser-to-backend HTTP boundary; +- one user-facing "off means off" promise that crosses UI, storage, network, and backend; +- two independent seeded defects that produce different evidence and different violation codes, so one flag cannot explain both and one fix cannot silence the other; +- a bounded GPT-5.6 investigation over a sanitized, versioned dossier; +- two allowlisted deterministic factual replays; +- an authentic, human-approved Codex source repair; +- exact, digest-bound human approval of the patch; +- application only in a fresh disposable git worktree; +- an unchanged deterministic verifier that alone decides PASS; +- a five-stage hosted walkthrough of the recorded run; +- external, evidence-only verification through a repository-local CLI; +- deterministic JSON and Markdown reports, bound to the evaluator source by digest; +- complete report reproduction that recomputes the report, not only its hashes; +- a hosted semantic tamper challenge that runs entirely in the browser; +- a reusable GitHub Action exercised on Windows, Ubuntu, and macOS runners; +- public judge and adoption documentation. + +Timeline, tag by tag: + +- **Jul 14** baseline, the deterministic initialization-race proof, and the propagation-failure defect (`build-week-baseline`, `milestone-02-deterministic-foundation`). +- **Jul 15** the bounded GPT-5.6 investigation layer (`milestone-03-gpt56-investigation`). +- **Jul 16** the authentic Codex repair with human approval (`milestone-04-authentic-repair-proof`, `submission-hardening-01`). +- **Jul 17** CI hardening and the judge experience (`ci-green-01`, `judge-rehearsal-01`, `judge-experience-01`). +- **Jul 20 to 21** the public product: hosted verifier and CLI (PR #1), the reusable GitHub Action (PR #2), and the judge package (PR #3), tagged `submission-rc-01` through `submission-rc-03`. + +**Why one week matters.** Building this in a single week demonstrates execution speed and architectural closure: the complete working lifecycle, its tests, the authentic repair proof, the hosted product, the CLI, and the CI integration were all built and verified inside the submission period. It is not a claim of production maturity or third-party adoption. The honest scope and limitations are recorded below and in the [README](README.md#limitations). + +Start here: + +- [JUDGE_START_HERE.md](JUDGE_START_HERE.md) +- [ADOPTION.md](ADOPTION.md) +- Hosted walkthrough: https://promiseproof.alex0paiva0.workers.dev/walkthrough/ +- Hosted verifier: https://promiseproof.alex0paiva0.workers.dev/verify/?judge=1 +- Pull requests: [#1](https://github.com/AlexPaiva/PromiseProof/pull/1), [#2](https://github.com/AlexPaiva/PromiseProof/pull/2), [#3](https://github.com/AlexPaiva/PromiseProof/pull/3) +- Submission tags: `submission-rc-01`, `submission-rc-02`, `submission-rc-03` + ## Project baseline — 2026-07-14 PromiseProof begins as a new OpenAI Build Week project. `AGENTS.md` is the canonical product and engineering specification. diff --git a/JUDGE_START_HERE.md b/JUDGE_START_HERE.md index 9c17b78..ffd9ca3 100644 --- a/JUDGE_START_HERE.md +++ b/JUDGE_START_HERE.md @@ -35,6 +35,18 @@ A bundled Action that runs the same verifier in CI with no `npm install`, no bro | Human | Yes | No | Yes | No | | Unchanged verifier | No | No | No | Yes | +## The model cannot award itself PASS + +This split is enforced in code and covered by tests, not merely asserted: + +- **No verdict field exists.** The final investigation contract in `src/investigation/contracts.ts` has no overall-verdict field; the only verdict-adjacent limitation code is `diagnostic_not_verdict` ("Diagnostic hypotheses do not determine the product promise verdict."). +- **Verdict language from the model is rejected.** `tests/investigation/investigation.unit.ts` asserts `PP_INV_VERDICT_LANGUAGE_REJECTED` when model output uses reserved verdict wording. +- **A claimed model verdict is ignored by the matrix.** `tests/judge/rehearsal.unit.ts`, "verification matrix rejects missing propagation control, browser errors, and model verdict claims," proves a supplied `modelVerificationVerdict` cannot turn a fail into a pass. +- **Approval is human and digest-bound.** `tests/repair/artifact-approval.unit.ts`, "accepts only the exact APPROVE or REJECT phrase for the current digest," proves the patch is approved by a human against its exact fingerprint, and the lifecycle records `modelVerdictUsed: false`. +- **The verdict path needs no model at all.** `tests/judge/rehearsal.unit.ts`, "offline rehearsal environment removes model credentials and Codex configuration," runs the whole verification with no model present. + +This is deterministic, code-level separation. It is not a formal or mathematical proof. + ## Recorded versus fresh - **Recorded and authentic:** the GPT-5.6 investigation, the Codex source repair, and the five-stage walkthrough that presents them. These happened once and are replayed, not re-run live. diff --git a/README.md b/README.md index 101b2e2..bdb31b7 100644 --- a/README.md +++ b/README.md @@ -6,15 +6,16 @@ **GPT-5.6 investigates. Codex repairs. Neither decides PASS.** -Carry a broken product promise all the way through bounded diagnosis, constrained repair, and independent verification, where a deterministic test, *never a model*, decides PASS. +A human approves the exact patch. An unchanged deterministic verifier decides whether the promise is actually fixed. +[![CI](https://github.com/AlexPaiva/PromiseProof/actions/workflows/submission-hardening.yml/badge.svg)](https://github.com/AlexPaiva/PromiseProof/actions/workflows/submission-hardening.yml) [![OpenAI Build Week 2026](https://img.shields.io/badge/OpenAI-Build_Week_2026-10a37f)](https://openai.com) [![License: MIT](https://img.shields.io/badge/License-MIT-3f6bf0.svg)](LICENSE) [![TypeScript](https://img.shields.io/badge/TypeScript-strict-3178c6)](https://www.typescriptlang.org/) [![Playwright](https://img.shields.io/badge/Playwright-verified-2fa968)](https://playwright.dev/) [![Cloudflare Workers](https://img.shields.io/badge/hosted-Cloudflare_Workers-f38020)](https://workers.cloudflare.com/) -**[▶ Live demo](https://promiseproof.alex0paiva0.workers.dev/)** · **[Judge Start Here](JUDGE_START_HERE.md)** · **[Judge walkthrough](https://promiseproof.alex0paiva0.workers.dev/walkthrough/)** · [How it works](#how-it-works) · [Quick start](#quick-start-the-judge-path) +**[▶ Live demo](https://promiseproof.alex0paiva0.workers.dev/)** · **[⚡ Challenge the proof](https://promiseproof.alex0paiva0.workers.dev/verify/?judge=1)** · **[Judge Start Here](JUDGE_START_HERE.md)** · **[Walkthrough](https://promiseproof.alex0paiva0.workers.dev/walkthrough/)** · [How it works](#how-it-works) · [Run it in CI](#try-it-yourself) ![PromiseProof: find the boundary that broke the promise](public/og-card.png) @@ -22,6 +23,36 @@ Carry a broken product promise all the way through bounded diagnosis, constraine --- +## The 30-second version + +1. **A user turned personalization off.** The interface, the browser storage, and the backend all reported the same thing: off. +2. **One identifiable request still crossed into recommendations.** The product had quietly broken the promise it showed the user. +3. **GPT-5.6 investigated within strict limits, Codex proposed a real source repair in isolation, and a human approved the exact change.** +4. **The same unchanged test that caught the break decided whether the repair worked, not a model.** You can challenge that verdict yourself in the browser, and drop the identical check into your CI. + +Most tools confirm that your tests pass. PromiseProof answers the one question a model should never answer about its own work: *is the fix actually real?* + +## See the proof + +Two states of the hosted verifier. Same evaluator, no model in the verdict path, nothing uploaded. + +![PromiseProof verifier returning PASS with five passing clauses](docs/screenshots/03-unchanged-verifier.png) + +*The unchanged evaluator, running in your browser, returns **PASS** bound to this evidence. Five clauses hold across OFF, reload, and the ON control. The verdict path makes no model call.* + +![PromiseProof verifier after a tamper, showing BROKEN_PROMISE and a stale sealed report](docs/screenshots/04-challenge-and-ci.png) + +*Change one load-bearing observation and the same evaluator flips to **BROKEN_PROMISE** with `PP_IDENTIFIABLE_EVENT_LEAK`, while the report you sealed a moment ago no longer reproduces (**STALE_OR_MISMATCH**). A PASS cannot be carried onto changed evidence.* + +The first two beats, the broken promise and the bounded investigation, play in order in the [five-stage walkthrough](https://promiseproof.alex0paiva0.workers.dev/walkthrough/). + + + ## The problem A user turns personalization **off**. The UI says off. Browser storage says off. The backend says off. And yet one *identifiable* request still crosses the boundary to the recommendation service. @@ -32,20 +63,85 @@ It turns an "off means off" promise into an executable check, helps locate the i ### Who it is for -PromiseProof is for product, QA, privacy, reliability, and platform engineers responsible for user-facing controls that cross browser, storage, network, and backend boundaries. It turns an ambiguous report like "OFF did not behave like OFF" into reproducible evidence and a bounded diagnostic action, so a team finds the responsible subsystem sooner. +PromiseProof is for product, QA, privacy, reliability, and platform engineers responsible for user-facing controls that cross browser, storage, network, and backend boundaries. It turns an ambiguous report like "OFF did not behave like OFF" into reproducible evidence and a bounded diagnostic action, so a team finds the responsible subsystem sooner. As coding agents start changing repositories on their own, it also gives those teams an acceptance gate that is inspectable, human-approved, and reproducible in the CI they already trust. + +## Why this is more than a privacy test + +The personalization toggle is the proving example, not the limit of the idea. User-facing promises routinely cross the interface, storage, network, and backend at once, and every one of those surfaces can look locally correct while the combined behavior is wrong. A monitor can surface that contradiction. PromiseProof is what happens next: it carries the contradiction through bounded diagnosis, a real source repair, exact human approval, independent acceptance, full report reproduction, and CI enforcement, without ever letting the model that proposed the fix certify it. It supports one contract family today, `activity-personalization/v1`, and the mechanism is built to generalize; the honest scope is stated in [Limitations](#limitations). -## What makes it different +## Who decides it is fixed > **A model may investigate and repair. Deterministic evidence keeps the verdict.** -The final investigation schema has **no verdict field**. GPT-5.6 proposes and ranks candidate causes and requests one *allowlisted* diagnostic replay; Codex proposes a constrained two-file repair in a disposable worktree; a human approves the exact patch by its fingerprint. Then an **unchanged Playwright journey and deterministic evaluator**, the very ones that caught the break, decide PASS or FAIL. +The final investigation schema has **no verdict field**. GPT-5.6 proposes and ranks candidate causes and requests one *allowlisted* diagnostic replay. Codex proposes a constrained two-file repair in a disposable worktree. A human approves the exact patch by its fingerprint. Then an **unchanged Playwright journey and deterministic evaluator**, the very ones that caught the break, decide PASS or FAIL. + +The AI never grades its own work. That is the whole point. The authority table and the tests that enforce it are in [JUDGE_START_HERE.md](JUDGE_START_HERE.md#the-model-cannot-award-itself-pass). + +## Why the patch is small + +The approved repair is intentionally small, and that is the point. Generating two changed files is not the hard part. The hard part is proving that the correct implementation boundary changed, that the restricted OFF behavior is now protected, that the permitted ON behavior still works, that the applied patch is exactly what the human approved, and that neither model was able to certify its own code. A one-line fix carried through that chain proves more than a large patch a model graded itself. + +| Verified fact | Current release | +| --- | --- | +| Distinguishable seeded failures | 2 | +| Allowlisted factual replays | 2 | +| Authentic approved Codex repair | 1 | +| Independent verification clauses | 5 | +| Developer surfaces | Browser, CLI, GitHub Action | +| Action runner operating systems | Windows, Ubuntu, macOS | +| Model-owned verdicts | 0 | +| Model calls in the verifier path | 0 | + +## Try it yourself + +Three ways in, none of them require an API key. + +**1. Watch the complete lifecycle.** The [five-stage walkthrough](https://promiseproof.alex0paiva0.workers.dev/walkthrough/) replays one authentic run: Observe, Investigate, Replay, Repair, Prove. The GPT-5.6 and Codex execution is authentic and recorded, not re-run live for remote judges. + +**2. Challenge the proof.** Open the **[hosted verifier](https://promiseproof.alex0paiva0.workers.dev/verify/?judge=1)** (no login, no key, everything runs in your browser): + +- On load: `PASS` and `BOUND_AND_REPRODUCED`. +- Press **Tamper OFF evidence**: the same evaluator returns `BROKEN_PROMISE` with `PP_IDENTIFIABLE_EVENT_LEAK`, and the report you sealed a moment ago becomes `STALE_OR_MISMATCH`. +- **Seal** the failing result: it binds honestly as `BROKEN_PROMISE`. A report can bind a failure; it cannot manufacture a PASS. +- **Download report.json / report.md**, or load your own OFF and ON bundles under "Bring your own evidence." +- **Reset** to return to the original PASS. + +**3. Run it in CI.** Drop the same verifier into any workflow with no `npm install`, no browser, and no key: + +```yaml +- uses: AlexPaiva/PromiseProof/.github/actions/verify@submission-rc-03 + with: + mode: gate + off_evidence: artifacts/personalization-off.json + on_evidence: artifacts/personalization-on.json + output_directory: artifacts/promiseproof +``` + +The step fails on `BROKEN_PROMISE` (exit 2) and still leaves `report.json` and `report.md` behind. + +**4. Reproduce a sealed receipt, with nothing to generate first.** The repo ships a committed report and the evidence it was computed from. After `npm ci`, regenerate that report byte for byte: + +```bash +npm run promiseproof -- check \ + --report artifacts/verify/passing-gate.report.json \ + --off artifacts/verify/passing-off.example.json \ + --on artifacts/verify/passing-on.example.json +# BOUND_AND_REPRODUCED (exit 0) +``` + +Swap in the committed `artifacts/verify/broken-off.example.json` and `gate` returns `BROKEN_PROMISE` (exit 2). No `init`, no scaffolding, no key. Full CLI, exit codes, and exact scope are in [PromiseProof Verify](#try-promiseproof-verify) below. + +## What was built in one Build Week -The AI never grades its own work. That is the whole point. +PromiseProof is new work, built July 14 to July 21, 2026, with a linear history a judge can walk from tag to tag: -## See it live +- **Jul 14** deterministic foundation: the synthetic app, the real browser-to-backend boundary, and the first seeded defect proven red. +- **Jul 15** bounded GPT-5.6 investigation over a sanitized dossier. +- **Jul 16** authentic, human-approved Codex source repair in a disposable worktree. +- **Jul 17** CI hardening and the judge walkthrough experience. +- **Jul 20 to 21** the public product: hosted verifier, external CLI, report reproduction, the cross-platform GitHub Action, and the judge and adoption guides. -- 🌐 **[promiseproof.alex0paiva0.workers.dev](https://promiseproof.alex0paiva0.workers.dev/)** hosts the landing page at `/` and the interactive judge walkthrough at `/walkthrough/`. -- The walkthrough is a **recorded run** of one concrete synthetic failure and its approved repair, with no live model calls. It is reproducible without an API key once dependencies are installed. +The full record, tags, and honest limitations are in [BUILD_WEEK.md](BUILD_WEEK.md). ## How it works @@ -227,7 +323,7 @@ The full milestone record, live-receipt details, and limitations live in [`BUILD - **Signal Shelf is synthetic.** It exists to make one broken promise visible and repairable end to end; it is not a real product, and PromiseProof makes no legal or regulatory compliance claim. - The integrity evidence is **internal and repository-level**, not external provider attestation. -- The current checkpoint is directly verified on **Windows 10 x64** with Node 22 and Playwright Chromium; macOS and Linux use the same cross-platform primitives but are not yet claimed as verified. +- The full investigation-and-repair lifecycle and the elevated repair sandbox are directly verified on **Windows 10 x64** (Node 22, Playwright Chromium); macOS and Linux use the same cross-platform primitives but are not yet claimed as verified for that lifecycle. The reusable verifier surface a team actually adopts, the bundled **GitHub Action**, is exercised on **Windows, Ubuntu, and macOS** runners. - The authentic Codex repair provider is intentionally restricted to a trusted, administrator-provisioned elevated Windows sandbox and **fails closed**, so it never falls back to a weaker backend. ## License diff --git a/docs/screenshots/03-unchanged-verifier.png b/docs/screenshots/03-unchanged-verifier.png new file mode 100644 index 0000000..649294e Binary files /dev/null and b/docs/screenshots/03-unchanged-verifier.png differ diff --git a/docs/screenshots/04-challenge-and-ci.png b/docs/screenshots/04-challenge-and-ci.png new file mode 100644 index 0000000..2104f38 Binary files /dev/null and b/docs/screenshots/04-challenge-and-ci.png differ