Skip to content

README.md

Latest commit

 

History

History
58 lines (41 loc) · 2.15 KB

README.md

File metadata and controls

58 lines (41 loc) · 2.15 KB
Raw

Benchmark

This is a small behavioral check, not an academic benchmark and not a claim that a prompt can prove arbitrary software correct.

It compares the same model, prompt, evidence, output schema, and reasoning setting in fresh workspaces:

  • baseline: no repository skill is available;
  • treatment: the committed skills/prove-it/SKILL.md is installed and explicitly invoked as $prove-it.

The twelve cases include misleading green signals, insufficient evidence, genuinely blocked verification, and three positive controls. Positive controls matter: a skill that always says NOT PROVEN is not useful.

Run

Use an exact model name and keep it in the published result:

node benchmark/check.mjs
node benchmark/run.mjs \
  --model gpt-5.6-luna \
  --reasoning low \
  --runs 1 \
  --mode both \
  --output benchmark/results/local-run.json

Run one case while iterating:

node benchmark/run.mjs \
  --model gpt-5.6-luna \
  --case weakened-assertion \
  --mode both

The runner records exact model and CLI versions, SHA-256 hashes of the skill and cases, raw structured outputs, and four intentionally simple metrics:

  • verdict accuracy;
  • false assurance rate;
  • decisive-signal recall;
  • positive-control accuracy.

Keyword matching is used only for case-specific decisive-signal recall. Read raw outputs before drawing conclusions. One run per cell is directional evidence, not a stable model-quality estimate.

Published result

The committed 2026-08-18 Codex result contains all 24 structured outputs.

Metric Baseline With Prove It
Verdict accuracy 75% 100%
False assurance rate 0% 0%
Decisive-signal recall 96% 96%
Falsification attempt rate 100% 100%
Positive-control accuracy 67% 100%

Environment: codex-cli 0.147.0, gpt-5.6-luna, low reasoning, one run per cell. The result records SHA-256 fingerprints for the exact skill and case set.

The cases were authored during skill development, so the result is not an independent evaluation. Treat it as a transparent, reproducible regression test for this repository.