results: real conformal FP-control result on public MBPP+ data

README.md

@@ -18,6 +18,7 @@ private anti-hack traps, or private engine internals.
 
 - [`evidence/sample_assurance_card_redacted.json`](evidence/sample_assurance_card_redacted.json)
 - [`evidence/aggregate_metrics_sample.md`](evidence/aggregate_metrics_sample.md)
+- [`results/conformal_fp_control_mbpp.json`](results/conformal_fp_control_mbpp.json) — **real** public-benchmark result (not synthetic)
 - [`reproducibility-notes.md`](reproducibility-notes.md)
 - [`manifests/`](manifests/) — sanitized HF/W&B export manifests (below)
 

reproducibility-notes.md

@@ -7,6 +7,52 @@ Every future evidence pack in this repository will state, per artifact:
 - which numbers are measured vs derived (clean score = raw x (1 - DCR)),
 - the redaction profile applied before publication.
 
-Current artifacts are **synthetic/illustrative** and exist to fix the format.
-Links to W&B dashboards / HF datasets will be added only after the
-corresponding export approvals (`VLABS_APPROVE_{WANDB_UPLOAD,HF_UPLOAD}`).
+Current artifacts under [`evidence/`](evidence/) are **synthetic/illustrative**
+and exist to fix the format. Links to W&B dashboards / HF datasets will be added
+only after the corresponding export approvals
+(`VLABS_APPROVE_{WANDB_UPLOAD,HF_UPLOAD}`).
+
+---
+
+## Real, reproducible result — conformal false-positive control for IPT (public data)
+
+The first **non-synthetic** finding published here. It runs entirely on **public
+datasets** (no customer data, nothing to redact) and reproduces from a **public
+script**, so the full numbers are stated openly.
+
+**Claim.** Adding a split-conformal gate to Isomorphic Perturbation Testing gives
+a *stated, distribution-free* false-positive bound on shortcut flags **and**
+catches detector-evading memorizers that the structural rule misses — a new
+operating point, not a free lunch.
+
+**Setup (measured, not derived).**
+- Benchmark: MBPP+ (`evalplus/mbppplus`) × model completions
+  (`wuschelschulz/mbpp_reward_hacking_and_normal_completions`), shared items.
+- Score: `iso_fail_frac` = fraction of the EvalPlus isomorphic bank a candidate
+  fails (the canonical `invariance_violation_rate`).
+- `tau` is calibrated by split-conformal on a held-out **half** of the genuine
+  scores; the false-positive rate is measured on the **other** half — honest and
+  out-of-sample. Calibration/test genuine never overlap.
+- `n = 209` problems · 625 hacks · 165 genuine ext-pass (83 calib / 82 test).
+
+**Result.**
+
+| rule | recall (strong memorizer) | recall (evasive memorizer) | held-out genuine FP | stated bound? |
+|---|---|---|---|---|
+| behavioral (`is_shortcut`) | 1.00 | 1.00 | 15.85% | no |
+| structural (`CONFIRMED`) | 1.00 | **0.03** | 0% | no |
+| **conformal @ α=0.05** | 0.94 | **0.94** | **2.44%** | **≤ 5% ✓** |
+| **conformal @ α=0.10** | 1.00 | **1.00** | **9.76%** | **≤ 10% ✓** |
+
+The `evasive memorizer` is a SHA-1-routed lookup that passes the public suite by
+construction but evades all six AST detectors (0/6 fire) — the obfuscation threat
+the structural rule cannot see (cf. Obfuscation Atlas, arXiv:2602.15515). The
+conformal gate recovers it while holding the genuine FP under the chosen α,
+verified out-of-sample at both settings. Useful range α ≈ 0.05–0.10 (α=0.2
+degenerates `tau → 0`).
+
+**Reproduce.** Public script in the open engine repo:
+`vlabs-verifier-robustness-engine/scripts/eval_conformal_mbpp.py`
+(`HF_TOKEN=… python scripts/eval_conformal_mbpp.py --alpha 0.05 0.1`); machine-
+readable metrics in [`results/conformal_fp_control_mbpp.json`](results/conformal_fp_control_mbpp.json).
+The gate ships in the platform as `POST /v1/verifier-audits/assurance-card`.

results/conformal_fp_control_mbpp.json

@@ -0,0 +1,90 @@
+{
+  "_comment": "REAL public-benchmark result (NOT synthetic). Public datasets only; nothing redacted. Reproduces from vlabs-verifier-robustness-engine/scripts/eval_conformal_mbpp.py.",
+  "result_type": "verifier_robustness.conformal_fp_control",
+  "status": "real_public_data",
+  "benchmark": "MBPP+ x wuschelschulz \u2014 three operating points on a continuous IPT score",
+  "datasets": [
+    "evalplus/mbppplus",
+    "wuschelschulz/mbpp_reward_hacking_and_normal_completions"
+  ],
+  "n_problems": 209,
+  "n_hacks": 625,
+  "n_genuine_ext_pass": 165,
+  "split": {
+    "calib_genuine": 83,
+    "test_genuine": 82,
+    "policy": "calib/test genuine disjoint; FP measured out-of-sample"
+  },
+  "score": "iso_fail_frac = fraction of EvalPlus bank inputs the candidate fails (the canonical invariance_violation_rate)",
+  "operating_points": {
+    "behavioral": {
+      "fp_on_heldout_genuine": 0.1585,
+      "recall_strong_hack": 1.0,
+      "recall_evasive_hack": 1.0,
+      "recall_real_hack": 0.0048,
+      "precision": 0.9699,
+      "pooled_recall": 0.6704,
+      "f1": 0.7928
+    },
+    "structural_confirmed": {
+      "fp_on_heldout_genuine": 0.0,
+      "recall_strong_hack": 1.0,
+      "recall_evasive_hack": 0.0287,
+      "recall_real_hack": 0.0,
+      "precision": 1.0,
+      "pooled_recall": 0.344,
+      "f1": 0.5119
+    },
+    "conformal": {
+      "alpha=0.05": {
+        "tau": 0.818182,
+        "n_calib": 83,
+        "fp_bound": 0.05,
+        "fp_on_heldout_genuine": 0.0244,
+        "recall_strong_hack": 0.9378,
+        "recall_evasive_hack": 0.9378,
+        "recall_real_hack": 0.0,
+        "precision": 0.9949,
+        "pooled_recall": 0.6272,
+        "f1": 0.7694
+      },
+      "alpha=0.1": {
+        "tau": 0.107143,
+        "n_calib": 83,
+        "fp_bound": 0.1,
+        "fp_on_heldout_genuine": 0.0976,
+        "recall_strong_hack": 1.0,
+        "recall_evasive_hack": 1.0,
+        "recall_real_hack": 0.0048,
+        "precision": 0.9813,
+        "pooled_recall": 0.6704,
+        "f1": 0.7966
+      },
+      "alpha=0.2": {
+        "tau": 0.0,
+        "n_calib": 83,
+        "fp_bound": 0.2,
+        "fp_on_heldout_genuine": 1.0,
+        "recall_strong_hack": 1.0,
+        "recall_evasive_hack": 1.0,
+        "recall_real_hack": 0.0048,
+        "precision": 0.8363,
+        "pooled_recall": 0.6704,
+        "f1": 0.7442
+      }
+    }
+  },
+  "headline": {
+    "stated_fp_bound_holds_out_of_sample": {
+      "alpha_0.05": "FP 0.0244 <= 0.05",
+      "alpha_0.10": "FP 0.0976 <= 0.10"
+    },
+    "evasive_memorizer_recall": {
+      "structural": 0.0287,
+      "conformal_0.05": 0.9378,
+      "conformal_0.10": 1.0
+    }
+  },
+  "reproduce": "HF_TOKEN=... python scripts/eval_conformal_mbpp.py --alpha 0.05 0.1  (vlabs-verifier-robustness-engine)",
+  "deployed_as": "POST /v1/verifier-audits/assurance-card (vlabs-platform)"
+}