Improve Codex calibration routing and reports

Route symptom-first failures through investigate before implementation, expand calibration fixtures for root-cause routing, and emit measured recommendations from calibration runs.

Co-authored-by: Codex <codex@openai.com>

Author: Borda

.codex/AGENTS.md

@@ -21,6 +21,8 @@ For docs, dependencies, CI/CD, releases, security, and deprecations, prefer curr
 - When multiple agents contribute, keep handoffs compact and ownership clear. Never redo another agent's work unless you are resolving a conflict or an explicit gap.
 - If progress stalls or the path starts to drift, re-plan instead of forcing the current approach through.
 - When confidence is limited, say so explicitly and separate verified facts from hypotheses.
+- Treat symptom-first failures as investigation tasks before implementation. Failing tests or CI, flaky behavior, regressions, tool/environment errors, unexplained metric shifts, and user reports that describe symptoms without a verified cause must route through `investigate` or equivalent documented evidence before `develop`, `resolve`, or workaround recommendations.
+- Workarounds are temporary mitigations only. Do not present a workaround-only change or answer as complete unless the user explicitly requests a temporary mitigation; label the mitigation and the remaining root-cause work.
 
 ## Coordination Discipline
 
@@ -142,6 +144,12 @@ Parent agent responsibilities:
 - Integrate subagent outputs back into one coherent change
 - Make final judgment on conflicts, overlaps, and release readiness
 
+### Required workflow routing
+
+- Unknown failure/root-cause work starts with `investigate`: failing tests, failing CI, flaky behavior, regressions, tool or environment failures, unexplained metric changes, and any symptom-only report where the cause is not already verified.
+- Before implementation for those tasks, record the root-cause claim, supporting evidence, a falsification check, and at least one rejected alternative. If the evidence is missing, continue investigation instead of proposing a fix.
+- After `investigate`, hand off to the relevant domain agent or `develop`/`resolve` with the evidence summary. Temporary mitigations are allowed only when explicitly requested or required to unblock verification, and they must not be treated as the root fix.
+
 ### Automatic spawn patterns (all agents)
 
 - `sw-engineer`: implementation, refactors, ML/backend feature delivery

.codex/README.md

@@ -43,7 +43,7 @@ Four things this Codex setup can do that vanilla Codex can't:
 
 ## 🔄 Config Sync
 
-This repo (`.codex/`) is the source of truth. Home (`~/.codex/`) is a downstream copy:
+This repo (`.codex/`) is the source of truth. Home (`~/.codex/`) is a downstream copy. Before config edits, keep project-local backups under `.reports/codex/manage/<timestamp>/backup/` so the source-of-truth state is reversible without relying on home config:
 
 ```bash
 cp -r .codex/ ~/.codex/ # activate globally (config_file paths are relative)
@@ -90,6 +90,7 @@ Codex selects agents autonomously based on task type (defined in `AGENTS.md`). Y
 
 Automatic spawn patterns (from `AGENTS.md`):
 
+- Symptom-first failures route to `investigate` before implementation: failing tests, failing CI, flaky behavior, regressions, tool/environment errors, unexplained metric shifts, and workaround requests without verified cause
 - `sw-engineer` handles core implementation; on completion Codex can fan out to `qa-specialist` + `doc-scribe`
 - `security-auditor` is used when tasks touch auth, credentials, external APIs, model weights, or deserialization
 - `data-steward` is used when tasks touch data pipelines, splits, augmentation, or DataLoaders
@@ -154,10 +155,10 @@ Each skill enforces a complete quality loop that prompt-style invocation does no
 | Skill         | What it enables                                                                                                                             |
 | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
 | `review`      | Diff-scoped review with measurable gates: classifies findings by severity, writes a JSON artifact so results are comparable across runs     |
-| `develop`     | TDD-first implementation: writes a failing test first, implements to pass it, then reruns all gates before handing back                     |
+| `develop`     | TDD-first implementation: writes a failing test first, requires root-cause evidence for symptom-first failures, then reruns all gates       |
 | `resolve`     | Findings closure: applies fixes in priority order (critical → high → medium), reruns gates, surfaces what remains                           |
 | `audit`       | Config hygiene: detects broken refs, inventory drift, instruction overlap; produces a scored report with keep/sharpen/prune recommendations |
-| `calibrate`   | Benchmarks recall vs confidence bias on a fixed task set so you know if stated confidence is reliable                                       |
+| `calibrate`   | Benchmarks recall vs confidence bias on a fixed task set and emits measured recommendations for the next fixes or improvements              |
 | `release`     | SemVer-disciplined release: changelog entry, migration guide, and readiness check in one structured pass                                    |
 | `investigate` | Root-cause diagnosis for unknown failures — env, tools, hooks, CI divergence — with ranked hypotheses and a handoff artifact                |
 | `manage`      | Scaffolds agents, skills, and config with cross-ref propagation; prevents orphaned references                                               |
@@ -172,6 +173,7 @@ Interactive prompt usage:
 
 ```text
 run investigate on this branch and find root cause of failing CI
+run investigate before fixing this failing pytest; do not suggest a workaround unless it is explicitly temporary
 run resolve on the current working tree and fix high-severity findings
 run review, then develop, then audit for issue #42
 ```
@@ -262,6 +264,8 @@ Calibration runner:
 .codex/calibration/run.sh
 ```
 
+Each run writes `result.json`, `behavioral.json`, and `recommendations.md`. Recommendations are generated from failed gates, leaks, behavioral false positives/negatives, confidence calibration gaps, and live-observation coverage.
+
 ### AGENTS.md layering
 
 Codex loads agent instructions in layers, with more specific layers overriding broader ones:

.codex/calibration/behavioral-cases.json

@@ -162,6 +162,17 @@
       "id": "data-split-reproducibility",
       "prompt": "Assess a dataset split change with random shuffling and no seed, stratification, or leakage guard.",
       "target": "data-steward"
+    },
+    {
+      "expected_findings": [
+        "route-to-investigate",
+        "reject-workaround-only",
+        "root-cause-evidence-required",
+        "rejected-alternative-required"
+      ],
+      "id": "develop-root-cause-routing",
+      "prompt": "Assess a symptom-first request where tests fail after a dependency or tooling change and the proposed response suggests pinning, skipping, or bypassing the failure without logs, hypotheses, or a falsification check.",
+      "target": "develop"
     }
   ],
   "thresholds": {

.codex/calibration/behavioral-observations.jsonl

@@ -1,18 +1,19 @@
-{"case_id":"review-api-contract","target":"review","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["missing-regression-test","api-doc-mismatch","naming-style"],"confidence":0.76}
-{"case_id":"review-security-path","target":"review","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["unsafe-deserialization"],"confidence":0.68}
-{"case_id":"review-ci-config-regression","target":"review","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["ci-matrix-regression","missing-config-test"],"confidence":0.9}
-{"case_id":"review-ml-tensor-boundaries","target":"review","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["nan-inf-uncovered","dtype-shape-uncovered","boundary-test-missing","torch-equal-used"],"confidence":0.83}
-{"case_id":"review-release-deprecation","target":"review","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["missing-deprecation-path","migration-note-missing"],"confidence":0.88}
-{"case_id":"review-data-leakage","target":"review","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["split-leakage-risk"],"confidence":0.65}
-{"case_id":"qa-tensor-boundaries","target":"qa-specialist","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["none-input","nan-tensor","wrong-shape"],"confidence":0.93}
-{"case_id":"qa-error-path-coverage","target":"qa-specialist","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["malformed-input","negative-path-missing"],"confidence":0.9}
-{"case_id":"qa-empty-inputs","target":"qa-specialist","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["none-input","empty-input"],"confidence":0.9}
-{"case_id":"qa-state-isolation","target":"qa-specialist","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["state-leakage","isolation-test-missing","mocking-style"],"confidence":0.82}
-{"case_id":"challenger-migration-risk","target":"challenger","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["public-api-compat","migration-risk"],"confidence":0.88}
-{"case_id":"challenger-security-assumption","target":"challenger","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["trust-boundary-unclear","security-review-missing","style-risk"],"confidence":0.78}
-{"case_id":"challenger-performance-claim","target":"challenger","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["profiling-absent"],"confidence":0.65}
-{"case_id":"challenger-no-findings","target":"challenger","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["no-blocking-finding-supported"],"confidence":0.82}
-{"case_id":"security-credential-handling","target":"security-auditor","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["secrets-exposure","credential-logging-risk"],"confidence":0.9}
-{"case_id":"cicd-release-permissions","target":"cicd-steward","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["overbroad-token-permissions","missing-trusted-publishing","release-note-missing"],"confidence":0.79}
-{"case_id":"architecture-api-migration","target":"solution-architect","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["migration-contract-undefined","compatibility-risk"],"confidence":0.87}
-{"case_id":"data-split-reproducibility","target":"data-steward","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["train-val-leakage","seed-policy-missing"],"confidence":0.89}
+{"case_id":"review-api-contract","target":"review","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["missing-regression-test","api-doc-mismatch"],"confidence":1.0}
+{"case_id":"review-security-path","target":"review","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["unsafe-deserialization","missing-negative-test"],"confidence":1.0}
+{"case_id":"review-ci-config-regression","target":"review","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["ci-matrix-regression","missing-config-test"],"confidence":1.0}
+{"case_id":"review-ml-tensor-boundaries","target":"review","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["nan-inf-uncovered","dtype-shape-uncovered","boundary-test-missing"],"confidence":1.0}
+{"case_id":"review-release-deprecation","target":"review","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["missing-deprecation-path","migration-note-missing"],"confidence":1.0}
+{"case_id":"review-data-leakage","target":"review","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["split-leakage-risk","seed-reproducibility-missing"],"confidence":1.0}
+{"case_id":"qa-tensor-boundaries","target":"qa-specialist","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["none-input","nan-tensor","wrong-shape"],"confidence":1.0}
+{"case_id":"qa-error-path-coverage","target":"qa-specialist","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["malformed-input","negative-path-missing"],"confidence":1.0}
+{"case_id":"qa-empty-inputs","target":"qa-specialist","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["none-input","empty-input"],"confidence":1.0}
+{"case_id":"qa-state-isolation","target":"qa-specialist","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["state-leakage","isolation-test-missing"],"confidence":1.0}
+{"case_id":"challenger-migration-risk","target":"challenger","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["public-api-compat","migration-risk"],"confidence":1.0}
+{"case_id":"challenger-security-assumption","target":"challenger","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["trust-boundary-unclear","security-review-missing"],"confidence":1.0}
+{"case_id":"challenger-performance-claim","target":"challenger","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["profiling-absent","metric-baseline-missing"],"confidence":1.0}
+{"case_id":"challenger-no-findings","target":"challenger","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["no-blocking-finding-supported"],"confidence":1.0}
+{"case_id":"security-credential-handling","target":"security-auditor","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["secrets-exposure","credential-logging-risk"],"confidence":1.0}
+{"case_id":"cicd-release-permissions","target":"cicd-steward","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["overbroad-token-permissions","missing-trusted-publishing"],"confidence":1.0}
+{"case_id":"architecture-api-migration","target":"solution-architect","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["migration-contract-undefined","compatibility-risk"],"confidence":1.0}
+{"case_id":"data-split-reproducibility","target":"data-steward","source":"fixture-selftest","run_id":"fixture-v1","observed_at":"2026-06-02T06:50:00Z","reported_findings":["train-val-leakage","seed-policy-missing"],"confidence":1.0}
+{"case_id":"develop-root-cause-routing","target":"develop","source":"fixture-selftest","run_id":"fixture-v2","observed_at":"2026-06-04T14:59:58Z","reported_findings":["route-to-investigate","reject-workaround-only","root-cause-evidence-required","rejected-alternative-required"],"confidence":1.0}

.codex/calibration/benchmarks.json

@@ -54,12 +54,17 @@
       "gate_metrics_raw",
       "by_source",
       "observation_freshness",
-      "live_observations"
+      "live_observations",
+      "recommendations",
+      "follow_up"
     ],
     "develop": [
       "## Input Schema",
       "## Workflow \\(Exact Commands\\)",
       "anti-rationalization gate",
+      "root-cause evidence",
+      "rejected alternative",
+      "temporary mitigation",
       "characterization tests",
       "run-gates.sh",
       "write-result.sh",