feat: harden autonomy and review gates

Co-Authored-By: Codex <noreply@openai.com>

Author: Pigbibi

cloudflare/ai-gateway-dash/src/index.mjs

@@ -115,6 +115,37 @@ POST /v1/ai/review
   → return {results, consensus, recommended_action}
 ```
 
+## Autonomy and Merge Safety
+
+AiGateway is designed as a supervised autopilot, not as an unrestricted
+self-merging agent. AI systems may diagnose failures, propose patches, create
+PRs, and collect evidence. Deterministic policy, CI, and repository protection
+own the final merge decision.
+
+Default autonomy boundaries:
+
+| Risk | Allowed default action | Human audit requirement |
+|---|---|---|
+| Low (`docs/`, `tests/`, README) | `auto_merge` after policy + CI pass | Not required unless CI/review flags an issue |
+| Medium (`scripts/`, report helpers) | `auto_pr` | Required before merge unless explicitly allowlisted by repo policy |
+| High (source, strategy, app logic) | `auto_pr` only | Required before merge |
+| Critical (secrets, credentials, policy/permission expansion) | `escalate` | Always required |
+
+Review and gate failures are risk-aware:
+
+- Low-risk docs/tests changes may pass through when AI review infrastructure is
+  unavailable, so routine maintenance is not blocked by the reviewer service.
+- Medium/high/critical changes fail closed or receive a human-review decision
+  when AI review cannot complete.
+- Auto-merge policy, GitHub Actions, secrets, OIDC, deployment, and permission
+  changes must remain human-audited because they can expand the system's own
+  authority.
+
+Closed-loop feedback records every autonomous change with action, risk,
+confidence, changed paths, and before/after metrics when available. Degraded
+outcomes set a rollback-review intent for auditability; the service does not
+directly roll back production state.
+
 ## Security
 
 - **Authentication**: GitHub Actions OIDC JWT with allowlisted repos, workflows,

docs/architecture.md

@@ -0,0 +1,150 @@
+# Autonomy Health Auto-Merge Hardening Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Make AIAuditBridge safer for mostly-AI-operated monitoring, remediation PR creation, and guarded auto-merge while keeping high-risk changes auditable by humans.
+
+**Architecture:** Keep the current layered model: AI proposes/fixes, deterministic policy gates decide autonomy, CI and GitHub branch protections own final merge, health/feedback observe outcomes. This plan avoids a rewrite and tightens risk boundaries, persistence, and dashboard explainability.
+
+**Tech Stack:** Python stdlib service/scripts, GitHub Actions, pytest, ruff, actionlint, Cloudflare Worker dashboard.
+
+---
+
+## File/Module Map
+
+- `service/autonomy.py`: single autonomy decision envelope, safer default matrix, policy loading/normalization helpers.
+- `.github/codex_auto_merge_policy.json`: repo-level default policy and review-failure behavior.
+- `scripts/run_monthly_codex_audit.py`: guarded auto-merge classification alignment and feedback registration metadata.
+- `scripts/run_codex_pr_review.py`: risk-aware fail-open/fail-closed behavior.
+- `scripts/gate_codex_app_review.py`: risk-aware Codex App timeout behavior.
+- `service/feedback.py`: durable effectiveness records and rollback-issue intent metadata.
+- `service/health.py`: keep online health separate from background job latency; expose decision-friendly health reasons.
+- Dashboard worker/static files: add Autonomy Decisions, Pending Human Audit, Effectiveness presentation if this repo owns dashboard UI; otherwise document target repo handoff.
+- `tests/`: cover autonomy matrix, policy alignment, review-failure behavior, feedback persistence, health status.
+- `docs/architecture.md` / README: document tiers and human-audit boundary.
+
+---
+
+### Task 1: Policy Inventory and Branch Protection Baseline
+
+**Files:**
+- Modify: `docs/superpowers/plans/2026-07-04-autonomy-health-auto-merge-hardening.md`
+- No production code change.
+
+- [ ] Confirm current PR/check/protection state for `QuantStrategyLab/AIAuditBridge`.
+- [ ] Record whether branch protection/rulesets are missing.
+- [ ] Produce exact branch protection recommendation; do not mutate GitHub settings without explicit user confirmation if tool requires admin-side irreversible policy change.
+
+Commands:
+```bash
+gh pr view 11 --json mergeStateStatus,statusCheckRollup,reviewDecision
+gh api repos/QuantStrategyLab/AIAuditBridge/branches/main/protection || true
+gh api repos/QuantStrategyLab/AIAuditBridge/rulesets || true
+```
+
+Expected:
+- PR checks green or pending identified.
+- Current protection gaps documented.
+
+---
+
+### Task 2: Safer Autonomy Policy Defaults
+
+**Files:**
+- Modify: `service/autonomy.py`
+- Modify: `.github/codex_auto_merge_policy.json`
+- Test: `tests/test_autonomy.py` or extend existing tests if present.
+
+- [ ] Add tests proving high/critical risk never returns `auto_merge` by default.
+- [ ] Add tests proving low-risk high-confidence can return `auto_merge`.
+- [ ] Add tests proving medium-risk defaults to `auto_pr` or `escalate` unless explicit policy override says otherwise.
+- [ ] Adjust `DEFAULT_DECISION_MATRIX` to remove high-risk auto-merge.
+- [ ] Ensure critical/secrets always escalate.
+- [ ] Document reason strings accurately.
+
+Validation:
+```bash
+python -m pytest tests/test_autonomy.py -q
+```
+
+---
+
+### Task 3: Risk-Aware Review Failure Behavior
+
+**Files:**
+- Modify: `scripts/run_codex_pr_review.py`
+- Modify: `scripts/gate_codex_app_review.py`
+- Test: `tests/test_run_codex_pr_review.py`
+- Test: `tests/test_gate_codex_app_review.py`
+
+- [ ] Add/adjust tests: docs/tests-only PR may pass if AI review unavailable.
+- [ ] Add/adjust tests: source/workflow/policy PR must not silently pass when AI review unavailable; should request human review or fail check depending current workflow contract.
+- [ ] Keep developer velocity for low-risk surfaces.
+- [ ] Avoid API/secret logging.
+
+Validation:
+```bash
+python -m pytest tests/test_run_codex_pr_review.py tests/test_gate_codex_app_review.py -q
+```
+
+---
+
+### Task 4: Feedback Loop Durability and Action Envelope
+
+**Files:**
+- Modify: `service/feedback.py`
+- Modify: `service/ai_gateway_service.py` if API response shape needs envelope fields.
+- Test: existing feedback tests or create `tests/test_feedback.py`.
+
+- [ ] Make shadow disagreement persistence durable under the same job dir pattern, or document why not possible in this pass.
+- [ ] Ensure every registered autonomous change includes action, risk, confidence, changed paths, policy version, source PR/issue URL when available.
+- [ ] When effect is degraded, store rollback-required metadata or issue intent; do not directly roll back.
+- [ ] Expose aggregated stats for dashboard without secrets.
+
+Validation:
+```bash
+python -m pytest tests/test_ai_gateway_service_get_routes.py tests/test_health.py -q
+```
+
+---
+
+### Task 5: Dashboard Information Architecture
+
+**Files:**
+- Modify dashboard-owned files if present in this repo.
+- If dashboard UI is generated/deployed from another repo, create a handoff note in `docs/architecture.md` instead of guessing.
+
+- [ ] Locate dashboard source ownership.
+- [ ] Add sections or data mapping for:
+  - Autonomy Decisions
+  - Pending Human Audit
+  - Effectiveness / Recent Auto Actions
+- [ ] Keep mobile header distributed and card empty states centered.
+- [ ] Verify in browser if dashboard source is editable here.
+
+Validation:
+```bash
+python -m pytest tests/test_ai_gateway_service_get_routes.py -q
+```
+Browser smoke test on dashboard URL if deploy is performed.
+
+---
+
+### Task 6: Final Validation and Delivery
+
+**Files:**
+- No new production changes unless fixing validation failures.
+
+- [ ] Run targeted tests.
+- [ ] Run full test suite if targeted tests pass.
+- [ ] Run lint/actionlint if available.
+- [ ] Inspect git diff for accidental secrets or unrelated changes.
+- [ ] Report changed files, validation results, risks, and any manual actions required.
+
+Commands:
+```bash
+python -m pytest -q
+ruff check .
+actionlint
+python -m compileall service scripts
+```

docs/superpowers/plans/2026-07-04-autonomy-health-auto-merge-hardening.md

@@ -62,6 +62,40 @@ def check_metadata(files: list[dict[str, Any]], policy: dict[str, Any]) -> list[
     return _check_metadata(files, policy)
 
 
+def fetch_pr_files(token: str, repo: str, pr_number: int) -> list[dict[str, Any]]:
+    files: list[dict[str, Any]] = []
+    page = 1
+    while True:
+        payload = github_request(
+            token,
+            "GET",
+            f"/repos/{repo}/pulls/{pr_number}/files?per_page=100&page={page}",
+        )
+        if not isinstance(payload, list) or not payload:
+            break
+        files.extend(payload)
+        if len(payload) < 100:
+            break
+        page += 1
+    return files
+
+
+def changed_files_are_low_risk(files: list[dict[str, Any]], policy: dict[str, Any]) -> bool:
+    low = policy.get("risk_policy", {}).get("low", {})
+    low_prefixes = tuple(low.get("prefixes", []))
+    low_exact = set(low.get("exact", []))
+    if not files:
+        return False
+    for file in files:
+        path = str(file.get("filename", "")).strip().lstrip("./")
+        if not path:
+            return False
+        if path in low_exact or any(path.startswith(prefix) for prefix in low_prefixes):
+            continue
+        return False
+    return True
+
+
 def env(name: str, default: str = "") -> str:
     return os.environ.get(name, default).strip()
 
@@ -180,6 +214,20 @@ def app_decision(review: dict[str, Any] | None) -> tuple[int, str, str]:
             f"Codex state `{state}` at {at}. Not blocking. [View review]({url})")
 
 
+def infra_failure_decision(files: list[dict[str, Any]], review_error: str) -> tuple[int, str, str]:
+    if changed_files_are_low_risk(files, load_policy(POLICY_PATH)):
+        return (
+            0,
+            "Codex: review infra unavailable — low-risk pass",
+            f"Codex review could not complete ({review_error}). Low-risk docs/tests-only change passed through.",
+        )
+    return (
+        1,
+        "Codex: review infra unavailable — human review required",
+        f"Codex review could not complete ({review_error}). Human review is required before merge.",
+    )
+
+
 # ─── main ────────────────────────────────────────────────────────────────────
 
 def main() -> int:
@@ -204,6 +252,7 @@ def main() -> int:
         return 0
 
     print(f"PR #{pr_number}  sha={head_sha[:12]}  event={event_name}")
+    policy = load_policy()
 
     # ── Phase 1: Static guard (skip on review-only events) ────────────
     if event_name != "pull_request_review":
@@ -226,16 +275,26 @@ def main() -> int:
         return rc
 
     # WAIT: poll for existing or upcoming review
+    changed_files = fetch_pr_files(token, repo, pr_number)
+    low_risk_change = changed_files_are_low_risk(changed_files, policy)
     try:
         existing = get_codex_review(token, repo, pr_number)
-    except RuntimeError:
+    except RuntimeError as exc:
         existing = None
+        review_error = str(exc)
+    else:
+        review_error = ""
 
     if existing is not None:
         rc, title, summary = app_decision(existing)
         print(f"EXISTING → exit={rc}: {title}")
         step_summary(f"## {title}\n\n{summary}")
         return rc
+    if review_error and not low_risk_change:
+        rc, title, summary = infra_failure_decision(changed_files, review_error)
+        print(f"INFRA → exit={rc}: {title}")
+        step_summary(f"## {title}\n\n{summary}")
+        return rc
 
     poll_s = env_int("CODEX_GATE_POLL_SECONDS", 30)
     max_w = env_int("CODEX_GATE_MAX_WAIT_MINUTES", 5)
@@ -246,7 +305,8 @@ def main() -> int:
         time.sleep(poll_s)
         try:
             review = get_codex_review(token, repo, pr_number)
-        except RuntimeError:
+        except RuntimeError as exc:
+            review_error = str(exc)
             continue
         if review is not None:
             rc, title, summary = app_decision(review)
@@ -255,9 +315,15 @@ def main() -> int:
             return rc
 
     # Timeout
-    print(f"TIMEOUT → Codex did not respond in {max_w}min; passing through")
-    step_summary(f"## Codex: timeout after {max_w}min\n\nPassed through to avoid blocking development.")
-    return 0
+    if low_risk_change:
+        print(f"TIMEOUT → Codex did not respond in {max_w}min; passing through")
+        step_summary(f"## Codex: timeout after {max_w}min\n\nPassed through to avoid blocking development.")
+        return 0
+
+    rc, title, summary = infra_failure_decision(changed_files, review_error or f"timeout after {max_w}min")
+    print(f"TIMEOUT → {title}")
+    step_summary(f"## {title}\n\n{summary}")
+    return rc
 
 
 if __name__ == "__main__":

scripts/gate_codex_app_review.py

@@ -193,6 +193,11 @@ def classify_file_risk(
     return ("high", "source code change")
 
 
+def changed_files_are_low_risk(paths: list[str], policy: dict[str, Any]) -> bool:
+    """Return True when every changed path is low-risk under the policy."""
+    return bool(paths) and all(classify_file_risk(path, policy)[0] == TASK_COMPLEXITY_LOW for path in paths)
+
+
 # ---------------------------------------------------------------------------
 # PR diff fetching
 # ---------------------------------------------------------------------------
@@ -901,9 +906,7 @@ def main() -> int:
         print(f"::warning::Policy errors: {policy['policy_errors']}")
 
     # First pass: classify files. If all files are low-risk, skip review.
-    all_low_risk = all(
-        classify_file_risk(p, policy)[0] == "low" for p in changed_paths
-    )
+    all_low_risk = changed_files_are_low_risk(changed_paths, policy)
     if all_low_risk and changed_paths:
         print("All changed files are low-risk (docs/tests). Skipping Codex review.")
         decision = {
@@ -936,16 +939,27 @@ def main() -> int:
         )
     except ReviewError as exc:
         print(f"::error::Codex review failed: {exc}", file=sys.stderr)
-        # Don't block on review failures — post a warning comment
+        if all_low_risk:
+            # Don't block low-risk docs/tests when the review infra is unavailable.
+            warning_body = (
+                "<!-- codex-pr-review -->\n"
+                "## 🤖 Codex PR Review\n\n"
+                "⚠️ **Review skipped**: The Codex review could not be completed.\n\n"
+                f"```\n{exc}\n```\n"
+            )
+            upsert_pr_comment(token, repo, pr_number, warning_body)
+            return 0
+
+        # High-risk changes should not fail open on review infrastructure errors.
         warning_body = (
             "<!-- codex-pr-review -->\n"
             "## 🤖 Codex PR Review\n\n"
-            "⚠️ **Review skipped**: The Codex review could not be completed.\n\n"
+            "⚠️ **Human review required**: The Codex review could not be completed.\n\n"
             f"```\n{exc}\n```\n\n"
             "Please ensure a human reviewer checks this PR before merging.\n"
         )
         upsert_pr_comment(token, repo, pr_number, warning_body)
-        return 0  # Don't block on infrastructure failures
+        return 1
 
     print(f"Codex output: {len(output)} chars")
 

scripts/run_codex_pr_review.py

@@ -947,6 +947,8 @@ def _handle_feedback_evaluate(self, payload: dict[str, Any]) -> None:
                 "status": "ok",
                 "effect": record.effect,
                 "effect_detail": record.effect_detail,
+                "rollback_issue_required": record.rollback_issue_required,
+                "rollback_intent": record.rollback_intent,
             })
         except FileNotFoundError:
             _json_response(self, HTTPStatus.NOT_FOUND, {"status": "error", "error": "change_id not found"})