Skip to content

RUNBOOK.md

Latest commit

 

History

History
721 lines (560 loc) · 26.4 KB

RUNBOOK.md

File metadata and controls

721 lines (560 loc) · 26.4 KB
Raw

App bug-hunt runbook

How to run, monitor, and recover the codex-driven bug-hunt orchestrator. This doc is gitignored (lives under .omx/) — it's a working reference, not a product doc.

What this does

A Python orchestrator that spawns codex exec subagents in parallel to scan the App codebase for bugs, optionally fixing them in-flight. It calibrates against a category whitelist (silent failure, anti-fab, wire-shape, correctness, staleness, slop, etc.) and routes per-task across multiple codex models based on complexity.

Script: .omx/subagent_orchestrator/scripts/run_bug_hunt.py

Prompts:

  • Discovery: .omx/subagent_orchestrator/prompts/subagent-prompt.md
  • Fixer: .omx/subagent_orchestrator/prompts/fixer-prompt.md

TL;DR — most-common invocation

The standard fluid-budget concurrent-discovery + concurrent-fix sweep with build verification and spark fallback:

python3 .omx/subagent_orchestrator/scripts/run_bug_hunt.py \
  --target 200 --max-rounds 80 \
  --max-agents 5 --agent-timeout 600 \
  --codex-model gpt-5.5 --codex-reasoning-effort high \
  --simple-codex-model gpt-5.3-codex-spark --simple-reasoning-effort xhigh \
  --simple-codex-fallback-model gpt-5.3-codex \
  --simple-codex-fallback-reasoning-effort xhigh \
  --commit-fixes --parallel-fix --fix-agents 2 --verify-build \
  --fix-timeout 1500

What this does:

  • 5 discovery scouts (spark/xhigh) + 2 fix workers (gpt-5.5/high) sharing a fluid 7-permit budget
  • Discovery and fix run alongside each other from round 1
  • Each fix: codex edits → orchestrator validates file-set → commits → xcodebuild / node --check verifies the build → reverts on failure
  • Sticky fallback: if spark hits its weekly quota, all subsequent simple tasks switch to gpt-5.3-codex (different quota pool)
  • Estimated wall time: ~1-6 hours depending on how much fix work the findings imply

Quick recipes

Discovery-only, no fixes

When you want findings but not auto-commits.

python3 .omx/subagent_orchestrator/scripts/run_bug_hunt.py \
  --target 200 --max-rounds 80 --max-agents 5 --agent-timeout 600 \
  --codex-model gpt-5.5 --codex-reasoning-effort high \
  --simple-codex-model gpt-5.3-codex-spark --simple-reasoning-effort xhigh \
  --simple-codex-fallback-model gpt-5.3-codex \
  --simple-codex-fallback-reasoning-effort xhigh \
  --no-fix --commit-fixes

--no-fix blocks the fixer entirely; --commit-fixes is still needed to spawn the cmux monitor (a quirk — see "Monitoring" below).

Fix-only, replay against prior findings

When discovery already happened and you just want to land fixes from a previous run's unique_findings.json.

python3 .omx/subagent_orchestrator/scripts/run_bug_hunt.py \
  --fix-only-from 20260519T024841Z \
  --codex-model gpt-5.5 --codex-reasoning-effort high \
  --commit-fixes --parallel-fix --fix-agents 2 --verify-build \
  --fix-timeout 1500

The --fix-only-from arg accepts either a run-id slug or an absolute path to a unique_findings.json.

Serial fix (safest, no concurrency)

When you don't trust the parallel-fix collision detection or want a known-good fix queue.

python3 .omx/subagent_orchestrator/scripts/run_bug_hunt.py \
  --target 100 --max-rounds 60 --max-agents 5 --agent-timeout 600 \
  --codex-model gpt-5.5 --codex-reasoning-effort high \
  --commit-fixes --verify-build --fix-timeout 1500

No --parallel-fix. Discovery still runs concurrent; fix runs one at a time after discovery completes.

Subscription-first with pi.dev backstop

Use codex's subscription quota for as long as possible, then transparently fall through to pi.dev's pay-per-token pool when the subscription is spent. Long runs survive the day instead of aborting at the spark wall.

python3 .omx/subagent_orchestrator/scripts/run_bug_hunt.py \
  --target 200 --max-rounds 80 --max-agents 5 --agent-timeout 600 \
  --simple-codex-model gpt-5.3-codex-spark --simple-reasoning-effort xhigh \
  --simple-codex-fallback-model gpt-5.3-codex \
  --simple-codex-fallback-reasoning-effort xhigh \
  --pi-fallback-model pi:openai-codex/gpt-5.3-codex-spark \
  --pi-fallback-reasoning-effort xhigh \
  --commit-fixes --parallel-fix --fix-agents 2 --verify-build \
  --fix-timeout 1500

The chain is sticky — once pi engages, both discovery and fix route through pi for the rest of the run (unless --fix-model pins fix elsewhere). Watch the [orchestrator] tier-transition log lines in the monitor's main pane.

Cheap-flash discovery, capable-pro fix (model split via pi)

Run all discovery on DeepSeek's flash model (1M context, cheap) and all fixes on DeepSeek's pro model (1M context, more capable). Reverse the two if you want the bigger model finding edge cases and the cheap model mechanically patching them.

python3 .omx/subagent_orchestrator/scripts/run_bug_hunt.py \
  --target 200 --max-rounds 80 --max-agents 5 --agent-timeout 600 \
  --discovery-model pi:deepseek/deepseek-v4-flash \
  --discovery-reasoning-effort high \
  --fix-model pi:deepseek/deepseek-v4-pro \
  --fix-reasoning-effort high \
  --commit-fixes --parallel-fix --fix-agents 2 --verify-build \
  --fix-timeout 1500

Phase overrides disable the fallback chain for each phase, so this is a flat "always-this-model" config. Pay-per-token billing applies for both phases — keep an eye on [orchestrator] cost lines (pi's JSON mode emits per-call cost; we don't surface it yet but --codex-panes lets you watch it live).

Same model, different runtimes for redundancy

Use codex CLI's spark as primary, pi.dev's gateway to the same spark model as the fallback. Two different gateways, two different quota pools, same model fingerprint — useful when you want to keep running without changing the model's behaviour after the quota wall hits.

python3 .omx/subagent_orchestrator/scripts/run_bug_hunt.py \
  --target 200 --max-rounds 80 --max-agents 5 --agent-timeout 600 \
  --simple-codex-model gpt-5.3-codex-spark --simple-reasoning-effort xhigh \
  --pi-fallback-model pi:openai-codex/gpt-5.3-codex-spark \
  --pi-fallback-reasoning-effort xhigh \
  --commit-fixes --parallel-fix --fix-agents 2 --verify-build

(No --simple-codex-fallback-model — chain goes spark-via-codex → spark-via-pi directly.)

Architecture

Discovery pool

  • Up to --max-agents (default 5) concurrent read-only codex spawns
  • Each task carries a complexity: simple|complex label set by the focus name (see "Model routing")
  • Tasks come from build_tasks(ROOT, rng) which produces:
    • 15 hand-coded cross-file scopes (build_cross_file_tasks())
    • 1000-line file chunks × 2 focuses per chunk (build_per_file_tasks())
    • Total typically 700-800 tasks
  • Each batch is --max-agents tasks; rounds run sequentially through the task list
  • Stops on: --target findings hit, --max-rounds cap, scope exhaustion, or quota wall (≥half a round hits codex usage-limit)

Fix pool (with --parallel-fix)

  • Up to --fix-agents (default 2) concurrent workspace-write codex spawns
  • Workers run as a ThreadPoolExecutor, consuming a shared queue.Queue that the discovery loop populates after each batch
  • Each fix attempt:
    1. Codex edits files (concurrent across workers — no lock)
    2. _FIX_SERIALIZER_LOCK acquired
    3. git diff --name-only compared to fixer's declared files_touched — mismatch → stash + reject
    4. Risk gate: risk: high → stash + reject
    5. _orchestrator_commit(declared, subject, body) lands the commit
    6. If --verify-build: auto-route by file ext (xcodebuild for .swift/.m/.h, node --check for .mjs, py_compile for .py). Failure → git reset --soft HEAD~1 + stash
    7. Lock released

Fluid agent budget

  • Single threading.BoundedSemaphore(--max-agents + --fix-agents) shared by every run_codex spawn
  • Discovery and fix workers both acquire permits before spawning codex
  • When all --fix-agents permits are held by fix work, discovery naturally caps at total - fix_in_flight
  • When fix is idle, discovery can burst to the full budget

Model routing

Two orthogonal axes: (a) which tier of the fallback chain handles a task, and (b) which phase (discovery vs fix) the task belongs to.

Runtimes

Model strings are runtime-tagged:

  • Plain id → invoked through the codex CLI (subscription quota, per-model: spark and non-spark draw from different pools).
  • pi:-prefixed id → invoked through the pi.dev CLI (pay-per-token via API keys; separate quota pool from codex CLI even for the same underlying model). Examples:
    • pi:openai-codex/gpt-5.3-codex-spark — pi's gateway to the same spark
    • pi:openai-codex/gpt-5.3-codex — pi's gateway to the non-spark codex
    • pi:deepseek/deepseek-v4-flash — DeepSeek's fast/cheap model (1M ctx)
    • pi:deepseek/deepseek-v4-pro — DeepSeek's capable model (1M ctx)

Anywhere a model id is accepted (--codex-model, --simple-codex-model, --simple-codex-fallback-model, --pi-fallback-model, --discovery-model, --fix-model), a pi: prefix swaps the runtime; reasoning effort uses the same {minimal,low,medium,high,xhigh} scale for both.

Three-tier fallback chain (discovery)

Tier Engaged when Configured by
0 — simple-primary Default for simple-complexity focuses --simple-codex-model + --simple-reasoning-effort
1 — simple-fallback Tier 0 returns a quota signal --simple-codex-fallback-model + --simple-codex-fallback-reasoning-effort
2 — pi-fallback Tier 1 (or tier 0 when no tier 1) returns a quota signal --pi-fallback-model + --pi-fallback-reasoning-effort

Each tier flips a sticky threading.Event (_SIMPLE_QUOTA_EXHAUSTED, _PI_FALLBACK_ENGAGED) on first quota hit and stays engaged for the rest of the run. The triggering task is retried in-process against the next tier; every subsequent task skips the spent tier with zero retry overhead.

Complexity is set per-focus in FOCUS_COMPLEXITY. Simple focuses (pattern-match heavy): silent failure, @MainActor sync I/O, Pre-launch creep, Smoke-fixture drift, Slop. Complex focuses (cross-file or semantic): wire-shape, anti-fab, perf-signature, correctness, staleness. Complex tasks always use --codex-model (no tier-0 simple-primary) but DO advance to the pi fallback when the codex tier hits its wall.

Phase-specific overrides

--discovery-model and --fix-model override the chain entirely for the named phase. Use case: cheap broad discovery on pi:deepseek/deepseek-v4-flash (1M context, fast) and capable fixes on pi:deepseek/deepseek-v4-pro — or the reverse if you want pro to find subtle edge cases that flash patches mechanically. When a phase override is set the fallback chain is disabled for that phase (the override is a single model, not a chain).

Once _PI_FALLBACK_ENGAGED is set during discovery, the fix phase also switches to the pi fallback by default — otherwise fixes would still try codex and immediately bounce. An explicit --fix-model takes precedence over that auto-routing.

CLI reference

Required-ish (almost every run)

Flag Purpose
--target N Stop after N unique findings. Default 100
--max-rounds N Hard cap on scan batches. Default 24 — too low; raise to 80+ for real runs
--max-agents N Concurrent discovery agents. Default 5
--agent-timeout SEC Per-discovery-agent codex timeout. Default 240; raise to 600 for gpt-5.5/high

Model routing

Flag Purpose
--codex-model NAME Complex tasks model. Default: codex's own default. Use gpt-5.5 or a pi:-prefixed id
--codex-reasoning-effort {minimal,low,medium,high,xhigh} For complex tasks
--simple-codex-model NAME Simple-tier-0 model. Use gpt-5.3-codex-spark
--simple-reasoning-effort {minimal,...,xhigh} For simple tasks. Use xhigh
--simple-codex-fallback-model NAME Simple-tier-1 fallback when spark quota exhausted. Use gpt-5.3-codex
--simple-codex-fallback-reasoning-effort Reasoning for tier-1 fallback. Use xhigh
--pi-fallback-model NAME Tier-2 fallback (pi.dev, pay-per-token). Must start with pi: — e.g. pi:openai-codex/gpt-5.3-codex-spark, pi:deepseek/deepseek-v4-pro
--pi-fallback-reasoning-effort Reasoning for tier-2 fallback. Use xhigh
--discovery-model NAME Override the model for discovery only. Disables the chain for discovery. Codex or pi:-prefixed
--discovery-reasoning-effort Reasoning when --discovery-model is set
--fix-model NAME Override the model for fix only. Codex or pi:-prefixed
--fix-reasoning-effort Reasoning when --fix-model is set

Fix mode

Flag Purpose
--commit-fixes Allow fix agents to actually edit files + commit. Without this, fix workers are read-only
--no-fix Skip fix phase entirely. Discovery-only
--parallel-fix Concurrent fix workers (default: serial after discovery)
--fix-agents N Concurrent fix workers when --parallel-fix is on. Default 2
--fix-timeout SEC Per-fix codex timeout. Default 480 — too short for gpt-5.5/high with verify-build, use 1500
--max-fixes N Hard cap on auto-fix attempts. Default unbounded
--verify-build Run xcodebuild / node --check / py_compile after each commit; revert on failure

Resume / replay

Flag Purpose
--fix-only-from RUN_ID_OR_PATH Skip discovery; load unique_findings.json from a prior run and just run the fix phase
--run-id RUN_ID Override the auto-generated timestamp run-id

Monitoring + cosmetic

Flag Purpose
--codex-panes Open a cmux pane per codex spawn tailing its live stdout. High visual churn. Requires --commit-fixes + cmux running
--no-cmux-monitor Skip auto-spawning the 3-pane cmux monitor workspace
--seed N Random seed for task shuffling. Default 42

Output structure

Every run produces a directory at .omx/subagent_orchestrator/runs/<run-id>/:

runs/<run-id>/
├── raw/                              # Per-agent JSON outputs
│   ├── cross-01.json                 # Cross-file scope agents
│   ├── file-0042-0.json              # Per-file chunk, slot 0
│   ├── file-0042-1.json              # Per-file chunk, slot 1 (different focus)
│   └── file-NNNN-N.json.stdouterr    # Debug companion (codex stdout/stderr capture)
├── merged/                           # Deduplicated findings (JSON)
│   └── unique_findings.json          # Source of truth for --fix-only-from
└── final/                            # Human-readable reports
    ├── run_log.md                    # Round-by-round progress + status
    ├── unique_findings.md            # Markdown findings report
    ├── needs_decision.md             # Findings with requires_decision: true
    ├── simple_findings.md            # requires_decision: false (auto-fixable)
    ├── simple_fixes.jsonl            # Fix-attempt receipts (one per line)
    └── fixes/                        # Per-fix codex receipts
        └── fix-NNN.json

Monitoring

A 3-pane cmux workspace auto-spawns when --commit-fixes is set (the gate is args.commit_fixes and not args.no_cmux_monitor — quirky but intentional; the monitor was built for fix runs and stuck).

The 3 panes:

  1. tail -f run_log.md — round progress + fix outcomes
  2. tail -f simple_fixes.jsonl — fix attempt receipts as they land
  3. git log --grep "Codex Auto-Fix" + git status — committed work so far + working tree state

Auto-rotate logic: panes follow the most-recent run dir, so a fresh orchestrator invocation rebinds the existing monitor workspace.

To check progress mid-run from a separate shell:

# Round + finding count
tail -30 .omx/subagent_orchestrator/runs/<run-id>/final/run_log.md

# Fix outcomes so far
ls .omx/subagent_orchestrator/runs/<run-id>/final/fixes/ | wc -l

# Commits landed since the sweep started
git log --oneline ^origin/main main | wc -l

# Real quota errors (vs substring matches)
grep -l -E '"error":"codex-usage-limit"|"error":"codex-rate-limit"' \
  .omx/subagent_orchestrator/runs/<run-id>/raw/*.json | wc -l

# Active codex children
pgrep -af "codex exec" | wc -l

# Active orchestrator
pgrep -af "run_bug_hunt.py" | head -3

Output schema

Discovery finding (in unique_findings.md)

{
  "status": "found",
  "agent_scope": "...",
  "agent_focus": "...",
  "finding": {
    "file": "ios/App/Core/AppStateStore.swift",
    "line": 12345,
    "category": "Silent failure",
    "issue": "Short noun phrase, under 80 chars",
    "evidence": ["LINE 12345:    try? fileManager.removeItem(at: url)"],
    "why": "Concrete failure mode produced",
    "impact": "UX | developer | correctness | security + sentence",
    "conditions": "Runtime state that triggers it",
    "confidence": "high | medium | low",
    "severity": "critical | high | medium | low",
    "fix_or_validation": "Minimal fix shape OR probe to confirm",
    "recommendation": "Concrete what-to-change",
    "minimum_fix_scope": "1 file (call site only)",
    "why_tests_do_not_already_cover_this": "...",
    "suggested_regression_test": "single-line test seed OR null",
    "requires_decision": false
  }
}

Fix receipt (from the fixer agent)

{
  "status": "ready_to_commit",
  "issue_anchor": "<file>:<line>",
  "files_touched": ["path/to/file.swift"],
  "commit_subject": "fix(<area>): ...",
  "commit_body": "...",
  "risk": "low | medium | high",
  "risk_reasoning": "one sentence",
  "notes": "optional"
}

OR:

{
  "status": "skipped",
  "issue_anchor": "...",
  "reason": "already_fixed | unsafe_scope | depends_on_decision | code_moved | other",
  "notes": "..."
}

Orchestrator fix outcome (in simple_fixes.jsonl)

status is one of:

  • committed — landed cleanly, build green
  • skipped — codex chose skip
  • skipped_but_dirty_stashed — codex skipped but had leftover edits
  • rejected_bad_receipt — codex's output JSON malformed
  • rejected_missing_fields — files_touched or commit_subject empty
  • rejected_file_set_mismatch — declared file set ≠ actual git diff
  • rejected_risk_too_high — fixer self-flagged risk: high
  • reverted_build_failed — committed then build failed, soft-reset
  • commit_failed — git commit step failed (hook? identity? lock?)

Common scenarios

Quota wall — spark exhausted mid-run

Before fallback (last night's pattern): Round 49: 3/5 spark agents hit codex-usage-limit → orchestrator aborts discovery with Aborting discovery: 3/5 agents this round hit a codex usage/rate limit.

With fallback (recommended): First spark task to hit quota triggers retry on gpt-5.3-codex AND flips _SIMPLE_QUOTA_EXHAUSTED. All subsequent simple tasks go straight to fallback. Run continues at slower per-task throughput against a separate quota pool.

If fallback ALSO hits quota (rare — pools are usually distinct), the 3-of-5 abort guard catches the next round.

Fix-set-mismatch storm (parallel-fix tradeoff)

In parallel-fix mode, two fixers can edit overlapping files before either commits. The lock-protected file-set validator rejects the loser. With 2 fixers and ~80 findings, expect ~15-30 mismatches per run. The work isn't lost — each mismatch is stashed under a labeled stash for recovery.

To recover stashes after a run, see "Stash recovery" below.

Working tree dirty at start

Orchestrator checks _working_tree_dirty() before the fix phase. If dirty:

  • Serial mode: tries one defensive stash; if still dirty, skips
  • Parallel mode: the file-set validator inside the lock rejects every fix that touches a file you've already edited

Always run with a clean tree. If you have in-flight work, stash it under a clear label first:

git stash push -m "in-flight-feature-X-pre-sweep" -- <files>

Build verification too slow

--verify-build runs full xcodebuild after every Swift fix. Each xcodebuild is 60-90s. For a 50-fix sweep, that's an extra 50-75 minutes just for verification.

Tradeoffs:

  • Drop --verify-build: faster but commits can break the build
  • Keep --verify-build: longer run, but tree is always green and bad fixes auto-revert

For overnight runs, keep it. For quick experiments, drop it and manually xcodebuild once at the end.

Stash recovery

Rejected/reverted fixes are stashed with descriptive labels:

bug-hunt-fix-NNN-<file_path_underscored>_<line>-<reason>

Reasons:

  • file-set-mismatch — concurrent edit collision
  • risk-too-high — fixer self-flagged
  • build-failed-reverted — build verify caught a regression
  • skip-but-dirty — codex chose skip but had leftover edits
  • commit-failed — git commit step failed
  • bad-status / missing-fields / bad-receipt — codex output malformed
  • tree_dirty_pre — tree was dirty entering the attempt (rare with self-heal)

To triage stashes after a sweep:

# List
git stash list

# Inspect
git stash show "stash@{N}" --stat
git stash show "stash@{N}" -p | head -80

# Recover (apply, build, commit, drop)
git stash apply "stash@{N}"
xcodebuild -project ios/App.xcodeproj -scheme App \
  -destination 'platform=iOS Simulator,name=iPhone 17 Pro' \
  -configuration Debug build 2>&1 | grep -E '^(.*error:|\*\* BUILD)'
# If green:
git add <files> && git commit -m "..." && git stash drop "stash@{N}"
# If conflicts or build fail:
git checkout -- <files>  # discard the apply
# Leave stash for later or drop with note

The risk-too-high stashes are usually genuine product decisions — keychain throw refactors, recording-lifecycle changes, signing/manifest edits — read each before applying.

Prompts

Discovery prompt: prompts/subagent-prompt.md

Edit when:

  • Adding a new finding category (also update FOCUSES in the script and FOCUS_COMPLEXITY)
  • Changing the finding JSON schema (also update render_markdown)
  • Tightening calibration anchors
  • Adding new cross-file invariants to probe

Placeholders the script fills: {{SCOPE}}, {{FOCUS_HINT}}, {{SKIP_LIST}}.

Fixer prompt: prompts/fixer-prompt.md

Edit when:

  • Changing the fix output schema (also update _attempt_one_fix parsing)
  • Changing risk calibration anchors
  • Adding new concurrency constraints

The fixer is told that 1 other fix agent may be concurrent in parallel-fix mode — re-read files before editing, skip with reason code_moved if context shifted.

Skip list (deferred — do not surface in new findings)

These patterns are known and intentional. The discovery prompt already excludes them; if they come back, downgrade or skip.

  • 'ORG' / APP_HOSTED_PROCESSING_DEMO_KEYWORD — active dev workflow
  • APP_BACKEND_ALLOW_NON_CLIENT_OWNED=1 flag — 11-test refactor deferred
  • try? in test cleanup / tearDown — acceptable
  • SSE event: line parsing — design pinned: JSON envelope is source of truth
  • PROCESSING_MANIFEST_NONCE_MAX_TTL_MS clamping — security guardrail, not a fix target
  • legacyKey parameter on appEnvironmentValue — explicit opt-in

Audit history

Past sweep findings live at:

.omx/subagent_orchestrator/runs/<run-id>/final/unique_findings.md

Significant runs:

  • audit-20260517T200007Z — original 8-finding privacy audit (all closed)
  • audit-20260517T222143Z — 23-finding privacy follow-up (22 closed, 1 deferred)
  • 20260518T010641Z — first 169-finding broad sweep (56 closed via multi-session triage)
  • 20260518T200701Z — 23-finding sweep (all closed via codex handoff)
  • 20260519T024841Z — fluid-budget parallel-fix sweep (80 findings, 46 auto-committed + 18 recovered from stashes = 64 closed)

When git log --grep "audit-" to find commits referencing a specific audit run.

Commit conventions

Auto-fix commits follow the existing style:

fix(<area>): <one-line subject under 70 chars>

<one paragraph: failure mode + fix shape>

Finding: <file:line> [<category>]

Co-Authored-By: Codex Auto-Fix <codex-auto-fix@noreply.app>

Manual recovery commits (you, me, deepseek) use:

fix(<area>): <subject>

<paragraph>

Recovered from stash <stash-label> (<reason>).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Area prefixes used so far: fix(privacy), fix(anti-fab), fix(silent-failures), fix(wire-shape), fix(correctness), fix(staleness), fix(perf-signature), fix(intelligence), fix(record), fix(settings), fix(server), fix(backend), fix(agentic), fix(wiki), fix(tests), fix(auth), chore(privacy), chore(api-client), chore(tests), chore(intelligence), chore: bump build number to N.

Adjacent docs

  • README.md (same dir) — original orchestrator overview
  • docs/PRIVACY.md (repo root) — privacy contract + audit closure table
  • Memory: ~/.claude/projects/-Users-user-Code-App/memory/ — user/project facts that carry across sessions

Troubleshooting

Orchestrator hangs at startup

Codex CLI may be missing or unauthenticated. Test:

echo '{"task": "ping"}' | codex exec --sandbox read-only -C $(pwd) "Return {}"

Should return within 30s. If it hangs, codex is the issue, not the orchestrator.

Codex children stuck at 0% CPU

Known issue: codex needs stdin=DEVNULL or it can stall waiting for input that never comes. The script already passes stdin=subprocess.DEVNULL, but if you see hung children:

ps -o pid,etime,command -p $(pgrep codex) | head -10

If etime exceeds your --agent-timeout and CPU is 0%, send SIGTERM:

pkill -TERM -f "codex exec"

Build verification breaks an unrelated file

--verify-build runs xcodebuild on the FULL project, not just changed files. If an unrelated file already has a build error (your in-flight work, a stash residue, etc.), every fix will revert. Solution: make sure xcodebuild passes on main before kicking off the sweep.

Stash list growing unboundedly

If you have many runs back-to-back without cleaning up, stashes accumulate. Drop them with:

# List all bug-hunt stashes
git stash list | grep "bug-hunt-fix-"

# Drop a specific one
git stash drop "stash@{N}"

# Nuclear: drop all bug-hunt stashes (keeps anything not labelled
# `bug-hunt-fix-`)
git stash list | grep "bug-hunt-fix-" | \
  awk -F: '{print $1}' | tac | xargs -I{} git stash drop {}

Always triage before dropping en masse — some stashes contain real work.

Want to redo a finding cluster

--fix-only-from <run-id> replays the fix phase against a prior run's unique_findings.json. The dedup signatures persist across runs so you can re-run the SAME sweep and skip already-fixed anchors via the prompt's {{SKIP_LIST}}.