Refactor/code cleanup

.github/workflows/ci.yml

@@ -26,5 +26,8 @@ jobs:
           python -m pip install --upgrade pip
           pip install -r requirements-dev.txt
 
+      - name: Lint
+        run: ruff check src/ hpo_cli.py hpo_mcp_server.py broker.py --select F
+
       - name: Run tests (SQLite)
         run: pytest tests/ -q

.github/workflows/integration.yml

@@ -1,32 +1,23 @@
 name: Integration Test
 
-on: [push, pull_request]
+on:
+  push:
+    branches: [main, master]
+  pull_request:
 
 jobs:
   test:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v3
-    - name: Set up Python 3.11
-      uses: actions/setup-python@v4
-      with:
-        python-version: "3.11"
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        pip install -r requirements.txt
-    - name: Run Unit Tests
-      run: |
-        pip install pytest httpx
-        pytest tests/ -v
-    - name: Generate Quickstart Study
-      run: |
-        printf '\n\n\n\n\n' | python hpo_cli.py quickstart
-    - name: Start Broker
-      run: |
-        python broker.py --daemon &
-    - name: Wait for Broker
-      run: sleep 2
-    - name: Run Dummy Worker
-      run: |
-        python quickstart_worker.py
+      - uses: actions/checkout@v4
+      - name: Set up Python 3.11
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements-dev.txt
+      - name: Run tests
+        run: pytest tests/ -q

.gitignore

@@ -40,3 +40,9 @@ pending_changes.json
 # Build artifacts
 *.egg-info/
 
+# Generated model cards
+studies/
+
+# Default database location
+.data/
+

AGENTS.md

@@ -21,7 +21,7 @@ coordinator without an explicit user request.**
 ## MCP setup (one time, shared across IDEs)
 
 All three IDEs use the same server `pathfinder` (`python hpo_mcp_server.py`) and the
-same database. See the README "Exposing Pathfinder to AI Agents" section for the exact config block.
+same database. See the README "IDE Setup (Agent-Driven Onboarding & Inspection)" section for the exact config block.
 Capability does not differ between Cursor, Antigravity, and Claude Code. Use one IDE at a time
 for writes; reviews are idempotent per trial window, so a second client will not double-write.
 
@@ -51,49 +51,41 @@ Trigger this when the user says "integrate HPO", "onboard my training script", "
 
 7. **Document the GPU side.** Tell the user to set `HPO_BROKER_URL` and `HPO_STUDY_NAME` on the training machine. The study name must match the manifest. Enable `HPO_SPARKLINES=1` if they want a Unicode performance curve printed on trial completion.
 
-The worker contract is exactly three calls; full reference in `docs/INTEGRATION.md`. Load the grill checklist from `hpo://prompts/grill` (not a separate integration-guide tool).
+The worker contract is exactly three calls; full reference in `docs/INTEGRATION.md`.
 
 ### Statistical confidence (caveat, not a gate)
 
 `get_study_data` includes `statistical_confidence` derived from completed-trial count:
 
 | Tier | Completed trials | Agent behavior |
 |------|------------------|----------------|
-| `low` | < 10 | Treat fANOVA/Spearman as noisy; use `estimated_score_improvement=-1.0` when uncertain |
+| `low` | < 10 | Treat fANOVA/Spearman as noisy; be cautious with interpretation |
 | `medium` | 10–19 | Signals stabilizing; stay cautious on large bound shifts |
 | `high` | ≥ 20 | Standard interpretation |
 
 Reviews are never hard-blocked at low confidence — the dashboard shows a banner only.
 
-## Coordinator procedure (episodic review)
+## Inspection flow
 
-When the dashboard shows a health warning (Watch or Intervene) and the brand-mark double pings (white for Watch, red for Intervene), run the 7-step review. The review prompt can be loaded from the MCP resource `hpo://prompts/review`.
+When the user asks about study progress, trial results, or health:
 
-1. Call `get_study_data(study_name)` to retrieve the compacted statistical and telemetry packet (`statistical_confidence`, `coordinator_accuracy`, `past_reviews`).
-2. Read dynamic metrics from `project_context` and evaluate fANOVA importances and Spearman correlations; heed `statistical_confidence` when low/medium.
-3. Perform a safety review of VRAM predictions (`bounds_oom_risk`) and review the last 3 `past_reviews` — **ignore reviews where `quality_flagged` is true**.
-4. **Coordinator accuracy self-regulation:** `coordinator_accuracy` tracks review forecasts vs. measured best-score deltas (not trial-suggest logs). If `insufficient_data` is true (`n_scored_reviews` < 3), do not self-regulate yet. If `mean_absolute_error` > 0.05 with `n_scored_reviews` ≥ 3, propose smaller bound shifts.
-5. Propose active search space adjustments via `update_search_space(study_name, space_config, apply=False)`.
-6. Submit with `submit_agent_review` — **required:** `estimated_score_improvement` (float) and `cited_best_trial` (int). Use `-1.0` when < 5 completed trials. Human approves via dashboard **Apply Proposal** or `update_search_space(apply=True)`.
-7. Call `generate_model_card(study_name)` to write the model card and index it in the database.
-
-### Coordinator accuracy semantics (honest correlational accountability)
-
-At review time the system snapshots `baseline_best_score`. After the human applies a bounds patch and **≥ 5 new COMPLETE trials** finish, `backfill_review_outcomes` sets `actual_score_improvement = new_best - baseline`. MAE uses only `outcome_status=measured` reviews (excludes sentinel `-1.0`, `inconclusive` prune-storm windows, and `quality_flagged` rows). This does **not** prove causation — it measures whether the coordinator's numeric forecast matched the best-score delta N trials later. Flag bad reviews: `python hpo_cli.py flag-review --id N` or the dashboard **Flag** button.
-
-## Guardrails (always)
-
-- Never block the GPU worker on an LLM; the suggest path stays TPE.
-- Never auto-invoke the coordinator from the broker, dashboard, or a hook - only on user request.
-- Do not write/modify local JSON files for search space or configuration (e.g. `active_search_space.json`, `hpo_config.json`); state must reside in SQLite.
-- Do not modify the root `colab_worker.py` unless the user owns the bridge-crack project. Cloners use `templates/`.
+1. Call `get_study_data(study_name)` to retrieve trial telemetry, health tier, fANOVA importances, best trials, and OOM patterns.
+2. Summarize: current best score, health status, stagnation indicators, and any OOM warnings.
+3. If health is `watch` or `intervene`, explain the reason and offer to explore the data further.
 
 ## IDE triggers & status polling (.hpo_status.json)
 
 When a completed trial is reported or the background daemon polls health, the system writes `.hpo_status.json` to the workspace root.
 
 **Agent Action (Cursor, Antigravity, and Claude Code)**:
 * Eagerly check for the existence of `.hpo_status.json` in the project root.
-* If `health_tier` is `"watch"` or `"intervene"`, immediately notify the user in your first response, explain the `health_reason`, and proactively offer to run the **Coordinator procedure (episodic review)** described above.
+* If `health_tier` is `"watch"` or `"intervene"`, immediately notify the user in your first response, explain the `health_reason`, and proactively offer to inspect the study data.
 * Offer to use the `/goal` slash command to help coordinate and automate resolution of search space stagnation or failures.
 
+## Guardrails (always)
+
+- Never block the GPU worker on an LLM; the suggest path stays TPE.
+- Never auto-invoke the coordinator from the broker, dashboard, or a hook - only on user request.
+- Do not write/modify local JSON files for search space or configuration (e.g. `active_search_space.json`, `hpo_config.json`); state must reside in SQLite.
+- Do not modify the root `colab_worker.py` unless the user owns the bridge-crack project. Cloners use `templates/`.
+- Never modify `archive/` files — they are historical reference only.

Dockerfile

@@ -12,5 +12,9 @@ COPY . .
 # Expose broker port
 EXPOSE 8000
 
+# Health check
+HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
+    CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1
+
 # Run uvicorn on 0.0.0.0
 CMD ["python", "broker.py", "--daemon", "--host", "0.0.0.0", "--port", "8000"]