P3.35: correct OS-process isolation claims to module/data-level independence

docs/ATO_EVIDENCE_PACKET.md

@@ -60,18 +60,18 @@ flowchart TB
         Op["Human operator<br/>(campaign start, category budget)"]
     end
 
-    subgraph ZoneA["Zone A -- Adversarial (Red Team process/context)"]
-        RT["Red Team Agent<br/>local model: huihui_ai/qwen2.5-abliterate:7b<br/>(CPU-only, num_gpu=0)"]
+    subgraph ZoneA["Zone A -- Adversarial (isolated module boundary)"]
+        RT["Red Team Agent<br/>local model: huihui_ai/qwen2.5-abliterate:7b<br/>(CPU-only, num_gpu=0); the only model-backed role"]
     end
 
     subgraph TargetBoundary["Target's own PHI boundary (synthetic fixtures only, this engagement)"]
         T["Phase 2 Clinical Co-Pilot<br/>agentforge-2-evidence-agent v2.0.0<br/>(seeded synthetic patients, docs/STAGE1_TARGET.md)"]
     end
 
-    subgraph ZoneB["Zone B -- Evaluative (separate isolated processes/contexts)"]
-        J["Judge Agent<br/>local instruct model"]
-        O["Orchestrator Agent<br/>local instruct model"]
-        D["Documentation Agent<br/>local instruct model"]
+    subgraph ZoneB["Zone B -- Evaluative (separate modules, single process)"]
+        J["Judge Agent<br/>rule-based detect(); model-optional, unused by default"]
+        O["Orchestrator Agent<br/>deterministic rule/threshold logic; no model call"]
+        D["Documentation Agent<br/>deterministic template; model-optional, unused by default"]
     end
 
     subgraph Store["Local, on-disk state (no network egress)"]
@@ -110,10 +110,13 @@ flowchart TB
 **Reading the boundary.** Three separate things are colored above and must
 not be conflated:
 
-1. **Trust-zone boundary** (Zone A vs. Zone B): enforced by process/context
-   isolation, per `docs/ARCHITECTURE.md` §2's load-bearing property — this
-   is the Judge-vs-Red-Team independence guarantee, not a data-egress
-   control.
+1. **Trust-zone boundary** (Zone A vs. Zone B): enforced today at the
+   module and data level — no shared import path, typed inputs only — per
+   `docs/ARCHITECTURE.md` §1/§2's load-bearing property; this is the
+   Judge-vs-Red-Team independence guarantee, not a data-egress control.
+   OS-process isolation between the zones is a stated design goal, not yet
+   implemented (`docs/ARCHITECTURE.md` §1) — all four components currently
+   run in one Python process.
 2. **Target's PHI boundary**: this engagement's own data is synthetic —
    three seeded fixture patients (`docs/STAGE1_TARGET.md` §4: Phil Belford,
    Susan Underwood, Wanda Moore), not real PHI, per `docs/THREAT_MODEL.md`
@@ -124,11 +127,12 @@ not be conflated:
    pulled *out* of the target into the red-team platform's own storage
    (`evals/recordings/`, the exploit DB, or the filed vuln reports) beyond
    what the target's own `/chat` response already contained.
-3. **Network egress boundary**: zero. Every one of the four AI roles runs
-   locally (`docs/ARCHITECTURE.md` §4 — no role calls a hosted API); the
-   platform's own dependency footprint (§3 below) has no HTTP client aimed
-   at anything but `localhost` (ollama at `:11434`, the target container via
-   `docker exec`, per `docs/STAGE1_TARGET.md` §5).
+3. **Network egress boundary**: zero. Of the four roles, only the Red Team
+   Agent calls a model, and that call is local (`docs/ARCHITECTURE.md` §4 —
+   no role calls a hosted API); the platform's own dependency footprint (§3
+   below) has no HTTP client aimed at anything but `localhost` (ollama at
+   `:11434`, the target container via `docker exec`, per
+   `docs/STAGE1_TARGET.md` §5).
 
 ---
 
@@ -259,13 +263,16 @@ directory listing: `contracts/v1/` is the only version directory present.
 
 ### 3.3 Local model runtimes
 
-All four AI roles run locally, per `docs/ARCHITECTURE.md` §4 (the owner's
-locked, no-cloud decision for every role) and confirmed at the code level:
+Only the Red Team Agent calls a model in the shipped default path, and that
+model runs locally, per `docs/ARCHITECTURE.md` §4 (the owner's locked,
+no-cloud decision for every role that calls one) and confirmed at the code
+level. The other three roles are deterministic and call no model at all
+(see the table below):
 
 | Role | Runtime | Model | Verified where |
 |---|---|---|---|
 | Red Team Agent generator | ollama, `http://localhost:11434`, **CPU-only** (`num_gpu: 0`, hardcoded default) | `huihui_ai/qwen2.5-abliterate:7b` | `redteam/agents/red_team.py` `DEFAULT_MODEL` constant + module docstring: validated to comply with offensive-security generation, ~7s/call, CPU-only |
-| Judge / Orchestrator / Documentation Agents | separate local instruct-model instances, separate processes/contexts | stock instruct models (no uncensored requirement — none of the three is a generation-under-refusal-pressure task) | `docs/ARCHITECTURE.md` §4 |
+| Judge / Orchestrator / Documentation Agents | none -- deterministic Python, no model instance, no model call, in the same process as every other role | N/A in the shipped default path; each exposes an optional model-backed seam (`scorer`/`ranker`/`narrator`) that a stock instruct model could later fill (no uncensored requirement -- none of the three is a generation-under-refusal-pressure task) | `docs/ARCHITECTURE.md` §4 |
 | Target's answer model | GPU-resident, target's own container | 8B-Q5 quantized model | `docs/STAGE1_TARGET.md` §1/§6, `nvidia-smi` confirms ~7.8/12 GB VRAM resident |
 | Target's document-ingestion VLM | ollama-only, GPU when loaded | `qwen2.5vl:7b` | `planning/PHASE3_KICKOFF_PROMPT.md` (cited by `docs/ARCHITECTURE.md`) |
 
@@ -319,9 +326,9 @@ explicitly documented as an arbitrary placeholder accepted by the target's
 own insecure-by-default validator (VULN-0001) — "safe to publish as-is" per
 that document's own text, not a real credential.
 
-`pytest tests/ -q` re-run for this packet: **339 passed** with the sibling
+`pytest tests/ -q` re-run for this packet: **340 passed** with the sibling
 Phase 2 checkout (`../agentforge-2-evidence-agent`, pinned `v2.0.0`)
-present locally (confirmed at PR time); **233 passed, 106 skipped** in CI
+present locally (confirmed at PR time); **234 passed, 106 skipped** in CI
 and for anyone without that sibling — CI (`.github/workflows/ci.yml`) does
 not check it out, so the 100 total sibling-checkout citation cases
 class-skip cleanly there: 40 `TestTraceCitationsAgainstPinnedTarget` cases
@@ -366,7 +373,7 @@ those changes included, not a pre-change baseline.
   evidence the project has previously demonstrated this discipline under
   pressure, not as a claim about this PR's own diff (which touches no
   secret-adjacent files).
-- **339 passing tests (233 passed, 106 skipped in CI), no live/network/GPU
+- **340 passing tests (234 passed, 106 skipped in CI), no live/network/GPU
   call in the default suite.** Every test file under `tests/`
   (`tests/contracts/`, `tests/redteam/`, `tests/test_cases.py`,
   `tests/test_case_sourceref_relevance.py`, `tests/test_runner_sse.py`,
@@ -388,18 +395,18 @@ those changes included, not a pre-change baseline.
   has moved across PRs that touch test-suite-relevant code (e.g. PR #40's
   own test plan: "177 passed (unchanged; no test-suite-relevant code
   touched)" at that point in the repo's history; this PR's own platform
-  changes plus its expanded citation-verification test set move it to 339
-  with the sibling checkout present, or 233 passed / 106 skipped without
+  changes plus its expanded citation-verification test set move it to 340
+  with the sibling checkout present, or 234 passed / 106 skipped without
   it, §5.1).
 
 ---
 
 ## 5. Eval-result evidence
 
-### 5.1 The 339-test suite (232 in CI)
+### 5.1 The 340-test suite (234 in CI)
 
-`pytest tests/ -q` → **339 passed** with the sibling Phase 2 checkout
-present, re-confirmed for this packet (§4.1); **233 passed, 106 skipped**
+`pytest tests/ -q` → **340 passed** with the sibling Phase 2 checkout
+present, re-confirmed for this packet (§4.1); **234 passed, 106 skipped**
 in CI (`.github/workflows/ci.yml` does not check out the sibling target)
 and for any clone lacking it. Organized across `tests/contracts/` (schema
 + uniqueness constraints), `tests/redteam/` (the six agents + campaign
@@ -487,7 +494,7 @@ to approve and nothing already filed.
   fatal (this is §6's postmortem subject); `max_iterations` input
   validation. Test count: 163 baseline → 171 (PR #35's own reported delta;
   the repo has since grown to 250 total with the sibling checkout present,
-  or 233 passed / 106 skipped without it, §5.1).
+  or 234 passed / 106 skipped without it, §5.1).
 
 ### 5.4 Load-test numbers
 
@@ -607,8 +614,8 @@ describes — not because it was dramatic.
   (Mermaid diagram, trust-zone framing), §2 Auth model (platform + target),
   §3 Versioned dependency list (`requirements-contracts.txt`, contracts
   versioning, model runtimes), §4 Self-scan results (commands run + process
-  evidence), §5 Eval-result evidence (339 tests with the sibling checkout
-  present / 233 passed, 106 skipped in CI, 3 criticals, live-campaign
+  evidence), §5 Eval-result evidence (340 tests with the sibling checkout
+  present / 234 passed, 106 skipped in CI, 3 criticals, live-campaign
   evidence, load-test numbers), §6 Sample incident and postmortem.
 - **Every section cites a real, already-committed artifact**, not an
   invented one: `docs/ARCHITECTURE.md`, `docs/THREAT_MODEL.md`,

docs/DEMO_SCRIPT.md

@@ -20,9 +20,9 @@ evidence table this script complements with runnable commands.
   immediately before and after any live call and confirm VRAM stays flat.
 - `pytest tests/ -q` green (deterministic — no live/network/GPU call in the
   default suite; confirmed while writing this doc). The printed count is
-  environment-dependent: **339 passed** when the sibling Phase 2 checkout
+  environment-dependent: **340 passed** when the sibling Phase 2 checkout
   (`../agentforge-2-evidence-agent`, pinned `v2.0.0`) is present locally;
-  **233 passed, 106 skipped** in CI and for anyone cloning this repo without
+  **234 passed, 106 skipped** in CI and for anyone cloning this repo without
   that sibling (the 106 skipped are `TestTraceCitationsAgainstPinnedTarget`
   (40 cases, `tests/test_dos_input_bound_resolution.py`) plus
   `TestCitationsAgainstPinnedTargets` (60 cases,
@@ -33,7 +33,7 @@ evidence table this script complements with runnable commands.
 
 ```
 $ pytest tests/ -q
-339 passed in 2.38s          # with the sibling Phase 2 checkout present
+340 passed in 2.38s          # with the sibling Phase 2 checkout present
 ```
 
 ---
@@ -140,10 +140,11 @@ the full 3-report table).
 `redteam/agents/judge.py` (`JudgeAgent.score`) scores a target response
 against the `AttackCase` that produced it into a contract-valid
 `judge_verdict` (`contracts/v1/judge_verdict.schema.json`), independent of
-the Red Team Agent by construction: the module imports nothing from
-`redteam.agents`/`redteam.harness` and holds no Red Team state — the
-"conflict of interest by design" separation (`docs/ARCHITECTURE.md` §3(2)/
-§6). Verified directly:
+the Red Team Agent at the module and data level: the module imports nothing
+from `redteam.agents`/`redteam.harness`/`redteam.observability` and holds no
+Red Team state — the "conflict of interest by design" separation
+(`docs/ARCHITECTURE.md` §3(2)/§6). This is a module/import boundary, not an
+OS-process boundary — see `docs/ARCHITECTURE.md` §1. Verified directly:
 
 ```
 pytest tests/redteam/test_judge_agent.py::test_independence_module_imports_no_red_team_or_sibling_agent_internals -v
@@ -333,14 +334,14 @@ here for completeness:
 CI (`.github/workflows/ci.yml`) runs the deterministic suite —
 `python -m pytest tests/ -q` — on every push to `main` and on every pull
 request. CI does not check out the sibling Phase 2 target, so its printed
-count is **233 passed, 106 skipped** (the 106 skipped are
+count is **234 passed, 106 skipped** (the 106 skipped are
 `TestTraceCitationsAgainstPinnedTarget` (40, issue #25/#54),
 `TestCitationsAgainstPinnedTargets` (60, issue #58), and
 `TestStandingUpTargetPathsExistInPinnedTarget` (6, issue #61), all of
 which class-skip cleanly when `../agentforge-2-evidence-agent` is absent). Live-model and target-stack
 runs remain manual, outside CI: every command in this script was run
 locally against the dev stack while writing this doc, with the sibling
-checkout present, giving **339 passed**. `pytest tests/ -q` is still the
+checkout present, giving **340 passed**. `pytest tests/ -q` is still the
 reproducibility bar — re-run it after pulling this branch to confirm
-nothing here has drifted: expect **339 passed** if you have the sibling
-Phase 2 checkout at `v2.0.0`, or **233 passed, 106 skipped** if you don't.
+nothing here has drifted: expect **340 passed** if you have the sibling
+Phase 2 checkout at `v2.0.0`, or **234 passed, 106 skipped** if you don't.

redteam/agents/judge.py

@@ -5,18 +5,23 @@
 
 ## Architectural independence (ARCHITECTURE.md §3(2)/§6)
 
-The Judge is **architecturally independent** from the Red Team Agent:
-separate process/context by construction, because this module imports
-nothing from a Red Team module and holds no Red Team state. It scores from
-exactly two inputs -- the ``evals.schema.AttackCase`` that produced the
-probe (specifically its own rule-based ``detect`` predicate) and the
+The Judge is **architecturally independent** from the Red Team Agent at the
+module and data level (NOT an OS-process boundary -- see
+``docs/ARCHITECTURE.md`` §1, which corrects an earlier claim that each role
+runs as its own OS process; ``redteam/campaign.py::run_campaign`` calls all
+four components in-process, in one loop): this module imports nothing from
+a Red Team module and holds no Red Team state. It scores from exactly two
+inputs -- the ``evals.schema.AttackCase`` that produced the probe
+(specifically its own rule-based ``detect`` predicate) and the
 ``evals.runner.ParsedResponse`` the target returned -- never from Red Team
 internals (there is no Red Team Agent module yet; this module also imports
-nothing from any *sibling* agent or the stateful Regression Harness, so
-independence holds going forward too -- see
+nothing from any *sibling* agent, the stateful Regression Harness, or the
+Observability layer, so independence holds going forward too -- see
 ``tests/redteam/test_judge_agent.py::test_independence_module_imports_no_red_team_or_sibling_agent_internals``,
-which ast-scans this file's imports for `redteam.agents`/`redteam.harness`
-prefixes).
+which ast-scans this file's imports for `redteam.agents`/`redteam.harness`/
+`redteam.observability` prefixes, including relative and aliased forms --
+that scan's own documented limits (`importlib`, dynamic strings, attribute
+access after a bare `import redteam`) are in its docstring).
 
 ## Outcome mapping (the deterministic core)
 

tests/redteam/test_judge_agent.py

@@ -124,23 +124,67 @@ def test_deterministic_default_path_is_reproducible_no_model_call():
 def test_independence_module_imports_no_red_team_or_sibling_agent_internals():
     """Architectural independence (ARCHITECTURE.md §3(2)/§6): the Judge must
     not import any Red Team module, nor any sibling agent module, nor the
-    Regression Harness's stateful internals -- it scores purely from
-    (case, response). There is no Red Team module yet, so this also guards
-    the module going forward: adding `import redteam.agents.red_team` (or
-    similar) here should fail this test the moment such a module exists."""
+    Regression Harness's stateful internals, nor the Observability layer
+    (whose action log carries full Red Team attempt payloads, per
+    ``campaign.py``'s ``action_log.append`` calls) -- it scores purely from
+    (case, response, attempt_id). There is no Red Team module yet, so this
+    also guards the module going forward: adding `import
+    redteam.agents.red_team` (or similar) here should fail this test the
+    moment such a module exists.
+
+    ## What this guard does NOT catch (issue #73)
+
+    This is an AST scan over ``import`` / ``from ... import`` statements in
+    ``judge.py`` only. It resolves both absolute and relative imports
+    (``from .red_team import X``, ``from . import red_team``, ``from
+    ..observability.action_log import ActionLog``) to the fully-qualified
+    module they refer to, and catches aliased forms too (``import
+    redteam.agents.red_team as rt``, ``from redteam.agents import red_team
+    as r2`` -- the ``alias.name``/``node.module`` checked here are the
+    imported dotted path, not the local ``as`` binding, so aliasing does
+    not evade it). It does not, and cannot without much heavier static
+    analysis, catch: ``import redteam`` followed by attribute access
+    (``redteam.agents.red_team.something``), ``from redteam import agents``,
+    dynamic loading via ``importlib.import_module("redteam.agents.red_team")``
+    or ``__import__``, or a forbidden import built as a runtime string. It
+    also only scans this one file -- a forbidden import added to a module
+    that ``judge.py`` itself imports from would not be caught here. State
+    this precisely rather than implying a stronger guarantee than the scan
+    delivers."""
     source = JUDGE_MODULE_PATH.read_text(encoding="utf-8")
     tree = ast.parse(source, filename=str(JUDGE_MODULE_PATH))
-    forbidden_prefixes = ("redteam.agents", "redteam.harness")
+    forbidden_prefixes = ("redteam.agents", "redteam.harness", "redteam.observability")
+    # judge.py's own package, as dotted parts -- needed to resolve relative
+    # imports (level > 0) to the fully-qualified module they refer to, the
+    # same way Python's own import machinery would.
+    judge_package_parts = ["redteam", "agents"]
+
+    def _resolve_relative(node: ast.ImportFrom) -> list[str]:
+        base_len = len(judge_package_parts) - (node.level - 1)
+        base = judge_package_parts[:base_len] if base_len > 0 else []
+        prefix = ".".join(base)
+        if node.module:
+            full = f"{prefix}.{node.module}" if prefix else node.module
+            return [full]
+        return [f"{prefix}.{alias.name}" if prefix else alias.name for alias in node.names]
+
     offending: list[str] = []
     for node in ast.walk(tree):
         if isinstance(node, ast.Import):
             for alias in node.names:
                 if alias.name.startswith(forbidden_prefixes):
                     offending.append(alias.name)
-        elif isinstance(node, ast.ImportFrom) and node.module:
-            if node.module.startswith(forbidden_prefixes):
-                offending.append(node.module)
-    assert offending == [], f"judge.py must not import Red Team / sibling-agent internals: {offending}"
+        elif isinstance(node, ast.ImportFrom):
+            if node.level > 0:
+                resolved = _resolve_relative(node)
+            elif node.module:
+                resolved = [node.module]
+            else:
+                resolved = []
+            offending.extend(m for m in resolved if m.startswith(forbidden_prefixes))
+    assert offending == [], (
+        f"judge.py must not import Red Team / sibling-agent / observability internals: {offending}"
+    )
 
 
 def test_judge_timeout_raised_when_scoring_exceeds_budget():