feat(investigation): deliver autonomous investigation engine

Makefile

@@ -1,10 +1,10 @@
 .PHONY: format lint typecheck test verify benchmark
 
 format:
-	python3 -m ruff format src/nexus_runtime tests/test_agent.py tests/test_end_to_end.py tests/test_persistence_replay.py tests/test_scheduler.py tests/test_tools.py tests/test_distributed_runtime.py
+	python3 -m ruff format src/nexus_runtime tests/test_agent.py tests/test_end_to_end.py tests/test_persistence_replay.py tests/test_scheduler.py tests/test_tools.py tests/test_distributed_runtime.py tests/test_investigation_intelligence.py tests/test_evidence_loop.py tests/test_investigation_application.py tests/test_investigation_cli.py
 
 lint:
-	python3 -m ruff check src/nexus_runtime tests/test_agent.py tests/test_end_to_end.py tests/test_persistence_replay.py tests/test_scheduler.py tests/test_tools.py tests/test_distributed_runtime.py
+	python3 -m ruff check src/nexus_runtime tests/test_agent.py tests/test_end_to_end.py tests/test_persistence_replay.py tests/test_scheduler.py tests/test_tools.py tests/test_distributed_runtime.py tests/test_investigation_intelligence.py tests/test_evidence_loop.py tests/test_investigation_application.py tests/test_investigation_cli.py
 
 typecheck:
 	python3 -m mypy src/nexus_runtime
@@ -18,3 +18,4 @@ verify:
 
 benchmark:
 	python3 -m nexus_runtime.distributed.benchmark
+	python3 -m nexus_runtime.investigation.benchmark

README.md

@@ -93,6 +93,25 @@ The Phase 3 milestone provides a provider-neutral distributed execution layer:
 The public service boundary is intentionally `KnowledgeService`; NEXUS does not read
 the knowledge engine's database.
 
+### Autonomous investigation (Phase 4)
+
+Phase 4 closes the epistemic loop: structured objectives are observed against current
+knowledge, measurable gaps become scored and budget-aware investigation plans, those
+plans execute through the distributed Agent Harness, and provenance-complete evidence
+is fused, verified, and applied through the knowledge service. Progress is measured
+before another bounded iteration is considered.
+
+```bash
+nexus-research create "What is Acme's verified status?" \
+  --criterion "two independent sources support the answer"
+nexus-research status SESSION_ID
+nexus-research explain SESSION_ID
+nexus-investigation-bench
+```
+
+See [autonomous investigation](docs/autonomous-investigation.md) and the
+[closed epistemic loop](docs/autonomous-loop.md).
+
 **Quick start**
 
 ```bash

docs/adr/0003-evidence-verification-boundary.md → docs/adr/0004-evidence-verification-boundary.md

@@ -1,4 +1,4 @@
-# ADR 0003: verify evidence before promoting knowledge
+# ADR 0004: verify evidence before promoting knowledge
 
 ## Context
 

docs/adr/0005-ready-wave-runtime-integration.md

@@ -0,0 +1,26 @@
+# ADR 0005: Submit investigation DAGs in dependency-ready waves
+
+## Status
+
+Accepted for Phase 4.
+
+## Context
+
+`InvestigationPlan` needs dependencies, while the Phase 3 `DistributedTask` contract
+intentionally owns placement and delivery but has no dependency field. Adding a second
+scheduler in the investigation layer or reaching into coordinator storage would break
+the subsystem boundary.
+
+## Decision
+
+Plans compile through the existing `TaskDAG`. `PlanExecutionController` persists the
+plan-to-AgentRun-to-distributed-task mapping and submits only nodes whose parent tasks
+have succeeded. The normal `RuntimeApplication.submit_task` boundary remains the only
+submission path. Capabilities, priorities, retries, worker assignment, leases,
+cancellation, and recovery remain owned by the distributed runtime.
+
+## Consequences
+
+Independent nodes are queued together and remain parallelizable. Dependent nodes are
+submitted in later waves. Permanent parent failure blocks descendants explicitly.
+Session restart can reconstruct the mapping without depending on Python object identity.

docs/api-contract.md

@@ -11,3 +11,11 @@ adapters call these operations without exposing domain internals:
 The initial `RuntimeAPI` exposes them in-process. Adapters propagate trace/correlation
 IDs, authenticate callers, authorize cancellation and creation, and translate
 `DomainError` to their transport's validation response.
+
+Phase 4 adds a separate `InvestigationApplication` boundary. Its explicit stage
+operations are `create`, `observe`, `generate`, `score`, `select`, `build_plan`,
+`start_execution`, `advance_execution`, `collect_execution_results`,
+`collect_evidence`, `evaluate`, `verify`, `update_knowledge`, and
+`finish_iteration`. `status`, `pause`, `resume`, `cancel`, `explain`, and
+`resume_iteration` provide lifecycle and recovery operations. The CLI delegates to
+this boundary and contains no planning, execution, or verification policy.

docs/autonomous-investigation.md

@@ -0,0 +1,105 @@
+# Autonomous Investigation Engine
+
+Phase 4 adds the intelligence layer between the existing knowledge engine and the
+existing distributed AgentRun runtime. It does not replace retrieval, GraphRAG, gap
+analysis, the Agent Harness, task leases, retries, or worker scheduling.
+
+```mermaid
+flowchart LR
+    K1["Knowledge Intelligence"] -->|"snapshot + KnowledgeGap"| I["Investigation Engine"]
+    I -->|"InvestigationPlan + ready task wave"| D["Distributed Runtime"]
+    D -->|"AgentRun"| H["Agent Harness"]
+    H -->|"InvestigationResult + Evidence"| I
+    I -->|"verified KnowledgeUpdate"| K1
+```
+
+## Public domain model
+
+- `ResearchObjective` records the question, scope, constraints, success criteria,
+  metadata, identifier, and creation time.
+- `InvestigationSession` is an explicit `PLANNING → EXECUTING → EVALUATING →
+  UPDATING` state machine with paused and terminal states.
+- `KnowledgeSnapshot` captures retrieval references, GraphRAG entities and relations,
+  gaps, contradiction keys, uncertainty, and observation metadata.
+- `CandidateInvestigation` is a deterministic transformation of a measurable gap. It
+  records the hypothesis, evidence need, capability requirements, gain, cost, time,
+  risk, and priority.
+- `InvestigationPlan` validates dependencies and compiles through the existing
+  `TaskDAG` and distributed task contracts.
+- `Evidence`, `Evaluation`, `VerificationReport`, `InvestigationKnowledgeUpdate`, and
+  `ProgressReport` make the epistemic result inspectable rather than concatenating
+  agent text.
+
+All identifiers and important state are serializable. `SQLiteInvestigationRepository`
+persists the objective, session, iteration, and ordered artifacts. A process restart
+can reload collected `InvestigationResult` and `EvidenceSet` values and continue
+evaluation without rerunning completed agents.
+
+## Application boundary
+
+`InvestigationApplication` exposes independently callable stages:
+
+1. `observe`
+2. `generate`
+3. `score`
+4. `select`
+5. `build_plan`
+6. `start_execution` / `advance_execution`
+7. `collect_evidence`
+8. `evaluate`
+9. `verify`
+10. `update_knowledge`
+11. `finish_iteration`
+
+`plan_iteration` and `process_results` are thin compositions of those services. They
+do not hide an unbounded research loop.
+
+The high-level CLI is also thin:
+
+```text
+nexus-research create QUESTION --criterion CRITERION
+nexus-research status SESSION_ID
+nexus-research pause SESSION_ID
+nexus-research resume SESSION_ID
+nexus-research cancel SESSION_ID
+nexus-research explain SESSION_ID
+nexus-research evidence SESSION_ID
+nexus-research gaps SESSION_ID
+nexus-research plan SESSION_ID
+nexus-research iterations SESSION_ID
+```
+
+## Execution contract
+
+The current distributed record intentionally has no dependency field. The
+investigation layer therefore submits only dependency-ready waves. It never assigns a
+worker, claims a lease, retries an attempt, or executes an agent. When a parent task
+succeeds, `PlanExecutionController` submits newly ready children through
+`RuntimeApplication`. A cancelled or dead-lettered parent marks descendants blocked
+in the persisted plan execution rather than pretending they ran.
+
+The first wave can contain many independent tasks, allowing the existing scheduler to
+place them concurrently according to required capabilities.
+
+## Explainability and observability
+
+`explain` reconstructs three decision chains:
+
+- gap → gain/importance/uncertainty/cost/risk → score → selection;
+- claim → evidence → source lineage → evaluation → verification;
+- termination reason → objective confidence → gaps → contradictions → budget.
+
+Structured events use the `investigation.*` namespace and preserve objective/session
+correlation. Metrics include sessions, iterations, gaps discovered and resolved,
+generated/executed investigations, evidence, contradictions, updates, information
+gain, and cost per resolved gap.
+
+Run the deterministic 10/50/10 baseline with:
+
+```bash
+nexus-investigation-bench
+```
+
+See [lifecycle](investigation-lifecycle.md), [planning](investigation-planning.md),
+[evidence](evidence-model.md), [provenance](provenance.md),
+[verification](verification.md), and [closed loop](autonomous-loop.md).

docs/autonomous-loop.md

@@ -0,0 +1,91 @@
+# Closed Epistemic Loop
+
+The investigation engine advances one explicit stage at a time. There is no
+`while True` research loop and no prompt that substitutes for control flow.
+
+```mermaid
+stateDiagram-v2
+    [*] --> PLANNING
+    PLANNING --> EXECUTING: selected plan
+    EXECUTING --> EVALUATING: complete or accepted partial results
+    EVALUATING --> UPDATING: verification report
+    UPDATING --> PLANNING: continue after progress measurement
+    PLANNING --> COMPLETED: no valuable work / objective satisfied
+    UPDATING --> COMPLETED: stopping criterion
+    PLANNING --> PAUSED
+    EXECUTING --> PAUSED
+    EVALUATING --> PAUSED
+    UPDATING --> PAUSED
+    PAUSED --> PLANNING
+    PAUSED --> EXECUTING
+    PAUSED --> EVALUATING
+    PAUSED --> UPDATING
+    PLANNING --> CANCELLED
+    EXECUTING --> CANCELLED
+    EVALUATING --> FAILED
+    UPDATING --> FAILED
+```
+
+## One iteration
+
+```mermaid
+flowchart TD
+    O["Observe retrieval, GraphRAG, gaps, contradictions"] --> G["Generate grounded candidates"]
+    G --> S["Score gain, importance, uncertainty, availability, cost, time, risk, redundancy"]
+    S --> P["Budget/capacity selection and TaskDAG plan"]
+    P --> R["Submit ready AgentRuns to distributed runtime"]
+    R --> E["Collect complete and partial evidence with lineage"]
+    E --> F["Fuse duplicates, support, contradictions, and low-quality evidence"]
+    F --> V["Verify provenance, independent sources, quality, confidence, conflict policy"]
+    V --> U["Commit eligible updates through KnowledgeEngine"]
+    U --> N["Observe new knowledge and calculate progress"]
+    N --> T{"Termination policy"}
+    T -->|"continue"| O
+    T -->|"bounded stop"| X["Completed / Cancelled / Failed"]
+```
+
+## Bounded termination
+
+Every session has maximum iterations, investigations, AgentRuns, cost, and execution
+time. The deterministic termination policy handles objective satisfaction, confidence,
+budget exhaustion, maximum iterations, no valuable candidate, unresolvable conflict,
+user cancellation, and system failure in a documented precedence order.
+
+An uncertain result is valid. Verification preserves `confirmed`, `probable`,
+`uncertain`, `contradicted`, and `insufficient_evidence`; only policy-eligible claims
+reach the knowledge-update boundary.
+
+## Partial results and failure
+
+Distributed retry and lease recovery stay in the coordinator. The investigation layer
+sees the eventual task state and can evaluate completed or partial
+`InvestigationResult` values while other tasks remain running. Failed dependencies are
+recorded as blocked investigations. They are not silently counted as success.
+
+Collected results are persisted before evaluation. After restart,
+`resume_collected_iteration` reloads the evidence, continues evaluation and update, and
+does not invoke the Agent Harness again. At-least-once distributed execution can still
+duplicate delivery; evidence fingerprints and stable result lineage make fusion
+idempotent and inspectable.
+
+`resume_iteration` also reconstructs sessions interrupted during `EVALUATING` or
+`UPDATING`. The update proposal is persisted before calling the knowledge service. If
+the process fails after that call but before the receipt is stored, recovery may submit
+the same proposal again. Claim and evidence identifiers remain stable, so repository
+adapters must provide idempotent save-by-ID behavior. This is an explicit at-least-once
+knowledge-update boundary; the application does not claim a distributed transaction
+across its SQLite record and an external knowledge store.
+
+## Progress
+
+`ProgressMeasurer` compares before/after gap and contradiction snapshots and reports:
+
+- gaps resolved, remaining, and newly discovered;
+- uncertainty reduced;
+- contradictions introduced and resolved;
+- evidence and updates;
+- information gain;
+- cost per resolved gap.
+
+This report is the input to the next stopping decision and is persisted with the full
+iteration history.

docs/distributed-execution.md

@@ -10,3 +10,9 @@ must exist and the graph is validated for cycles. Independent ready tasks can le
 different workers concurrently. The scheduler enforces backpressure, worker
 concurrency, timeouts, retry policy, cancellation, and failed-dependency cancellation.
 On restart it reloads snapshots and deliberately reclaims former worker leases.
+
+The worker invokes the Agent Harness with an immutable `HarnessExecutionContext`:
+`run_id`, `correlation_id`, `task_id`, `attempt_id`, `lease_id`, `worker_id`, and the
+task metadata produced by the investigation plan. Harness implementations use this
+public context to construct evidence lineage; they do not query coordinator storage or
+infer the active attempt from in-process worker state.

docs/events.md

@@ -5,6 +5,27 @@ trace ID, correlation ID, optional causation ID, and structured payload. Stable
 namespaces are `agent`, `task`, `tool`, `worker`, `investigation`, `hypothesis`,
 `experiment`, and `knowledge_update`.
 
+The autonomous investigation application emits these stable lifecycle events:
+
+- `investigation.session_created`
+- `investigation.planning_started`
+- `investigation.gaps_identified`
+- `investigation.candidates_generated`
+- `investigation.selected`
+- `investigation.plan_created`
+- `investigation.execution_started`
+- `investigation.evidence_collected`
+- `investigation.evaluated`
+- `investigation.verification_started`
+- `investigation.knowledge_updated`
+- `investigation.iteration_completed`
+- `investigation.completed`
+- `investigation.failed`
+
+Their payloads include the session, objective, and iteration. The session is the trace
+identifier and the objective is the correlation identifier, preserving a stable chain
+from planning through distributed execution and knowledge update.
+
 `EventBus` is broker-independent: `publish`, `subscribe`, `acknowledge`, and
 `dead_letter`. `InMemoryEventBus` is deterministic for tests. Production adapters must
 define consumer-group, retention, authentication, ordering, acknowledgement, retry, and

docs/leases.md

@@ -17,7 +17,7 @@ sequenceDiagram
     TaskStore-->>Worker: attempt_id + lease_id
     Worker->>Coordinator: start(lease_id)
     Worker->>Coordinator: heartbeat + renew
-    Worker->>Harness: execute_or_resume(run_id)
+    Worker->>Harness: execute_or_resume(HarnessExecutionContext)
     Worker->>Coordinator: complete(lease_id)
     Coordinator->>TaskStore: atomic completion
 ```

docs/provenance.md

@@ -30,3 +30,8 @@ knowledge subsystem's chunk, document, source, and evidence identifiers.
 The knowledge update adapter never writes repositories or graph storage. It builds the
 existing `KnowledgeUpdate` service contract and submits it with
 `KnowledgeEngine.commit_knowledge_update`.
+
+Before submission, `KnowledgeEngine.validate_evidence_provenance` verifies that the
+referenced source, document, and chunk exist, form the declared source → document →
+chunk chain, and that the source reference matches. A structurally complete but
+dangling lineage is rejected rather than being written as trusted knowledge.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "nexus"
-version = "0.3.0"
+version = "0.4.0"
 description = "NEXUS autonomous knowledge-discovery platform: knowledge intelligence engine and fault-tolerant research runtime"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -31,6 +31,8 @@ dev = ["ruff>=0.6", "mypy>=1.11", "pytest>=7.4", "pytest-cov>=4.1"]
 nexus-knowledge = "nexus_knowledge.cli:main"
 nexus-runtime = "nexus_runtime.distributed.cli:main"
 nexus-runtime-bench = "nexus_runtime.distributed.benchmark:main"
+nexus-research = "nexus_runtime.investigation.cli:main"
+nexus-investigation-bench = "nexus_runtime.investigation.benchmark:main"
 
 [tool.setuptools.packages.find]
 where = ["src"]
@@ -59,3 +61,7 @@ select = ["E", "F", "I", "UP", "B"]
 python_version = "3.11"
 strict = true
 files = ["src/nexus_runtime"]
+
+[[tool.mypy.overrides]]
+module = ["nexus_knowledge.*", "numpy.*"]
+follow_imports = "skip"

src/nexus_knowledge/service/engine.py

@@ -271,6 +271,26 @@ def provenance(self, claim_id: str) -> ProvenanceResponse:
             source_references={sid: sources[sid].reference for sid in source_ids if sid in sources},
         )
 
+    def validate_evidence_provenance(
+        self,
+        source_id: str,
+        document_id: str,
+        chunk_id: str,
+        source_reference: str,
+    ) -> bool:
+        """Validate a source → document → chunk chain through the public service boundary."""
+        source = self.repository.sources.get(source_id)
+        document = self.repository.documents.get(document_id)
+        chunk = self.repository.chunks.get(chunk_id)
+        return bool(
+            source is not None
+            and document is not None
+            and chunk is not None
+            and source.reference == source_reference
+            and document.source_id == source.id
+            and chunk.document_id == document.id
+        )
+
     # -- transactions ---------------------------------------------------
     def commit_knowledge_update(self, update: KnowledgeUpdate) -> KnowledgeUpdateReceipt:
         """Atomically apply a batch of knowledge changes."""