[AI-stack hardening] production agent lifecycle, completion policy, MCP routing, and conformance

[rickylabs]

Context

Part of #238 and a production-hardening follow-up to the closed E3 agent-loop slice #242 and MCP slice #244. Related, but distinct from multi-agent composition in #247, durable UI integration in #943, and telemetry in #248.

The migration target is the current eis-chat runtime. Recent production debugging exposed edge cases that a future migration to @netscript/ai must absorb centrally; otherwise the application would have to retain stream surgery, completion fallbacks, MCP wrappers, and provider-specific cap logic alongside the framework.

The most serious reproduced failure came from confusing a provider iteration boundary with the end of the complete agent run. With TanStack's real event order, an iteration may emit RUN_FINISHED after tool-call arguments while the MCP wrapper executes afterward and emits TOOL_CALL_RESULT, followed by the next model iteration. Starting a tool-free synthesis pass at that intermediate RUN_FINISHED replayed pending calls against an empty registry, produced Unknown tool, duplicated call traces, and never reached the MCP server.

The existing createAgentLoop() is a good bounded typestate foundation, but its provider-neutral event model and tests do not yet specify these production lifecycle semantics or the policy flexibility required by local, hosted, subscription, and OpenAI-compatible providers.

Objective

Make @netscript/ai the single production-grade owner of agent-turn lifecycle, completion guarantees, tool/MCP execution, bounds, provider delegation, persistence hooks, and conformance testing. An application should compose policies and registries, not rewrite or post-process the agent stream.

Required runtime invariants

1. Iteration and run lifecycle

• Distinguish iteration finished from agent run exhausted/settled in types and state transitions.
• Treat exhaustion of the outer agent iterable/state machine—not a provider-level RUN_FINISHED chunk—as the authoritative terminal boundary.
• Define tool-call arguments complete separately from tool execution complete; in TanStack terms, TOOL_CALL_END is not proof of execution, while TOOL_CALL_RESULT is.
• Preserve real event order, call IDs, and exactly-once semantics across multiple model iterations.
• Support multiple calls in one iteration, parallel or out-of-order results, tool errors, late results, cancellation, and provider errors.
• Never replay pending tool calls against a missing/empty registry and never synthesize duplicate START/ARGS/END/RESULT sequences.
• A terminal done event must carry an explicit outcome (answered, tool-only, max-steps, output-limit, provider-eof, aborted, errored) rather than making all terminal states look successful.

2. Final-answer guarantee

• Provide an opt-in CompletionPolicy that can guarantee a user-facing answer after completed tool use.
• Only run final synthesis after the original agent stream has genuinely exhausted, at least one tool has a completed result, and no subsequent assistant text was produced.
• Do not synthesize while a next model iteration is already running or capable of answering.
• The synthesis transcript may include completed calls/results but must exclude unresolved pending calls.
• The synthesis pass must not expose/execute tools unless explicitly configured; it must not replay historical calls as new work.
• Reserve a configurable output budget for the final answer.
• If synthesis cannot produce text, surface a typed recoverable tool-only completion outcome rather than silently ending after a tool card.
• Handle provider EOF, output-token termination, mid-stream errors, and structured-output finalization distinctly.

3. Tool and MCP registry semantics

• Compose app-owned tools, eager MCP tools, and lazy discovery without one source unregistering or shadowing another accidentally.
• Keep a stable public/routed name and native MCP name. Server prefixes are routing metadata and must map back to the native tool exactly once.
• Detect and report name collisions deterministically; allow per-server namespacing/priority rather than relying on renaming servers.
• A discovery/connect failure must fail soft for unrelated local tools and healthy servers.
• Support enabled/disabled servers, stdio and Streamable HTTP, authenticated endpoints, keepalive/reconnect, stale session recovery, and explicit disposal.
• Expose hooks/spans for discovery, route selection, native invocation, result/error, and reconnect.
• Preserve UI/resource results and typed tool payloads without coercing everything to plain text.

4. Policy and provider flexibility

Introduce composable runtime policy with global defaults and per-agent/per-run overrides:

• maxSteps / iterations, independently configurable tool-call budget, history-token budget, output-token budget, timeouts, cancellation, retry policy, and reserved final-answer budget.
• Each cap supports an explicit mode: concrete value, framework default, or provider/unbounded delegation where appropriate.
• Hosted subscription providers and local runtimes such as LM Studio/Ollama can retain native limits instead of receiving an arbitrary framework cap.
• Mandatory runaway-loop safety remains explicit and cannot be disabled accidentally; delegating output limits must not imply an unbounded tool loop.
• Provider adapters normalize, without leaking SDK types, OpenAI/Responses output options, OpenRouter reasoning/options, Anthropic thinking/output controls, and OpenAI-compatible/local-provider behavior.
• Report effective policy and which layer supplied each value (runtime default, agent override, run override, provider-native).

5. Context, prompt, attachments, and history

• Supply a compact prompt-assembly seam for tools, skills, memory, and channel context without repeating large descriptions each iteration.
• Preserve complete assistant-tool-call/tool-result groups during history truncation; never retain one side of the pair.
• Keep system framing and the newest relevant context while reserving room for tool results and the final answer.
• Allow newly attached/OCR-processed content to be injected into the same turn once ingestion reaches ready; no mandatory second user turn.
• Make attachment readiness/ingestion failure explicit so the agent cannot hallucinate access to an image or call a tool that was not registered for the turn.

6. Durable execution and observability

• Provide settle hooks suitable for durable chat persistence: text, reasoning, calls, arguments, results, UI parts, errors, usage, and terminal outcome.
• Persistence must settle exactly once on success, abort, output limit, provider EOF, or error, and cold reload must reproduce the same visible trace.
• Emit run, iteration, discovery, and tool spans with stable IDs and real provider usage/cost metadata; align with [AI-stack E9] @netscript/ai: OTel GenAI/MCP semconv telemetry adapter (./otel) #248.
• Expose enough state to diagnose whether a failure happened before routing, during transport, in the tool, in the next model iteration, or in final synthesis.

Proposed public seams

Exact naming is open, but the framework needs equivalent responsibilities:

createAgentRuntime({ providers, tools, mcp, defaults, telemetry })
runtime.runTurn(input, {
  agentPolicy,
  completionPolicy,
  contextPolicy,
  signal,
  onEvent,
  onSettle,
})

• AgentPolicy: independently typed loop, tool, context, timeout, and provider-cap policies.
• CompletionPolicy: none | requireTextAfterTools | custom with reserved-budget settings.
• CompositeToolRegistry: app + MCP + lazy sources, collision policy, routed/native identity.
• A canonical runtime event/state reducer that adapters map into; applications do not interpret raw provider iteration markers.
• Testing helpers: scripted provider, in-memory registry, fake MCP transport, event collector, and live MCP conformance harness.

Conformance matrix / acceptance gates

• Real ordering: tool call → iteration RUN_FINISHED → tool result → next iteration → text answer; no premature synthesis.
• Tool call/result/answer normal path settles once and preserves order/IDs.
• Tool-only true exhaustion invokes completion policy once and returns visible text or typed failure.
• Several calls in one iteration, including parallel/out-of-order success and failure.
• Missing tool, schema error, tool exception, MCP disconnect/reconnect, discovery failure, stale session.
• App tool + multiple MCP servers + lazy discovery coexist; disabling one server does not remove other tools.
• Routed prefixed MCP wrapper invokes the native remote tool once and returns its real result; no Unknown tool replay.
• Max-step exhaustion after a completed tool result, independently from output cap.
• Provider-native/unlimited output policy with a still-bounded tool loop.
• Output limit, provider EOF, thrown provider error, cancellation during streaming, and cancellation during tool execution each settle correctly.
• History truncation preserves system framing and complete tool-call/result groups.
• Structured output and client-side approval/waiting-tool states do not trigger premature completion.
• Durable reload during an in-flight and completed tool turn reproduces the same trace.
• Deterministic fake-provider suite plus a live Streamable-HTTP MCP smoke test are release gates.

Migration gate for eis-chat

Migration is safe when the app can delete its custom completion guards, raw TanStack lifecycle interpretation, MCP prefix wrappers/pool orchestration, provider-specific cap branching, and related fallback tests while retaining all current behavior and configuration flexibility.

Evidence/reference implementation: rickylabs/eis-chat#152, especially the production ordering regression and live legacy-archeo-v2 MCP validation documented there.

Non-goals

• Named multi-agent strategies (router, supervisor, debate) remain in [AI-stack E8] @netscript/ai: orchestration primitives (fan-out + bounded-cycle) #247.
• Chat presentation components remain in fresh-ui/[frontend-contrib S21] AI frontend (durable chat route + assist launcher) #943.
• This issue does not require a centralized AI gateway; the runtime must work in-process first.

Dependencies / relationships

• Parent: epic: NetScript AI Stack — first-class AI runtime, chat & plugin seams (anchor #219) #238
• Extends: [AI-stack E3] @netscript/ai: createAgentLoop() state machine (maxIterations, history truncation, real usage) #242, [AI-stack E5] @netscript/ai: McpTransportPort (stdio + reconnectable Streamable-HTTP + auth modes) #244
• Coordinates with: [AI-stack E9] @netscript/ai: OTel GenAI/MCP semconv telemetry adapter (./otel) #248, [frontend-contrib S21] AI frontend (durable chat route + assist launcher) #943
• Orthogonal to: [AI-stack E8] @netscript/ai: orchestration primitives (fan-out + bounded-cycle) #247