Skip to content

fix: dedupe MCP server entries and retry sidecar connections on startup - #60

Merged
DominikPinsel merged 1 commit into
mainfrom
fix/mcp-startup-retry-dedupe
Aug 12, 2026
Merged

fix: dedupe MCP server entries and retry sidecar connections on startup#60
DominikPinsel merged 1 commit into
mainfrom
fix/mcp-startup-retry-dedupe

Conversation

@DominikPinsel

Copy link
Copy Markdown
Owner

Two MCP wiring reliability issues surfaced while debugging agent chat (fix in #59 + deployment PR AInsel/ainsel-deployment#14):

1. Duplicate MCP_SERVERS entries

The controller builds MCP_SERVERS from Agent.spec.enabledMCPs discovery and AgentImage.spec.mcpServers independently, so an MCP declared on both (e.g. mem0 on the dev cluster) reached the runtime twice and was connected/registered twice:

MCP_SERVERS=mem0=http://mcp-mem0...:8080/mcp,mem0=http://mcp-mem0...:8080/mcp,forgejo-...,chat=...

Fix: mcpservers.DedupeEntries (first occurrence wins) applied before emitting the env var, plus defensive first-wins dedupe in the pi extension's parseServers.

2. Startup race — sidecar connections skipped forever

pi-ainsel-mcp connected to each server exactly once at process start. If a pod sidecar (e.g. chat on localhost:8081) was not listening yet, the server was skipped for the pod's entire lifetime:

pi-ainsel-mcp: server unreachable; skipping { server: 'chat', err: 'fetch failed' }
pi-ainsel-mcp: registered 117 MCP tools   <- no chat tools, agent can never reply

Observed live on agent-a-14b9fb30 after a rollout; a pod restart "fixed" it by winning the race.

Fix: per-server retry with exponential backoff in Catalog.connect (4 attempts, 250ms base: 0.25+0.5+1+2s), bounded by the existing 10s startup budget. Each attempt is logged (connect failed; retrying).

Verification

  • go test ./internal/controller/mcpservers/ + go build ./... (operators/agent) ✅
  • golangci-lint run ./internal/controller/... — 0 issues ✅
  • Node smoke tests (type-stripped): dedupe drops duplicates; retry loop against a closed port logs 3 retries + final skip; a server that starts listening after ~400ms is recovered (connected, tool registered) ✅

Deployment notes

  • The operator-side dedupe takes effect with the next operator image build (mutable dev tag, digest-aware restarts).
  • The pi extension retry requires a new dpinsel/ainsel-pi-go agent image; the dev cluster currently pins 1.24 via its AgentImage CRs, so those need a tag bump to pick this up.

Two MCP wiring reliability issues surfaced while debugging agent chat:

1. Duplicate MCP_SERVERS entries: the controller builds MCP_SERVERS from
   Agent.spec.enabledMCPs discovery and AgentImage.spec.mcpServers
   independently, so an MCP declared on both (e.g. mem0) reached the
   runtime twice and was connected/registered twice. Dedupe by server
   name (first occurrence wins) before emitting the env var, and drop
   duplicates at parse time in the pi extension as well.

2. Startup race: pi-ainsel-mcp connected to each server exactly once at
   process start; if a pod sidecar (e.g. chat on localhost:8081) was not
   listening yet, that server was skipped for the pod's entire lifetime
   and the agent silently lost its tools. Retry per server with
   exponential backoff (4 attempts, 250ms base) inside the existing
   startup budget.
@DominikPinsel

Copy link
Copy Markdown
Owner Author

@review-agent please have a look

@ainsel-reviewer ainsel-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

Overall: Approve

Focused, well-diagnosed fix for two real MCP wiring bugs; the operator-side dedupe is properly unit-tested, the runtime-side dedupe is placed correctly after validation, and the retry loop stays inside the existing startup budget. CI fully green on the head SHA (5/5 runs).

Blockers

  • None.

Suggestions

  • pi/pi-extensions/ainsel-mcp/catalog.ts:84 — The retry loop never re-checks deadline.aborted after await sleep(delayMs), and sleep itself is not deadline-aware. Worst case, connect() starts one extra attempt after the startup budget has expired and overshoots it by up to the last backoff delay (~2s). Bounded and harmless in practice, but a deadline.aborted check after the sleep would make the 10s budget strict.
  • pi/pi-extensions/ainsel-mcp/catalog.ts:62 / parse.ts:35 — No checked-in automated tests for the retry loop or the runtime dedupe (the PR body documents manual Node smoke tests for both). This area of the repo has no TS test harness and CI only runs node --check, so this is not blocking — but the async backoff loop is exactly the kind of code that regresses silently; worth considering a minimal test setup for the extension as follow-up work.
  • pi/pi-extensions/ainsel-mcp/parse.ts:35 — The runtime-side dedupe drops duplicates silently. A debug-level log line ("duplicate MCP server entry dropped") would make future operator-side regressions easier to spot in pod logs.

Nits

  • operators/agent/internal/controller/mcpservers/discover.go:79DedupeEntries keys on the untrimmed prefix e[:i] while parseServers trims before deduping. The controller never emits whitespace-padded entries so this can't diverge in practice — just noting it for anyone tempted to reuse DedupeEntries on arbitrary input.

What's Done Well

  • Defense in depth with matching semantics: dedupe at the source (operator) and defensively at runtime (parseServers), both first-wins.
  • Retry logic cleanly extracted into connectServer, with each attempt observable — both the retry and the final skip log lines carry attempt metadata.
  • Good test coverage for DedupeEntries (first-wins order, nil input, no-= entries); the i > 0 guard handles leading-= gracefully.
  • Excellent PR description: root cause, observed symptoms, per-side fix, verification, and deployment notes (AgentImage tag bump on the dev cluster).

Testing

  • Operator: three new unit tests in discover_test.go cover the new function; go test + lint pass in CI.
  • Pi extension: no checked-in tests (no TS harness exists in the repo); the PR documents smoke tests covering dedupe and the retry/recovery path, and CI (node --check, both image builds, secret scan) is green on head dc7b680.

@ainsel-reviewer

Copy link
Copy Markdown

@DominikPinsel LGTM — approved.

@DominikPinsel
DominikPinsel merged commit 3a6f280 into main Aug 12, 2026
5 checks passed
@DominikPinsel
DominikPinsel deleted the fix/mcp-startup-retry-dedupe branch August 12, 2026 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant