Architecture: Multi-Project Session Isolation

[ketema]

Problem Statement

Serena's current architecture uses a global active project model:

• SerenaAgent._active_project is shared across all MCP clients
• activate_project("/repo/A") from Agent-1 switches Agent-2's workspace
• Not compatible with multi-agent, multi-worktree workflows

Proposed Architecture

┌─────────────────────────────────────┐
│     SerenaMCPServer (HTTP)          │
│  ┌───────────────────────────────┐  │
│  │    SessionRegistry            │  │  ← Map session_id → SessionContext
│  │    {session_id: SessionCtx}   │  │
│  └───────────────────────────────┘  │
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│         SessionContext              │  ← Per-client state
│  - session_id                       │
│  - workspace_root (project path)    │
│  - modes                            │
│  - lsp_workspace_folders[]          │
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│      LSPProcessPool (SHARED)        │  ← One process per language
│  - python_lsp (shared)              │
│  - rust_lsp (shared)                │
│  - typescript_lsp (shared)          │
│  + workspace_folders tracking       │
└─────────────────────────────────────┘

Key Insights

LSP Already Supports Multi-Workspace

LSP is inherently multi-client and multi-workspace:

• Each connection has its own rootUri / workspaceFolders
• Documents are scoped by URI, not "project"
• Servers maintain per-workspace caches and indexes

HTTP Sessions Provide Isolation

MCP streamable-http transport provides Mcp-Session-Id header - this IS the session boundary.

Design Decisions

Decision	Rationale
One LSP process per language	LSP natively supports workspaceFolders[]
Session = MCP session_id	MCP streamable-http already provides session header
Lazy workspace registration	workspace/didChangeWorkspaceFolders on project activation
Path validation per session	Tools reject paths outside session's workspace_root

Implementation Phases

Phase 1: Session Isolation (No LSP Changes)

• SessionContext dataclass with session_id, workspace_root, modes
• SessionRegistry mapping session_id → SessionContext
• Extract session_id from MCP request context

Phase 2: Session-Aware Tools

• Modify tool dispatch to use session context
• Validate paths are under session's workspace_root
• Route to correct LSP workspace

Phase 3: LSP Workspace Multiplexing

• LSPPool with shared processes per language
• Dynamic workspace/didChangeWorkspaceFolders calls
• Track workspace registrations per language

Open Questions

1. Session creation trigger: First tool call? Explicit activate_project? MCP initialize?
2. Session cleanup: When does a session expire? LRU eviction? Explicit shutdown?
3. LSP workspace limits: Some LSPs may have limits on workspace folders
4. Backward compatibility: Single-project mode should still work

Related

• Branch: feature/multi-project-support
• Upstream issue Feature Request: Auto-activate project based on MCP client working directory oraios/serena#895: Auto-activate project based on CWD
• Polyglot PR Add Multi-Language (Polyglot) Project Support oraios/serena#703 experience (closed, maintainer preference for simpler impl)

[ketema]

Status Update: Multi-Project Support Refactoring (Issue #6)

This update provides the definitive roadmap, architectural corrections, and new requirements identified during a progress audit.

1. Architectural Correction: The "Singleton" Conflict

I identified that the codebase (SerenaAgent, Project) enforces a Single-Session Architecture that conflicts with the new Multi-Project Contracts.

• Conflict: SerenaAgent._activate_project actively destroys the previous session.
• Fix: SerenaAgent must become stateless. Session state moves to SessionRegistry.
• Plan: See Multi-Project Refactoring Roadmap (v2) below.

2. LSP Compatibility Findings

An audit of 48 supported LSPs revealed that a binary "Single vs. Multi-Root" model is insufficient. We are introducing a Policy-Based Adapter to handle critical outliers:

• TypeScript: Requires resource management policy (memory limits).
• Clangd: Requires cache isolation policy.
• Terraform: Requires forced isolation despite claiming multi-root support.

3. Observability Requirements

To manage the complexity of concurrent sessions and shared LSPs, we are adding a full-stack observability layer:

• Backend: New API endpoints for session/LSP state.
• Frontend: New "Sessions" and "LSP Resources" dashboard views.

---

Artifacts

A. Refactoring Roadmap (v2)

Multi-Project Refactoring Roadmap (v2)

Date: 2026-01-12
Status: DEFINITIVE GUIDE
Supersedes:

• multi-project-implementation-plan-v1.md (Integration section)
• option-c-sync-refactor-plan.md (Integration section)
• multi-project-integration-complete.md (Invalidated - claims rejected)

Executive Summary

This roadmap defines the mandatory refactoring phases required to bridge the gap between the verified contracts and the actual application entry points.

---

Phase 1: Architectural Decoupling (The "Unwiring")

Goal: Remove stateful assumptions from core classes to prepare for multi-session routing.

REQ-1: Make SerenaAgent Stateless

WHAT: Remove _active_project and _current_session_id instance variables from SerenaAgent.
WHY: These variables enforce a Singleton pattern. Storing state on the Agent prevents it from handling multiple concurrent sessions (Issue #6).
EXPECTED:

• SerenaAgent methods must no longer rely on self._active_project.
• Any method requiring project context must resolve it dynamically via SessionRegistry using the current context (ContextVar).

REQ-2: Convert Project to Configuration Holder

WHAT: Remove LanguageServerManager ownership from the Project class.
WHY: Project currently owns LSP lifecycles. This prevents GlobalLanguageServerPool from managing shared LSPs across sessions. If Project owns the LSP, opening the same project in two sessions creates two competing LSPs (or errors), violating resource efficiency requirements.
EXPECTED:

• Project class acts solely as a configuration loader (paths, ignore patterns, settings).
• Project does not start or stop subprocesses.

---

Phase 2: Core Wiring (The "Rewiring")

Goal: Connect the dormant MCPSessionBridge and GlobalLanguageServerPool to the application lifecycle.

REQ-3: Initialize Global Services in Factory

WHAT: Update SerenaMCPFactory (in mcp.py) to instantiate singleton instances of SessionRegistry, MCPSessionBridge, and GlobalLanguageServerPool at startup.
WHY: These components currently exist only in tests. They must exist in the production runtime to manage state.
EXPECTED:

• The FastMCP server lifespan context initializes these services.
• SerenaAgent receives references to these registries via dependency injection (constructor).

REQ-4: Implement Session-Aware Activation

WHAT: Rewrite SerenaAgent.activate_project to use SessionRegistry.bind_session.
WHY: The current implementation calls unbind_session on the previous session, actively destroying concurrency.
EXPECTED:

• Calling activate_project binds the current session ID (from ContextVar) to the target project path.
• It does NOT affect other active sessions.
• It returns success/failure based on path validation.

REQ-5: Route LSPs via Global Pool

WHAT: Replace direct LanguageServerManager calls in SerenaAgent with SessionAwareToolDispatch.
WHY: Tools must request LSPs from the global pool to enable sharing and idle reclamation.
EXPECTED:

• When a tool (e.g., find_symbol) is called, the system:
  1. Identifies the current session.
  2. Resolves the session's workspace root.
  3. Acquires an LSP from GlobalLanguageServerPool.
  4. Executes the request.
  5. Releases the LSP reference (start idle timer).

---

Phase 3: Verification & Stabilization

Goal: Prove concurrency works with adversarial tests.

REQ-6: Implement Real Integration Tests

WHAT: Create a new test suite that initializes the full SerenaMCPFactory stack.
WHY: Current tests are unit-level mocks. We need to prove that Client A can work on Project A while Client B works on Project B.
EXPECTED:

• Test: Connect Client A, Activate Project A.
• Test: Connect Client B, Activate Project B.
• Test: Verify Client A cannot read Project B files.
• Test: Verify Client A disconnect does not affect Client B.

REQ-7: De-Stub GlobalLanguageServerPool

WHAT: Replace the _create_lsp stub in GlobalLanguageServerPool with actual SolidLanguageServer instantiation logic.
WHY: The current implementation is a mock that crashes if run.
EXPECTED:

• The pool correctly instantiates SolidLanguageServer with configuration derived from the target Project.

---

Phase 4: Cleanup

REQ-8: Remove Legacy Single-Session Code

WHAT: Delete LanguageServerManager (the old class) if it is fully superseded by GlobalLanguageServerPool, or refactor it to be a stateless utility used by the Pool.
WHY: Dead code creates confusion and maintenance burden.
EXPECTED:

• Clean codebase with one clear way to manage LSPs.

---

Critical Checkpoints

1. Stop Unbinding: The immediate priority is to stop SerenaAgent from unbinding the previous session during activation.
2. Context Injection: Ensure mcp.py properly injects the new components into SerenaAgent.

This roadmap serves as the authoritative source for the remaining work on Issue #6.

B. LSP Compatibility Analysis

LSP Compatibility Analysis for Multi-Project Support

Date: 2026-01-12
Status: DRAFT - Findings from Adversarial Audit
Author: Gemini (Adversarial Mode)

Executive Summary

The transition to multi-project support via GlobalLanguageServerPool relies on the assumption that LSPs fall into two clean categories:

1. Multi-Root: Shareable instance (e.g., rust-analyzer, pylsp).
2. Single-Root: Cheap, isolated instances per project (e.g., tsserver treated as single-root).

An audit of the 48 supported LSPs reveals that this binary categorization is insufficient. Three critical outliers break the resource management and isolation guarantees:

1. TypeScript (tsserver/vtsls): Resource intensive, stateful, and resistant to simple process isolation without explicit configuration.
2. Clangd: Relies on global/project-local caches that may conflict if multiple instances run for related projects without unique cache path injection.
3. TerraformLS: Claims multi-root capability but operates on directory-based module state, risking cross-project contamination if treated as a shared instance.

Detailed Findings

1. TypeScript (TypeScriptLanguageServer) - The Resource Hog

Contract: TsServerAdapterContract (Single-Root / multi_root_support=NONE).
Impact: Spawns 1 process per project.
Analysis:

• tsserver is memory-heavy (often 300MB-1GB+ per instance).
• A user with 5 active TS projects (common in microservices) will spawn 5 heavy node processes.
• Violation: CON-3: Memory Constraints.
• Technical Detail: tsserver technically supports workspaceFolders but implementation details (ProcessLaunchInfo in typescript_language_server.py) hardcode rootPath and rootUri. It doesn't expose tsserver's internal "project" abstraction effectively to the LSP client in a way that allows safe sharing.

2. Clangd (ClangdLanguageServer) - The Cache Corruptor

Contract: ClangdAdapterContract (Single-Root / multi_root_support=NONE).
Impact: Spawns 1 process per project.
Analysis:

• clangd relies on .cache/clangd or global caches.
• If multiple instances run for projects that share common headers/libraries (monorepo), they might race on cache writes.
• Violation: REQ-1: Session Isolation.
• Missing Feature: The current factory does not inject unique --cache-path arguments to isolate these instances.

3. Terraform (TerraformLS) - The False Friend

Contract: Detected as Multi-Root (capabilities["workspace"]["workspaceFolders"] = True).
Impact: Shared instance across all Terraform projects.
Analysis:

• Terraform state is directory-specific (.terraform).
• terraform-ls might handle didChangeWorkspaceFolders, but Terraform operations (init, plan) are strictly directory-bound.
• Risk: A shared instance might leak module state or confuse terraform init contexts between unrelated projects.
• Recommendation: Treat as Single-Root despite capabilities to ensure safety.

Architectural Proposal: Capabilities-Driven Policy Injection

The binary "Multi/Single" distinction is too coarse. We need a Policy-based approach.

New Concept: LSPPoolingPolicy

Instead of just is_multi_root(), the Adapter should return a Policy:

1. SHARED_INSTANCE (Default for Multi-Root)

   • Stateless or capable of managing multiple roots safely.
   • Used for: Rust, Python, Go.

2. ISOLATED_PROCESS (Default for Single-Root)

   • Separate process per root.
   • Used for: Generic single-root LSPs.

3. ISOLATED_WITH_RESOURCE_MANAGEMENT (New)

   • Separate process, but managed carefully.
   • TypeScript: Needs memory limit flags (--max-old-space-size) injected.
   • Clangd: Needs --cache-path injection for isolation.

4. FORCED_ISOLATION (New)

   • Claims multi-root but unsafe to share.
   • Terraform: Force separate process even if capability exists.

Implementation Strategy

Refactor LSPCapabilityAdapter to include:

• get_pooling_policy(language) -> PoolingPolicy
• get_launch_arguments(root) -> list[str] (To inject cache paths/memory limits)

This moves the complexity of "how to run safely" into the Adapter, keeping GlobalLanguageServerPool generic.

C. Observability Requirements

Multi-Project Observability Requirements (v2)

Date: 2026-01-12
Status: APPROVED - Backend + Frontend Full Stack Plan
Feature: Multi-Project Support (Issue #6)

1. Overview

With the transition to a multi-project architecture (SessionRegistry, GlobalLanguageServerPool), the system state becomes more complex. Users and administrators need visibility into concurrent sessions, shared LSP resources, and isolation boundaries. This document defines the requirements for logging, backend API, and the frontend dashboard UI.

2. Structured Logging Requirements

The existing MemoryLogHandler captures formatted strings. To maintain compatibility while improving debuggability:

LOG-1: Session Context Prefix

All logs generated within a session context MUST include the session ID and workspace name.

• Format: [Session: <short_id> | <project_name>] <message>
• Example: [Session: a1b2 | backend-api] Tool dispatch: find_symbol

LOG-2: LSP Lifecycle Events

Critical LSP pool events must be logged at INFO level with resource details.

• Acquire: [LSP-Pool] Acquired python LSP for /abs/path/to/project (Session: a1b2)
• Release: [LSP-Pool] Released python LSP for /abs/path/to/project (Ref Count: 0)
• Reclaim: [LSP-Pool] Reclaiming idle python LSP (Idle: 3601s > 3600s)
• Crash: [LSP-Pool] LSP Crashed: rust-analyzer (PID: 1234). Restarting...

LOG-3: Path Validation Failures

Security boundaries must be observable.

• Violation: [Security] Path Traversal Blocked: /etc/passwd escapes /abs/path/to/project (Session: a1b2)

3. Dashboard API Requirements

The SerenaDashboardAPI (src/serena/dashboard.py) must be extended to expose the new architectural components.

API-1: Active Sessions Endpoint

Endpoint: GET /get_session_overview
Response Model:

{
  "active_sessions": [
    {
      "session_id": "uuid-string",
      "workspace_root": "/abs/path/to/project",
      "project_name": "project-name",
      "connected_at": "timestamp",
      "client_id": "optional-client-info",
      "acquired_lsps": ["python", "rust"]
    }
  ],
  "total_sessions": 2
}

API-2: LSP Pool Statistics

Endpoint: GET /get_lsp_pool_stats
Response Model:

{
  "active_lsps": [
    {
      "language": "python",
      "workspace_root": "/abs/path/to/project",
      "pid": 12345,
      "memory_mb": 150,  # Stretch goal
      "ref_count": 1,
      "active_sessions": ["uuid-string"],
      "status": "running"
    }
  ],
  "total_memory_usage_mb": 450
}

4. Frontend Requirements (UI/UX)

The frontend (dashboard.js, index.html) must be updated to visualize the new multi-project state.

UI-1: Sessions View (New Section)

Goal: Visualize concurrent MCP clients.

• Location: page-overview > New "Sessions" section (below "Executions").
• Visual: Table or Card list.
• Data:
  • Session ID (shortened).
  • Project Name.
  • Connected Time.
  • List of active LSPs (badges).
• Action: "Terminate Session" button (calls unbind_session).

UI-2: LSP Resource Monitor (New Page)

Goal: Monitor resource usage of the global pool.

• Location: New Menu Item: "LSP Resources" (page-lsp).
• Visual:
  • Summary Cards: Total Active LSPs, Total Memory (est), Idle LSPs.
  • LSP Table:
    • Language (Icon/Text).
    • Workspace Path.
    • Status (Running/Idle/Reclaiming).
    • Ref Count (Number of attached sessions).
    • PID.
• Action: "Restart LSP" button per row.

UI-3: Config Overview Updates

• Update: "Active Project" section is currently a singleton. Update it to show "Primary Session Project" or list all active projects if multiple exist.

5. Implementation Plan

Phase 1: Backend API

1. Add get_session_overview() method to SessionRegistry.
2. Add get_pool_stats() method to GlobalLanguageServerPool.
3. Implement Flask routes in dashboard.py.

Phase 2: Frontend Logic (dashboard.js)

1. Add loadSessions() and loadLspStats() polling functions.
2. Implement displaySessions(sessions) renderer.
3. Implement displayLspStats(stats) renderer.
4. Add navigation logic for new "LSP Resources" page.

Phase 3: Frontend Structure (index.html)

1. Add HTML container for <div id="active-sessions-display">.
2. Add HTML template for <div id="page-lsp">.
3. Add Menu Item in <div id="menu-dropdown">.

Phase 4: Styling (dashboard.css)

1. Add styles for session-card and lsp-table.
2. Ensure dark mode compatibility for new elements.

[ketema]

Update: Roadmap Refined (v2.1). Added Security (SEC-5), Thread-Safety (DCL), and Observability requirements based on AI Panel feedback. Proceeding with Phase 1 Implementation.

[opcode81]

@ketema, with oraios#1104, we are adding support for querying other projects to Serena, which is loosely related to this issue.

[ketema]

Awesome! I’m sure your implementation will be better than what I hacked together. Can’t wait

…

On Sat, Feb 28, 2026 at 18:37 Dominik Jain ***@***.***> wrote: *opcode81* left a comment (ketema/serena#6) <#6 (comment)> @ketema <https://github.com/ketema>, with oraios#1104 <oraios#1104>, we are adding support for querying other projects to Serena. — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB7TGHS2NP4TKWY7QDMF3D4ONZ2XAVCNFSM6AAAAACRKD26GGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSNZYGYZDANBYGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>