From 63d862a732749840ec2bc6723bc12170fb6bae69 Mon Sep 17 00:00:00 2001 From: Archit Mittal Date: Mon, 10 Aug 2026 18:05:20 +0530 Subject: [PATCH] docs: add development guide docs/DEVELOPMENT.md and track design docs (Fixes #28) --- .gitignore | 2 +- docs/AGENTS.md | 462 ++++++++++++++++++++++++++++++++++++++++++++ docs/DESIGN.md | 233 ++++++++++++++++++++++ docs/DEVELOPMENT.md | 141 ++++++++++++++ docs/PRD.md | 145 ++++++++++++++ docs/RULES.md | 155 +++++++++++++++ docs/SCHEMA.md | 399 ++++++++++++++++++++++++++++++++++++++ 7 files changed, 1536 insertions(+), 1 deletion(-) create mode 100644 docs/AGENTS.md create mode 100644 docs/DESIGN.md create mode 100644 docs/DEVELOPMENT.md create mode 100644 docs/PRD.md create mode 100644 docs/RULES.md create mode 100644 docs/SCHEMA.md diff --git a/.gitignore b/.gitignore index b131846..813dcd0 100644 --- a/.gitignore +++ b/.gitignore @@ -52,5 +52,5 @@ htmlcov/ *.tgz # Developer & AI Agent Docs -docs/ +# docs/ diff --git a/docs/AGENTS.md b/docs/AGENTS.md new file mode 100644 index 0000000..1be588e --- /dev/null +++ b/docs/AGENTS.md @@ -0,0 +1,462 @@ +# AgentScope: Guidelines for AI Coding Agents + +AgentScope is a lightweight, open-source, self-hosted observability dashboard for multi-agent AI pipelines. This `AGENTS.md` file serves as guidance for AI coding agents (like GitHub Copilot, Cursor, Claude, etc.) working on this codebase. + +## 1. Project Overview for AI Agents + +- AgentScope is a three-module system: Python SDK, FastAPI Server, Next.js UI +- Data flows one direction: **SDK → Server → UI** +- The SDK instruments LangChain pipelines via callback handlers +- The Server receives events via WebSocket, persists to SQLite, broadcasts to UI +- The UI renders real-time event feeds, agent DAG graphs, token counters, and session history + +```mermaid +graph LR + SDK[Python SDK\nLangChain Callback] -->|WebSocket Events| Server[FastAPI Server] + Server <-->|Persist| DB[(SQLite WAL)] + Server -->|WebSocket Broadcast| UI[Next.js UI] +``` + +## 2. Repository Structure + +```text +agentscope/ +├── packages/ +│ ├── sdk/ # Python SDK — LangChain callback handler + WebSocket client +│ │ ├── agentscope/ +│ │ │ ├── __init__.py +│ │ │ ├── callback.py # Core: AsyncCallbackHandler subclass +│ │ │ ├── client.py # WebSocket client with reconnection +│ │ │ ├── models.py # Pydantic v2 event models +│ │ │ ├── decorators.py # @trace decorator +│ │ │ └── _pricing.py # Token cost lookup +│ │ ├── tests/ +│ │ ├── setup.py +│ │ └── pyproject.toml +│ ├── server/ # FastAPI server — REST + WebSocket + SQLite +│ │ ├── main.py +│ │ ├── ws_manager.py # Dual-pool WebSocket manager +│ │ ├── database.py # SQLAlchemy engine + session factory +│ │ ├── models.py # SQLAlchemy ORM models +│ │ ├── schemas.py # Pydantic response schemas +│ │ ├── graph_layout.py # Dagre-based node positioning +│ │ ├── routers/ +│ │ │ ├── sessions.py +│ │ │ └── events.py +│ │ ├── tests/ +│ │ └── requirements.txt +│ └── ui/ # Next.js 14 dashboard — React Flow + Recharts + Zustand +│ ├── app/ # App Router pages +│ ├── components/ # React components +│ ├── hooks/ # Custom hooks (useWebSocket, useSession, useEvents) +│ ├── store/ # Zustand state management +│ ├── lib/ # Utilities and API client +│ ├── types/ # TypeScript type definitions +│ └── package.json +├── examples/ # Runnable integration examples +├── docs/ # Project documentation +├── docker-compose.yml +├── Makefile +└── README.md +``` + +## 3. Critical Rules for AI Agents + +These are hard rules that must NEVER be violated: + +> [!CAUTION] +> 1. **Use LangChain's `run_id` as `event_id`** — Never generate new UUIDs for events. LangChain provides `run_id` (UUID) and `parent_run_id` on every callback. Use them directly as `event_id` and `parent_event_id`. +> 2. **SDK must NEVER block the agent pipeline** — All event emission is fire-and-forget via asyncio.Queue + background task. Callbacks must return immediately. Never raise exceptions into the calling agent code. + +> [!IMPORTANT] +> 3. **Server computes graph layout, not the UI** — Dagre layout runs server-side. The UI receives `{ nodes: [{id, position, data}], edges: [] }` and passes it to React Flow. +> 4. **Data flows SDK → Server → UI only** — The UI never writes data to the server except for session deletion (DELETE). +> 5. **Token cost is estimated, not exact** — Use `LLMResult.llm_output.token_usage` when available. Never implement tiktoken or similar. Show '—' when data is unavailable. +> 6. **SQLite with WAL mode** — Always `PRAGMA journal_mode=WAL` for concurrent read/write. +> 7. **No heavy component libraries** — Tailwind CSS + shadcn/ui only. No MUI, Chakra, Ant Design. +> 8. **React Flow, not D3** — For the agent DAG graph. + +## 4. Module-Specific Guidance + +### SDK (Python) +- All callback methods follow the pattern: construct `AgentEvent` → call `self._emit(event)` +- The WebSocket client runs in a background asyncio task +- Events are buffered in an `asyncio.Queue` if the server is unreachable +- Reconnection uses exponential backoff (1s → 2s → 4s → 8s → max 30s) +- Dependencies: `websockets`, `pydantic` (required); `langchain-core` (optional) +- Test with: `pytest`, mock WebSocket connections + +### Server (Python) +- FastAPI with uvicorn, runs on port `8765` +- Two WebSocket pools: SDK (ingest) and UI (broadcast) +- Event flow: SDK sends → validate (Pydantic) → persist (SQLAlchemy) → broadcast (WebSocket) +- Graph layout uses dagre/networkx, computed on demand at `/graph` endpoint +- CORS: allow all origins (local dev tool) +- Test with: `pytest`, `httpx.AsyncClient` for API tests + +### UI (TypeScript/Next.js) +- Next.js 14 with App Router, TypeScript strict mode +- State: Zustand store for sessions + live events +- Data fetching: SWR for REST, custom `useWebSocket` hook for real-time +- Components: `AgentGraph` (React Flow), `EventFeed`, `EventInspector`, `TokenCounter` +- Styling: Tailwind CSS, shadcn/ui primitives +- No default exports for components (use named exports) + +## 5. Common Tasks for AI Agents + +### Adding a new event type +1. Add to `EventType` enum in `packages/sdk/agentscope/models.py` +2. Add Pydantic model in same file +3. Add callback method in `packages/sdk/agentscope/callback.py` +4. Add SQLAlchemy handling in `packages/server/models.py` +5. Add to TypeScript types in `packages/ui/types/index.ts` +6. Add rendering in `packages/ui/components/EventFeed.tsx` + +### Adding a new API endpoint +1. Add route in appropriate router (`packages/server/routers/`) +2. Add Pydantic schema in `packages/server/schemas.py` +3. Add TypeScript type in `packages/ui/types/index.ts` +4. Add API client function in `packages/ui/lib/api.ts` +5. Add SWR hook if needed in `packages/ui/hooks/` + +### Adding a new UI component +1. Create in `packages/ui/components/` +2. Use TypeScript interface for props +3. Use Tailwind for styling +4. Named export only +5. Add tests in colocated `*.test.tsx` file + +## 6. Environment Setup + +```bash +# Server +cd packages/server +pip install -r requirements.txt +uvicorn main:app --host 0.0.0.0 --port 8765 + +# UI +cd packages/ui +npm install +npm run dev + +# SDK (development) +cd packages/sdk +pip install -e . + +# Full stack +docker compose up +``` + +## 7. Key Files to Understand First + +1. [`packages/sdk/agentscope/callback.py`](file:///Users/architmittal/Desktop/CODE/AGENTSCOPE/packages/sdk/agentscope/callback.py) — The core integration point +2. [`packages/sdk/agentscope/client.py`](file:///Users/architmittal/Desktop/CODE/AGENTSCOPE/packages/sdk/agentscope/client.py) — WebSocket client with buffering +3. [`packages/sdk/agentscope/models.py`](file:///Users/architmittal/Desktop/CODE/AGENTSCOPE/packages/sdk/agentscope/models.py) — Canonical data models +4. [`packages/server/main.py`](file:///Users/architmittal/Desktop/CODE/AGENTSCOPE/packages/server/main.py) — Server entry point +5. [`packages/server/ws_manager.py`](file:///Users/architmittal/Desktop/CODE/AGENTSCOPE/packages/server/ws_manager.py) — Dual-pool WebSocket manager +6. [`packages/server/database.py`](file:///Users/architmittal/Desktop/CODE/AGENTSCOPE/packages/server/database.py) — Database setup +7. [`packages/server/models.py`](file:///Users/architmittal/Desktop/CODE/AGENTSCOPE/packages/server/models.py) — ORM models +8. [`packages/ui/store/sessionStore.ts`](file:///Users/architmittal/Desktop/CODE/AGENTSCOPE/packages/ui/store/sessionStore.ts) — Client state management +9. [`packages/ui/hooks/useWebSocket.ts`](file:///Users/architmittal/Desktop/CODE/AGENTSCOPE/packages/ui/hooks/useWebSocket.ts) — Real-time connection +10. [`packages/ui/components/AgentGraph.tsx`](file:///Users/architmittal/Desktop/CODE/AGENTSCOPE/packages/ui/components/AgentGraph.tsx) — DAG visualization + +## 8. Testing Guidance + +- Always write tests for new functionality +- SDK tests should mock WebSocket connections +- Server tests should use in-memory SQLite +- UI tests should mock API calls and WebSocket +- Run: `make test` from project root + +--- + +## 9. Git & Pull Request Workflow (MANDATORY) + +> [!CAUTION] +> **NEVER commit directly to `main`.** Every change — no matter how small — goes through a feature branch and a Pull Request. No exceptions. + +### 9.1 Branch Rules + +1. **Always create a new branch** before starting any work: + ```bash + git checkout main + git pull origin main + git checkout -b / + ``` +2. **Branch naming convention:** + | Type | Pattern | Example | + |---|---|---| + | Feature | `feat/` | `feat/websocket-reconnection` | + | Bug fix | `fix/` | `fix/event-broadcast-race` | + | Docs | `docs/` | `docs/api-contract-update` | + | Refactor | `refactor/` | `refactor/simplify-ws-manager` | + | Tests | `test/` | `test/sdk-callback-coverage` | + | Chores | `chore/` | `chore/update-dependencies` | + +3. **One branch per logical change.** Don't mix unrelated changes in a single branch. + +### 9.2 Pre-PR Verification Checklist (MANDATORY) + +Before creating a Pull Request, **verify ALL of the following**: + +``` +Step 1: Ensure you're on the correct branch (NOT main) + → git branch --show-current + → Must NOT be "main" + +Step 2: Check for uncommitted changes + → git status + → Commit or stash everything + +Step 3: Run linting / formatting + → Python: black . && ruff check . + → TypeScript: npx prettier --check . && npx eslint . + +Step 4: Run tests + → make test (or module-specific: pytest / npm test) + → ALL tests must pass. Do NOT create a PR with failing tests. + +Step 5: Verify the build compiles + → Server: python -c "from main import app" (no import errors) + → UI: npm run build (no TypeScript errors) + +Step 6: Review your own diff + → git diff main..HEAD --stat + → Confirm only intended files are changed + → No debug prints, no commented-out code, no TODO hacks +``` + +> [!WARNING] +> If ANY verification step fails, **fix it before creating the PR**. Do not create a PR that you know is broken. + +### 9.3 Commit Messages + +Use **Conventional Commits** format — every commit message must follow this pattern: + +``` +: + +[optional body with context] +``` + +| Type | When to use | +|---|---| +| `feat` | New feature or functionality | +| `fix` | Bug fix | +| `docs` | Documentation only | +| `refactor` | Code restructuring, no behavior change | +| `test` | Adding or updating tests | +| `chore` | Build config, dependencies, tooling | +| `perf` | Performance improvement | + +Examples: +``` +feat: add real-time event feed WebSocket hook +fix: handle SDK reconnection when server restarts mid-session +docs: add mem0 memory protocol to AGENTS.md +test: add unit tests for session PATCH endpoint +chore: pin fastapi to >=0.110.0,<1.0 +``` + +### 9.4 Creating the Pull Request + +When creating a PR: + +1. **Title** follows Conventional Commits format: `feat: add event inspector panel` +2. **Description** must include: + - **What** was changed (brief summary) + - **Why** it was changed (motivation / context) + - **How** it was tested (commands run, manual verification) +3. **Link to issue** — if this PR addresses a GitHub Issue, connect it: + - Use `Closes #` or `Fixes #` in the PR body + - This auto-closes the issue when the PR is merged + - If it partially addresses an issue, use `Related to #` instead + + ```markdown + ## Summary + Added the EventInspector slide-in panel that shows full event payload + when clicking any event in the feed. + + ## Changes + - Created `EventInspector.tsx` component with syntax-highlighted JSON + - Added Sheet component from shadcn/ui + - Connected click handler in EventFeed to open inspector + + ## Testing + - `npm run build` passes with no errors + - Manually tested with mock event data + - Added `EventInspector.test.tsx` with 4 test cases + + Closes #42 + ``` + +### 9.5 After PR Creation — Summary to User + +After creating a PR, **always give the user a crisp summary** in this format: + +``` +PR # created: + Branch: <branch-name> → main + Issue: Closes #<issue> (or "No linked issue") + Changes: <1-2 sentence summary of what changed> + Files: <count> files changed + Link: <PR URL> +``` + +Example: +``` +PR #23 created: feat: add real-time event feed + Branch: feat/event-feed → main + Issue: Closes #12 + Changes: Added EventFeed component with WebSocket streaming and auto-scroll. Integrated with Zustand store. + Files: 5 files changed + Link: https://github.com/user/agentscope/pull/23 +``` + + +> [!TIP] +> Keep the summary **short and scannable**. The user should know in 5 seconds what the PR does, whether it solves an issue, and where to find it. No walls of text. + +### 9.6 Quick Reference — The Full Git Flow + +``` +1. git checkout main && git pull origin main +2. git checkout -b feat/my-feature +3. ... make changes ... +4. Run verification checklist (section 9.2) +5. git add -A && git commit -m "feat: my feature description" +6. git push -u origin feat/my-feature +7. Create PR → link issue → add description +8. Give user the crisp summary (section 9.5) +``` + +--- + +## 10. Mem0 Memory — Project Continuity + +> [!IMPORTANT] +> This project uses **mem0** (MCP server) to persist project context across agent sessions. Every AI agent working on this codebase **MUST** follow the memory protocol below to maintain continuity. + +### 10.1 Why This Matters + +Without mem0 memory, every new agent session starts from scratch — the agent has to re-read files, re-discover project structure, and re-learn past decisions. Mem0 eliminates this by storing structured project knowledge that agents can retrieve instantly. + +### 10.2 Memory Categories + +The following memory categories are stored under `user_id: "architmittal"` with `metadata.project: "agentscope"`: + +| Category | What It Contains | +|---|---| +| `overview` | Project location, description, license, module structure, data flow direction | +| `architecture` | SDK internals (AsyncCallbackHandler, WebSocket client, asyncio.Queue), Server (FastAPI, dual-pool WS, SQLite WAL, dagre), UI (Next.js 14, React Flow, Zustand, SWR) | +| `design-decisions` | All 7 key decisions (run_id as event_id, server-side dagre, fire-and-forget SDK, no tiktoken, WAL mode, React Flow over D3, Tailwind-only) | +| `status` | Current project phase, what's been completed, what's next | +| `mvp-scope` | All 10 MVP features with descriptions | +| `tech-stack` | Full dependency list + environment variables for all modules | + +### 10.3 Start-of-Session Protocol (MANDATORY) + +When starting work on this project, **before writing any code**, do the following: + +``` +Step 1: Search mem0 for project context +che + → search_memories(query="AgentScope project", user_id="architmittal") + +Step 2: Search for current status + → search_memories(query="AgentScope status current phase", user_id="architmittal") + +Step 3: Search for recent work + → search_memories(query="AgentScope recent changes implementation", user_id="architmittal") +``` + +This gives you: +- Where the project is located on disk +- Full architecture and design decisions +- What's already been built +- What phase the project is in +- What needs to be done next + +> [!TIP] +> If mem0 returns no results for AgentScope, the memories may have been cleared. Fall back to reading this `AGENTS.md` file and the other docs in `docs/` to rebuild context. Then **re-save** the context to mem0 following the end-of-session protocol. + +### 10.4 End-of-Session Protocol (MANDATORY) + +Before ending your session, **always save your progress** to mem0: + +``` +Step 1: Update project status + → add_memory( + text="AgentScope status: [describe what was completed, what's in progress, what's blocked]", + user_id="architmittal", + metadata={"category": "status", "project": "agentscope"} + ) + +Step 2: Save any new decisions or changes + → add_memory( + text="AgentScope: [describe any new architectural decisions, files created, bugs found, etc.]", + user_id="architmittal", + metadata={"category": "changes", "project": "agentscope"} + ) + +Step 3: Save blockers or TODOs (if any) + → add_memory( + text="AgentScope TODO: [describe pending items, known issues, next steps]", + user_id="architmittal", + metadata={"category": "todos", "project": "agentscope"} + ) +``` + +### 10.5 When to Update Memory Mid-Session + +Update mem0 immediately (don't wait for end-of-session) when: + +- **A major milestone is completed** (e.g., "SDK callback.py fully implemented and tested") +- **A critical bug is discovered** (e.g., "WebSocket reconnection fails when server restarts during active session") +- **An architectural decision is made or changed** (e.g., "Switched from networkx to grandalf for Python dagre layout") +- **A new dependency is added** (e.g., "Added `grandalf` to server requirements.txt for graph layout") + +### 10.6 Memory Search Patterns + +Use these specific queries for targeted retrieval: + +| Need | Search Query | +|---|---| +| Full project context | `"AgentScope project"` | +| Architecture details | `"AgentScope architecture SDK Server UI"` | +| Design decisions | `"AgentScope design decisions"` | +| Current status / progress | `"AgentScope status current phase"` | +| MVP features | `"AgentScope MVP features scope"` | +| Tech stack / dependencies | `"AgentScope tech stack dependencies"` | +| Recent changes | `"AgentScope recent changes"` | +| Known bugs / TODOs | `"AgentScope TODO blockers issues"` | + +### 10.7 Memory Hygiene Rules + +> [!WARNING] +> - **Never delete** existing memories unless they are explicitly wrong or outdated +> - **Be specific** in memory text — "implemented SDK callback.py" is better than "did some work" +> - **Always include** `user_id: "architmittal"` and `metadata.project: "agentscope"` for proper scoping +> - **Don't duplicate** — search before adding to check if the info already exists +> - **Update, don't pile up** — if the project status changed, add a new status memory (mem0 handles deduplication) +> - **Include file paths** when relevant — "Created packages/server/main.py with FastAPI app, CORS, and lifespan handler" + +--- + +## 11. Session Continuity Checklist + +Use this checklist at the start and end of every coding session: + +### Starting a Session +- [ ] Search mem0 for `"AgentScope project"` to load context +- [ ] Search mem0 for `"AgentScope status"` to find current progress +- [ ] Read this `AGENTS.md` file if mem0 results are sparse +- [ ] Confirm the workspace path: `/Users/architmittal/Desktop/CODE/AGENTSCOPE` +- [ ] Check what files already exist: `find packages/ -type f | head -50` + +### Ending a Session +- [ ] Save progress summary to mem0 with category `status` +- [ ] Save any new decisions/changes to mem0 with category `changes` +- [ ] Save any pending TODOs to mem0 with category `todos` +- [ ] Ensure all new files are saved to disk +- [ ] Run tests if code was written: `make test` diff --git a/docs/DESIGN.md b/docs/DESIGN.md new file mode 100644 index 0000000..e28562b --- /dev/null +++ b/docs/DESIGN.md @@ -0,0 +1,233 @@ +# AgentScope Technical Design Document + +## 1. System Overview + +AgentScope is a lightweight, open-source, self-hosted observability dashboard for multi-agent AI pipelines. It provides developers with real-time insights into agent execution, LLM calls, tool usage, and overall pipeline structure. + +The system consists of three core modules: +1. **Python SDK**: A lightweight tracing library that instruments agent pipelines (primarily LangChain) and pushes events to the server. +2. **FastAPI Server**: The backend that ingests events, persists them to a local SQLite database, processes data (like graph layout), and serves it to the UI. +3. **Next.js UI**: The frontend dashboard that visualizes agent sessions, execution graphs, real-time event feeds, and aggregated statistics. + +**Data Flow**: Data flows in one direction from the SDK to the Server, and then to the UI. The UI only reads data, with the exception of deleting sessions. + +**Technology Choices & Rationale**: +- **Python (SDK & Server)**: Native language for AI/LLM development (LangChain, etc.). FastAPI provides high performance for API and WebSocket handling. +- **SQLite**: Zero-configuration, local-first database perfect for a self-hosted developer tool. +- **Next.js & React**: Industry standard for building interactive dashboards. +- **WebSockets**: Essential for real-time observability without polling overhead. + +## 2. Architecture Diagram + +```mermaid +graph TD + subgraph "Python SDK (packages/sdk/)" + A[Agent Pipeline] -->|Traces| B[Callback Handler / Decorator] + B -->|Events| C[Async Event Queue] + C -->|JSON| D[WebSocket Client] + end + + subgraph "FastAPI Server (packages/server/)" + D -->|WS Events| E[WebSocket Manager] + E -->|SDK Pool| F[Event Validator] + + F --> G[(SQLite DB)] + F --> H[UI Pool Broadcast] + + I[REST API] --> G + J[Graph Layout Engine] --> G + K[Stats Aggregator] --> G + end + + subgraph "Next.js UI (packages/ui/)" + H -->|Live Events| L[Zustand Store] + I -->|Historical Data| L + J -->|Nodes/Edges| M[AgentGraph UI] + K -->|Aggregations| N[Stats UI] + + L --> M + L --> N + L --> O[EventFeed / Inspector] + end +``` + +## 3. Module Deep Dives + +### 3.1 Python SDK (`packages/sdk/`) + +- **Callback Handler**: Subclasses LangChain's `AsyncCallbackHandler`. Each method (`on_chain_start`, `on_llm_start`, `on_tool_start`, etc.) creates an `AgentEvent` and calls `_emit()`. Uses LangChain's native `run_id` as the `event_id` and `parent_run_id` as the `parent_event_id` to effortlessly reconstruct the execution tree. +- **WebSocket Client**: Runs as a background `asyncio` task. Maintains a persistent connection to the FastAPI server. Uses an `asyncio.Queue` for event buffering to handle bursts. Implements exponential backoff for reconnection (1s, 2s, 4s, 8s, max 30s). All event emissions are fire-and-forget—the SDK never blocks the agent pipeline. +- **`@trace` Decorator**: Used for instrumenting non-LangChain functions. Creates a span by generating a start event when the function is called and an end event upon return. Captures exceptions automatically as error events. +- **Session Management**: Automatically creates a new session via `POST /sessions` on the first event emission. Sends a `PATCH` request to update the session status upon pipeline completion or error. + +### 3.2 Server (`packages/server/`) + +- **FastAPI Application**: Mounts REST routers for sessions and events. Configured with CORS middleware to allow all origins for local development. Uses a lifespan handler for robust database initialization and teardown. +- **WebSocket Manager (`ws_manager.py`)**: Maintains two distinct connection pools: + - *SDK Pool*: Receives incoming events from SDK instances. + - *UI Pool*: Sends real-time event broadcasts to connected browser clients. + Upon receiving an SDK event, it validates the payload, persists it to the database, and broadcasts it to UI clients subscribed to that specific `session_id`. +- **Database Layer**: Built with SQLAlchemy 2.0 and an async engine. Uses SQLite in WAL (Write-Ahead Logging) mode to support concurrent reads while writing. Key tables: `sessions`, `events`. Indexes are placed on `session_id`, `timestamp`, and `event_type` for query performance. +- **Graph Layout (`graph_layout.py`)**: Responsible for computing the DAG layout server-side using the `dagre` algorithm (or a Python equivalent like `networkx`). It reads session events, builds an adjacency graph using `parent_event_id`, computes X/Y coordinates for each node, and returns a format directly compatible with React Flow nodes and edges. This ensures deterministic layouts across UI clients. +- **Stats Aggregation**: Queries the `events` table to calculate token sums, latency averages, and error counts, grouped by agent or step. Computes token usage timelines for charting. + +### 3.3 UI (`packages/ui/`) + +- **State Management**: Uses a Zustand store to hold the session list, the active session details, and a live events map. A custom WebSocket hook listens for incoming broadcasts and dispatches updates to the store. +- **Data Fetching**: Utilizes SWR for interacting with REST endpoints. Configured to auto-revalidate running sessions every 2 seconds. Merges historical data fetched via REST with live events streaming via WebSockets. +- **Pages**: + - `/` — Session list view with a data table, status badges (Running, Completed, Error), and total cost/token information. + - `/sessions/[id]` — Detailed session layout with a persistent header showing name, status, and token counter. Includes tab navigation for sub-views. + - `/sessions/[id]/graph` — Full-width canvas rendering the React Flow DAG. + - `/sessions/[id]/events` — Filterable, chronological event feed with a slide-in inspector panel. + - `/sessions/[id]/stats` — Dashboards powered by Recharts (token usage bar charts, latency timelines). +- **Key Components**: + - `AgentGraph.tsx` — React Flow implementation with custom node types (`ChainNode`, `LLMNode`, `ToolNode`, `RetrieverNode`). + - `EventFeed.tsx` — Real-time scrolling list. Includes an auto-scroll mechanism with a pause toggle. + - `EventInspector.tsx` — Slide-in panel displaying the full raw event JSON and syntax-highlighted prompt/response texts. + - `TokenCounter.tsx` — Sticky header component that updates live as events arrive. + +## 4. Key Design Decisions + +1. **Use LangChain's `run_id` as `event_id`** + - *What*: We map LangChain's UUIDs directly to our event IDs. + - *Why*: Avoids building an ID mapping layer and makes tree reconstruction trivial since LangChain natively tracks `parent_run_id`. + - *Alternatives*: Generating our own UUIDs, which would require maintaining state maps during SDK execution. + +2. **Server-side graph layout (dagre)** + - *What*: X/Y positions for DAG nodes are computed on the FastAPI server before being sent to the UI. + - *Why*: Ensures deterministic layouts regardless of the client, keeps the UI stateless regarding layout math, and avoids browser CPU spikes on large graphs. + - *Alternatives*: Client-side layout (e.g., dagre.js in the browser), which can cause layout shifts and UI thread blocking. + +3. **SDK never blocks agent pipeline** + - *What*: SDK uses an `asyncio.Queue` and background task for fire-and-forget event emission. + - *Why*: Observability should never impact the performance or stability of the primary application. + - *Alternatives*: Synchronous HTTP calls, which would drastically increase LLM pipeline latency. + +4. **Token cost estimated, not exact** + - *What*: We rely on LLM response metadata (e.g., OpenAI's `usage` object) to track tokens. If absent, we display '—' rather than trying to calculate it exactly. + - *Why*: Calculating exact tokens requires heavy dependencies like `tiktoken` in the SDK, which bloats the package and requires model-specific logic. + - *Alternatives*: Bundling tokenizers in the SDK or server. + +5. **SQLite with WAL mode** + - *What*: Using local SQLite configured with `PRAGMA journal_mode=WAL;`. + - *Why*: Perfect for a zero-config local developer tool. WAL mode allows the UI to run heavy read queries while the SDK is simultaneously streaming hundreds of write events. + - *Alternatives*: PostgreSQL (requires user setup via Docker) or in-memory DB (loses history across restarts). + +6. **React Flow over D3** + - *What*: Using the React Flow library for the DAG visualization. + - *Why*: Purpose-built for rendering node-edge graphs in React. It handles zooming, panning, and selection out of the box, saving weeks of development time. + - *Alternatives*: D3.js (powerful but requires manual DOM manipulation in React) or specialized DAG libraries (less customizable). + +7. **Tailwind + shadcn/ui only** + - *What*: Building UI components using Tailwind CSS and Radix primitives (shadcn/ui). + - *Why*: Keeps the bundle small and avoids the visual baggage and performance overhead of heavy component libraries. + - *Alternatives*: Material UI (MUI) or Chakra UI. + +8. **WebSocket dual pool** + - *What*: The Server maintains separate logic/pools for SDK publishers and UI subscribers. + - *Why*: Clean separation of concerns. SDKs only push data, UIs only pull data. Prevents accidental broadcasting of events back to an SDK. + - *Alternatives*: A single generic PubSub pool. + +## 5. Sequence Diagrams + +### Session Lifecycle +```mermaid +sequenceDiagram + participant SDK as Python SDK + participant API as FastAPI Server + participant DB as SQLite + participant UI as Next.js UI + + SDK->>API: POST /sessions (Create Session) + API->>DB: INSERT session + API-->>SDK: Session ID returned + + rect rgb(240, 248, 255) + note right of SDK: Pipeline Execution + SDK->>API: WS: Stream Events + API->>DB: INSERT events + end + + SDK->>API: PATCH /sessions/{id} (Status: Completed) + API->>DB: UPDATE session status + + UI->>API: GET /sessions + API->>DB: SELECT sessions + DB-->>API: rows + API-->>UI: JSON list +``` + +### Real-time Event Flow +```mermaid +sequenceDiagram + participant Agent as LangChain Agent + participant SDK as Python SDK (Queue) + participant WS_MGR as WS Manager (Server) + participant DB as SQLite + participant UI as Next.js UI + + UI->>WS_MGR: Connect & Subscribe (session_id) + Agent->>SDK: on_llm_start() + Note over SDK: Event added to asyncio.Queue + SDK->>WS_MGR: Send Event JSON via WS + WS_MGR->>WS_MGR: Validate Payload + WS_MGR->>DB: Persist Event + WS_MGR->>UI: Broadcast Event JSON + UI->>UI: Update Zustand State + UI->>UI: Render new event +``` + +### Graph Rendering +```mermaid +sequenceDiagram + participant UI as Next.js UI + participant API as FastAPI Server + participant Layout as Graph Layout Engine + participant DB as SQLite + + UI->>API: GET /sessions/{id}/graph + API->>DB: SELECT all events for session + DB-->>API: Event records + API->>Layout: Pass events + Note over Layout: Build Adjacency List<br>Run Dagre Algorithm<br>Compute X,Y coordinates + Layout-->>API: ReactFlow Nodes & Edges + API-->>UI: JSON payload + UI->>UI: Render React Flow canvas +``` + +## 6. Error Handling Strategy + +- **SDK**: All callbacks are wrapped in broad `try/except` blocks. Errors are caught, logged as warnings (`logging.warning`), and never propagated back to the underlying agent pipeline. The agent must run even if observability fails. +- **Server**: Standard HTTP error responses using appropriate status codes (400 for bad payloads, 404 for missing entities, 422 for validation errors, 500 for server errors) containing detailed error body descriptions. +- **WebSocket**: Client implements exponential backoff reconnection logic. During disconnection periods, events are buffered in memory up to a configurable limit before being dropped. +- **UI**: Utilizes React Error Boundaries to prevent full app crashes. Displays toast notifications for transient API failures and renders friendly empty/error states for major data loading failures. + +## 7. Performance Considerations + +- **SDK**: The combination of `asyncio.Queue` and a background drain task ensures that the tracing callback returns in < 1ms, preventing any slowdown to LLM generation. +- **Server**: SQLite WAL mode allows concurrent read/write operations. Indexes on hot columns (`session_id`, `timestamp`) keep queries fast. Pagination is implemented on list endpoints to handle thousands of past sessions. +- **UI**: The event feed uses a virtualized list to efficiently render thousands of events without DOM bloat. React Flow is natively optimized for handling large graphs. SWR caching prevents redundant API calls on navigation. + +## 8. Security Model + +- **Scope**: Designed as a local-only developer tool. There is no authentication or authorization in v1. +- **CORS**: Configured to allow all origins to facilitate painless local development across different localhost ports. +- **Data Storage**: SQLite file is stored locally with standard user-level permissions. +- **Network**: The server makes no external network calls; everything runs locally. +- **Privacy Warning**: Traced events will contain sensitive information, including full LLM prompts, tool inputs, and potentially API keys if passed in prompts. Users are responsible for securing their local environment and database file. + +## 9. Deployment Architecture + +- **Docker Compose**: The recommended deployment method groups the Server (FastAPI running on Uvicorn) and the UI (Next.js standalone build) as two services. +- **Volumes**: The SQLite database (`agentscope.db`) is mounted to a host filesystem volume for persistence across container restarts. +- **Ports**: Exposes port `8765` for the FastAPI server and `3000` for the Next.js UI. +- **Dependencies**: Fully self-contained. No external services (no Redis, no Postgres, no cloud accounts) are required to run the stack. + +## 10. Future Architecture Considerations + +- **PostgreSQL Adapter**: Support swapping SQLite for PostgreSQL via a `DATABASE_URL` environment variable for team deployments. +- **Multi-framework Support**: Adapters for CrewAI, AutoGen, and raw OpenAI SDKs, expanding beyond LangChain. +- **Live DAG Updates**: Pushing incremental node/edge updates via WebSockets rather than requiring REST refreshes for the graph view. +- **Session Diffing**: Infrastructure to compare two runs side-by-side (e.g., comparing token usage or latency before and after a prompt change). +- **Plugin System**: Allow users to register custom event types and provide custom UI components to render them in the inspector. diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md new file mode 100644 index 0000000..51bd87b --- /dev/null +++ b/docs/DEVELOPMENT.md @@ -0,0 +1,141 @@ +# AgentScope Developer Setup & Contribution Guide + +Welcome to the **AgentScope** development repository. This document outlines step-by-step instructions for configuring your local environment, executing test suites, running example pipelines, and contributing code changes. + +--- + +## 1. Project Architecture Overview + +AgentScope is structured as a monorepo containing three core packages: + +* **`packages/sdk/`**: Python-based telemetry and monitoring SDK. Contains the `@trace` decorator and LangChain tracking callbacks. +* **`packages/server/`**: FastAPI-based observability server, persisting sessions and telemetry log events to SQLite. +* **`packages/ui/`**: Next.js (Tailwind CSS, Zustand, React Flow) developer dashboard displaying live telemetry streams and session analytics. + +--- + +## 2. Local Environment Setup + +### Prerequisites +* **Python**: Version `3.10` or higher. +* **Node.js**: Version `18` or higher (with `npm`). +* **Git**: For version control. + +### Python Backend & SDK Setup +1. Create and activate a virtual environment in the repository root: + ```bash + python -m venv venv + source venv/bin/activate # On Windows, use: venv\Scripts\activate + ``` +2. Install dependencies for the SDK: + ```bash + cd packages/sdk + pip install -e . + cd ../.. + ``` +3. Install dependencies for the Server: + ```bash + cd packages/server + pip install -r requirements.txt -r requirements-dev.txt + cd ../.. + ``` + +### Node.js UI Dashboard Setup +1. Navigate to the UI directory: + ```bash + cd packages/ui + ``` +2. Install frontend dependencies: + ```bash + npm install + ``` +3. Build the dashboard to verify compilation: + ```bash + npm run build + ``` +4. Navigate back to root: + ```bash + cd ../.. + ``` + +--- + +## 3. Running Services Locally + +To verify end-to-end telemetry flows, boot up the observability server followed by the UI client. + +### Step 1: Start the Observability Server +Launch Uvicorn in the server folder (exposing port `8765` by default): +```bash +cd packages/server +uvicorn main:app --reload --port 8765 +``` + +### Step 2: Start the UI Dev Server +Launch the Next.js development server (runs on `http://localhost:3000` by default): +```bash +cd packages/ui +npm run dev +``` + +--- + +## 4. Running the Local Test Suite + +We use `pytest` to validate Python modifications and maintain code quality. We provide helper commands in the root `Makefile` to simplify validation. + +* **Run all tests (SDK & Server)**: + ```bash + make test + ``` +* **Run SDK tests only**: + ```bash + cd packages/sdk + pytest + ``` +* **Run Server tests only**: + ```bash + cd packages/server + pytest + ``` + +--- + +## 5. Telemetry Pipeline Verification + +To verify that your setup intercepts agent executions and streams them to the local server, execute a simple Python telemetry script: + +```python +import time +import asyncio +import agentscope.decorators as as_deco + +# 1. Configure telemetry client to send records to the server +as_deco.configure( + host="127.0.0.1", + port=8765, + session_name="Manual Verification Session" +) + +# 2. Instrument functions using the @trace decorator +@as_deco.trace(name="MathChain", agent_type="chain") +def run_math_computation(base_val: int) -> int: + time.sleep(0.5) + return base_val * 20 + +@as_deco.trace(name="GreetingAgent", agent_type="agent") +async def async_greeting(user_name: str) -> str: + await asyncio.sleep(0.3) + return f"Hello, {user_name}!" + +# 3. Execute instrumented methods +if __name__ == "__main__": + print("Launching instrumented pipeline...") + res_sync = run_math_computation(5) + print(f"Sync Result: {res_sync}") + + res_async = asyncio.run(async_greeting("Developer")) + print(f"Async Result: {res_async}") + print("Verification execution complete. Open http://localhost:3000 to view session events.") +``` +Ensure the server and UI are running, execute this script, and confirm that the session name and logged events appear on the dashboard in real-time. diff --git a/docs/PRD.md b/docs/PRD.md new file mode 100644 index 0000000..073a7c7 --- /dev/null +++ b/docs/PRD.md @@ -0,0 +1,145 @@ +# Product Requirements Document (PRD) - AgentScope + +## 1. Executive Summary + +**AgentScope** is a lightweight, open-source, self-hosted observability dashboard for multi-agent AI pipelines. + +Debugging multi-agent AI pipelines is notoriously difficult. Developers currently lack visibility into which agent called which tool, the exact prompt and completion text, total token costs, and the overall call hierarchy of complex executions. AgentScope solves this by providing a lightweight, local observability dashboard. With just two lines of Python code, developers can instrument their LangChain applications and immediately visualize real-time event feeds, agent execution graphs, and token usage metrics—all self-hosted with no accounts or cloud dependencies required. + +**Target Audience:** AI/ML developers building and debugging multi-agent pipelines. + +## 2. Goals & Objectives + +### Primary Goals +- Give developers instant visibility into multi-agent pipeline execution. +- Provide clear, real-time insights into prompts, completions, and tool executions. +- Visualize agent call hierarchies to understand complex chains. + +### Secondary Goals +- Persist session history for retrospective debugging and analysis. +- Provide token usage and cost estimations per session. + +### Non-Goals +- Cloud deployment or SaaS offering. +- Team-based features or collaboration tools. +- User authentication or authorization. +- Production monitoring at high scale (focused on dev/staging). + +## 3. User Personas + +- **Solo AI Developer:** Building rapid prototypes with LangChain, needs fast iteration and instant feedback on why an agent failed or hallucinates. +- **ML Engineer:** Debugging complex, multi-layered multi-agent chains in dev/staging environments, requiring deep dives into intermediate steps and token consumption. +- **Open Source Contributor:** Looking to extend observability integrations beyond LangChain (e.g., CrewAI, AutoGen) and contribute to the dashboard ecosystem. + +## 4. User Stories + +| ID | As a... | I want to... | So that I can... | +|---|---|---|---| +| 1 | Developer | Add 2 lines of code to my LangChain setup to see events in a dashboard | Instrument my pipeline with minimal friction | +| 2 | Developer | See the exact prompt and completion text for each LLM call | Debug unexpected outputs or hallucinations | +| 3 | Developer | View token usage and estimated cost per session | Optimize my pipeline's resource consumption | +| 4 | Developer | See a DAG visualization of my agent call hierarchy | Understand complex multi-agent interactions | +| 5 | Developer | Filter events by agent name and event type | Quickly find relevant information in large sessions | +| 6 | Developer | Have sessions persist in a local database | Debug past runs without having to re-execute them | +| 7 | Developer | Watch real-time event streaming as my pipeline runs | Monitor execution progress and identify hangs | +| 8 | Developer | Click on any event and see its full payload | Inspect raw data, tool inputs, and intermediate states | +| 9 | Developer | Run everything locally with one command (docker-compose) | Setup the environment quickly without manual configuration | +| 10 | Developer | See error events clearly highlighted in the dashboard | Find and fix pipeline failures fast | + +## 5. Functional Requirements + +### 5.1 Python SDK (`packages/sdk/`) +- **Integration:** Hook into LangChain via custom `BaseCallbackHandler`. +- **Callbacks Implemented:** + - `on_llm_start`, `on_llm_end`, `on_llm_error` + - `on_chain_start`, `on_chain_end`, `on_chain_error` + - `on_tool_start`, `on_tool_end`, `on_tool_error` + - `on_agent_action`, `on_agent_finish` +- **Communication:** Establish a WebSocket connection to the Server to stream events asynchronously. +- **Resilience:** Buffer events in-memory if the WebSocket connection drops, and attempt reconnection with exponential backoff. + +### 5.2 Server (`packages/server/`) +- **Framework:** FastAPI application. +- **Database:** SQLite for persistent storage of sessions and events. +- **Endpoints:** + - `GET /api/sessions`: List all historical sessions. + - `GET /api/sessions/{id}`: Retrieve full details and events for a specific session. + - `DELETE /api/sessions/{id}`: Delete a session. + - `GET /api/stats`: Retrieve aggregate statistics (total tokens, total runs, etc.). +- **WebSocket Route:** + - `WS /ws/events`: Receive events from the Python SDK, persist them to SQLite, and immediately broadcast them to connected UI clients. + +### 5.3 UI Dashboard (`packages/ui/`) +- **Framework:** Next.js 14 (App Router) + React. +- **Components:** + - **Session List Sidebar:** Displays past and current sessions, sortable by date. + - **Real-time Event Feed:** A scrolling list of events, auto-updating via WebSocket. Highlights errors in red. + - **Event Inspector (Drawer/Modal):** Displays the full JSON payload of a selected event (prompts, responses, tool inputs). + - **Agent DAG Graph:** Uses React Flow to visualize the execution hierarchy (chains -> agents -> tools -> LLMs). + - **Metrics Header:** Displays session duration, token count, and estimated cost. + - **Filters:** Dropdowns to filter the event feed by Event Type or Agent Name. + +## 6. Non-Functional Requirements + +- **Performance:** The Python SDK must never block the main agent pipeline execution (fire-and-forget asynchronous networking). +- **Latency:** Events emitted by the SDK should appear in the UI dashboard within 100ms. +- **Storage:** Use SQLite configured in WAL (Write-Ahead Logging) mode to support concurrent reads (UI) and writes (Server receiving events). +- **Reliability:** The SDK must buffer events if the server is temporarily unreachable, preventing data loss during local network hiccups. +- **Security:** Local-only deployment. The server binds to `localhost` or `127.0.0.1` by default. No authentication required for the initial V1 release. + +## 7. MVP Scope + +The Minimum Viable Product (MVP) consists of the following 10 features: + +1. **LangChain Integration:** Python SDK with `BaseCallbackHandler`. +2. **Real-time Event Feed:** WebSocket streaming to UI. +3. **Token Counter:** Tracking prompt and completion tokens. +4. **Session Persistence:** Storing execution history in SQLite. +5. **Agent DAG Graph:** Basic visualization using React Flow. +6. **Event Inspector:** Viewing raw event payloads in the UI. +7. **Session List:** Browsing historical runs. +8. **Basic Stats:** Aggregate metrics dashboard. +9. **Docker Compose:** One-click local deployment. +10. **CLI Launcher:** Alternative `agentscope start` command to spin up services. + +## 8. Post-MVP / Out of Scope + +The following features are explicitly deferred to post-MVP updates: +- Integrations for CrewAI, AutoGen, or LlamaIndex. +- Live, dynamic DAG updates (DAG will initially render statically per session state). +- PostgreSQL or other external database support. +- Authentication, RBAC, or team collaboration modes. +- Cloud deployment configurations. +- Cost budget alerts or rate-limiting. +- Prompt diff view across iterations. +- Exporting session data to JSON/CSV. + +## 9. Success Metrics + +- **Ease of Use:** Instrumentation requires exactly 2 lines of code (`from agentscope import AgentScopeCallback; callbacks=[AgentScopeCallback()]`). +- **Responsiveness:** < 1 second latency from event emission to UI rendering under normal load. +- **Time to Value:** < 5 minutes from `git clone` to viewing the first event in the running dashboard. +- **Performance Impact:** Zero measurable impact (blocking) on the primary agent pipeline execution time. + +## 10. Technical Constraints + +- **Language:** Python 3.10+ required for the SDK and Server. +- **Frontend UI:** Next.js 14 utilizing the App Router and strict TypeScript configuration. +- **Database:** SQLite (WAL mode) as the default and only supported storage engine. +- **Transport:** WebSocket protocol for all real-time communication. +- **Dependencies:** Minimal third-party dependencies to ensure easy installation and reduced conflict surface. + +## 11. Risks & Mitigations + +| Risk | Impact | Mitigation Strategy | +|---|---|---| +| **LangChain API changes** | SDK callbacks break | Pin the integration to `langchain-core` and utilize the most stable callback interfaces. | +| **WebSocket reliability** | Dropped events | Implement an in-memory buffer and robust reconnection logic with exponential backoff in the SDK. | +| **Large sessions overwhelming SQLite** | UI/Server sluggishness | Implement pagination for event retrieval and a `MAX_SESSIONS` pruning mechanism to delete old data. | + +## 12. Suggested Timeline + +- **Week 1-2:** Develop Python SDK (LangChain callbacks) + FastAPI Server core (SQLite setup, WebSocket ingestion). +- **Week 3-4:** Build Next.js 14 UI dashboard (Event Feed, React Flow DAG, Event Inspector). +- **Week 5:** Integration testing, robust error handling, example projects, and Docker Compose setup. +- **Week 6:** Finalize documentation (README, setup guides), overall polish, and V1 release. diff --git a/docs/RULES.md b/docs/RULES.md new file mode 100644 index 0000000..c3ec3f0 --- /dev/null +++ b/docs/RULES.md @@ -0,0 +1,155 @@ +# AgentScope Project Rules & Conventions + +This document outlines the core principles, coding standards, architectural guidelines, and processes for the AgentScope project. Adhering to these rules ensures consistency, high code quality, and maintainability across the Python SDK, FastAPI Server, and Next.js UI modules. + +## 1. Project Philosophy + +- **Developer experience first**: AgentScope should require only 2 lines to instrument and offer zero configuration to get started. +- **Local-first**: No cloud dependencies and no accounts required. Everything runs locally. +- **Minimal dependencies**: Each module should have the fewest possible dependencies to reduce bloat and compatibility issues. +- **Non-blocking SDK**: The SDK must **NEVER** block the agent pipeline. +- **Unidirectional Data Flow**: Data flows strictly in one direction: SDK → Server → UI. + +## 2. Code Style & Formatting + +### Python (SDK + Server) + +- **Version**: Python 3.10+ required. +- **Formatter**: `black` with default settings (line length 88). +- **Linter**: `ruff` with pyflakes + pycodestyle + isort enabled. +- **Type hints**: Required on all public functions, optional (but encouraged) on internal functions. +- **Docstrings**: Google style is required on all public classes and functions. +- **Imports**: Formatted using `isort` with a black-compatible profile. Order: stdlib → third-party → local. +- **Naming Conventions**: + - Functions/Variables: `snake_case` + - Classes: `PascalCase` + - Constants: `UPPER_SNAKE` +- **Standard Library usage**: + - Prefer `pathlib.Path` over `os.path`. + - Use f-strings over `.format()` or `%` formatting. +- **Frameworks**: + - Use Pydantic v2 models for all data validation. + - Use SQLAlchemy 2.0 style (avoid legacy 1.x patterns). +- **Control Flow & Error Handling**: + - Prefer async functions for I/O-bound operations. + - No bare `except:` — always catch specific exceptions. + - Use the built-in `logging` module for all output; never use `print()`. + +### TypeScript (UI) + +- **Language**: TypeScript strict mode is required. +- **Formatter**: Prettier (default configuration). +- **Linter**: ESLint with Next.js configuration. +- **Component Pattern**: Functional components with hooks only. No class components. +- **Types/Props**: Define props as TypeScript interfaces (no inline types). +- **Naming Conventions**: + - Variables/Functions: `camelCase` + - Components/Types: `PascalCase` + - Constants: `UPPER_SNAKE` +- **File Naming**: + - Components: PascalCase (e.g., `AgentGraph.tsx`) + - Hooks: camelCase (e.g., `useWebSocket.ts`) + - Utilities: camelCase (e.g., `api.ts`) +- **Declarations & Exports**: + - Use `const` by default. Use `let` only when reassignment is needed. Never use `var`. + - Prefer named exports over default exports. +- **Styling**: Tailwind utility classes only. Use `shadcn/ui` for primitives. **No** MUI, Chakra, or Ant Design. + +## 3. Architecture Rules + +- **Data Flow**: SDK → Server → UI is the ONLY allowed data flow direction. The UI never writes data to the server (with the exception of DELETE operations for session cleanup). +- **Identifiers**: Use LangChain's `run_id` directly as `event_id`. Never generate new UUIDs for events. +- **Layout Computation**: The server computes the graph layout using `dagre`, not the UI. +- **SDK Telemetry**: All WebSocket transmissions in the SDK must be fire-and-forget, running in a background task or thread. +- **Token Costs**: Estimate token cost from LLM response metadata *only* when available; do not estimate when unavailable (display '—'). +- **Database**: Use SQLite with WAL mode (`PRAGMA journal_mode=WAL`). +- **Graph UI**: Use React Flow for graph visualization, not D3. + +## 4. Git Conventions + +- **Branch Naming**: + - `feature/short-description` + - `fix/short-description` + - `docs/short-description` +- **Commit Messages**: Follow the [Conventional Commits](https://www.conventionalcommits.org/) format. + - Examples: + - `feat: add real-time event feed component` + - `fix: handle WebSocket reconnection on network drop` + - `docs: add integration guide for LangChain` + - `chore: update dependencies` + - `refactor: simplify event broadcasting logic` + - `test: add unit tests for session router` +- **Pull Requests**: PR titles must follow the same commit message convention. +- **Merging**: Use Squash merge to main. +- **Main Branch**: The `main` branch must remain in a consistently deployable state. + +## 5. Testing Rules + +### Python +- **Framework**: `pytest` +- **Coverage**: Minimum 80% for SDK, 70% for Server. +- **Structure**: Place tests in a `tests/` directory within each package. +- **Naming**: `test_<function_name>_<scenario>` +- **Mocking**: Mock all external dependencies (e.g., WebSockets, database). +- **Async**: Use `pytest-asyncio` for async testing. +- **Fixtures**: Define fixtures in `conftest.py`. + +### TypeScript +- **Framework**: Jest + React Testing Library. +- **Scope**: Test component rendering and user interactions. +- **Mocking**: Mock WebSocket connections and API calls. +- **Structure**: Test files must be colocated with the source code (e.g., `*.test.tsx` or `*.test.ts`). + +## 6. Dependency Rules + +| Module | Allowed/Expected Dependencies | +| ------ | ----------------------------- | +| **SDK** | `websockets`, `pydantic`. `langchain-core` as an optional extra. | +| **Server**| `fastapi`, `uvicorn`, `sqlalchemy`, `pydantic`, `websockets`, `python-dotenv` | +| **UI** | `next`, `react`, `reactflow`, `recharts`, `zustand`, `swr`, `tailwindcss` | + +- **General Rule**: No dependency should be added without strong justification. +- **Versioning**: Pin major versions but allow minor/patch updates. +- **Maintenance**: Review and update dependencies on a monthly basis. + +## 7. Error Handling Rules + +- **SDK**: NEVER raise exceptions into the agent pipeline. Catch all errors, log a warning, and continue execution. +- **Server**: Return appropriate HTTP status codes (e.g., 400, 404, 422, 500) complete with descriptive error bodies. +- **UI**: Always display user-friendly error states (never blank screens). Use toast notifications for transient errors. +- **Logging**: All errors across the stack must be logged with sufficient context (e.g., `session_id`, `event_id`). + +## 8. Security Rules (v1) + +- **Authentication**: None required (this is a local dev tool). +- **CORS**: Allow all origins (for local development only). +- **Data Privacy**: No PII storage by design. Note that events contain LLM prompts/completions which may inherently contain user data. +- **File System**: SQLite file permissions should be restricted to user-only read/write. +- **Network**: No external network calls from the server or UI (except to localhost). + +## 9. Documentation Rules + +- **Code Comments**: All public APIs must have docstrings (Python) or JSDoc (TypeScript). +- **Readmes**: + - The root `README.md` must include a 5-minute quickstart guide. + - Each module (`sdk`, `server`, `ui`) must have its own detailed `README.md`. +- **Changelog**: `CHANGELOG.md` must be updated with every release. +- **Examples**: The `examples/` directory must contain code that is runnable with minimal setup. + +## 10. Performance Rules + +- **SDK Telemetry**: SDK callback methods must return in **< 1ms** (employ async fire-and-forget). +- **Server Throughput**: Server event ingestion must be capable of handling **1000 events/second**. +- **UI Rendering**: The UI must be able to render up to **10,000 events per session** without frame drops or jank. +- **Graph Layout**: Graph layout computation must complete in **< 2 seconds for 500 nodes**. +- **API Latency**: REST API responses must complete in **< 200ms** for typical queries. + +## 11. Release Process + +- **Versioning Strategy**: Use Semantic Versioning (`MAJOR.MINOR.PATCH`). + - Use `v0.x.x` during initial development. +- **Releases**: Tag releases on GitHub. +- **Distribution**: + - Publish the SDK to PyPI. + - Docker images should be tagged with the exact version. +- **Changelog**: `CHANGELOG.md` must be updated prior to every release. diff --git a/docs/SCHEMA.md b/docs/SCHEMA.md new file mode 100644 index 0000000..df816d6 --- /dev/null +++ b/docs/SCHEMA.md @@ -0,0 +1,399 @@ +# AgentScope Data Schema Reference + +> [!NOTE] +> This document defines all data models, database schemas, API request/response shapes, and WebSocket message formats used across the three AgentScope modules (SDK, Server, UI). It serves as the canonical reference for ALL data structures in the system. + +## 1. Overview + +AgentScope relies on a consistent set of core data models across its stack: +- **TypeScript interfaces** for the frontend UI and SDK. +- **Pydantic models** for server-side validation and API contracts. +- **SQLAlchemy ORM models** for data persistence. + +The system communicates via REST APIs for static data loading and WebSockets for real-time event streaming and ingestion. + +## 2. Core Data Models + +### 2.1 Session + +A Session represents a single execution run of a multi-agent pipeline. + +```typescript +interface Session { + session_id: string // UUID v4 + name: string // human-readable, e.g. "research_pipeline_run_3" + status: "running" | "completed" | "failed" + started_at: string // ISO 8601 + ended_at: string | null + total_tokens: number + total_cost_usd: number + error_count: number + agent_count: number // distinct agent names seen + metadata: Record<string, any> // arbitrary user-supplied tags +} +``` + +### 2.2 AgentEvent + +An AgentEvent represents a single observable action, state change, or output from any component within the agent pipeline. + +```typescript +interface AgentEvent { + event_id: string // maps directly to LangChain's run_id (UUID) + session_id: string + parent_event_id: string | null // maps to LangChain's parent_run_id + event_type: EventType + agent_name: string + agent_type: "chain" | "llm" | "tool" | "retriever" | "agent" | "custom" + timestamp: string // ISO 8601 + latency_ms: number | null // only on *_end events + status: "running" | "completed" | "error" + payload: LLMPayload | ToolPayload | ChainPayload | RetrieverPayload +} + +type EventType = + | "chain_start" | "chain_end" | "chain_error" + | "llm_start" | "llm_end" | "llm_token" | "llm_error" + | "tool_start" | "tool_end" | "tool_error" + | "agent_action" | "agent_finish" + | "retriever_start" | "retriever_end" +``` + +### 2.3 Payload Types + +```typescript +interface LLMPayload { + model: string + prompts: string[] + completion: string | null + prompt_tokens: number | null + completion_tokens: number | null + total_tokens: number | null + temperature: number | null + streaming: boolean +} + +interface ToolPayload { + tool_name: string + tool_description: string | null + input: string + output: string | null + error: string | null +} + +interface ChainPayload { + chain_type: string + inputs: Record<string, any> + outputs: Record<string, any> | null + error: string | null +} + +interface RetrieverPayload { + query: string + documents: Array<{ content: string; metadata: Record<string, any> }> | null +} +``` + +## 3. Database Schema (SQLite / SQLAlchemy) + +The server utilizes SQLite for local persistent storage. + +> [!TIP] +> SQLite is configured with `PRAGMA journal_mode=WAL;` and `PRAGMA foreign_keys=ON;` for better concurrency and integrity. + +```sql +-- Enable WAL mode +PRAGMA journal_mode=WAL; +PRAGMA foreign_keys=ON; + +CREATE TABLE sessions ( + session_id TEXT PRIMARY KEY, + name TEXT NOT NULL, + status TEXT NOT NULL, + started_at DATETIME NOT NULL, + ended_at DATETIME, + total_tokens INTEGER DEFAULT 0, + total_cost_usd REAL DEFAULT 0.0, + error_count INTEGER DEFAULT 0, + agent_count INTEGER DEFAULT 0, + metadata JSON +); + +CREATE TABLE events ( + event_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL, + parent_event_id TEXT, + event_type TEXT NOT NULL, + agent_name TEXT NOT NULL, + agent_type TEXT NOT NULL, + timestamp DATETIME NOT NULL, + latency_ms INTEGER, + status TEXT NOT NULL, + payload JSON, + FOREIGN KEY (session_id) REFERENCES sessions (session_id) ON DELETE CASCADE, + FOREIGN KEY (parent_event_id) REFERENCES events (event_id) ON DELETE SET NULL +); + +-- Indexes for performance +CREATE INDEX idx_events_session_id ON events(session_id); +CREATE INDEX idx_events_parent_event_id ON events(parent_event_id); +``` + +## 4. API Request/Response Schemas + +| Method | Path | Request | Response | +|---|---|---|---| +| POST | `/sessions` | `{ "name": "...", "metadata": {} }` | `Session` | +| GET | `/sessions` | `?page=1&limit=10` | `{ "sessions": Session[], "total": 100, "page": 1 }` | +| GET | `/sessions/{id}` | — | `Session` | +| PATCH | `/sessions/{id}` | `{ "status": "completed", "ended_at": "..." }` | `Session` | +| DELETE | `/sessions/{id}` | — | `{ "deleted": true }` | +| GET | `/sessions/{id}/events` | `?type=llm_start&agent=Agent1&limit=50` | `{ "events": AgentEvent[] }` | +| GET | `/sessions/{id}/events/{event_id}`| — | `AgentEvent` | +| GET | `/sessions/{id}/graph` | — | `GraphResponse` | +| GET | `/sessions/{id}/stats` | — | `StatsResponse` | +| GET | `/health` | — | `{ "status": "ok", "version": "1.0.0" }` | + +### Derived API Response Types + +```typescript +interface GraphResponse { + nodes: ReactFlowNode[] + edges: ReactFlowEdge[] +} + +interface ReactFlowNode { + id: string + type: "ChainNode" | "LLMNode" | "ToolNode" | "RetrieverNode" + position: { x: number; y: number } + data: { + label: string + agentName: string + eventType: string + durationMs: number | null + tokenCount: number | null + status: "completed" | "error" | "running" + } +} + +interface ReactFlowEdge { + id: string + source: string + target: string + type: "default" | "error" +} + +interface StatsResponse { + total_tokens: number + total_cost_usd: number + total_duration_ms: number + event_count: number + error_count: number + agents: Array<{ + name: string + type: string + call_count: number + total_tokens: number + avg_latency_ms: number + error_count: number + }> + token_timeline: Array<{ timestamp: string; cumulative_tokens: number }> +} +``` + +## 5. WebSocket Message Formats + +WebSockets are utilized for fast streaming of events from the SDK to the Server, and then broadcasting those to the UI. + +### 5.1 Connection Handshake +- **UI Client:** `ws://localhost:8765/ws?client_type=ui&session_id={id}` +- **SDK Client:** `ws://localhost:8765/ws?client_type=sdk` + +### 5.2 SDK → Server (Event Ingestion) +```json +{ + "type": "event", + "session_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d", + "event": { + "event_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d", + "event_type": "llm_start", + "agent_name": "Writer", + "agent_type": "llm", + "status": "running", + "timestamp": "2026-07-21T15:10:00Z" + } +} +``` + +### 5.3 Server → UI (Broadcast) +Broadcasts include the same event shapes as ingestion, but also add `session_update` events when session states change globally. +```json +{ + "type": "session_update", + "session_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d", + "session": { + "status": "completed", + "ended_at": "2026-07-21T15:10:00Z", + "total_tokens": 4821 + } +} +``` + +## 6. Pydantic Models (Python) + +```python +from pydantic import BaseModel, Field +from typing import Optional, Dict, Any, List, Union +from datetime import datetime +from enum import Enum + +class SessionStatus(str, Enum): + RUNNING = "running" + COMPLETED = "completed" + FAILED = "failed" + +class Session(BaseModel): + session_id: str + name: str + status: SessionStatus + started_at: datetime + ended_at: Optional[datetime] = None + total_tokens: int = 0 + total_cost_usd: float = 0.0 + error_count: int = 0 + agent_count: int = 0 + metadata: Dict[str, Any] = Field(default_factory=dict) + +class EventType(str, Enum): + CHAIN_START = "chain_start" + CHAIN_END = "chain_end" + CHAIN_ERROR = "chain_error" + LLM_START = "llm_start" + LLM_END = "llm_end" + LLM_TOKEN = "llm_token" + LLM_ERROR = "llm_error" + TOOL_START = "tool_start" + TOOL_END = "tool_end" + TOOL_ERROR = "tool_error" + AGENT_ACTION = "agent_action" + AGENT_FINISH = "agent_finish" + RETRIEVER_START = "retriever_start" + RETRIEVER_END = "retriever_end" + +class LLMPayload(BaseModel): + model: str + prompts: List[str] + completion: Optional[str] = None + prompt_tokens: Optional[int] = None + completion_tokens: Optional[int] = None + total_tokens: Optional[int] = None + temperature: Optional[float] = None + streaming: bool + +class ToolPayload(BaseModel): + tool_name: str + tool_description: Optional[str] = None + input: str + output: Optional[str] = None + error: Optional[str] = None + +class ChainPayload(BaseModel): + chain_type: str + inputs: Dict[str, Any] + outputs: Optional[Dict[str, Any]] = None + error: Optional[str] = None + +class Document(BaseModel): + content: str + metadata: Dict[str, Any] + +class RetrieverPayload(BaseModel): + query: str + documents: Optional[List[Document]] = None + +class AgentEvent(BaseModel): + event_id: str + session_id: str + parent_event_id: Optional[str] = None + event_type: EventType + agent_name: str + agent_type: str + timestamp: datetime + latency_ms: Optional[int] = None + status: str + payload: Union[LLMPayload, ToolPayload, ChainPayload, RetrieverPayload] +``` + +## 7. SQLAlchemy ORM Models (Python) + +```python +from typing import Any, Dict +from datetime import datetime +from sqlalchemy import Column, String, Integer, Float, DateTime, ForeignKey, JSON +from sqlalchemy.orm import declarative_base, relationship + +Base = declarative_base() + +class SessionModel(Base): + __tablename__ = 'sessions' + + session_id = Column(String, primary_key=True) + name = Column(String, nullable=False) + status = Column(String, nullable=False) + started_at = Column(DateTime, nullable=False, default=datetime.utcnow) + ended_at = Column(DateTime, nullable=True) + total_tokens = Column(Integer, default=0) + total_cost_usd = Column(Float, default=0.0) + error_count = Column(Integer, default=0) + agent_count = Column(Integer, default=0) + metadata_ = Column("metadata", JSON, default=dict) + + events = relationship("EventModel", back_populates="session", cascade="all, delete-orphan") + + +class EventModel(Base): + __tablename__ = 'events' + + event_id = Column(String, primary_key=True) + session_id = Column(String, ForeignKey('sessions.session_id', ondelete='CASCADE'), index=True, nullable=False) + parent_event_id = Column(String, ForeignKey('events.event_id', ondelete='SET NULL'), index=True, nullable=True) + event_type = Column(String, nullable=False) + agent_name = Column(String, nullable=False) + agent_type = Column(String, nullable=False) + timestamp = Column(DateTime, nullable=False) + latency_ms = Column(Integer, nullable=True) + status = Column(String, nullable=False) + payload = Column(JSON, nullable=True) + + session = relationship("SessionModel", back_populates="events") + parent = relationship("EventModel", remote_side=[event_id]) +``` + +## 8. Token Pricing Table + +> [!NOTE] +> This pricing table is hardcoded and used for estimated cost calculations during an active session based on LLM usage. + +| Model | Input Price ($ / 1M tokens) | Output Price ($ / 1M tokens) | +|---|---|---| +| `gpt-4o` | $2.50 | $10.00 | +| `gpt-4` | $30.00 | $60.00 | +| `gpt-3.5-turbo` | $0.50 | $1.50 | +| `claude-3-5-sonnet` | $3.00 | $15.00 | +| `gemini-1.5-pro` | $1.25 | $5.00 | + +## 9. Enums & Constants + +- **Session Statuses**: `running`, `completed`, `failed` +- **Agent Types**: `chain`, `llm`, `tool`, `retriever`, `agent`, `custom` +- **Event Statuses**: `running`, `completed`, `error` +- **Event Types**: + - `chain_start`, `chain_end`, `chain_error` + - `llm_start`, `llm_end`, `llm_token`, `llm_error` + - `tool_start`, `tool_end`, `tool_error` + - `agent_action`, `agent_finish` + - `retriever_start`, `retriever_end` + +--- + +*This document is auto-generated as the canonical reference for AgentScope data structures.*