Claude Computer Use Agent - Session Management Backend

Author: Jaimin Patel

A scalable backend system for managing Claude Computer Use agent sessions with real-time streaming, VNC integration, and concurrent session support.

Demo

demo.mp4

Overview

This project implements a FastAPI-based backend that replaces the experimental Streamlit interface from Anthropic's computer-use-demo with a robust, production-ready API system. It enables Claude to control a virtual computer through a web interface, with support for multiple concurrent sessions.

Features

Session Management: Create, manage, and terminate agent sessions via REST API
Real-time Streaming: WebSocket-based communication for live progress updates
Concurrent Sessions: Dynamic worker spawning with isolated display environments
VNC Integration: Web-based VNC access to each session's virtual display
Database Persistence: SQLite storage for chat history and session data
Bedrock Fallback: Optional AWS Bedrock API support as fallback

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                      Single Docker Container                         │
├─────────────────────────────────────────────────────────────────────┤
│  ┌───────────────────────────────────────────────────────────────┐  │
│  │                     FastAPI Backend (:8000)                    │  │
│  │   REST: /sessions, /messages    WebSocket: /ws/{session_id}   │  │
│  └───────────────────────────────────────────────────────────────┘  │
│                              │                                       │
│  ┌───────────────────────────────────────────────────────────────┐  │
│  │          Session Manager (asyncio + multiprocessing)           │  │
│  │   - Dynamic process spawning per session                       │  │
│  │   - Display allocation (:1, :2, :N via Xvfb)                  │  │
│  │   - VNC port allocation (5901, 5902, 590N)                    │  │
│  │   - IPC via multiprocessing.Queue                             │  │
│  └───────────────────────────────────────────────────────────────┘  │
│           │                │                │                        │
│           ▼                ▼                ▼                        │
│   ┌─────────────┐  ┌─────────────┐  ┌─────────────┐                │
│   │ Worker 1    │  │ Worker 2    │  │ Worker N    │                │
│   │ DISPLAY=:1  │  │ DISPLAY=:2  │  │ DISPLAY=:N  │                │
│   │ Xvfb+x11vnc │  │ Xvfb+x11vnc │  │ Xvfb+x11vnc │                │
│   │ Firefox     │  │ Firefox     │  │ Firefox     │                │
│   │ Claude Loop │  │ Claude Loop │  │ Claude Loop │                │
│   └─────────────┘  └─────────────┘  └─────────────┘                │
│                                                                      │
│  ┌───────────────────────────────────────────────────────────────┐  │
│  │                    noVNC Proxy (:6080)                         │  │
│  │   Routes to correct VNC port based on session token           │  │
│  └───────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────┘

Installation

Prerequisites

Docker (version 20.10 or later)
Docker Compose (version 2.0 or later)
Anthropic API Key (get one at https://console.anthropic.com)

Step 1: Clone the Repository

git clone https://github.com/Jaiminp007/claude-computer-use-fastapi.git
cd claude-computer-use-fastapi

Step 2: Configure Environment Variables

# Copy the example environment file
cp .env.example .env

# Edit the .env file and add your API key
# On macOS/Linux:
nano .env
# Or use any text editor

Add your Anthropic API key to the .env file:

ANTHROPIC_API_KEY=sk-ant-api03-your-key-here
DEBUG=false

Step 3: Build and Run with Docker

# Build and start the container
docker-compose -f docker/docker-compose.yml up --build

# Or run in detached mode (background)
docker-compose -f docker/docker-compose.yml up -d --build

Step 4: Access the Application

Once the container is running:

Service	URL
Web UI	http://localhost:8000
API Docs	http://localhost:8000/docs
Health Check	http://localhost:8000/health

Configuration

Environment Variables

Create a .env file in the project root with the following variables:

Variable	Description	Required	Default
`ANTHROPIC_API_KEY`	Your Anthropic API key	Yes	-
`DEBUG`	Enable debug logging	No	`false`
`MODEL`	Claude model to use	No	`claude-haiku-3-5-20241022`
`DATABASE_URL`	SQLite database path	No	`sqlite+aiosqlite:///./data/sessions.db`
`BEDROCK_API_KEY`	AWS Bedrock API key (fallback)	No	-

Example `.env` File

# Required
ANTHROPIC_API_KEY=sk-ant-api03-xxxxx

# Optional
DEBUG=false
MODEL=claude-haiku-3-5-20241022

# Optional: Bedrock fallback (if Anthropic API has issues)
BEDROCK_API_KEY=bedrock-api-key-xxxxx

Usage

Creating a Session

Open http://localhost:8000 in your browser
Click "New Session" to create an agent session
Wait for the VNC display to connect (shows the virtual desktop)
Type a task in the chat input, e.g., "Search the weather in Tokyo"
Watch the agent automate Firefox in the VNC viewer

Example Tasks

"Search the weather in Dubai"
"Go to YouTube and search for 'cooking tutorials'"
"Open Google and search for 'latest tech news'"

API Reference

REST Endpoints

Method	Endpoint	Description
`POST`	`/api/sessions`	Create new session
`GET`	`/api/sessions`	List all sessions
`GET`	`/api/sessions/{id}`	Get session details
`DELETE`	`/api/sessions/{id}`	Terminate session
`GET`	`/api/sessions/{id}/messages`	Get chat history
`GET`	`/api/sessions/{id}/vnc`	Get VNC connection info
`GET`	`/api/sessions/status`	Get system status

WebSocket Endpoint

WS /ws/{session_id}

Send Messages:

{"type": "user_message", "content": "Search the weather in Dubai"}
{"type": "ping"}
{"type": "get_history"}

Receive Messages:

{"type": "status", "data": {"status": "processing"}}
{"type": "progress", "data": {"type": "text", "text": "Opening Firefox..."}}
{"type": "tool_use", "data": {"tool": "computer", "input": {"action": "screenshot"}}}
{"type": "complete", "data": {"final_response": {...}}}

Testing Concurrent Sessions

The system supports multiple concurrent sessions running in parallel.

Test Procedure

Open two browser tabs to http://localhost:8000
Create a session in each tab by clicking "New Session"
Simultaneously send tasks:
- Tab 1: "Search weather in Tokyo"
- Tab 2: "Search weather in New York"
Verify parallel execution:
- Both VNC displays should show Firefox opening at the same time
- Neither session should block the other

How It Works

Each session runs in an isolated environment:

Separate OS process (multiprocessing.Process)
Unique X display (:1, :2, etc.)
Separate VNC server (ports 5901, 5902, etc.)
Independent Firefox instance

Project Structure

claude-computer-use-fastapi/
├── backend/
│   ├── main.py                 # FastAPI app entry point
│   ├── api/
│   │   ├── sessions.py         # Session CRUD endpoints
│   │   ├── messages.py         # Message endpoints
│   │   └── websocket.py        # WebSocket handlers
│   ├── core/
│   │   ├── config.py           # App configuration
│   │   ├── session_manager.py  # Process management
│   │   ├── display_manager.py  # Display allocation
│   │   └── worker.py           # Worker process
│   ├── db/
│   │   ├── database.py         # SQLite connection
│   │   ├── models.py           # SQLAlchemy models
│   │   └── crud.py             # CRUD operations
│   └── agent/
│       └── loop.py             # Claude agent loop
├── frontend/
│   ├── index.html              # Main UI
│   ├── app.js                  # Frontend logic
│   └── styles.css              # Styling
├── docker/
│   ├── Dockerfile
│   ├── docker-compose.yml
│   └── scripts/
│       └── entrypoint.sh
├── anthropic-quickstarts/      # Computer use tools
├── requirements.txt
├── .env.example
└── README.md

Troubleshooting

Container fails to start

# Check container logs
docker-compose -f docker/docker-compose.yml logs -f

# Rebuild from scratch
docker-compose -f docker/docker-compose.yml down
docker-compose -f docker/docker-compose.yml up --build --force-recreate

VNC not connecting

Wait 5-10 seconds after creating a session for VNC to initialize
Check that ports 6080 and 5901-5920 are not blocked by firewall
Try refreshing the VNC iframe with the refresh button

API key errors

Verify your ANTHROPIC_API_KEY is set correctly in .env
Check that the API key has sufficient credits
Try configuring a Bedrock API key as fallback in the settings

Session stuck or not responding

Delete the session and create a new one
Check backend logs for error messages
Restart the Docker container

Local Development (Without Docker)

For development without Docker (requires Linux with X11):

# Install system dependencies (Ubuntu/Debian)
sudo apt-get install xvfb x11vnc mutter firefox-esr novnc websockify

# Create virtual environment
python3.11 -m venv venv
source venv/bin/activate

# Install Python dependencies
pip install -r requirements.txt

# Set environment variables
export ANTHROPIC_API_KEY=your-key
export PYTHONPATH=$(pwd)

# Run the server
uvicorn backend.main:app --reload --port 8000

License

This project is for evaluation purposes.

Built with FastAPI, Anthropic Claude, and Python

Name		Name	Last commit message	Last commit date
Latest commit History 16 Commits
anthropic-quickstarts/computer-use-demo/computer_use_demo		anthropic-quickstarts/computer-use-demo/computer_use_demo
backend		backend
docker		docker
frontend		frontend
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
demo.mp4		demo.mp4
pytest.ini		pytest.ini
requirements.txt		requirements.txt
setup.sh		setup.sh

Folders and files

Latest commit

History

Repository files navigation