diff --git a/docs/aws_tools.md b/docs/aws_tools.md new file mode 100644 index 0000000..55c7669 --- /dev/null +++ b/docs/aws_tools.md @@ -0,0 +1,309 @@ +# AWS Tools Reference + +All tools are read-only. They wrap boto3 calls and return structured dicts. Every tool +is decorated with `@tool_cached` (2-minute TTL) and `with_cap()` (response size limit). +Error responses always include an `"error"` key so the agent can handle failures gracefully. + +--- + +## CloudWatch + +### `get_alarms(state?)` +List CloudWatch alarms, optionally filtered by state. + +| Param | Type | Default | Description | +|---|---|---|---| +| `state` | `string` | none | Filter: `OK`, `ALARM`, or `INSUFFICIENT_DATA` | + +**Returns:** `{ alarms: [{name, state, reason, metric, namespace, updated_at}], count }` + +--- + +### `get_alarm_history(alarm_name, hours?)` +Fetch state-change history for a specific alarm. + +| Param | Type | Default | Description | +|---|---|---|---| +| `alarm_name` | `string` | required | CloudWatch alarm name | +| `hours` | `int` | `24` | How far back to look | + +**Returns:** `{ alarm_name, history: [{timestamp, summary}], count }` + +--- + +### `get_metric_data(namespace, metric, dimensions, period?, hours?, stat?)` +Fetch raw CloudWatch metric datapoints. + +| Param | Type | Default | Description | +|---|---|---|---| +| `namespace` | `string` | required | e.g. `AWS/Lambda` | +| `metric` | `string` | required | e.g. `Errors` | +| `dimensions` | `list` | required | e.g. `[{"Name": "FunctionName", "Value": "my-fn"}]` | +| `period` | `int` | `300` | Period in seconds | +| `hours` | `int` | `3` | How far back to look | +| `stat` | `string` | `Sum` | `Sum`, `Average`, or `Maximum` | + +**Returns:** `{ namespace, metric, stat, datapoints: [{timestamp, value}], count }` + +--- + +### `get_log_events(log_group, log_stream?, filter_pattern?, hours?, limit?)` +Fetch recent log events from a CloudWatch Logs group. + +| Param | Type | Default | Description | +|---|---|---|---| +| `log_group` | `string` | required | CloudWatch log group name | +| `log_stream` | `string` | none | Specific stream to read | +| `filter_pattern` | `string` | none | CloudWatch filter pattern, e.g. `ERROR` | +| `hours` | `int` | `1` | How far back to look | +| `limit` | `int` | `100` | Max events to return | + +**Returns:** `{ log_group, events: [{timestamp, message, stream}], count }` + +> **Tip:** Always call `describe_log_groups` first to discover the real log group name — never guess it. + +--- + +### `describe_log_groups(prefix?)` +List CloudWatch log groups, optionally filtered by name prefix. + +| Param | Type | Default | Description | +|---|---|---|---| +| `prefix` | `string` | none | Filter by log group name prefix | + +**Returns:** `{ log_groups: [{name, retention_days, stored_bytes}], count }` + +--- + +### `query_logs_insights(log_group, query, hours?, limit?)` +Run a CloudWatch Logs Insights structured query. Supports the full query language: +`fields`, `filter`, `stats`, `sort`, `limit`. + +| Param | Type | Default | Description | +|---|---|---|---| +| `log_group` | `string` | required | Log group name (e.g. `/aws/lambda/my-fn`) | +| `query` | `string` | required | Logs Insights query string | +| `hours` | `int` | `1` | Time window to search | +| `limit` | `int` | `100` | Max results | + +**Returns:** `{ log_group, query, results: [{}], count, scanned_mb }` + +Example queries: +``` +fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 50 +stats count(*) as errors by bin(5m) | sort errors desc +``` + +--- + +## CloudTrail + +### `lookup_cloudtrail_events(hours?, resource_name?, event_name?, limit?)` +Look up recent API events to find deployments, config changes, and permission changes. + +| Param | Type | Default | Description | +|---|---|---|---| +| `hours` | `int` | `2` | How far back to look | +| `resource_name` | `string` | none | Filter by resource name (e.g. Lambda function name) | +| `event_name` | `string` | none | Filter by API event (e.g. `UpdateFunctionCode`) | +| `limit` | `int` | `50` | Max events (capped at 50 by AWS) | + +**Returns:** `{ events: [{event_name, event_time, username, resources, event_id}], count }` + +Common events to look for: `UpdateFunctionCode`, `UpdateFunctionConfiguration`, +`PutFunctionConcurrency`, `CreateDeployment`, `UpdateService`. + +--- + +## ECS + +> **Note:** Always call `list_ecs_clusters` first to discover real cluster names — never guess them. + +### `list_ecs_clusters()` +List all ECS clusters in the region with status and task counts. + +**Returns:** `{ clusters: [{name, arn, status, active_services, running_tasks, pending_tasks}], count }` + +--- + +### `list_ecs_services(cluster)` +List services in a cluster with their desired/running/pending task counts. + +| Param | Type | Description | +|---|---|---| +| `cluster` | `string` | Cluster name or ARN | + +**Returns:** `{ cluster, services: [{name, status, desired, running, pending, task_definition}], count }` + +--- + +### `describe_ecs_service(cluster, service)` +Get detailed info about an ECS service including recent events and deployments. + +| Param | Type | Description | +|---|---|---| +| `cluster` | `string` | Cluster name or ARN | +| `service` | `string` | Service name or ARN | + +**Returns:** `{ name, status, desired, running, pending, deployments: [{status, desired, running, failed, created_at, task_definition}], events: [{created_at, message}] }` + +--- + +### `get_ecs_task_logs(cluster, task_id, log_group, limit?)` +Fetch stdout/stderr logs for a specific ECS task from CloudWatch Logs. + +| Param | Type | Default | Description | +|---|---|---|---| +| `cluster` | `string` | required | Cluster name | +| `task_id` | `string` | required | Task ID (short or full ARN) | +| `log_group` | `string` | required | CloudWatch log group, often `/ecs/` | +| `limit` | `int` | `100` | Max log lines | + +**Returns:** `{ task_id, log_group, events: [{message}], count }` + +--- + +## Lambda + +### `list_lambda_functions()` +List all Lambda functions in the region. + +**Returns:** `{ functions: [{name, runtime, memory_mb, timeout_s, last_modified}], count }` + +--- + +### `get_lambda_function_config(name)` +Get detailed configuration for a Lambda function. + +| Param | Type | Description | +|---|---|---| +| `name` | `string` | Lambda function name | + +**Returns:** `{ name, runtime, memory_mb, timeout_s, last_modified, env_var_keys, layers, vpc_enabled, reserved_concurrency }` + +> `reserved_concurrency: 0` means the function is fully throttled (intentional or misconfiguration). + +--- + +### `get_lambda_error_rate(name, hours?)` +Get error count, throttle count, and invocation count from CloudWatch. + +| Param | Type | Default | Description | +|---|---|---|---| +| `name` | `string` | required | Lambda function name | +| `hours` | `int` | `3` | How far back to look | + +**Returns:** `{ function, hours, total_invocations, total_errors, total_throttles, error_rate_pct, errors_by_period, throttles_by_period }` + +--- + +## EC2 + +### `describe_ec2_instances(filters?)` +List EC2 instances with state, type, and tags. + +| Param | Type | Default | Description | +|---|---|---|---| +| `filters` | `list` | none | EC2 filters, e.g. `[{"Name": "instance-state-name", "Values": ["running"]}]` | + +**Returns:** `{ instances: [{instance_id, name, type, state, az, private_ip, launch_time}], count }` + +--- + +### `get_ec2_system_status(instance_id)` +Get EC2 instance status checks (system reachability and instance reachability). + +| Param | Type | Description | +|---|---|---| +| `instance_id` | `string` | EC2 instance ID | + +**Returns:** `{ instance_id, instance_state, system_status, instance_status, system_events }` + +--- + +## RDS + +### `describe_rds_instances()` +List RDS DB instances with status, engine, class, and multi-AZ configuration. + +**Returns:** `{ instances: [{identifier, status, engine, class, multi_az, storage_gb, endpoint}], count }` + +--- + +### `get_rds_events(hours?, db_identifier?)` +Fetch RDS events log for recent database activity, failovers, and maintenance. + +| Param | Type | Default | Description | +|---|---|---|---| +| `hours` | `int` | `24` | How far back to look | +| `db_identifier` | `string` | none | Filter by specific DB instance identifier | + +**Returns:** `{ events: [{source, message, date, categories}], count }` + +--- + +## IAM / STS + +### `get_caller_identity()` +Return the current AWS caller identity — confirms which account and role the agent is using. + +**Returns:** `{ account_id, arn, user_id }` + +--- + +### `get_iam_role_policies(role_name)` +List policies (attached and inline) on an IAM role. + +| Param | Type | Description | +|---|---|---| +| `role_name` | `string` | IAM role name | + +**Returns:** `{ role, attached_policies: [{name, arn}], inline_policy_names }` + +--- + +## Bash Execution Tool + +### `run_bash_command(command)` +Run a read-only shell command. Covers the full AWS CLI surface for services not +available via the structured boto3 tools (S3, DynamoDB, SNS, SQS, Route53, ACM, +Secrets Manager, SSM, etc.), plus kubectl and docker. + +**Allowed commands:** +- `aws ` where operation starts with: `describe-*`, `list-*`, `get-*`, `lookup-*`, `filter-*`, `search-*`, `scan-*`, `query*`, `batch-get-*` +- `kubectl get / describe / logs` +- `docker ps / logs / inspect` + +**Rules:** No chaining (`&&`, `|`, `;`), no state-modifying commands. Max command length: 2000 chars. Hard timeout: 30s. Output capped at 4000 chars. + +**Returns:** `{ success, output, error, command, blocked }` + +If `blocked: true` the command was rejected by the allowlist — do not retry with a variation. + +--- + +## History Tools + +### `get_investigation_history(days?)` +Cross-session analytics: top alarms investigated, top Lambda functions, recurring +tool errors, and daily investigation frequency. Data is aggregated at the DB level — +no raw message content is read. + +| Param | Type | Default | Description | +|---|---|---|---| +| `days` | `int` | `30` | How many days to look back | + +**Returns:** `{ days, top_alarms, top_lambdas, recurring_errors, trend }` + +--- + +### `search_past_investigations(query, limit?)` +Search past session titles and messages by keyword. Returns session summaries with +short snippets — never full message bodies. + +| Param | Type | Default | Description | +|---|---|---|---| +| `query` | `string` | required | Keyword or phrase (e.g. alarm name, service, error type) | +| `limit` | `int` | `10` | Max results (capped at 20) | + +**Returns:** `{ results: [{id, title, last_active_at, model, snippet}], count }` diff --git a/docs/caching.md b/docs/caching.md new file mode 100644 index 0000000..60a1db0 --- /dev/null +++ b/docs/caching.md @@ -0,0 +1,40 @@ +# Tool Response Caching + +## What it does + +Every AWS tool function is decorated with `@tool_cached`. Repeated calls with the +same arguments within a 2-minute window return the cached result instantly — no +AWS API call is made. This matters for investigations where the agent calls the same +tool multiple times (e.g. `get_alarms` at the start and again after a hypothesis). + +## How it works + +`src/tools/_cache.py` wraps `cachetools.TTLCache`: + +- **Max entries:** 256 +- **TTL:** 120 seconds (2 minutes) +- **Cache key:** function name + AWS profile + AWS region + serialised arguments + +Including the AWS profile and region in the key means: +- Zero-argument functions like `list_lambda_functions()` never collide across functions +- Results from different AWS accounts or regions are never mixed (multi-tenant safe) + +## Configuration + +The cache is always on — there are no env vars to configure it. It is entirely +in-process and does not require Redis or any external service. + +To swap the cache backend to Redis later, only `src/tools/_cache.py` needs to change — +tool files don't need to be touched. + +## Interaction with tool response capping + +The cap (`TOOL_RESPONSE_MAX_CHARS`) is applied **after** the cache is checked. +The raw full result is cached; the cap is applied on the way out to the LLM. This +means the cache always stores the full response regardless of the current cap setting. + +## When caching is NOT useful + +- First call in a session always hits AWS +- Results older than 2 minutes are always re-fetched +- The cache is lost on process restart (in-memory only) diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 0000000..f56f9ce --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,140 @@ +# CLI Reference + +The `devops-agent` CLI is powered by Typer and Rich. All commands share the same +LLM and AWS configuration from `.env`. + +```bash +uv run devops-agent --help +``` + +--- + +## `investigate` + +Investigate an AWS incident and produce a structured root cause report. + +```bash +uv run devops-agent investigate "" [OPTIONS] +``` + +| Option | Short | Description | +|---|---|---| +| `--alarm` | `-a` | CloudWatch alarm name to investigate | +| `--service` | `-s` | Service name (e.g. ECS service name) | +| `--region` | `-r` | AWS region override | +| `--json` | | Output raw JSON instead of the Rich panel | + +**Examples:** + +```bash +# Basic incident description +uv run devops-agent investigate "high error rate on my payment Lambda" + +# With alarm and service hints +uv run devops-agent investigate "latency spike" --alarm HighLatencyAlarm --service api-service + +# Machine-readable output +uv run devops-agent investigate "ECS tasks keep crashing" --json +``` + +**Output (Rich panel):** +- Root cause category and summary +- Confidence level (HIGH / MEDIUM / LOW) +- Evidence list +- Mitigation steps +- Validation steps +- Services affected +- Recommended follow-up + +**JSON output fields:** `root_cause_category`, `root_cause_summary`, `evidence`, +`mitigation_steps`, `validation_steps`, `confidence`, `services_affected`, +`recommended_follow_up`, `tool_calls_made`. + +--- + +## `ask` + +Ask a freeform question about your AWS environment. No tool calls — direct LLM answer. + +```bash +uv run devops-agent ask "" +``` + +**Examples:** + +```bash +uv run devops-agent ask "why would a Lambda function suddenly start throttling?" +uv run devops-agent ask "what CloudWatch metrics should I watch for RDS failovers?" +``` + +Output is rendered as Markdown in the terminal via Rich. + +--- + +## `report` + +Generate a daily ops health summary showing CloudWatch alarm states. + +```bash +uv run devops-agent report [OPTIONS] +``` + +| Option | Short | Description | +|---|---|---| +| `--region` | `-r` | AWS region override | + +**Output:** +- Summary table: OK / ALARM / INSUFFICIENT_DATA counts +- Table of currently firing alarms with metric name and state reason + +```bash +uv run devops-agent report +``` + +--- + +## `ui` + +Launch the web UI chat interface locally. + +```bash +uv run devops-agent ui [OPTIONS] +``` + +| Option | Default | Description | +|---|---|---| +| `--host` | `127.0.0.1` | Bind host | +| `--port` / `-p` | `8000` | Port to listen on | + +```bash +# Default +uv run devops-agent ui + +# Expose to LAN +uv run devops-agent ui --host 0.0.0.0 --port 8080 +``` + +Opens the full React UI + FastAPI backend in one process. The frontend is served +as static files from the same process — no separate Vite server needed. + +--- + +## `mcp` + +Start the agent as an MCP (Model Context Protocol) server, exposing `investigate`, +`ask`, and `list_sessions` tools to MCP-compatible clients like Claude Desktop or Cursor. + +```bash +uv run devops-agent mcp [OPTIONS] +``` + +See [`mcp_server.md`](mcp_server.md) for full setup instructions and transport options. + +--- + +## Global behaviour + +- **Logging** — level controlled by `LOG_LEVEL` in `.env` (default `INFO`); logs go to stderr +- **Timeout** — all LLM calls respect `INVESTIGATION_TIMEOUT` (default 120s) +- **AWS profile** — all commands use `AWS_PROFILE` from `.env` if set +- **Model** — all commands use `LLM_MODEL` from `.env` diff --git a/docs/dashboard.md b/docs/dashboard.md new file mode 100644 index 0000000..d39137a --- /dev/null +++ b/docs/dashboard.md @@ -0,0 +1,79 @@ +# Dashboard + +The dashboard is available at `http://localhost` (Docker) or `http://localhost:5173` +(local dev). It shows aggregated analytics across all sessions and requires SQLite or +Postgres — the memory backend returns live counts only (lost on restart). + +All data comes from `GET /stats`. + +--- + +## Stat cards (top row) + +| Card | What it shows | +|---|---| +| **Sessions** | Total non-deleted sessions | +| **Tool calls** | Total tool calls across all sessions · avg per session · error % | +| **Total cost** | Sum of `cost_usd` from all usage events · total tokens (input + output) | +| **Avg latency** | Average response latency in seconds · total query count | + +## Context management row + +Appears only when at least one summarization has fired. + +| Card | What it shows | +|---|---| +| **Sessions compacted** | Number of times conversation summarization ran | +| **Context saved** | Estimated tokens recovered (`chars_compacted / 4`) · raw char count | + +See [`conversation_summarization.md`](conversation_summarization.md) for how this data is collected. + +## Activity chart + +Bar chart of session count per day over the last 14 days. Hover any bar for the +exact date and count. Empty days render as a thin grey bar. + +## Calls by service + +Horizontal bar chart showing how many tool calls each AWS service received, as a +percentage of total tool calls. Services: CloudWatch, Lambda, CloudTrail, RDS, ECS, +EC2, IAM, Agent (= `submit_investigation` calls). + +## Recurring incident types + +Shows the distribution of root cause categories across all completed investigations +(sessions where `submit_investigation` was called with a `root_cause_category`): + +| Category | Colour | +|---|---| +| System change | Amber | +| Input anomaly | Blue | +| Resource limit | Orange | +| Component failure | Red | +| Dependency issue | Purple | +| Unknown | Grey | + +## Top tools (no UI panel — API only) + +`GET /stats` returns a `top_tools` list (up to 12 entries) with call count and error +count per tool. Not currently rendered in the UI but available for custom dashboards +or monitoring. + +## Recent sessions + +Last 6 sessions sorted by `last_active_at`, showing title, model, query count, tool +call count, and cost. Click any row to open the session in the chat page. + +--- + +## Data source + +All dashboard data comes from three tables: + +| Table | Used for | +|---|---| +| `sessions` | Session counts, activity chart, recent sessions | +| `tool_calls` | Tool call counts, error counts, service breakdown, root causes | +| `usage_events` | Token counts, cost, latency, summarization stats | + +See [`schema.md`](schema.md) for full table definitions. diff --git a/docs/ui.md b/docs/ui.md new file mode 100644 index 0000000..b65cb00 --- /dev/null +++ b/docs/ui.md @@ -0,0 +1,97 @@ +# Web UI + +The web UI is a React + Vite single-page application served by the FastAPI backend. + +**URLs:** +- Docker Compose: `http://localhost` (frontend on port 80, backend on port 8000) +- Local dev: `http://localhost:5173` (Vite dev server with HMR) + +--- + +## Chat page + +The main investigation interface. + +**Input area** — type a message and press Enter or click Send. The agent starts +streaming its response immediately. A Stop button cancels an in-flight request mid-stream. + +**Streaming response** — agent tokens appear in real time as the LLM reasons. +Tool calls appear as collapsible cards as they complete, showing the tool name, +arguments, and result. A contextual status label ("Digging through CloudTrail…", +"Checking Lambda metrics…") appears while each tool runs. + +**Tool call inspector** — each tool call card is collapsed by default; click to expand +and see the full arguments and result JSON. + +**Cost card** — after each agent response, a collapsible card shows: +- Input tokens / Output tokens +- Per-component USD cost breakdown +- Total cost for the turn +- Latency in milliseconds + +**Session continuity** — the session ID is stored in the URL. Refreshing the page +or sharing the URL resumes the same conversation from where it left off (SQLite/Postgres +backends only — memory backend loses history on restart). + +--- + +## Session history sidebar + +Lists all past sessions sorted by last activity. Each entry shows: +- Session title (first user message, truncated) +- Time since last active (relative, e.g. "3 hours ago") +- Model used + +**Actions:** +- Click any session to load it in the chat view with full history restored +- **New chat** button starts a fresh session with a new UUID +- **Delete** (trash icon) soft-deletes the session — it disappears from the list + immediately but data is preserved in the database + +--- + +## History page + +Accessible from the sidebar. Shows a searchable list of all past sessions. + +**Search** — filters sessions by keyword matching the title or any user message. +Results show a short content snippet from the first matching message. + +**Session list** — displays title, last active time, model, and a content snippet. +Click any row to open the session in the chat page. + +--- + +## Dashboard page + +See [`dashboard.md`](dashboard.md) for full details on each panel. + +--- + +## Settings page + +Currently shows the active configuration read from the backend: + +- Active LLM model +- Storage backend type (memory / sqlite / postgres) +- AWS region +- Tool response cap limit +- Summarization settings + +Settings are read-only in the UI — change them by editing `.env` and restarting +the server. + +--- + +## Dark mode + +The UI respects the system dark/light mode preference automatically. No toggle is needed. + +--- + +## Theme and design + +- Font: system monospace stack +- Framework: React 18 + Tailwind CSS +- Icons: Lucide React +- Build tool: Vite with HMR in development