Acme Operations Copilot

Enterprise agentic assistant for Acme Operations — grounded customer support via LangGraph tool calls, PostgreSQL, Redis, Keycloak, and MCP.

Quick start

cp .env.example .env   # add OPENAI_API_KEY or use Ollama
make demo              # stack + demo seed data

Service	URL
Command Centre	http://localhost:5173/dashboard
Web UI	http://localhost:5173
API	http://localhost:8000/docs
Dashboard API	http://localhost:8000/api/dashboard/summary
Keycloak	http://localhost:8080 (`bob` / `bob123`)
Jaeger (traces)	http://localhost:16686
Grafana (LGTM)	http://localhost:3002 (admin / `GRAFANA_ADMIN_PASSWORD`)
Prometheus	http://localhost:9090
Langfuse	http://localhost:3001
Arize Phoenix	http://localhost:6006

See docs/architecture.md and docs/tradeoffs.md.

Repository layout

acmeops-copilot/
├── apps/
│   ├── api/                 # FastAPI + LangGraph agent
│   │   ├── main.py
│   │   ├── config.py
│   │   ├── auth/            # keycloak, rbac, dependencies
│   │   ├── agent/           # graph, tool_router, runner
│   │   ├── prompts/         # versioned YAML prompts (Git source of truth)
│   │   ├── services/        # chat_service, prompt_service
│   │   ├── routers/         # chat controller (HTTP layer)
│   │   ├── skills/          # customer_escalation_summary
│   │   ├── memory/          # redis_store, checkpoint, conversations
│   │   ├── telemetry/       # tracing, metrics, logging, audit
│   │   ├── schemas/         # chat, customer, issue
│   │   ├── support/         # db, middleware, exceptions, evidence
│   │   └── tests/
│   ├── mcp-server/          # Acme MCP tools (SSE)
│   │   ├── server.py
│   │   ├── tools/           # customers, issues, next_actions
│   │   ├── db.py
│   │   └── telemetry.py
│   └── web/                 # React UI (demo)
├── infra/
│   ├── postgres/            # init.sql, seed.sql, migrations
│   ├── keycloak/            # realm-export.json
│   ├── redis/
│   ├── knowledge/           # SLA docs (filesystem MCP)
│   ├── mcp/                 # filesystem, postgres, search MCP servers
│   ├── otel/
│   └── kubernetes/          # K8s manifests, Ingress LB, Headlamp — see infra/kubernetes/README.md
├── evals/
│   ├── dataset.json
│   ├── eval_questions.json
│   ├── run_eval.py
│   └── eval_results.md
├── docs/
│   ├── architecture.md
│   ├── architecture-diagram.mmd
│   ├── ai-usage-notes.md
│   ├── threat-model.md
│   └── tradeoffs.md
├── docker-compose.yml
├── Makefile
└── README.md

Commands

make up          # docker compose up --build
make demo        # bootstrap + seed PostgreSQL demo data
make reseed-db   # reload demo customers/issues
make eval        # run evals/run_eval.py (10-question suite)
make logs        # follow api logs
make logs-otel   # collector + jaeger + phoenix + api trace logs
make otel-ui     # print Jaeger URL
make phoenix-ui  # print Phoenix URL

Kubernetes (production infra)

Load-balanced deployment on Kubernetes with NGINX Ingress, cert-manager TLS, HPA, NetworkPolicies, and Headlamp (replaces deprecated Kubernetes Dashboard):

make k8s-bootstrap    # kind cluster + ingress-nginx + cert-manager + Headlamp
make k8s-secrets      # generate random DB/Redis passwords (gitignored)
make k8s-deploy       # build images + apply local overlay (HTTPS via mkcert)

Full guide: infra/kubernetes/README.md.

Testing & quality

See docs/CONTRIBUTING.md for the full merge bar.

Layer	Command	Notes
All quality gates	`make quality`	lint + typecheck + coverage + mypy + security audit
Ruff (Python)	`make lint`	`ruff check` + `ruff format --check`
ESLint (web)	`make lint`	`apps/web` — `npm run lint`
Coverage	`make test-coverage`	Python HTML → `htmlcov/`, web → `apps/web/coverage/`
Diff coverage	`make diff-coverage`	≥80% on changed Python lines (PR gate)
Mypy	`make mypy`	No new errors vs `mypy-baseline.txt`
Pre-commit	`make pre-commit-install`	Hooks on commit/push
API unit	`make test`	FastAPI + infra MCP unit tests
API integration	`make test-integration`	Postgres + Redis
Web unit	`make test-web`	Vitest + Testing Library
E2E	`make test-e2e`	Playwright + `docker-compose.e2e.yml`

python3 -m venv .venv && .venv/bin/pip install -r requirements-dev.txt
cd apps/web && npm install && npx playwright install chromium
make pre-commit-install
make quality

CI runs pre-commit, security scans (Bandit, Semgrep, Gitleaks, pip/npm audit), SBOM, coverage + Codecov, diff coverage on PRs, mypy baseline, integration tests, and Playwright E2E.

Observability

Four tracing backends are supported, each independently enable/disable via .env:

Provider	Env flag	UI
OpenTelemetry → Jaeger + Tempo	`ENABLE_OTEL=true`	http://localhost:16686 (Jaeger), Grafana → Tempo
Grafana LGTM	Docker stack (default)	http://localhost:3002
Langfuse (self-hosted v3)	`ENABLE_LANGFUSE=true`	http://localhost:3001
LangSmith (cloud)	`ENABLE_LANGSMITH=true` + `LANGSMITH_API_KEY`	https://smith.langchain.com
Arize Phoenix	`ENABLE_PHOENIX=true`	http://localhost:6006

OBSERVABILITY_PROVIDER filters which providers are active:

all (default) — all enabled flags apply
both — legacy: Langfuse + LangSmith only
comma-separated, e.g. langfuse,otel or phoenix,langsmith

What gets traced: HTTP requests, outbound HTTP, Postgres, Redis, agent runs, tool calls (with latency), RBAC checks, and errors. Local JSONL logs are written to evals/logs/traces-*.jsonl.

Disable examples:

ENABLE_OTEL=false          # no Jaeger spans
ENABLE_LANGFUSE=false      # no Langfuse callbacks
ENABLE_LANGSMITH=false     # no LangSmith export
ENABLE_PHOENIX=false       # no Phoenix / OpenInference
OBSERVABILITY_PROVIDER=otel  # only OpenTelemetry when enabled

OpenTelemetry

Traces flow API → OTLP collector → Jaeger + Tempo (+ Phoenix). Metrics and logs are available in Grafana (see Grafana LGTM). Enabled by default in Docker (ENABLE_OTEL=true).

Start the stack: make demo (or docker compose up -d jaeger otel-collector api)
Open Jaeger: http://localhost:16686
Select service acme-assistant-api, click Find Traces
Send a chat message in the UI (or curl /api/health) to generate spans

What gets traced: HTTP requests (FastAPI), outbound HTTP (httpx), Postgres (asyncpg), Redis, agent runs (agent.run), and tool calls (tool.*).

MCP (agent integration)

Four MCP servers run in Docker; the chat agent loads their tools when ENABLE_MCP_AGENT_TOOLS=true (default):

Server	Port	Agent tool prefix (examples)
Custom Acme	8090	`acme-custom_acme_get_customer_by_name`, `acme-custom_acme_list_open_issues`, …
Filesystem	8091	`filesystem_fs_read_file`, `filesystem_fs_list_directory`
PostgreSQL	8092	`postgres_postgres_query` (SELECT only)
Search (RAG)	8093	`search-mock_search_knowledge_semantic`, `search-mock_get_knowledge_chunk`, `search-mock_rag_index_status`

Native tools (get_open_issues, create_next_action, …) remain for RBAC, HITL, and audit. MCP tools add policies, runbooks, SQL exploration, and semantic RAG (Qdrant-backed knowledge search).

RAG (Qdrant)

Step	Command / location
Vector DB	Docker service `qdrant` (internal only); K8s `qdrant` StatefulSet
Ingest	`make rag-ingest` or automatic on `mcp-search` startup when index is empty
MCP tools	`search_knowledge_semantic`, `get_knowledge_chunk`, `rag_index_status`
Config	`QDRANT_URL`, `QDRANT_API_KEY`, `EMBEDDING_MODEL`, `EMBEDDING_DIMENSION`

Without Azure/OpenAI embeddings, the ingest/search path uses a deterministic hash embedding for local demos.

Status: GET /api/mcp/status (SSE reachability + agent tool load count)
Disable MCP in agent: ENABLE_MCP_AGENT_TOOLS=false

Correlate a request: API responses include an X-Trace-Id header. Paste that trace ID into Jaeger’s Trace ID search.

In the UI: Assistant sidebar → Trace status shows OTel/Jaeger links. Evaluations page has the same observability panel.

Disable: set ENABLE_OTEL=false in .env and restart the API.

Grafana (LGTM stack)

Unified metrics, logs, and traces for platform performance monitoring. LLM-specific debugging remains in Phoenix/Langfuse.

Component	URL	Role
Grafana	http://localhost:3002	Dashboards (metrics + logs + traces)
Prometheus	http://localhost:9090	Scrapes `/metrics` + OTel collector
Loki	internal `:3100`	Container log aggregation (Promtail)
Tempo	internal `:3200`	Distributed traces (OTLP from collector)

Quick start

make demo                    # starts prometheus, loki, promtail, tempo, grafana
make grafana-ui              # print Grafana URL + credentials hint

Open Grafana: http://localhost:3002 (default admin / admin, override with GRAFANA_ADMIN_PASSWORD)
Open dashboard Acme API — Performance (folder: Acme Operations)
Send chat traffic — panels update for HTTP latency, agent runs, tool calls
Use Explore → Loki for JSON logs (trace_id, request_id labels)
Click TraceID in logs to jump to Tempo (or use Jaeger datasource)

Configuration

GRAFANA_UI_URL=http://localhost:3002
GRAFANA_ADMIN_PASSWORD=admin
LOG_FORMAT=json              # structured logs for Loki (default in Docker)

Data flow: API → OTLP collector → Jaeger + Tempo + Phoenix; Promtail → Loki; Prometheus scrapes API /metrics and OTel :8888.

Kubernetes: kube-prometheus-stack Grafana + Loki (make k8s-bootstrap) with Tempo in acme-ops namespace. Port-forward: make k8s-grafana.

Arize Phoenix

Phoenix traces LLM/agent workflows using arize-phoenix-otel (phoenix.otel.register) and OpenInference auto-instrumentation. Enabled by default in Docker (ENABLE_PHOENIX=true).

Configuration

ENABLE_PHOENIX=true
PHOENIX_COLLECTOR_ENDPOINT=http://phoenix:6006/v1/traces   # Docker
# PHOENIX_COLLECTOR_ENDPOINT=https://phoenix.acme.local/v1/traces  # local HTTPS ingress
PHOENIX_UI_URL=http://localhost:6006
PHOENIX_PROJECT_NAME=acme-ops

PHOENIX_COLLECTOR_ENDPOINT must be the fully qualified HTTP OTLP URL (.../v1/traces). If you pass only the host (e.g. https://phoenix.acme.local), the API appends /v1/traces automatically.

How it works

On startup, telemetry/phoenix_tracing.py calls phoenix.otel.register():

from phoenix.otel import register

tracer_provider = register(
    project_name=settings.phoenix_project_name,
    endpoint=settings.phoenix_collector_endpoint,
    auto_instrument=True,
    batch=True,
)

tracer = tracer_provider.get_tracer("acme.agent")

Agent entry points are wrapped with @tracer.chain (via phoenix_chain):
- run_agent — full chat/agent run
- invoke_agent_graph — single ReAct LangGraph path
- invoke_multi_agent — supervisor multi-agent path
When both Phoenix and OpenTelemetry are enabled, Phoenix registers the global TracerProvider first; Jaeger export is attached to the same provider (dual export: Phoenix UI + Jaeger).

View traces

Start Phoenix: make demo (or docker compose up -d phoenix api)
Open Phoenix UI: http://localhost:6006 (or make phoenix-ui)
Select project acme-ops (or your PHOENIX_PROJECT_NAME)
Send a chat message — spans appear for agent chains, LangChain/LangGraph calls, and tools

Kubernetes (local HTTPS): Phoenix UI at https://phoenix.acme.local — set PHOENIX_UI_URL and collector endpoint in infra/kubernetes/base/config/app-config.yaml.

Kubernetes Grafana LGTM: https://grafana.acme.local (or make k8s-grafana). Deploy runs make k8s-sync-observability automatically.

Disable: ENABLE_PHOENIX=false or OBSERVABILITY_PROVIDER=otel,langfuse (exclude phoenix).

Evaluation

Run the 10-question assessment suite (tool selection, groundedness, RBAC, next-action quality):

make eval
# or: python evals/run_eval.py

Questions live in evals/eval_questions.json. Results are written to evals/results.json and evals/eval_results.md.

Metrics measured per question:

Tool selection — expected tools invoked for the query
Groundedness — response tied to database tool results
RBAC — role-based access respected (deny / approval flows)
Next actions — recommended follow-ups are reasonable when requested

Local API (outside Docker): point export at the collector on localhost:

ENABLE_OTEL=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
JAEGER_UI_URL=http://localhost:16686

Demo prompts

Sign in as bob / bob123 and try:

"Show me open customer issues for AsteriaPay Global, summarise the latest status, and suggest the next action."
"Who is the account owner for AsteriaPay Global?"
"Run customer escalation summary for AsteriaPay Global."

Demo data: 12+ customers, 35+ issues, and pending next actions across FinTech, healthcare, retail, travel, and utilities. Flagship account AsteriaPay Global (ASTERIAPAY) has ISSUE-1001–1003 (infra/postgres/seed.sql + 07-extended-demo-seed.sql). Schema design: docs/database-schema.md.

Azure: Set AZURE_OPENAI_* in .env for chat (DEFAULT_PROVIDER=azure), voice (mic in assistant), and image generation. See .env.example.

Assessment mapping

Requirement	Location
LLM agent + dynamic tools	`apps/api/agent/`
Reusable skills (5 workflows)	`apps/api/skills/` — escalation, SLA, triage, handoff, health
MCP server	`apps/mcp-server/` + `infra/mcp/`
PostgreSQL schema & seed	`infra/postgres/`, `docs/database-schema.md`
Keycloak RBAC	`infra/keycloak/`, `apps/api/auth/`
Evaluation	`evals/` (10-question suite)
Observability	Langfuse, LangSmith, Phoenix, OTel/Jaeger/Tempo, Grafana LGTM — `apps/api/telemetry/`
Architecture & AI notes	`docs/`

Name		Name	Last commit message	Last commit date
Latest commit History 25 Commits
.cursor/skills/langfuse		.cursor/skills/langfuse
.github		.github
apps		apps
deliverables		deliverables
docs		docs
evals		evals
evaluation/logs		evaluation/logs
infra		infra
scripts		scripts
.env.example		.env.example
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
Architecture.png		Architecture.png
CHANGELOG.md		CHANGELOG.md
DB-Diagram.png		DB-Diagram.png
Makefile		Makefile
README.md		README.md
codecov.yml		codecov.yml
dashboard-2.png		dashboard-2.png
dashboard.png		dashboard.png
docker-compose.e2e.yml		docker-compose.e2e.yml
docker-compose.yml		docker-compose.yml
mypy-baseline.txt		mypy-baseline.txt
pyproject.toml		pyproject.toml
requirements-dev.txt		requirements-dev.txt
uv.lock		uv.lock

Navigation Menu

Repository files navigation

Acme Operations Copilot

Quick start

Repository layout

Commands

Kubernetes (production infra)

Testing & quality

Observability

OpenTelemetry

MCP (agent integration)

RAG (Qdrant)

Grafana (LGTM stack)

Quick start

Configuration

Arize Phoenix

Configuration

How it works

View traces

Evaluation

Demo prompts

Assessment mapping

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

Acme Operations Copilot

Quick start

Repository layout

Commands

Kubernetes (production infra)

Testing & quality

Observability

OpenTelemetry

MCP (agent integration)

RAG (Qdrant)

Grafana (LGTM stack)

Quick start

Configuration

Arize Phoenix

Configuration

How it works

View traces

Evaluation

Demo prompts

Assessment mapping

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages