Skip to content

PROGRESS.md

Latest commit

 

History

History
250 lines (205 loc) · 11.3 KB

PROGRESS.md

File metadata and controls

250 lines (205 loc) · 11.3 KB
Raw

ACSA — Progress Log

Paste this entire file at the start of every new Claude session to resume exactly where you left off.

Project

Autonomous Customer Support Agent with Escalation (ACSA) — multi-agent AI platform.

Status: ✅ ALL PHASES + IMPROVEMENTS COMPLETE

Phases

Phase Name Status
0 Foundation + Knowledge Base ✅ Complete
1 Intent Classifier + Router ✅ Complete
2 Autonomous ReAct Agent + Tools ✅ Complete
3 Confidence Checker + Escalation ✅ Complete
4 Eval Dashboard + Frontend ✅ Complete

Production Improvements

# Improvement Status
1 Chat history in Redis + session persistence ✅ Complete
2 Human queue viewer in Dashboard ✅ Complete
3 Better /health — checks Redis, ChromaDB, Groq ✅ Complete
4 Rate limiting with slowapi ✅ Complete
5 GitHub Actions CI — ruff + tests on every push ✅ Complete

Completed Steps

Phase 0 — Foundation + Knowledge Base

  • ✅ Project scaffold — all folders + __init__.py files created
  • ✅ uv sync --extra dev — 122 packages installed
  • ✅ .env + backend/config.py — Pydantic Settings, all config values typed
  • ✅ backend/main.py — FastAPI app with /health endpoint verified
  • ✅ backend/db/chroma.py — ChromaDB singleton client + search_faq() helper
  • ✅ scripts/ingest_faq.py — 75 FAQ entries ingested across 15 topics
  • ✅ backend/db/redis_client.py — sync Redis client, queue + metrics + chat history helpers
  • ✅ GitHub repo created and all code pushed

Phase 1 — Intent Classifier + Router

  • ✅ backend/agents/state.py — AgentState TypedDict with all fields
  • ✅ backend/agents/classifier.py — Groq LLM call returning topic + confidence score
  • ✅ backend/agents/graph.py — LangGraph with conditional routing (≥0.7 → agent, <0.7 → escalate)
  • ✅ Verified: clear messages → 0.9 confidence → AGENT, gibberish → 0.2 → HUMAN QUEUE

Phase 2 — Autonomous ReAct Agent + Tools

  • ✅ backend/tools/kb_search.py — ChromaDB semantic search as LangChain tool
  • ✅ backend/tools/order_status.py — mock order lookup by order ID
  • ✅ backend/tools/ticket_creator.py — support ticket creation stored in Redis
  • ✅ backend/agents/react_agent.py — ReAct loop, autonomously chains tools, chat history context
  • ✅ backend/agents/graph.py — real ReAct agent wired in, placeholder removed
  • ✅ Verified: agent chained 3 tools autonomously on complaint query

Phase 3 — Confidence Checker + Escalation

  • ✅ backend/agents/critic.py — Groq LLM scores draft response quality (0.0–1.0)
  • ✅ backend/agents/escalator.py — packages full context to Redis human queue
  • ✅ backend/agents/graph.py — full pipeline wired: classify → agent → critic → pass/escalate
  • ✅ Verified: critic scored 0.9 and 0.8 on clear queries (PASS), gibberish escalated with correct reason

Phase 4 — WebSocket API + Streamlit Frontend

  • ✅ backend/api/metrics.py — GET /metrics + GET /queue endpoints
  • ✅ backend/api/websocket.py — WebSocket endpoint running full LangGraph pipeline
  • ✅ backend/main.py — metrics and websocket routers wired in, rate limiting
  • ✅ backend/db/redis_client.py — switched to sync Redis, no more event loop conflicts
  • ✅ frontend/app.py — Streamlit chat UI + performance dashboard + human queue viewer
  • ✅ Verified: full end-to-end pipeline working — chat, order lookup, escalation, metrics

Production Improvements

  • ✅ Chat history stored in Redis per session — agent remembers full conversation context
  • ✅ Session ID persists across Streamlit reruns via st.session_state
  • ✅ Human Queue tab shows full escalated conversation with history, tools used, timestamps
  • ✅ /health checks Redis, ChromaDB, and Groq key status
  • ✅ Rate limiting: /health 60/min, /metrics 60/min, /queue 30/min
  • ✅ GitHub Actions CI — ruff + ChromaDB tests run on every push (2 min, green)
  • ✅ Full test suite: scripts/run_all_tests.py — 25/25 tests passing
  • ✅ ReAct agent retry logic — falls back to no-history on tool call format errors

Current Folder Structure

ACSA/
├── .github/
│   └── workflows/
│       └── ci.yml            ✅ done
├── backend/
│   ├── agents/
│   │   ├── __init__.py
│   │   ├── state.py          ✅ done
│   │   ├── classifier.py     ✅ done
│   │   ├── graph.py          ✅ done
│   │   ├── react_agent.py    ✅ done (chat history + retry logic)
│   │   ├── critic.py         ✅ done
│   │   └── escalator.py      ✅ done
│   ├── tools/
│   │   ├── __init__.py
│   │   ├── kb_search.py      ✅ done
│   │   ├── order_status.py   ✅ done
│   │   └── ticket_creator.py ✅ done
│   ├── api/
│   │   ├── __init__.py
│   │   ├── websocket.py      ✅ done (session_id from client)
│   │   └── metrics.py        ✅ done (/metrics + /queue, rate limited)
│   ├── db/
│   │   ├── __init__.py
│   │   ├── chroma.py         ✅ done
│   │   └── redis_client.py   ✅ done (sync, chat history, queue viewer)
│   ├── __init__.py
│   ├── config.py             ✅ done
│   └── main.py               ✅ done (health check, rate limiting)
├── frontend/
│   └── app.py                ✅ done (Chat + Dashboard + Human Queue tabs)
├── data/
│   ├── sample_faq.json       ✅ 75 entries, 15 topics
│   └── mock_orders.json      ✅ 20 orders
├── scripts/
│   ├── ingest_faq.py         ✅ done
│   ├── test_search.py        ✅ done
│   ├── test_redis.py         ✅ done
│   ├── test_classifier.py    ✅ done
│   ├── test_graph.py         ✅ done
│   ├── test_tools.py         ✅ done
│   ├── test_react_agent.py   ✅ done
│   ├── test_queue.py         ✅ done
│   ├── clear_queue.py        ✅ done
│   └── run_all_tests.py      ✅ done (25/25 passing)
├── tests/
│   └── unit/
├── PROGRESS.md
├── pyproject.toml
└── .env

Gotchas (never repeat these)

  1. PowerShell: does not support && — always run commands together on separate lines
  2. PowerShell: does not support brace expansion for mkdir — use explicit paths
  3. uv PATH: run $env:Path = "C:\Users\suraj\.local\bin;$env:Path" in every new terminal
  4. ChromaDB: runs fully local, no Docker needed — already installed via uv sync
  5. ChromaDB: PersistentClient is a function not a class — don't use it in type hints with | syntax; just use = None untyped
  6. Groq free tier: 100k tokens/day limit — switch API key if limit hit
  7. Redis on Windows: installed as a Windows service via winget — already running, don't run redis-server manually
  8. Redis on Windows: old version doesn't support HELLO command — always set protocol=2 in both sync and async Redis clients
  9. scripts/: always add sys.path.insert(0, str(Path(__file__).parent.parent)) at top — scripts can't find backend otherwise
  10. ruff: always run uv run ruff check --fix before committing — E501 line length errors need manual fixing
  11. LangGraph state: must use TypedDict, not Pydantic
  12. LangGraph: always call builder.compile() — validates all edges and nodes
  13. sync Redis: use sync redis everywhere — asyncio.run() inside FastAPI/LangGraph event loop causes "attached to different loop" error
  14. LangGraph recursion_limit: pass config={"recursion_limit": 10} to agent.invoke() to prevent infinite tool loops
  15. Streamlit session_id: use st.session_state for session_id — do NOT use st.rerun() with query params, it resets state
  16. WebSocket session_id: backend must read session_id from message data, not generate it at connection time
  17. Groq tool_use_failed: happens when chat history confuses LLM tool call format — retry without history as fallback
  18. GitHub Secrets: GROQ_API_KEY must be added as Repository secret AND Environment secret for CI to work

Key Decisions

  • Confidence threshold for escalation: 0.7 (set in config.py, tunable)
  • Critic pass threshold: 0.6 (below this → escalate mid-flow)
  • Human queue stored in Redis as a list using LPUSH / LRANGE pattern
  • Human queue Redis key: acsa:human_queue
  • Chat history Redis key: acsa:chat:{session_id} — expires after 24 hours
  • Chat history max: 10 messages per session (last 6 passed to agent)
  • No Docker for this project — everything runs locally
  • ChromaDB replaces Qdrant — simpler, no container
  • Groq replaces Gemini as primary LLM — faster, no daily limit (100k tokens/day)
  • LangGraph state uses TypedDict (not Pydantic)
  • Embedding model: all-MiniLM-L6-v2 (local, no API key, ~90MB, cached after first run)
  • Frontend: Streamlit (chosen over React + TypeScript — faster, Python-native)
  • Rate limits: /health 60/min, /metrics 60/min, /queue 30/min
  • Classifier valid topics: refund, shipping, order, payment, account, product, complaint, returns, loyalty_and_rewards, invoice_and_gst, privacy_and_data, app_and_technical, seller_and_marketplace, gift_and_special_orders, sustainability_and_packaging, general

How to Resume Local Dev After a Break

# 1. Reload uv PATH (if needed)
$env:Path = "C:\Users\suraj\.local\bin;$env:Path"

# 2. Redis is a Windows service — check it's running
Get-Service -Name Redis

# 3. Start backend (Terminal 1)
uv run uvicorn backend.main:app --reload

# 4. Start frontend (Terminal 2)
uv run streamlit run frontend/app.py

# 5. Run full test suite
uv run python scripts/run_all_tests.py

Service URLs

Service URL
Backend API docs http://localhost:8000/docs
Health check http://localhost:8000/health
Metrics API http://localhost:8000/metrics
Human Queue API http://localhost:8000/queue
WebSocket ws://localhost:8000/ws/chat
Frontend http://localhost:8501
GitHub Actions https://github.com/codewithleo1/ACSA/actions

Full Pipeline

User → Streamlit UI (💬 Chat tab)
     → WebSocket → Classifier (Groq)
                 → confidence ≥ 0.7 → ReAct Agent (tools: kb_search, order_status, ticket_creator)
                                     → Critic (Groq) → score ≥ 0.6 → Response to User
                                                      → score < 0.6 → Escalator → Redis Queue
                 → confidence < 0.7 → Escalator → Redis Queue
                                                 → Human Queue tab (📊 Dashboard)

Remaining (Deployment — free tier)

  • Backend — Railway or Render free tier
  • Redis — Redis Cloud free tier (30MB)
  • ChromaDB — keep local or Chroma hosted free tier
  • Frontend — Streamlit Community Cloud (completely free)

Last updated: All phases + 5 production improvements complete. 25/25 tests passing. CI green. Deployment pending.