🐤 Agent Canary

Uptime Robot, but for AI chatbot quality.

Monitor your AI chatbots and agents by sending canary queries and checking response quality using semantic similarity scoring.

Quick Start

# Install deps
pip install -r requirements.txt

# Add a target chatbot
python canary.py add-target --url "https://api.example.com/chat" --name "My Chatbot"

# Add canary queries with expected answers
python canary.py add-canary --target "My Chatbot" \
  --query "What are your hours?" \
  --expected "We are open 9-5 Monday to Friday"

python canary.py add-canary --target "My Chatbot" \
  --query "How do I reset my password?" \
  --expected "Go to Settings > Account > Reset Password"

# Run a one-time check
python canary.py check --target "My Chatbot"

# Continuous monitoring (every 5 minutes)
python canary.py monitor --interval 5

# View quality report
python canary.py report --target "My Chatbot"

CLI Commands

Command	Description
add-target	Add a chatbot/agent endpoint to monitor
list-targets	List all configured targets
add-canary	Add a canary query with expected answer
list-canaries	List canary queries for a target
check	Run canary checks once
monitor	Continuous monitoring at interval
report	Show quality scores over time
add-alert	Configure alert channel (email/webhook/slack)
score	Quick similarity test between two texts

Alerting

Configure alerts to fire when quality degrades:

# Slack webhook
python canary.py add-alert --channel slack \
  --config '{"webhook_url": "https://hooks.slack.com/services/xxx"}'

# Generic webhook
python canary.py add-alert --channel webhook \
  --config '{"url": "https://hooks.example.com/canary"}'

# Email (SMTP)
python canary.py add-alert --channel email \
  --config '{"smtp_host":"smtp.gmail.com","smtp_port":587,"smtp_user":"x","smtp_pass":"x","from_addr":"canary@x.com","to_addr":"admin@x.com"}'

How It Works

1. Sends canary queries to your chatbot's API endpoint
2. Captures the response and measures latency
3. Scores response quality using semantic similarity (sentence-transformers)
4. Tracks scores over time in SQLite
5. Detects quality degradation via moving average
6. Fires alerts when quality drops below threshold

Architecture

canary.py      — CLI interface (click)
checker.py     — Core check engine
scorer.py      — Semantic similarity scoring
alerts.py      — Email, webhook, Slack alerts
db.py          — SQLite database layer
canary.db      — SQLite database (auto-created)