AI Operations Monitor & Process Performance Dashboard

A self-contained system that runs an LLM-based agent pipeline against a simulated business process (invoice validation), watches its accuracy in real time, detects anomalies, fires alerts, and automatically rewrites its own prompt when quality drops.

Everything here runs in Docker. Nothing needs a paid API key or a paid tool. The default LLM is Groq (free, no card, dedicated inference hardware — doesn't throw the "model overloaded / 503" errors free Gemini tiers can throw under load); Gemini is available as an alternate provider via one .env setting.

What it does

1. Simulator generates synthetic invoices (app/simulator). Most are clean. A configurable share carry a known defect (amount mismatch, duplicate ID, missing PO number, suspicious vendor, currency mismatch). The correct label is stored as ground truth, but never shown to the model.
2. Pipeline (app/pipeline, built with LangGraph) sends each invoice through: parse -> validate with the active LLM provider (app/llm/router.py dispatches to Groq or Gemini) -> enforce a governance rule (low confidence is forced to "review", never auto-approved) -> score against ground truth -> log.
3. Monitoring (app/monitoring) tracks rolling accuracy and confidence across batches. When accuracy drops below a threshold, it raises an alert and flags the batch for optimization.
4. Optimization loop (app/optimization) reacts to that flag: it pulls the invoices the model got wrong, builds a corrected few-shot block from them, and asks the active LLM provider to fold that into a tightened instruction. The new prompt is stored as a new version and used for the next batch. Every version is kept, with the accuracy it produced, so you can see the lineage.
5. Governance log (app/governance + Postgres) stores every decision the model made, the prompt version used, its confidence, and whether it was later graded correct.
6. Observability: every LLM call (Groq or Gemini) is traced in Arize Phoenix (latency, tokens, input/output). Every batch's metrics are exported to Prometheus and shown in a pre-built Grafana dashboard.
7. Reports (app/reports) auto-generate a Markdown and PDF run report per batch and expose them over the API.

Why this maps to the AI Operations & Optimization Analyst JD

JD requirement	Where it lives
Monitor daily AI/process performance	Prometheus metrics + Grafana dashboard, /pipeline/run batch loop
Identify & resolve AI issues	app/monitoring/anomaly.py, app/monitoring/alerts.py
Improve prompt accuracy	app/optimization/prompt_optimizer.py, prompt_versions table
Business process context	app/simulator/generator.py (invoice validation)
Responsible AI compliance	app/governance/logger.py (confidence floor, full audit trail)
Documentation	auto-generated run reports in app/reports

Stack

• FastAPI (API layer)
• LangGraph (agent pipeline orchestration)
• Ollama (default LLM provider — fully local, self-hosted in its own container, no API key, no rate limits, no deprecation risk), or Groq / Gemini as alternate cloud providers (one .env line to switch)
• Arize Phoenix, self-hosted (LLM observability, traces all three providers)
• Prometheus + Grafana (metrics dashboard)
• PostgreSQL (process data + governance log + prompt registry)
• APScheduler (runs a batch automatically every N minutes, like a daily job)

Every one of these is open source and self-hosted in Docker. With the default Ollama provider, the app makes zero external network calls — the whole system runs offline once the model is pulled. With Groq/Gemini, the only external call is to that provider, using your own free key.

Setup

Default provider is Ollama — fully local, no API key, no rate limits, no external dependency. This is the recommended path if you just want it to work.

1. Copy .env.example to .env (the defaults already point at Ollama, nothing to fill in):

   cp .env.example .env
   

2. Build and start everything:

   docker compose up --build
   

   On first run, the ollama-pull service downloads the model (~2GB for the default llama3.2:3b) into a Docker volume — this can take a few minutes depending on your connection. Watch it with:

   docker compose logs -f ollama-pull
   

   Wait for it to print success before triggering a batch — the app itself starts immediately, but /pipeline/run will fail until the model finishes pulling.

3. Open:

   • API docs: http://localhost:8000/docs
   • Grafana: http://localhost:3000 (user: admin, pass: admin)
   • Prometheus: http://localhost:9090
   • Phoenix (traces): http://localhost:6006

4. Kick off a batch:

   curl -X POST http://localhost:8000/pipeline/run -H "Content-Type: application/json" -d "{\"batch_size\": 20}"
   

   A batch also runs automatically every SCHEDULER_INTERVAL_MINUTES (default 10) once the stack is up — see app/scheduler/jobs.py.

5. Watch it self-correct: lower ACCURACY_THRESHOLD in .env (e.g. to 0.95) so a normal batch trips it, run a couple of batches, and check GET /governance/prompt-versions to see a new prompt version appear with the correction baked in.

6. Pull a report for any batch:

   curl http://localhost:8000/reports/<batch_id>          # markdown
   curl -O -J http://localhost:8000/reports/<batch_id>/pdf # PDF download
   

Want faster / higher-quality responses instead of local?

Switch to Groq or Gemini any time — both are free, just less reliable than a model that lives on your own machine:

• Groq: https://console.groq.com/keys (no credit card) → set LLM_PROVIDER=groq and GROQ_API_KEY=<your key> in .env.
• Gemini: https://aistudio.google.com/apikey (no credit card) → set LLM_PROVIDER=gemini and GEMINI_API_KEY=<your key> in .env.

Then docker compose up -d --build app (no need to rebuild the whole stack).

Switching providers or models later

Everything is env-var driven — no code changes needed:

# .env
LLM_PROVIDER=ollama          # or "groq" or "gemini"
OLLAMA_MODEL=llama3.2:3b
GROQ_MODEL=openai/gpt-oss-120b
GEMINI_MODEL=gemini-flash-latest

Then docker compose up -d --build app. app/llm/router.py is the only place that knows which provider is active — everything else (the LangGraph pipeline, the optimizer, the governance logger, Phoenix tracing) is provider-agnostic.

A note on model names

Cloud free-tier model catalogs change every few months and models get deprecated without much warning — this is exactly what caused the llama-3.3-70b-versatile 404 error if you hit that: Groq retired it in June 2026. No provider name is ever hardcoded outside .env:

• OLLAMA_MODEL defaults to llama3.2:3b, pulled into your own Docker volume — it stays available for as long as you keep the volume, immune to any provider's catalog changes. Browse other local models at https://ollama.com/library.
• GROQ_MODEL defaults to openai/gpt-oss-120b. Check https://console.groq.com/docs/models for the current free-tier catalog, and https://console.groq.com/docs/deprecations if a call ever 404s like the one above.
• GEMINI_MODEL defaults to gemini-flash-latest, an alias Google keeps pointed at whatever current free-tier Flash model is live. For a pinned name, check https://aistudio.google.com/.

Whichever provider is active, app/llm/router.py -> the matching client module runs a one-time startup check that lists your available models and fails fast with a clear message if the configured one isn't on the list, instead of failing confusingly mid-batch. If you change GROQ_MODEL or LLM_PROVIDER in .env, remember docker compose up -d --build app — editing .env alone doesn't apply to an already-running container.

Free-tier friendly by design

• Ollama (default): no tier at all — it's your own hardware. No rate limit, no key, no deprecation risk. The trade-off is speed (CPU inference is slower than Groq/Gemini's dedicated hardware) and a smaller model (3B params vs. 70B-120B on the cloud options), so expect noticeably simpler judgment calls on ambiguous invoices.
• Groq (alternate): free developer tier, no credit card, ~30 requests/minute per model. app/llm/groq_client.py throttles to GROQ_MAX_RPM (default 28) and retries on 429s automatically.
• Gemini (alternate): Google's free tier caps requests per minute per model — commonly 5 RPM for Flash models, reported exactly in the 429 error if you hit it. app/llm/gemini_client.py throttles itself to GEMINI_MAX_RPM (default 4) so a batch spaces its calls out instead of firing them all at once — a batch is noticeably slower than on Groq as a result (batch_size 20 takes roughly (20 x 15s) ≈ 5 minutes at the default rate). If you hit occasional Gemini 503 "model overloaded" errors, that's Google's shared free-tier capacity under load, not a bug here — switching LLM_PROVIDER=groq avoids it entirely.
• The prompt optimizer only calls the LLM extra times when a batch actually trips the accuracy threshold, not on every run.
• Phoenix, Prometheus, and Grafana are all self-hosted community editions with no usage caps.

Project layout

ai-ops-monitor/
├── docker-compose.yml
├── .env.example
├── .gitignore
├── app/                    # FastAPI service
│   ├── main.py
│   ├── config.py
│   ├── db/                 # SQLAlchemy models + session
│   ├── simulator/           # synthetic invoice generator
│   ├── llm/                 # router.py (provider dispatch) + ollama_client.py + groq_client.py + gemini_client.py
│   ├── pipeline/            # LangGraph agent graph + prompt templates
│   ├── monitoring/          # Prometheus metrics, anomaly/drift detection, alerts, Phoenix setup
│   ├── optimization/         # prompt self-correction loop
│   ├── governance/           # decision logging
│   ├── reports/              # Markdown + PDF run report generators
│   ├── scheduler/            # periodic batch job
│   └── routes/                # API endpoints
└── monitoring/               # Prometheus + Grafana provisioning

Security note

.env is gitignored — keep your real keys only there, never in .env.example or committed to version control. If a key is ever accidentally shared or committed, revoke and regenerate it immediately at the provider's console (https://console.groq.com/keys or https://aistudio.google.com/apikey).