An open, locally-hosted reference implementation that given one alarm (with its sensor frame and asset metadata) it gathers context, runs specialist checks, and issues a structured evidence package that a policy gate routes to either an automatic work-order or a human technician. The tools and the intelligence are what make it usable — a per-alarm job is fast enough to call from an operator UI.
This implementation keeps that architecture but replaces the closed, vendor-locked stack with open equivalents:
| role in the original pattern | open source used here |
|---|---|
| Agent orchestration / tracing / HTTP | OpenAI Agents SDK (Agent, Runner, as_tool, trace processors, FastAPI) |
| Reasoning + light-triage brain | google/gemma:12b (gemma4:12b-mlx) via Ollama's OpenAI-compatible endpoint |
| Reasoning and fast tier (one family) | one open model, two temperature tiers |
| Document parsing / OCR (scanned playbooks) | Tesseract (+ optional Ollama vision model glm-ocr) |
| Unstructured RAG (procedure manuals) | Ollama embeddings (qwen3-embedding:0.6b) + a numpy vector store |
| Semi-structured search (past remedy tickets) | same vector store, second corpus |
| Specialist signal analysis | SciPy (signal.welch, FFT) + NumPy z-score anomaly detection |
| Text-to-SQL over the warehouse | an LLM-driven / template SQL tool over SQLite |
| Secure sandboxed runtime + declarative policy | a YAML policy engine (policy.yaml) gating every tool call and action |
| Content-safety model | a transparent, auditable heuristic gate (injection / exfiltration / policy) |
| Agent evaluator over traces | a trace-based evaluation script reading the SDK's span log |
| accelerated data handling (reference design runs on GPU) | CPU NumPy/Pandas stand-ins (swap-in ready) |
Nothing leaves the machine except the model's own calls to the local Ollama server. The brain is a 12B open model; the specialists are classical DSP tools.
Design note: the reference design mentions a larger “super” model for complex reasoning and a small “nano” model for light tasks. With a single local 12B model we express the same two tiers as the same model at different temperatures / roles rather than two different checkpoints.
Machinery is increasingly connected — compressors, motors, bearings — so technicians are flooded with data yet must still filter it to find the signal that matters. Hundreds of alarms per hour, thousands of sensor readings, and a mix of documentation formats. For each important alarm a technician asks:
- Has this asset/class hit this alarm before? What was done? What fixed it?
- What does the specific playbook say for this error code, plant and location?
- Is the signal real? A “temperature high” — anomaly or trend? Maintenance in progress?
Then they must write a recommendation (usually a work-order). The per-alarm analysis agent automates exactly that generic loop: filter, gather, confirm with a specialist, and draft a reusable recommendation — so a human can focus on the hard cases.
The system is a single-call, per-alarm analysis pipeline. Every stage below lives in its own box; the numbered arrows are the order a request flows through them. Nothing crosses the machine boundary except the model's own calls to the local Ollama server.
Each box is one component; → is the direction of data/control.
┌──────────────────────────┐ POST /analyze ┌────────────────────────────┐
│ UPSTREAM CONSUMERS │ 1 alarm payload │ FastAPI SERVER (server.py) │
│ • SCADA / IIoT stream │ ──────────────────▶ │ • AlarmPayload (pydantic) │
│ • operator-UI button │ {event, asset, │ • POST /analyze │
│ • batch pre-analyzer │ alarm_code, value} │ • GET /policy , /health │
└──────────────────────────┘ └─────────────┬──────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ PIPELINE pipeline.py :: analyze_alarm() │
│ orchestrate: agent → extract → gate → route │
└───┬───────────────┬───────────────────────┬───────────┘
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ AGENT RUN │ │ GATE │ │ ROUTING │
│ (see §2 + §3) │→ │ (see §4) │→ │ (decision → tgt) │
└───────────────┘ └──────────────────┘ └────────┬────────┘
│ │
▼ ▼
┌───────────────────┐ ┌────────────────────────┐
│ TRACE PROCESSOR │ read by │ OUTGOING ROUTES │
│ trace.py (1 JSON │ ─────────────▶ │ • work_order_system │
│ file / run) │ ┌────────────┐ │ • technician_queue │
└───────────────────┘ │ EVALUATOR │ └────────────────────────┘
│ eval.py │
│ trace-based │
└──────────────┘
The whole thing is exposed as one HTTP endpoint (POST /analyze).
Anything upstream — an in-stream filter, an on-click button in the operator UI,
or a batch pre-analyzer — calls it the same way. The pipeline
(pipeline.py:analyze_alarm) is the only thing that server, batch driver, and
tests call; it returns the full response
{alarm, evidence, evidence_package, gate, routing, raw_model_output, traces_dir, wall_time_s}.
Three agents, one open model (gemma4:12b-mlx via Ollama). The root
investigator owns Phase 1 and Phase 3 as its own tool calls; Phase 2 is a
handoff to two specialist subagents exposed to the investigator as
agent-as-tool wrappers (as_tool). Evidence accumulates on the run
AlarmContext, never in the prompt.
┌───────────────────────────────────────────────┐
│ OLLAMA (local, OpenAI-compatible /v1) │
│ gemma4:12b-mlx qwen3-embedding:0.6b │
│ glm-ocr (opt) ← nothing leaves the box │
└───────────────▲───────────────▲───────────────┘
│ reasoner tier │ fast tier
│ │
┌──────────────────────────────────────────────────────────────────────────┐
│ INVESTIGATOR (root agent) model = reasoner max_turns = 12 │
│ alarm_agent/agents.py :: build_agents() │
│ │
│ PHASE 1 — gather evidence (its direct tool calls) │
│ ├─ gather_alarm_history SQL over event/time-series warehouse │
│ ├─ text_to_sql NL → SQL (read-only warehouse) │
│ ├─ search_playbooks RAG over OCR'd procedure manuals │
│ └─ search_remedy_history RAG over past remedy / work-order tix │
│ │
│ PHASE 2 — specialist analysis (handoff to subagents, as tools) │
│ ├─ consult_vibration_specialist ─▶ VIBRATION SPECIALIST ─▶ analyze_vibration │
│ └─ consult_anomaly_specialist ─▶ ANOMALY SPECIALIST ─▶ analyze_anomaly │
│ │
│ PHASE 3 — action synthesis (its direct tool call, run ONCE) │
│ └─ synthesize_evidence_package ─▶ {observation, root_cause, │
│ remedy, recommended_action, │
│ action_type, confidence} (JSON) │
└───────────────┬───────────────────────────────────────┬───────────────────┘
│ handoff + JSON findings │ evidence_package (JSON)
▼ ▼
┌──────────────────────────────┐ ┌────────────────────────────────────────┐
│ VIBRATION SPECIALIST (sub) │ │ ANOMALY SPECIALIST (sub) │
│ model = reasoner │ │ model = fast │
│ tool: analyze_vibration │ │ tool: analyze_anomaly │
│ • SciPy signal.welch (FFT) │ │ • NumPy z-score vs recent history │
│ • BPFO / BPFI defect peaks │ │ • outlier(real) vs artefact/trend │
│ • rotor balance / bearing │ │ • verdict + mean/std/z │
└──────────────────────────────┘ └────────────────────────────────────────┘
Why agents-as-tools: a 12B model is unreliable with strict output schemas, so
every structured boundary is a tool that returns JSON text instead of an
output schema the model must honor. synthesize_evidence_package also records the
package onto the context, which is the preferred source extract_package reads
(agents.py:extract_package).
Phase 1 — gather evidence (tools.py). The investigator answers
“what happened before, and what do the playbooks say?”, pulling from two data
stores. Every call first passes the policy allow-list (_gate).
┌─────────────────────────┐ ┌───────────────────────────┐
│ EVENT / TIME-SERIES │ gather_alarm_history │ VECTOR STORE (numpy) │
│ WAREHOUSE (SQLite) │ ─────────────────────▶ │ data/index │
│ alarm_event, asset, │ text_to_sql │ ├─ corpus: playbooks │
│ sensor_frame │ (NL → SQL, read-only) │ │ ← OCR'd manuals │
└─────────────────────────┘ │ └─ corpus: remedies │
▲ │ ← past work-order tix │
│ seed.py └──────────────▲──────────────┘
│ (synthetic plant, embed, OCR) │ embed / search
│ qwen3-embedding
┌───────────┴──────────┐
│ EMBEDDER + OCR │
│ Ollama / Tesseract │
└──────────────────────┘
Phase 2 — specialist analysis (tools.py). A generic alarm
(“Temperature High”, “Vibration Spike”) is confirmed in context by a specialist
before any recommendation:
consult_vibration_specialist→analyze_vibration: FFT of the accelerometer series, extracts bearing defect frequencies (BPFO/BPFI) → “is the bearing actually failing, or a sensor artefact / rotor-balance issue?”.consult_anomaly_specialist→analyze_anomaly: z-score of the reported value against its history → “outlier (real) or within the normal band (artefact/ trend)?”.
Phase 3 — action generation (tools.py::synthesize_evidence_package). The
investigator assembles a structured package — observation, root-cause
hypothesis, remedy, recommended action, action_type, confidence — as JSON, then
that package moves to the gate. action_type is constrained to
[raise_work_order, schedule_maintenance, escalate_to_technician].
The gate (gate.py::evaluate) decides what happens to the package. Order
matters: action policy → content safety → confidence. Each stage is its own
decision box; the first hit wins.
evidence_package
│
▼
┌──────────────────────────────────┐ action_type in deny-list?
│ ① ACTION POLICY │ ─── yes ──▶ ┌──────────────────────────┐
│ policy.check_action() │ │ REFUSE │
│ deny: actuate_equip, │ ◀── no ──── │ blocked_action set, │
│ reset_interlock, │ │ escalate to technician │
│ override_alarm │ │ (with evidence attached) │
└───────────────┬──────────────────┘ └──────────────────────────┘
│ pass
▼
┌──────────────────────────────────┐ injection / exfiltration / offensive?
│ ② CONTENT SAFETY │ ─── yes ──▶ ┌──────────────────────────┐
│ policy.check_content() │ │ REFUSE │
│ (transparent heuristics; no │ ◀── no ──── │ refuse, escalate to tech │
│ proprietary model) │ └──────────────────────────┘
└───────────────┬──────────────────┘
│ pass
▼
┌──────────────────────────────────┐ confidence ≥ 0.70 & not escalate?
│ ③ CONFIDENCE │ ─── yes ──▶ ┌──────────────────────────┐
│ auto_dispatch_min_confidence │ │ AUTO-DISPATCH │
│ = 0.70 ; escalate below │ ◀── no ──── │ → work_order_system │
│ + escalate_to_technician always │ └──────────────────────────┘
│ routes to a human │
└───────────────┬──────────────────┘
│ confidence < 0.70 / agent asked for human
▼
┌──────────────────────────────────┐
│ ESCALATE │ ──▶ technician_queue (evidence pre-attached)
│ decision = "escalate" │
└──────────────────────────────────┘
Everything in §2–§4 is enforced by one policy engine (policy.py, backed by
policy.yaml). It is the secure runtime: the model can only ask; the
policy decides, before any tool runs and before any action dispatches.
┌───────────────────────────────────────────────────────────────────────────┐
│ POLICY ENGINE policy.py :: Policy (loaded from policy.yaml) │
│ │
│ ① TOOL ALLOW-LIST allowed_tools (7) — anything else denied │
│ ② DATA-SOURCE SCOPE warehouse / playbooks / remedies │
│ read-only + NO outbound network (no exfiltration) │
│ ③ ACTION ALLOW/DENY allow: raise_work_order, schedule_maintenance, │
│ escalate_to_technician │
│ deny: actuate_equip, reset_interlock, │
│ override_alarm (physical/destructive) │
│ ④ CONTENT SAFETY block: prompt_injection, exfiltration, │
│ offensive ; max_output_chars = 4000 │
└───────────────┬───────────────────────────────────────────────┬───────────┘
│ check_tool() before every tool │ check_action /
│ (tools.py::_gate wraps all 7 tools) │ check_content after
▼ ▼
[ Phase 1 & 2 tools ] [ Phase 3 package → gate ]
Every run writes one JSON trace file to data/traces/
(trace.py), capturing generations, tool calls, and handoffs — the open
substitute for a proprietary trace/eval backend. The evaluator
(eval.py) reads those traces to score the agent.
run_alarm() trace.py eval.py
──────────▶ data/traces/alarm_<id>.json ──────▶ score traces
(Runner.run_sync, tracing on) (root cause / action / gate)
The learning loop: the “past remedy” corpus (remedies) is the agent's
growing memory. Each time a diagnosis proves right it can be added to remedies,
improving future search_remedy_history hits — so the agent improves in
cost/performance as more alarms pass through it.
Running the demo over a 3-event critical stream (MTR-101 vibration spike, CMP-101 temperature high, CMP-102 pressure low):
ANALYZING E1001 A-VIB-SPIKE @ MTR-101
root_cause : Rotor imbalance / coupling misalignment (175.78 Hz dominant)
remedy : Prior 11.2 mm/s cases resolved by rotor re-balancing (rm-003)
action : schedule_maintenance (confidence 0.90)
DECISION : AUTO_DISPATCH -> work_order_system
ANALYZING E1002 A-TEMP-HIGH @ CMP-101
root_cause : reading judged a sensor artefact by the anomaly specialist
remedy : clean heat exchanger / repair lube oil pump
action : schedule_maintenance (confidence 0.85)
DECISION : AUTO_DISPATCH -> work_order_system
ANALYZING E1003 A-PRESS-LOW @ CMP-102
root_cause : seal leak / discharge-valve fault
remedy : per past remedy rm-004, discharge seal leak repaired
action : raise_work_order (confidence 0.85)
DECISION : AUTO_DISPATCH -> work_order_system
All three events reach a gated decision with evidence attached; each trace shows
all three phases plus RAG and specialist checks ran. The deterministic
gate_demo additionally shows escalate (low confidence) and refuse
(denied physical action / exfiltration) so every path is visible.
-
Ollama running, with these models pulled:
ollama pull gemma4:12b-mlx # brain (reasoner + fast tier) ollama pull qwen3-embedding:0.6b # semantic search ollama pull glm-ocr:latest # optional: vision OCR
-
Tesseract on
PATH(brew install tesseracton macOS). -
Python 3.11+ and a network connection to the Ollama HTTP endpoint (
http://11434by default; it exposes an OpenAI-compatible/v1).
./run.shIt creates a venv, installs deps, applies the openai-agents/openai compat patch, seeds the synthetic plant, runs the gate matrix, the end-to-end demo, and the evaluator.
python3 -m venv .venv && ./.venv/bin/pip install -r requirements.txt
./.venv/bin/python scripts/patch_agents_sdk.py # openai>=2.19 compat
./.venv/bin/python -m alarm_agent.seed # build warehouse + indexes + OCR
./.venv/bin/python -m alarm_agent.gate_demo # deterministic gate matrix
./.venv/bin/python -m alarm_agent.demo # end-to-end over event stream
./.venv/bin/python -m alarm_agent.eval # trace-based evaluation
./.venv/bin/python -m alarm_agent.server # serve HTTP endpointcurl -X POST localhost:8080/analyze -H 'Content-Type: application/json' \
-d '{
"event_id":"E1", "asset_id":"MTR-101",
"alarm_code":"A-VIB-SPIKE", "alarm_name":"Vibration Spike",
"location":"unit_3", "serial_group":"SERIAL-GRP-1",
"value":11.2, "unit":"mm/s"
}'Returns the evidence package, the gate decision, the routing target, and the raw
model output. GET /policy prints the live allow/deny lists; GET /health is a
liveness probe.
config.yaml Open model endpoints, gate thresholds, paths
policy.yaml Declarative security policy (the sandbox)
run.sh One-command setup + demo
scripts/patch_agents_sdk.py openai-agents/openai version-compat shim
alarm_agent/
config.py Config loader
llm.py Model factory (Ollama) + embeddings + OCR (tesseract/vision)
data_store.py SQLite warehouse + text-to-SQL
vector_store.py numpy vector store (playbooks + remedies)
seed.py Synthetic plant + OCR'd scanned playbook + indexes
tools.py The seven agent tools (3 phases) with policy gating
agents.py investigator + 2 specialist subagents (agent-as-tool)
pipeline.py analyze_alarm(): agent -> extract -> gate -> route
gate.py Policy/confidence/content-safety decision
gate_demo.py Deterministic decision-matrix demo
trace.py File-based trace processor (eval input)
eval.py Trace-based evaluator
server.py FastAPI HTTP endpoint
demo.py End-to-end batch driver
data/ generated warehouse.db, indexes, traces/, logs/
| original concept | this implementation |
|---|---|
| agent orchestration library | OpenAI Agents SDK |
| closed reasoning model family | google/gemma:12b via Ollama (OpenAI-compatible) |
| fast/nano vs super tiers | one model, two temperature tiers |
| document parse model | Tesseract (+ optional Ollama vision OCR) |
| GPU vector index (reference design) | numpy cosine store + Ollama embeddings (CPU) |
| GPU signal/anomaly libs (reference design) | SciPy FFT + NumPy z-score (CPU) |
| text-to-sql tool | SQLite + LLM/template SQL |
| secure runtime + YAML policy | policy.yaml gate engine |
| content-safety model | auditable heuristic gate |
| trace/eval backend | file trace processor + eval.py |
The agent is the part of the system that improves in cost and performance as more alarms are put through it.