FaultCast — Efficient LLM Work-Order Surveillance

Abid Hasan · github.com/abidhasanzim

Free-text maintenance work orders → structured fault records → per-asset-family anomaly detection → alerts. Qwen2.5-3B with an optional QLoRA fine-tune, FastAPI backend, one-command pipeline.

Surveillance runs on the gold records (solid arrow from records.jsonl) so detection is not confounded by extraction noise; the dotted arrow shows that extracted records can feed it too. taxonomy.py is shared: it assigns the asset family used for aggregation and normalises the fault mode used for the alert recommendation.

Synthetic fields. MaintIE masks equipment ids and dates as <id>/<date>, so asset_id, timestamp and severity are fabricated by prepare_data.py, and the anomaly the detectors find is a deliberately planted burst (45% of the hydraulic family moved into a 5-day window). Only text, item, fault_mode and action come from real annotations. The detection results are a test of the detector, not a statement about real equipment.

Results

Detection

Extraction (80 held-out records, JSON validity 100% after one retry)

configuration	item F1	item EM	fault_mode F1	action* F1
prompted (2-shot), fp16	0.791	55.0%	0.891	0.640
prompted (2-shot), nf4 — control	0.755	51.2%	0.876	0.520
QLoRA, nf4	0.967	92.5%	0.983	1.000

* restricted to the 25 records with a non-null gold action; 55 of 80 gold actions are null and both-empty set-F1 scores 1.0, so the unrestricted figure mostly measures agreement on absence.

QLoRA targeted one diagnosed failure: prompting returns the parent assembly (air conditioner thermostat) where the annotations mark the failed part (thermostat). 458 training records, ~4 minutes, item exact-match 51.2% → 92.5%.

Contamination check. The split is disjoint by exact text, but work orders are formulaic — 21% of test records have a ≥0.80-similar training text. On the 53 records that are both textually dissimilar and carry an unseen (item, fault_mode) pair:

field (53 novel records)	prompted nf4	QLoRA
item F1	0.826	0.956
item exact match	62.3%	90.6%
fault_mode F1	0.892	0.975
action* F1 (15 records)	0.600	1.000

Gains shrink but hold. Four test verbs (change, drill out, testing, inspection) never occur in training and are still correct — the model learned to copy the verb span verbatim rather than normalise it. Single seed, no hyperparameter search: treat the effect size as indicative.

Efficiency

50 held-out work orders, greedy, 64 new tokens, median of 3 runs. Measured on RTX 3090, sm_86, 82 SMs, torch 2.9.1+cu126, read from torch.cuda.get_device_properties(0) at measurement time.

variant	latency s/rec	p90 s	peak VRAM	vs fp16
fp16	0.58	0.62	6.21 GB	—
int8	2.40	2.57	3.49 GB	4.15× slower, 0.56× memory
nf4	0.79	0.85	2.14 GB	1.37× slower, 0.35× memory
nf4 + QLoRA	1.20	1.33	2.31 GB	2.07× slower, 0.37× memory

int8 loses to nf4 on both axes measured — ~3× slower and 63% larger. The LLM.int8() mixed-precision decomposition costs more per matmul than 4-bit dequantise-and-matmul at batch size 1. (int8 accuracy was never measured, so this is not strict dominance.) The tuned nf4 model is the most accurate configuration and the second smallest.

On-disk size is 6.17 GB for every row — bitsandbytes quantises at load, so quantisation buys runtime memory, not storage. Peak VRAM is the allocator high-water mark, not total device usage.

Detector caveats

5 alerts, all on the planted burst days, none elsewhere. Two honest qualifications:

• The Poisson flag set is a strict subset of the CUSUM set, so requiring both detectors gives exactly what the Poisson test gives alone. It only suppresses the 6 CUSUM-only alarms an OR-rule would raise.
• CUSUM is mis-calibrated at k=0.5, h=5.0: measured in-control ARL0 ≈200 days against a nominal ≈950, so its clean-family alarms are consistent with noise.

The forecaster is worse than trivial: backtested per family, predicting zero wins on 5 of 9 and the historical mean on 2, against Holt's 2. Daily counts are too sparse and trendless for it to help.

Data

MaintIE (MIT) — expert-annotated mining work orders. gold_release.json holds 1,076 records; 538 carry both an item and a fault mode and form the corpus. Fields are derived from the entity/relation annotations:

field	rule
item	the hasPatient or hasAgent target of the fault entity, else first PhysicalObject
fault_mode	longest State/Process/Property-Undesirable span
action	longest Activity span, null if absent (68% of the corpus)

Both participant roles matter: "blown brake hose" links via hasPatient, but "leaking taps" via hasAgent — the taps do the leaking. Following only hasPatient mislabels the parent assembly. Fault modes span State and Process, which lifts item+fault coverage 33.6% → 50.0%.

Setup

./setup_env.sh          # conda env `exp`, pinned deps
conda activate exp
python -c "from huggingface_hub import snapshot_download; snapshot_download('Qwen/Qwen2.5-3B-Instruct')"

Run

python src/prepare_data.py                       # MaintIE → data/processed/records.jsonl
python run_pipeline.py                           # end to end
python run_pipeline.py --no-llm                  # skip extraction (no GPU)
python src/evaluate.py                           # both result tables

python src/finetune.py                           # QLoRA → checkpoints/lora
python run_pipeline.py --quant nf4 --adapter checkpoints/lora

FAULTCAST_QUANT=nf4 FAULTCAST_ADAPTER=checkpoints/lora uvicorn src.api:app --port 8000
curl -X POST localhost:8000/extract -H 'Content-Type: application/json' \
     -d '{"text":"<id> hydraulic hose leaking on boom cylinder"}'

Layout

src/schema.py         pydantic FaultRecord
src/taxonomy.py       14 fault modes + 9 asset families, from MaintIE frequencies
src/prepare_data.py   annotations → records.jsonl, injects the synthetic fields
src/model.py          Qwen2.5-3B at fp16 / int8 / nf4, optional LoRA adapter
src/extract.py        work order → validated FaultRecord, one retry
src/surveillance.py   daily counts per family, Poisson + CUSUM
src/forecast.py       Holt linear trend, trailing-mean fallback
src/alert.py          flags → escalations
src/api.py            POST /extract
src/finetune.py       QLoRA (nf4 base + LoRA r=16)
src/evaluate.py       both tables
run_pipeline.py       end-to-end CLI

Credits

Built by Abid Hasan — github.com/abidhasanzim.

• Data: MaintIE, MIT.
• Model: Qwen2.5-3B-Instruct, Apache 2.0.
• Detection uses standard SPC (one-sided Poisson excess test, standardized tabular CUSUM) with Holt linear-trend forecasting via statsmodels.