Version: 1.0.0
A small, locally-deployable LLM pipeline for insurance document classification, extraction, and memo generation — built as a cost-efficient alternative to frontier enterprise models for high-volume document intake.
Insurance operations (claims intake, underwriting submissions, policy servicing) generate huge volumes of structured-but-messy documents — ACORD forms, loss notices, adjuster reports, correspondence. Processing these with frontier API models (GPT-4/Claude-class) works, but at volume it's expensive and requires sending documents to a third party. This project explores whether a small, specialized, locally-hosted pipeline can match "good enough" accuracy at a fraction of the cost, fully in-house.
Rather than one large model trying to do everything, this pipeline splits the problem into three specialized components:
- Classification — a small fine-tuned encoder (DeBERTa-v3 class) identifies document type and category
- Extraction — a layout-aware model (LayoutLMv3/Donut class) pulls structured fields from forms, including scanned/OCR'd documents
- Summarization — a small fine-tuned generative LLM (7–8B, quantized) writes adjuster-style memos and analysis from the extracted content
All three run offline, on modest hardware, with no per-token API costs.
Package version: 1.0.0 — see CHANGELOG.md
for the incremental history from 0.1.0 through this beta.
Covers Phases 0–3 and 5, DICIE (5b), sample corpus store, Discord bot, eval/cost harness, and classical/ViT classifiers (Phase 4 summarizer LoRA still pending):
- Characteristic profiles from public document/layout/legal-style priors
- Synthetic skeleton → document → memo → OCR-noise pipeline (optional OpenRouter LLM; auto-routes to free models on credit exhaustion, else template fallback)
- Document-type classifier train/eval (text DeBERTa + optional ViT image path + TF-IDF Random Forest baseline)
- Field extraction train/eval with noisy stress reporting
- DICIE (Fig. 1): document processing → classification → information extraction → aggregated response (
src/docie/) for medical bills and salvage claims - Single-action analysis chain: to_markdown → classify → extract → vision_llm → summarize (
src/pipeline/) - PNG/PDF → structured markdown before LLM stages (token + context optimization)
- Sample corpus store: queryable SQLite house for synthetic medical / salvage docs (
src/storage/) - Discord bot (Chloride):
/analyzememo chain plus notes, STT, DJ/vibes, chat - Frontier vs. local eval harness + cost-model spreadsheet
Phase 4 (fine-tuned summarizer LoRA) still pending — the chain currently uses a template memo grounded in upstream extraction/vision outputs, with a hook for a local generative model when configured.
Docs: 📚 Live Quarto site (Posit Connect Cloud) · Usage · Architecture · CHANGELOG · Local preview / re-publish
This project uses no real insurance company data. All training data is either:
- Publicly available (ACORD form templates, public document-layout datasets, public insurance distribution shapes, legal writing used for vocabulary/reasoning style only), or
- Synthetically generated (fictional claims, documents, and memos produced from randomized skeletons)
Legal corpora never become classification labels. See docs/data_provenance.md.
Every synthetic record is logged in data/provenance_log.jsonl with its generation source.
# install
pip install -e ".[dev]"
# create local secrets file (gitignored) — paste keys into .env
python scripts/setup_env.py
# then edit .env:
# OPENROUTER_API_KEY=... # https://openrouter.ai/keys
# WANDB_API_KEY=... # https://wandb.ai/authorize
# HF_TOKEN=... # optional, https://huggingface.co/settings/tokens
python scripts/setup_env.py --status # confirms which secrets are set (never prints values)
# refresh profiles (optional Hub ingest: add --ingest)
python -m src.generation.run_seed_pipeline --n 240
# prepare + train classifier (smoke uses DistilBERT on CPU)
python -m src.classification.prepare_dataset --in data/synthetic/documents/documents_from_skeletons_n240_seed42.jsonl
python -m src.classification.train_classifier --prepared data/synthetic/documents/classification_prepared --smoke
python -m src.classification.eval --model-dir models/classifier_smoke --prepared data/synthetic/documents/classification_prepared
# classical baseline: TF-IDF + Random Forest on typed text + handwriting/OCR noise
python -m src.classification.train_random_forest
# or interactively:
# pip install -e ".[notebooks]"
# jupyter notebook notebooks/random_forest_text_handwriting_classification.ipynb
# ViT document-image classifier (Kaggle/HF-style image classification on rendered pages)
python -m src.extraction.render_forms --in data/synthetic/documents/documents_from_skeletons_n240_seed42.jsonl --out data/synthetic/documents/rendered
python -m src.classification.prepare_image_dataset --in data/synthetic/documents/rendered/rendered.jsonl
python -m src.classification.train_vit_classifier --prepared data/synthetic/documents/rendered/vit_classification_prepared --smoke
python -m src.classification.eval_vit --model-dir models/vit_classifier_smoke --prepared data/synthetic/documents/rendered/vit_classification_prepared
# render forms + train extractor (smoke path)
python -m src.extraction.render_forms --in data/synthetic/documents/documents_from_skeletons_n240_seed42.jsonl --out data/synthetic/documents/rendered
python -m src.extraction.prepare_dataset --in data/synthetic/documents/rendered/rendered.jsonl
python -m src.extraction.train_extractor --prepared data/synthetic/documents/rendered/extraction_prepared --smoke
python -m src.extraction.eval --model-dir models/extractor_smoke --prepared data/synthetic/documents/rendered/extraction_prepared
# --- paper Fig. 1 DICIE pipeline (process → classify → extract → respond) ---
# Docs: src/docie/README.md · docs/docie_pipeline.md
# Medical bills / salvage claims applications from Raj et al.
python -m src.docie \
--application salvage_claims \
--text "LETTER OF GUARANTEE
Claim Number: CLM-2024-100200
VIN: 1HGCM82633A004352
Year: 2018
Make: Honda
Model: Accord" \
--response-only
python -m src.docie --application medical_bills --pdf path/to/bill.pdf
python -m src.docie --application salvage_claims --image path/to/log.png
python -m src.docie \
--application salvage_claims \
--in tests/fixtures/sample_docie_documents.jsonl \
--out data/pipeline/docie/salvage_demo.jsonl
# Optional REST server (paper §VI FastAPI shape)
# pip install -e ".[serve]"
# python -m src.docie.serve --application salvage_claims --port 8080
# --- one action: full document analysis chain (memo path) ---
# PNG/PDF are converted to structured markdown before LLM stages (token-efficient).
python -m src.pipeline.orchestrator \
--in data/synthetic/documents/documents_from_skeletons_n240_seed42.jsonl \
--out data/pipeline/analysis.jsonl \
--vision --limit 20
python -m src.pipeline.orchestrator --image path/to/scan.png --vision
python -m src.pipeline.orchestrator --pdf path/to/claim.pdf --vision
python -m src.pipeline.batch_runner \
--in data/synthetic/documents/documents_from_skeletons_n240_seed42.jsonl \
--out-dir data/pipeline/batch_demo \
--vision --limit 20
# ad-hoc single document
python -m src.pipeline.orchestrator --vision --text "LOSS NOTICE\nClaim Number: CLM-1\nDate of Loss: 2024-01-15\n..."
# --- Discord bot (Chloride) ---
# Docs + notes + STT + DJ/vibes + chat agent
pip install -e ".[discord]"
# optional voice DJ: pip install -e ".[discord,discord-voice]" && brew install ffmpeg
python scripts/setup_env.py # set DISCORD_TOKEN (+ OPENROUTER_API_KEY; OPENAI_API_KEY for Whisper)
cp discord/smol-doc-analyzer/config.yaml.example discord/smol-doc-analyzer/config.yaml
python -m src.discord_bot
# autostart at login (macOS):
./scripts/install_discord_bot_loginitem.sh
# See discord/smol-doc-analyzer/README.md
# --- Changelog + incremental version bump (Keep a Changelog) ---
python scripts/update_changelog.py # cut release + bump +0.0.1 (e.g. 1.0.0 → 1.0.1)
python scripts/update_changelog.py --bump major # milestone: → X.5.0 or next whole N.0.0
python scripts/update_changelog.py --dry-run --print-unreleased
python scripts/update_changelog.py --bump none # refresh [Unreleased] only
./scripts/install_changelog_launchagent.sh # Wed 11pm America/Chicago (LaunchAgent)
# ./scripts/install_changelog_launchagent.sh --crontab # optional user-crontab instead
# full-scale generation (when ready)
python -m src.generation.skeleton_sampler --n 5000 --out data/synthetic/skeletons/
python -m src.generation.stage_a_document_gen --in data/synthetic/skeletons/skeletons_n5000_seed42.jsonl
python -m src.generation.stage_b_memo_gen --in data/synthetic/documents/documents_from_skeletons_n5000_seed42.jsonlFor GPU training, omit --smoke and use the default DeBERTa-v3 / ViT / LayoutLMv3 model names.
On high-RAM local hosts, point VISION_LLM_MODEL_PATH at a downloaded Qwen2-VL
(or similar) checkpoint and set VISION_LLM_LOAD=1 to refine extraction from
page images inside the same chain.
Guides, architecture notes, and notebooks are published as a Quarto website
under docs/ (docs/_quarto.yml).
# install Quarto CLI: https://quarto.org/docs/get-started/
cd docs && quarto preview # live local portal (run inside docs/)
./scripts/preview_docs_site.sh # same from repo root, with a PATH check
cd docs && quarto render # static build → docs/_site/How-to: docs/how-to/launch-quarto-site.qmd. Quick start page: docs/quick-start.qmd. Notebook portal: docs/notebooks/index.qmd.
See docs/architecture.qmd. Full command reference: docs/usage.md.
Paper Fig. 1 path for medical bills and salvage claims: process → classify → extract → aggregated response. Image-first (page OCR), with optional FastAPI serving.
- Module README (CLI, Python API, REST, schema): src/docie/README.md
- Design notes: docs/docie_pipeline.md
- Taxonomies:
taxonomy/medical_bills.yaml,taxonomy/salvage_claims.yaml - Tests:
pytest tests/test_docie_pipeline.py
This is separate from the markdown → memo chain in src/pipeline/ (used by
the Discord bot). Prefer DICIE for paper-aligned workflow apps; prefer
src.pipeline.orchestrator when you need adjuster-style memos.
Queryable SQLite store for synthetic medical bills and salvage documentation (Letters of Guarantee, salvage sales receipts, towing/storage attachments) patterned after AmFam-style intake — without proprietary insurer data.
# Seed realistic samples + optional JSONL export
python -m src.storage seed --seed 42 --also-export
# Export for DICIE / classification / extraction training
python -m src.storage export --format docie --application salvage_claims \
--out data/sample_corpus/exports/salvage_docie.jsonl- Design notes: docs/sample_document_corpus.md
- Schemas:
data/schemas/medical_bill_skeleton.schema.json,data/schemas/salvage_document_skeleton.schema.json - Tests:
pytest tests/test_document_store.py tests/test_storage_training.py - Notebooks:
# optional: regenerate the three sample-corpus notebooks
python scripts/build_sample_corpus_notebooks.py
# interactive:
# pip install -e ".[notebooks]"
# jupyter notebook notebooks/sample_corpus_train_test_pipeline.ipynbQueryable SQLite index for the public
aharley/rvl_cdip dataset
(400k document images, 16 classes). All Hub downloads and the DB stay under
.venv/rvl_cdip/. The default build fetches only the small label files
(~17 MB); the ~38 GB image archive is opt-in.
python -m src.rvl_cdip build
python -m src.rvl_cdip summary
python -m src.rvl_cdip list --split train --label invoice --limit 5
python -m src.rvl_cdip query \
"SELECT l.name AS label, COUNT(*) AS n
FROM documents d JOIN labels l ON l.label_id = d.label_id
GROUP BY l.name ORDER BY n DESC"
# optional — explicit acknowledgement required (~38 GB)
# python -m src.rvl_cdip download-images --i-understand-large-download- Design notes: docs/rvl_cdip_sql.md
- Tests:
pytest tests/test_rvl_cdip_store.py
Fixed-size 10 images × 16 classes @ 1024×1024 classification experiments on
Braintrust (DSHB_amfam_capstone_2026), with OpenRouter reasoning-token capture
(Kimi K3) and DeepSeek R1 prompt improvement.
pip install -e ".[braintrust]"
python -m src.braintrust_eval build-dataset --placeholder # or full RVL sample
python -m src.braintrust_eval upload-dataset # needs BRAINTRUST_API_KEY
python -m src.braintrust_eval run-eval --model moonshotai/kimi-k3
python -m src.braintrust_eval cost-estimate- Guide: docs/braintrust_rvl_eval.md
- Secrets:
BRAINTRUST_API_KEY+OPENROUTER_API_KEYin.env
Optional Discord front-end via Chloride
(Coral AI agent). Install the discord extra, set DISCORD_TOKEN, and run
python -m src.discord_bot.
The bot is an all-purpose server agent with an insurance-docs specialty:
- Docs:
/analyze,/analyze_url, toolanalyze_insurance_document - Notes / STT:
/note,/transcribe,/remind - DJ / vibes:
/play,/queue,/vibe(voice optional via.[discord-voice]) - Chat: mention the bot or
--prefix for free-form Chloride agent replies - Utils:
/poll,/status,/help,/ping
Details: discord/smol-doc-analyzer/README.md.
Per-model reports land in evaluation/reports/ (classification_report.*,
vit_classification_report.*, random_forest_classification_report.md,
extraction_report.json, failure_modes.md).
Compare Anthropic / OpenAI (via OpenRouter) against local pipeline models on the same held-out set across classification, extraction, and memo generation:
# plan calls without spending API budget
python -m evaluation.eval_harness \
--eval-set data/eval/eval_set.jsonl \
--tasks classification extraction memo_generation \
--backends anthropic openai local \
--n-samples 50 \
--output-dir evaluation/results/eval_run_2026-07-13 \
--dry-run
# live run (requires OPENROUTER_API_KEY for frontier backends)
python -m evaluation.eval_harness \
--eval-set data/eval/eval_set.jsonl \
--backends anthropic openai local \
--output-dir evaluation/results/eval_run_2026-07-13Outputs: eval_results.jsonl (source of truth) + eval_results.csv (cost-model
spreadsheet feed). Pricing lives in evaluation/pricing.yaml.
Score a completed run into the spreadsheet "Eval Results" summary:
python -m evaluation.metrics \
--results evaluation/results/eval_run_2026-07-13/eval_results.jsonl \
--output evaluation/results/eval_run_2026-07-13/summary.csv
# or score immediately after a live harness run
python -m evaluation.eval_harness ... --output-dir evaluation/results/eval_run_2026-07-13 --scoreClassification → accuracy + macro F1; extraction → field micro-F1 (fuzzy fields optional); memo generation → rubric coverage (LLM-judge scores merge in when present on the JSONL rows).
Formula-driven workbook comparing frontier vs. local $/doc and monthly cost at
volume. Paste scores from summary.csv into the Eval Results sheet:
pip install -e ".[cost-model]" # openpyxl
python -m evaluation.build_cost_model
# writes evaluation/cost_model/cost_model.xlsxSheets: Legend, Assumptions, Eval Results, Cost Per Doc, Scaling Projection, Dashboard (with volume chart). Blue/yellow cells are editable inputs; green cells are cross-sheet links; black cells are formulas.
Training, evaluation, and the seed generation pipeline log to Weights & Biases by default:
- Train: Hugging Face Trainer metrics (
loss, eval accuracy / F1), run config,train_meta.jsonartifacts (text DeBERTa, ViT image, and LayoutLMv3 extractors) - Eval: summary metrics, per-class / field tables, confusion matrix (classifier + ViT), report + failure-mode artifacts
- Generation: stage progress and output path summaries for
run_seed_pipeline
# copy env and set your key (https://wandb.ai/authorize)
python scripts/setup_env.py
# WANDB_API_KEY=... WANDB_PROJECT=smol-doc-analyzer
# offline / no key still works (local wandb/ cache)
WANDB_MODE=offline python -m src.classification.train_classifier --prepared ... --smoke
# disable for a single invocation
python -m src.classification.train_classifier --prepared ... --smoke --no-wandbUseful flags on train/eval/seed CLIs: --wandb, --no-wandb, --wandb-project, --wandb-run-name.
Set WANDB_MODE=disabled or WANDB_DISABLED=true to turn tracking off globally.
Document taxonomy references publicly available ACORD form structures (acord.org). This project is not affiliated with or endorsed by ACORD.
Discord agent integration uses Chloride (fork of Coral), MIT licensed.