Add claim outcome prediction as a tracked accuracy metric

README.md

@@ -131,7 +131,17 @@ Details: [discord/smol-doc-analyzer/README.md](discord/smol-doc-analyzer/README.
 
 ## Evaluation
 
-Reports land in `evaluation/reports/` (`classification_report.*`, `vit_classification_report.*`, `extraction_report.json`, `failure_modes.md`).
+Reports land in `evaluation/reports/` (`classification_report.*`, `vit_classification_report.*`, `extraction_report.json`, `outcome_prediction_report.*`, `failure_modes.md`).
+
+Claim **outcome prediction** accuracy is a complementary metric to classification / extraction:
+
+```bash
+python -m src.pipeline.eval_outcome \
+  --in data/synthetic/documents/documents_from_skeletons_n240_seed42.jsonl \
+  --no-wandb --limit 50
+```
+
+Gold `expected_outcome` is written on synthetic skeletons and scored against the `predict_outcome` pipeline stage.
 
 ## Experiment tracking (Weights & Biases)
 

data/profiles/insurance_distributions.json

@@ -57,6 +57,7 @@
     "ambiguous": 0.22,
     "fraud_flagged": 0.08
   },
+  "expected_outcome_notes": "Gold expected_outcome is derived deterministically from narrative_complexity + financials + injuries (see src/pipeline/outcome.py). Tracked in evaluation/reports/outcome_prediction_report.*",
   "police_report_rate": 0.42,
   "injuries_reported_rate": 0.18,
   "acord_form_by_document_type": {

data/schemas/claim_skeleton.schema.json

@@ -3,7 +3,7 @@
   "title": "ClaimSkeleton",
   "description": "Structured intermediate representation used to generate synthetic insurance documents and memos. This is entirely fictional data -- see docs/data_provenance.md.",
   "type": "object",
-  "required": ["claim_id", "document_type", "policy", "loss_event", "parties", "financials", "narrative_complexity"],
+  "required": ["claim_id", "document_type", "policy", "loss_event", "parties", "financials", "narrative_complexity", "expected_outcome"],
   "properties": {
     "claim_id": {
       "type": "string",
@@ -82,6 +82,17 @@
       "enum": ["clean", "standard", "ambiguous", "fraud_flagged"],
       "description": "Controls how straightforward vs. messy/edge-case the generated document and memo should be."
     },
+    "expected_outcome": {
+      "type": "string",
+      "enum": [
+        "pay_full",
+        "pay_partial",
+        "deny",
+        "investigate",
+        "close_without_payment"
+      ],
+      "description": "Synthetic supervisory label for predicted claim disposition. Deterministic from skeleton features so pipeline outcome accuracy tracks end-to-end feature recovery."
+    },
     "multi_doc_group_id": {
       "type": ["string", "null"],
       "description": "If set, links this skeleton to other skeletons that belong to the same claim file (e.g. loss notice + repair estimate + adjuster memo all sharing a claim_id)."

docs/architecture.md

@@ -10,7 +10,8 @@ execute chronologically; each stage reacts to prior stage outputs:
 2. **Classification** — DeBERTa-v3 encoder (heuristic fallback) maps document text → taxonomy label; optionally a **ViT** image classifier maps rendered page images → the same taxonomy
 3. **Extraction** — LayoutLMv3 / token classifier (heuristic fallback) pulls structured fields; conditioned on the predicted document type
 4. **Vision LLM refine** — markdown-first local multimodal/text model (default target: Qwen2-VL class) corrects fields using classify+extract context; optional page image via `VISION_LLM_USE_IMAGE=1`
-5. **Summarization** — generative LLM or template memo grounded in upstream markdown + payloads (not ground-truth skeletons)
+5. **Predict outcome** — deterministic claim-disposition prediction (`pay_full` / `pay_partial` / `deny` / `investigate` / `close_without_payment`) from extracted features; gold `expected_outcome` on synthetic skeletons enables accuracy tracking alongside classification / extraction metrics
+6. **Summarization** — generative LLM or template memo grounded in upstream markdown + payloads + predicted outcome (not ground-truth skeletons)
 
 Entry points:
 
@@ -70,7 +71,7 @@ which runs the same chronological pipeline as the CLI. See
 Public corpora → profiles → skeletons → documents (+ noisy) → classifier (text and/or ViT on renders) / extractor
                                          └→ memos (Phase 4 training targets)
 
-Inbound PNG/PDF/text → to_markdown → classify → extract → vision_llm → summarize
+Inbound PNG/PDF/text → to_markdown → classify → extract → vision_llm → predict_outcome → summarize
                               ↓
                      structured markdown (LLM context)
 ```
@@ -86,6 +87,7 @@ reorders by name. Each stage receives an accumulating `AnalysisContext`:
 | classify | markdown plain_text (preferred) | `classification.document_type`, confidence |
 | extract | markdown + classification | `extraction.fields*`, optional page render |
 | vision_llm | markdown (+ optional image) + classify + extract | `vision.refined_fields` (merged into extraction) |
+| predict_outcome | extraction (+ vision refine) + text cues | `outcome.expected_outcome`, confidence, optional gold compare |
 | summarize | markdown + all prior payloads | `summary.memo` |
 
 Low-confidence stages append flags (`low_confidence_classification`, etc.)

evaluation/reports/outcome_prediction_report.json

@@ -0,0 +1,117 @@
+{
+  "n": 40,
+  "n_skipped_no_gold": 0,
+  "accuracy": 0.4,
+  "macro_f1": 0.3111111111111111,
+  "per_class": {
+    "pay_full": {
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1-score": 0.0,
+      "support": 19.0
+    },
+    "pay_partial": {
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1-score": 0.0,
+      "support": 2.0
+    },
+    "deny": {
+      "precision": 1.0,
+      "recall": 1.0,
+      "f1-score": 1.0,
+      "support": 1.0
+    },
+    "investigate": {
+      "precision": 0.38461538461538464,
+      "recall": 1.0,
+      "f1-score": 0.5555555555555556,
+      "support": 15.0
+    },
+    "close_without_payment": {
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1-score": 0.0,
+      "support": 3.0
+    },
+    "accuracy": 0.4,
+    "macro avg": {
+      "precision": 0.27692307692307694,
+      "recall": 0.4,
+      "f1-score": 0.3111111111111111,
+      "support": 40.0
+    },
+    "weighted avg": {
+      "precision": 0.16923076923076924,
+      "recall": 0.4,
+      "f1-score": 0.23333333333333334,
+      "support": 40.0
+    }
+  },
+  "confusion_matrix": [
+    [
+      0,
+      0,
+      0,
+      19,
+      0
+    ],
+    [
+      0,
+      0,
+      0,
+      2,
+      0
+    ],
+    [
+      0,
+      0,
+      1,
+      0,
+      0
+    ],
+    [
+      0,
+      0,
+      0,
+      15,
+      0
+    ],
+    [
+      0,
+      0,
+      0,
+      3,
+      0
+    ]
+  ],
+  "label_order": [
+    "pay_full",
+    "pay_partial",
+    "deny",
+    "investigate",
+    "close_without_payment"
+  ],
+  "gold_distribution": {
+    "pay_full": 19,
+    "investigate": 15,
+    "pay_partial": 2,
+    "close_without_payment": 3,
+    "deny": 1
+  },
+  "pred_distribution": {
+    "investigate": 39,
+    "deny": 1
+  },
+  "n_correct": 16,
+  "chain": [
+    "to_markdown",
+    "classify",
+    "extract",
+    "vision_llm",
+    "predict_outcome",
+    "summarize"
+  ],
+  "metric_family": "claim_outcome_prediction",
+  "n_detail_rows": 40
+}

evaluation/reports/outcome_prediction_report.md

@@ -0,0 +1,27 @@
+# Claim outcome prediction report
+
+- N scored: **40**
+- Skipped (no gold): 0
+- Accuracy: **0.4**
+- Macro F1: **0.3111111111111111**
+- Chain: `to_markdown → classify → extract → vision_llm → predict_outcome → summarize`
+
+## Label distribution (gold)
+
+- `close_without_payment`: 3
+- `deny`: 1
+- `investigate`: 15
+- `pay_full`: 19
+- `pay_partial`: 2
+
+## Per-class F1
+
+- `pay_full`: precision=0.000 recall=0.000 f1=0.000 support=19.0
+- `pay_partial`: precision=0.000 recall=0.000 f1=0.000 support=2.0
+- `deny`: precision=1.000 recall=1.000 f1=1.000 support=1.0
+- `investigate`: precision=0.385 recall=1.000 f1=0.556 support=15.0
+- `close_without_payment`: precision=0.000 recall=0.000 f1=0.000 support=3.0
+
+## Notes
+
+Gold `expected_outcome` is a deterministic function of skeleton features (complexity, injuries, damage vs deductible/reserve). Accuracy therefore tracks how well upstream extraction recovers those features for the predictive disposition rule — complementary to classification accuracy and extraction field F1.

src/discord_bot/formatters.py

@@ -10,6 +10,7 @@ def compact_analysis(result: dict[str, Any], *, max_memo_chars: int = 1200) -> d
     classification = result.get("classification") or {}
     extraction = result.get("extraction") or {}
     vision = result.get("vision") or {}
+    outcome = result.get("outcome") or {}
     summary = result.get("summary") or {}
     markdown = result.get("markdown") or {}
 
@@ -44,6 +45,13 @@ def compact_analysis(result: dict[str, Any], *, max_memo_chars: int = 1200) -> d
         or classification.get("label")
         or classification.get("predicted_label"),
         "classification_confidence": classification.get("confidence"),
+        "expected_outcome": result.get("expected_outcome")
+        or outcome.get("expected_outcome")
+        or outcome.get("outcome_label"),
+        "outcome_confidence": outcome.get("confidence"),
+        "outcome_description": outcome.get("description"),
+        "gold_outcome": outcome.get("gold_outcome"),
+        "outcome_correct": outcome.get("correct"),
         "fields": fields,
         "memo": memo,
         "flags": result.get("flags") or [],
@@ -66,6 +74,17 @@ def format_discord_summary(compact: dict[str, Any]) -> str:
         rid = compact.get("claim_id") or compact.get("record_id")
         lines.append(f"**Record:** `{rid}`")
 
+    outcome = compact.get("expected_outcome")
+    if outcome:
+        oconf = compact.get("outcome_confidence")
+        oconf_s = f" ({float(oconf):.0%})" if isinstance(oconf, (int, float)) else ""
+        lines.append(f"**Predicted outcome:** `{outcome}`{oconf_s}")
+        if compact.get("outcome_description"):
+            lines.append(f"-# {compact['outcome_description']}")
+        if compact.get("gold_outcome") is not None:
+            mark = "✓" if compact.get("outcome_correct") else "✗"
+            lines.append(f"**Gold outcome:** `{compact['gold_outcome']}` {mark}")
+
     fields = compact.get("fields") or {}
     if fields:
         lines.append("")

src/discord_bot/runner.py

@@ -61,9 +61,8 @@ def _overlay_secrets(raw: dict) -> dict:
                 ("google-gla:", "google-vertex:", "openai:", "anthropic:", "xai:", "groq:")
             ) and "/" not in model.split(":", 1)[-1]
             if chloride_native or not model:
-                data["AI_MODEL_NAME"] = os.getenv(
-                    "DISCORD_AI_MODEL", "anthropic/claude-sonnet-4.5"
-                )
+                override = os.getenv("DISCORD_AI_MODEL", "").strip()
+                data["AI_MODEL_NAME"] = override or "anthropic/claude-sonnet-4.5"
 
     return data
 

src/generation/skeleton_sampler.py

@@ -17,6 +17,7 @@
 from src.utils.config import Config
 from src.utils.io import read_json, write_json, write_jsonl
 from src.utils.provenance import ProvenanceRecord, log_provenance
+from src.pipeline.outcome import derive_expected_outcome, features_from_skeleton
 
 logger = logging.getLogger(__name__)
 
@@ -126,6 +127,7 @@ def sample_skeleton(
         "multi_doc_group_id": multi_doc_group_id,
         "target_outputs": {"document_text": None, "memo_text": None},
     }
+    skeleton["expected_outcome"] = derive_expected_outcome(features_from_skeleton(skeleton))
     return skeleton
 
 

src/pipeline/batch_runner.py

@@ -79,6 +79,7 @@ def run_batch(
         "review_queue_path": str(review_path),
         "by_document_type": _count_by_type(results),
         "flag_counts": _count_flags(results),
+        "outcome_metrics": _outcome_metrics(results, rows),
     }
     write_json(summary_path, summary)
     log_provenance(
@@ -123,10 +124,45 @@ def _count_flags(results: list[dict[str, Any]]) -> dict[str, int]:
     return counts
 
 
+def _outcome_metrics(
+    results: list[dict[str, Any]], rows: list[dict[str, Any]]
+) -> dict[str, Any]:
+    """Accuracy of predicted claim outcomes vs gold when labels are available."""
+    from src.pipeline.outcome import derive_expected_outcome, features_from_skeleton
+
+    y_true: list[str] = []
+    y_pred: list[str] = []
+    for row, result in zip(rows, results):
+        gold = row.get("expected_outcome")
+        skeleton = row.get("skeleton")
+        if not gold and isinstance(skeleton, dict):
+            gold = skeleton.get("expected_outcome") or derive_expected_outcome(
+                features_from_skeleton(skeleton)
+            )
+        pred = (result.get("outcome") or {}).get("expected_outcome")
+        if not gold or not pred:
+            continue
+        y_true.append(str(gold))
+        y_pred.append(str(pred))
+
+    if not y_true:
+        return {
+            "n_scored": 0,
+            "accuracy": None,
+            "note": "no gold expected_outcome on inputs",
+        }
+    correct = sum(1 for a, b in zip(y_true, y_pred) if a == b)
+    return {
+        "n_scored": len(y_true),
+        "n_correct": correct,
+        "accuracy": correct / len(y_true),
+    }
+
+
 def main() -> None:
     logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
     parser = argparse.ArgumentParser(
-        description="Batch-run the chained to_markdown→classify→extract→vision→summarize pipeline"
+        description="Batch-run the chained to_markdown→classify→extract→vision→predict_outcome→summarize pipeline"
     )
     parser.add_argument("--in", dest="inp", type=Path, required=True)
     parser.add_argument("--out-dir", type=Path, default=None)