SMS classification intelligence evaluation and targeted model improvement

.gitattributes

@@ -7,3 +7,4 @@ src/mBERT/training/model-training/mbert_ots_model_2.1.pth filter=lfs diff=lfs me
 src/mBERT/training/model-training/mbert_ots_model_2.5.pth filter=lfs diff=lfs merge=lfs -text
 *.npz filter=lfs diff=lfs merge=lfs -text
 *.pth filter=lfs diff=lfs merge=lfs -text
+evals/assets/bert-base-multilingual-cased-vocab.txt filter=lfs diff=lfs merge=lfs -text

.github/workflows/tests.yml

@@ -0,0 +1,46 @@
+name: Unit Tests
+
+# Runs the fast, offline unit tests (no model weights / LFS needed). The
+# batching + normalisation tests use stubs and stdlib; settings tests construct
+# the config object directly.
+
+# Least-privilege GITHUB_TOKEN: this workflow only reads the repo to run tests.
+permissions:
+  contents: read
+
+on:
+  push:
+    branches: ["**"]
+    paths:
+      - "src/api_interface/**"
+      - ".github/workflows/tests.yml"
+  pull_request:
+    paths:
+      - "src/api_interface/**"
+      - ".github/workflows/tests.yml"
+
+jobs:
+  unit-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          lfs: false  # tests stub the model; weights not required
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+
+      - name: Install test dependencies
+        run: |
+          python -m pip install --upgrade pip
+          # Targeted deps for the unit suite (CPU torch wheel on Linux).
+          # fastapi + httpx are required by test_metrics_endpoint's TestClient.
+          pip install "torch>=2.3" "transformers>=4.40" \
+                      "pydantic>=2.7" pydantic-settings \
+                      fastapi "httpx>=0.27" \
+                      pytest pytest-asyncio
+
+      - name: Run unit tests
+        run: pytest src/api_interface/tests/ -v

evals/.gitignore

@@ -0,0 +1,7 @@
+__pycache__/
+*.pyc
+# Large per-sample prediction dumps are reproducible via run_eval.py.
+# Keep the compact summary_*.json (the evidence) under version control.
+# Catch-all (covers uci/imc25/mishra/fable5/ots60 and any future dataset).
+results/predictions_*.json
+results/summary.json

evals/TIER1.md

@@ -0,0 +1,132 @@
+# Tier-1 Improvement Runbook (Apple Silicon / M1)
+
+Goal: retrain the classifier from the v2.5 base using the Tier-1 improvements
+(GPU/MPS, validation split, best-checkpoint selection, class-weighted loss) and
+verify the gains against the public benchmarks. On an M1 this takes minutes, not
+the ~1 hour the CPU proof-of-concept took.
+
+Everything needed is already on this branch: the v2.5 base model (Git LFS), the
+synthetic dataset, the original corpus, the vendored tokenizer vocab, and the
+benchmark loaders. No Hugging Face download is required.
+
+## 0. One-time setup
+
+```bash
+# from the repo root, on the sms-classification-intelligence-eval branch
+git pull origin sms-classification-intelligence-eval     # get the Tier-1 scripts
+
+python3.12 -m venv ots && source ots/bin/activate
+pip install --upgrade pip
+# Broad pins so this installs against current stable wheels (don't pin to an
+# unreleased torch). Newer minors are fine.
+pip install "torch>=2.3,<3.0" "transformers>=4.40,<5.0" "tokenizers>=0.19,<1.0" scikit-learn
+
+# confirm Apple Silicon GPU is visible
+python -c "import torch; print('MPS available:', torch.backends.mps.is_available())"
+```
+
+Make sure the LFS model is actually present (not a pointer):
+
+```bash
+git lfs pull
+ls -la src/mBERT/training/model-training/mbert_ots_model_2.5.pth   # should be ~680 MB
+```
+
+## 1. Download the two public benchmarks (Mishra & Soni already ships in-repo)
+
+```bash
+curl -sL -o /tmp/uci.tsv   https://raw.githubusercontent.com/justmarkham/DAT8/master/data/sms.tsv
+curl -sL -o /tmp/imc25.csv https://raw.githubusercontent.com/reportsmishing/Smishing-Dataset-IMC25/main/dataset/final_dataset_output.csv
+```
+
+## 2. Baseline — score v2.5 (so you have a clean before)
+
+```bash
+python evals/run_eval.py \
+  --model src/mBERT/training/model-training/mbert_ots_model_2.5.pth \
+  --dataset uci:/tmp/uci.tsv \
+  --dataset mishra:evals/datasets/mishra_soni_5971.csv \
+  --dataset imc25:/tmp/imc25.csv:8000 \
+  --dataset fable5 \
+  --tag v2.5
+```
+
+## 3. Tier-1 fine-tune (runs on the M1 GPU automatically)
+
+```bash
+python evals/finetune_tier1.py \
+  --base   src/mBERT/training/model-training/mbert_ots_model_2.5.pth \
+  --synthetic src/mBERT/training/model-training/dataset/synthetic_fable5_v1.csv \
+  --original  src/mBERT/training/model-training/dataset/sms_spam_phishing_dataset_v2.4_combined.csv \
+  --out    src/mBERT/training/model-training/mbert_ots_model_2.7.pth \
+  --epochs 2 --batch-size 32 --loss plain --lr 1e-5
+```
+
+This is the **shipped recipe** for model 2.7: a gentle fine-tune (plain
+cross-entropy, lr 1e-5, 2 epochs) continued from the v2.5 base over the synthetic
+set plus a 2,500/class rehearsal sample. Earlier heavier runs (class-weighted
+loss, 4 epochs, lr 2e-5) scored higher on the in-domain validation split but
+*overfit* — they regressed the real-world IMC25 block rate. Backing off to this
+gentler configuration recovered generalization and beat v2.5 on every benchmark.
+
+It prints `Device: Apple Silicon MPS`, a per-epoch validation line, and saves the
+**best** epoch (by validation macro-F1) — plus a `*.trainlog.json` with the curve.
+
+> ⚠️ The ~99% macro-F1 in the trainlog is on the **in-domain validation split**,
+> not the public benchmarks. It is an optimistic signal — do not interpret it as
+> real-world accuracy. The numbers that matter are the public-benchmark scores
+> from step 4 (and the IMC25 block rate in particular). See `evals/REPORT.md`.
+
+Knobs worth trying (departures from the shipped recipe — validate against IMC25
+before trusting any gain, since the in-domain split is an optimistic signal):
+- `--loss class_weighted` / `--loss focal` — weight the rare phishing class; both
+  raised validation F1 here but hurt real-world generalization.
+- `--epochs 4` / `--lr 2e-5` — more/larger steps; the same overfitting caveat.
+- `--orig-per-label 4000` — more rehearsal data (better protects classic spam,
+  slower).
+
+## 4. Score the new model (the after)
+
+```bash
+python evals/run_eval.py \
+  --model src/mBERT/training/model-training/mbert_ots_model_2.7.pth \
+  --dataset uci:/tmp/uci.tsv \
+  --dataset mishra:evals/datasets/mishra_soni_5971.csv \
+  --dataset imc25:/tmp/imc25.csv:8000 \
+  --dataset fable5 \
+  --tag v2.7
+
+# side-by-side
+python evals/compare_runs.py \
+  --before evals/results/summary_v2.5.json \
+  --after  evals/results/summary_v2.7.json
+```
+
+## 5. (Optional) Calibrate the phishing/spam boundary — no retraining
+
+If phishing is still being routed into spam, find a per-class logit bias on a
+held-out set and verify it on a *different* set (calibrate and verify on separate
+data — calibrating and scoring on the same set is optimistic):
+
+```bash
+# calibrate on Mishra
+python evals/calibrate_thresholds.py \
+  --model src/mBERT/training/model-training/mbert_ots_model_2.7.pth \
+  --data evals/datasets/mishra_soni_5971.csv:mishra \
+  --objective macro_f1 --out evals/results/calibration.json
+
+# verify the chosen bias on the adversarial set (held-out)
+python evals/run_eval.py \
+  --model src/mBERT/training/model-training/mbert_ots_model_2.7.pth \
+  --dataset fable5 --logit-bias 0.0,-1.0,2.0      # use the bias it printed
+```
+
+## 6. Promote
+
+If the after numbers hold up (UCI block ≥ 99%, IMC25 block up, no surprise
+regressions), rename `mbert_ots_model_2.7.pth` to the version you ship, commit it
+via Git LFS, and update `default_mbert_version` / the model path in
+`src/api_interface/config/settings.py`.
+
+The obfuscation hardening (homoglyph + zero-width normalisation in the live
+batching path) is already wired in on this branch — no model change needed for it.

evals/assets/ATTRIBUTION.md

@@ -0,0 +1,33 @@
+# Third-party assets — attribution
+
+## bert-base-multilingual-cased-vocab.txt
+
+- **Source:** Hugging Face model `bert-base-multilingual-cased`
+  (https://huggingface.co/bert-base-multilingual-cased), originally published by
+  Google Research as part of BERT (https://github.com/google-research/bert).
+- **What it is:** the WordPiece vocabulary (119,547 tokens) for the multilingual
+  cased BERT tokenizer. Vendored here so the offline evaluation harness
+  (`evals/run_eval.py`, `evals/calibrate_thresholds.py`) can construct the
+  tokenizer without any network access to the Hugging Face Hub.
+- **License:** Apache License 2.0. The vocabulary file is redistributed
+  unmodified under the terms of that license; see
+  https://www.apache.org/licenses/LICENSE-2.0 for the full text.
+- **Storage:** tracked via Git LFS (see `.gitattributes`).
+
+## mishra_soni_5971.csv (evals/datasets/)
+
+- **Source:** "SMS Phishing Dataset for Machine Learning and Pattern Recognition"
+  by Sandhya Mishra and Devpriya Soni, Mendeley Data V1
+  (https://data.mendeley.com/datasets/f45bkkt8pr/1), `Dataset_5971.csv`.
+- **What it is:** 5,971 SMS messages labelled ham / spam / smishing, used by the
+  offline evaluation harness as a phishing-vs-spam benchmark
+  (`evals/run_eval.py --dataset mishra:...`).
+- **License:** CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).
+  Redistributed here with attribution as required by the license. Please cite:
+  Mishra, S., Soni, D. (2023), Mendeley Data, doi:10.17632/f45bkkt8pr.1.
+- **Changes:** re-encoded to UTF-8 CSV with `text,label` columns; otherwise
+  content is unmodified.
+
+No other changes have been made to the vendored files. This project
+(OpenTextShield) is the work of TelecomsXChange (TCXC); the attributions above
+cover only the third-party artifacts named.

evals/calibrate_thresholds.py

@@ -0,0 +1,140 @@
+#!/usr/bin/env python3
+"""
+Per-class logit-bias calibration for the OpenTextShield mBERT classifier.
+
+The model collapses much of `phishing` into `spam` (see evals/REPORT.md). A cheap,
+training-free fix is to add a small additive bias to each class's logit before the
+argmax: nudging the phishing logit up recovers phishing recall, and the spam bias
+can be tuned to keep the spam/ham balance. This searches a small grid of (spam,
+phishing) biases on a held-out CSV and reports the combination that maximises a
+chosen objective, plus the before/after metrics.
+
+The chosen bias vector is written to JSON and can be passed straight to
+run_eval.py via --logit-bias to verify the gain on the public benchmarks, or
+applied in production by adding it to the logits before argmax.
+
+Objectives:
+  macro_f1     - balanced 3-class quality (default)
+  block_then_f1 - maximise block accuracy first, break ties by macro-F1
+  phishing_recall - maximise phishing recall (use with care; can hurt spam)
+
+Usage:
+  python evals/calibrate_thresholds.py \
+    --model src/mBERT/training/model-training/mbert_ots_model_2.7.pth \
+    --data  evals/datasets/mishra_soni_5971.csv:mishra \
+    --out   evals/results/calibration.json
+"""
+
+import argparse
+import csv
+import json
+from pathlib import Path
+
+import torch
+from transformers import BertConfig, BertForSequenceClassification, BertTokenizerFast
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+VOCAB_FILE = REPO_ROOT / "evals/assets/bert-base-multilingual-cased-vocab.txt"
+LABELS = ["ham", "spam", "phishing"]
+MAX_LEN = 96
+csv.field_size_limit(10 * 1024 * 1024)
+
+# Reuse the dataset loaders from the eval harness so calibration and evaluation
+# read data identically. They live in loaders.py (no argparse / no side effects),
+# so a normal import is safe. Insert this file's own directory (from __file__,
+# not cwd) so the import resolves whether run as a script or imported elsewhere.
+import sys
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from loaders import LOADERS  # noqa: E402
+
+
+def load_logits(model_path, samples, device="cpu", batch_size=32):
+    tok = BertTokenizerFast(vocab_file=str(VOCAB_FILE), do_lower_case=False)
+    model = BertForSequenceClassification(BertConfig(vocab_size=119547, num_labels=3))
+    model.load_state_dict(torch.load(model_path, map_location=device, weights_only=True))
+    model.eval().to(device)
+    texts = [s["text"] for s in samples]
+    order = sorted(range(len(texts)), key=lambda i: len(texts[i]))
+    logits = [None] * len(texts)
+    with torch.inference_mode():
+        for start in range(0, len(order), batch_size):
+            idx = order[start:start + batch_size]
+            enc = tok([texts[i] for i in idx], padding=True, truncation=True,
+                      max_length=MAX_LEN, return_tensors="pt").to(device)
+            out = model(**enc).logits.float().cpu()
+            for j, i in enumerate(idx):
+                logits[i] = out[j]
+    return torch.stack(logits)
+
+
+def metrics_for(logits, golds, bias):
+    preds = (logits + torch.tensor(bias)).argmax(dim=1).tolist()
+    n = len(golds)
+    acc = sum(p == g for p, g in zip(preds, golds)) / n
+    block = sum((g != 0) == (p != 0) for p, g in zip(preds, golds)) / n
+    # macro-F1
+    f1s = []
+    for c in range(3):
+        tp = sum(p == c and g == c for p, g in zip(preds, golds))
+        fp = sum(p == c and g != c for p, g in zip(preds, golds))
+        fn = sum(p != c and g == c for p, g in zip(preds, golds))
+        prec = tp / (tp + fp) if tp + fp else 0.0
+        rec = tp / (tp + fn) if tp + fn else 0.0
+        f1s.append(2 * prec * rec / (prec + rec) if prec + rec else 0.0)
+    macro_f1 = sum(f1s) / 3
+    gold2 = sum(g == 2 for g in golds)
+    phish_recall = sum(p == 2 and g == 2 for p, g in zip(preds, golds)) / gold2 if gold2 else 0.0
+    return {"accuracy": round(acc, 4), "block_accuracy": round(block, 4),
+            "macro_f1": round(macro_f1, 4), "phishing_recall": round(phish_recall, 4)}
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--model", required=True)
+    ap.add_argument("--data", required=True, help="path:loader (e.g. evals/datasets/mishra_soni_5971.csv:mishra)")
+    ap.add_argument("--out", default=str(REPO_ROOT / "evals/results/calibration.json"))
+    ap.add_argument("--objective", choices=["macro_f1", "block_then_f1", "phishing_recall"],
+                    default="macro_f1")
+    ap.add_argument("--grid", type=float, default=2.0, help="max abs bias to search")
+    ap.add_argument("--steps", type=int, default=9, help="grid points per class")
+    args = ap.parse_args()
+
+    path, loader = args.data.rsplit(":", 1)
+    samples = LOADERS[loader](Path(path))
+    golds = [LABELS.index(s["gold"]) for s in samples]
+    print(f"Loaded {len(samples)} samples via '{loader}'")
+
+    logits = load_logits(args.model, samples)
+    baseline = metrics_for(logits, golds, [0.0, 0.0, 0.0])
+    print(f"Baseline (no bias): {baseline}")
+
+    grid = [round(-args.grid + i * (2 * args.grid) / (args.steps - 1), 3) for i in range(args.steps)]
+    best = None
+    for sb in grid:
+        for pb in grid:
+            m = metrics_for(logits, golds, [0.0, sb, pb])
+            if args.objective == "macro_f1":
+                key = (m["macro_f1"],)
+            elif args.objective == "block_then_f1":
+                key = (m["block_accuracy"], m["macro_f1"])
+            else:
+                key = (m["phishing_recall"], m["macro_f1"])
+            if best is None or key > best["key"]:
+                best = {"key": key, "bias": [0.0, sb, pb], "metrics": m}
+
+    print(f"Best bias (ham/spam/phishing): {best['bias']}")
+    print(f"Calibrated: {best['metrics']}")
+    result = {"model": args.model, "data": args.data, "objective": args.objective,
+              "baseline": baseline, "best_bias": best["bias"],
+              "calibrated": best["metrics"]}
+    Path(args.out).parent.mkdir(parents=True, exist_ok=True)
+    with open(args.out, "w") as f:
+        json.dump(result, f, indent=2)
+    print(f"Saved to {args.out}")
+    print(f"\nVerify on a benchmark with:\n"
+          f"  python evals/run_eval.py --model {args.model} "
+          f"--dataset {loader}:{path} --logit-bias {','.join(map(str, best['bias']))}")
+
+
+if __name__ == "__main__":
+    main()