SMS classification intelligence evaluation and targeted model improvement

[ajamous]

Summary

Evaluates the OpenTextShield mBERT classifier against public SMS-classification benchmarks plus a new multilingual adversarial suite, identifies where its "intelligence" breaks down, and closes those gaps with a targeted synthetic dataset, an incremental fine-tune (shipped as model 2.7), and a set of Tier-1 training/serving improvements — with no regression on classic spam.

All evaluation is offline and reproducible: weights are loaded directly, the mBERT vocab is vendored, and no API server or external model download is required.

Findings (v2.5 baseline)

Benchmark	n	3-class acc	Block acc	Phishing recall
UCI SMS Spam (classic)	5,574	99.3%	99.3%	n/a (no phishing class)
Mishra & Soni SMS Phishing	5,971	88.8%	99.2%	2.7%
IMC 2025 smishing (modern)	8,007	20.4%	69.9%	17.1%
Adversarial suite (new, decontaminated)	127	48.0%	75.6%	11.9%

• Strong on classic SMS spam (UCI). Contamination note: ~42% of unique UCI texts (and ~36% of Mishra & Soni) appear verbatim in the v2.4 training corpus, so absolute scores on those two are not leaderboard-comparable — they serve as internal regression gates, and the informative number is the v2.5 → v2.7 delta on identical data (details in evals/REPORT.md §2).
• The phishing class is collapsed into spam on Mishra & Soni — 617 of 638 smishing messages mislabelled as spam.
• On modern, real-world smishing (IMC25), ~30% gets an allow decision and ~25% of phishing is silently labelled ham. Conversational ("hi mum"), brand-impersonation (toll/delivery fee, bank lock), vishing-callback, and obfuscated attacks evade detection; non-English block rate falls to ~52–56%.

Improvement (shipped model 2.7)

Continues training from v2.5 on a targeted synthetic dataset (modern smishing archetypes + hard-negative ham twins, 15 languages) mixed with a rehearsal sample of the original corpus to prevent catastrophic forgetting. Numbers below are the shipped 2.7 weights, from evals/results/summary_v2.5.json / summary_v2.7.json:

Benchmark	Block acc (v2.5 → 2.7)	Phishing recall (v2.5 → 2.7)
UCI SMS Spam (classic)	99.3% → 99.5%	n/a
Mishra & Soni Phishing	99.2% → 99.3%	2.7% → 6.1%
IMC25 smishing (modern)	69.9% → 72.4%	17.1% → 45.7%
Adversarial suite (decontaminated)	75.6% → 96.9%	11.9% → 80.6%

Block rate improves on every benchmark with zero classic-spam regression. Per-language and per-category breakdowns are in evals/REPORT.md §5.

Evidence integrity (2026-07-01 update)

• The adversarial suite was decontaminated against the fine-tune training set: 23 of 150 rows matching synthetic_fable5_v1.csv verbatim or at ≥0.8 token overlap were removed, and both models re-scored on the clean 127-row suite. The gains are virtually unchanged (block 96.9% vs 96.7% pre-decontamination), confirming they are not memorization.
• UCI / Mishra & Soni training-corpus overlap is now disclosed in REPORT.md, and both are framed as regression gates rather than independent confirmations. The cleanest external signal is the IMC25 block-rate gain.
• Known limitation: the suite's hard-negative ham is thin on transactional A2P. A dedicated multilingual false-positive benchmark is in progress (data: FP benchmark + gap-fix training set for transactional A2P #209) and shows over-blocking of branded transactional ham (measured 36.4% FP for 2.7 vs 30.7% for v2.5 on that draft benchmark, concentrated in government-sender messages) — the target of the next fine-tune iteration.

Tier-1 improvements (this branch)

• evals/finetune_tier1.py — Apple Silicon MPS / CUDA / CPU auto-detection, stratified train/validation split with per-epoch metrics, best-checkpoint selection by validation macro-F1, and class-weighted / focal loss options.
• evals/calibrate_thresholds.py + run_eval.py --logit-bias — training-free per-class logit-bias calibration. On the adversarial set this alone lifts block rate and phishing recall substantially on the unchanged v2.5 weights.
• Obfuscation hardening (production path) — strips zero-width / invisible formatting characters in the preprocessor and runs homoglyph + zero-width normalisation inside the dynamic-batching path (previously only the per-request path normalised, so the production default never did).
• evals/TIER1.md — runbook to reproduce all of the above on Apple Silicon in minutes.

What's included

• evals/run_eval.py — offline evaluation harness (3-class accuracy, block rate, per-class P/R/F1, confusion matrix, per-language/category breakdowns; optional logit-bias)
• evals/datasets/fable5_adversarial_v1.csv — 127-message multilingual adversarial suite (decontaminated against the fine-tune set)
• evals/datasets/mishra_soni_5971.csv — Mishra & Soni SMS Phishing benchmark + loader (CC BY 4.0, see evals/assets/ATTRIBUTION.md)
• evals/generate_synthetic.py + dataset/synthetic_fable5_v1.csv — targeted synthetic data generator and output
• evals/finetune_incremental.py / evals/finetune_tier1.py — fine-tune scripts
• evals/calibrate_thresholds.py, evals/compare_runs.py — calibration and comparison utilities
• evals/REPORT.md, evals/TIER1.md — written report and Apple-Silicon runbook
• evals/results/ — baseline and post-fine-tune metric summaries

Model weights

The fine-tuned weights mbert_ots_model_2.7.pth (~680 MB) are committed to this branch via Git LFS and the API default points at them. They are also fully reproducible from the included scripts. The shipped recipe is a gentle, overfitting-aware fine-tune (plain loss, lr 1e-5, 2 epochs) trained on Apple Silicon MPS with best-checkpoint selection — heavier configs won the in-domain validation split but regressed the real-world IMC25 block rate. Follow evals/TIER1.md or:

python evals/finetune_tier1.py \
    --base   src/mBERT/training/model-training/mbert_ots_model_2.5.pth \
    --synthetic src/mBERT/training/model-training/dataset/synthetic_fable5_v1.csv \
    --original  src/mBERT/training/model-training/dataset/sms_spam_phishing_dataset_v2.4_combined.csv \
    --out    src/mBERT/training/model-training/mbert_ots_model_2.7.pth \
    --epochs 2 --batch-size 32 --loss plain --lr 1e-5

Rollback: set OTS_MBERT_MODEL_PATH to a previous checkpoint (e.g. the v2.5 weights) to switch models via env with no code change.

Notes / next steps

• Close the transactional-A2P false-positive gap surfaced by data: FP benchmark + gap-fix training set for transactional A2P #209 (next fine-tune iteration, model 2.8), ideally rehearsing from the deduplicated dataset in Fix train/test leakage from duplicate dataset rows #195 rather than the duplicated v2.4 corpus.
• Bigger gains (especially non-English) would come from real labelled smishing feeds (operator 7726 reports, full IMC25 with a clean split) and from upgrading the mBERT backbone to XLM-RoBERTa / mDeBERTa-v3.

[Frhnfaya]

Hi @ajamous I Went through the eval harness closely — solid, security-relevant work, and wiring normalisation into the batching path is the right production catch.

Building on the automated review rather than repeating it: the three Critical items aren't independent — they all distort the same number REPORT.md leans on hardest (IMC25 block / phishing recall). In that light:

1. load_imc25 (run_eval.py:116) — gold = "spam" if scam_type == "spam" else "phishing" forces every non-spam row (empty / legit / unmapped) into phishing, inflating the headline phishing class. The fix is an explicit map + skip-and-count for unknowns, but the map should come from the data, not a guess:

from collections import Counter
print(Counter((r.get("scam_type") or "").strip().lower() for r in csv.DictReader(open(IMC25_PATH))))

Happy to send a small PR with the corrected loader once we confirm whether IMC25 has a legit/ham class (which should map to ham, not be skipped).

2. Circular eval — worth stating outright in REPORT.md: fable5 was authored by the same model that generated the fine-tune data, so the 98% adversarial figure is in-distribution. IMC25's +8.3pts is the honest number to lead with.

3. v2.7 loss mismatch — if v2.7 shipped with plain loss rather than the documented --loss class_weighted, the report's phishing-recall gains may not be in the merged model. I can re-run finetune_tier1.py with the intended settings + a held-out split on GPU and post validation numbers before merge if useful.

[Frhnfaya]

Saw the v2.10.0-rc.1 / model 2.7 release — solid. The "gentle recipe beats class-weighted because the heavier config overfit IMC25" result is a clean finding worth keeping in REPORT.md, and the explicit "treat 72.4% as validation, UCI/Mishra/adversarial as independent" framing resolves the circular-eval point well.

Correction on my earlier point #1: I pulled the public IMC25 set to check the scam_type values before assuming. It's banking / delivery / government / telecom / others / spam / wrong number / hey-mum-dad — no ham/legit class. So gold = "spam" if scam_type=="spam" else "phishing" is correct for this corpus; my "inflating phishing" concern doesn't hold. The only edge case is ~81 rows (0.2%) with an empty scam_type silently falling to phishing — a continue-and-count would just make that explicit. Minor.

The one methodology gap that does stand: IMC25 is all-attack, so the multilingual eval has no legitimate-message control. Block rate there measures attack recall, but there's no signal on non-English false positives — whether the model over-blocks legit non-English SMS, which is the weakest area in the per-language breakdown. A small multilingual ham control set alongside it would close that. Happy to put one together if useful.

[ajamous]

Saw the v2.10.0-rc.1 / model 2.7 release — solid. The "gentle recipe beats class-weighted because the heavier config overfit IMC25" result is a clean finding worth keeping in REPORT.md, and the explicit "treat 72.4% as validation, UCI/Mishra/adversarial as independent" framing resolves the circular-eval point well.

Correction on my earlier point #1: I pulled the public IMC25 set to check the scam_type values before assuming. It's banking / delivery / government / telecom / others / spam / wrong number / hey-mum-dad — no ham/legit class. So gold = "spam" if scam_type=="spam" else "phishing" is correct for this corpus; my "inflating phishing" concern doesn't hold. The only edge case is ~81 rows (0.2%) with an empty scam_type silently falling to phishing — a continue-and-count would just make that explicit. Minor.

The one methodology gap that does stand: IMC25 is all-attack, so the multilingual eval has no legitimate-message control. Block rate there measures attack recall, but there's no signal on non-English false positives — whether the model over-blocks legit non-English SMS, which is the weakest area in the per-language breakdown. A small multilingual ham control set alongside it would close that. Happy to put one together if useful.

@Frhnfaya Thanks for pulling the IMC25 set to verify rather than assuming, agreed, no ham/legit class in that corpus, so the non-spam→phishing mapping holds.

I’ll have the loader skip-and-count the ~81 empty scam_type rows so it’s explicit rather than silent.
The false-positive point is the one that matters. IMC25 only measures attack recall, so we have no read on whether v2.7 over-blocks legitimate non-English SMS and the per-language breakdown is exactly where that risk lives.

Yes to the multilingual ham control set. Even a couple hundred legit messages across the weaker languages would let us track per-language false-positive rate next to block rate. If you put it together I’ll wire it into run_eval as its own benchmark and report FP rates per language alongside the existing metrics.

On the loss config: plain / lr 1e-5 / 2 epochs was deliberate — class-weighted at 4 epochs overfit IMC25 and dropped block rate to ~65%. I’ll make REPORT.md and TIER1.md state that explicitly so it doesn’t get “corrected” back to the overfitting recipe.

[Frhnfaya]

Saw the v2.10.0-rc.1 / model 2.7 release — solid. The "gentle recipe beats class-weighted because the heavier config overfit IMC25" result is a clean finding worth keeping in REPORT.md, and the explicit "treat 72.4% as validation, UCI/Mishra/adversarial as independent" framing resolves the circular-eval point well.
Correction on my earlier point #1: I pulled the public IMC25 set to check the scam_type values before assuming. It's banking / delivery / government / telecom / others / spam / wrong number / hey-mum-dad — no ham/legit class. So gold = "spam" if scam_type=="spam" else "phishing" is correct for this corpus; my "inflating phishing" concern doesn't hold. The only edge case is ~81 rows (0.2%) with an empty scam_type silently falling to phishing — a continue-and-count would just make that explicit. Minor.
The one methodology gap that does stand: IMC25 is all-attack, so the multilingual eval has no legitimate-message control. Block rate there measures attack recall, but there's no signal on non-English false positives — whether the model over-blocks legit non-English SMS, which is the weakest area in the per-language breakdown. A small multilingual ham control set alongside it would close that. Happy to put one together if useful.

@Frhnfaya Thanks for pulling the IMC25 set to verify rather than assuming, agreed, no ham/legit class in that corpus, so the non-spam→phishing mapping holds.

I’ll have the loader skip-and-count the ~81 empty scam_type rows so it’s explicit rather than silent. The false-positive point is the one that matters. IMC25 only measures attack recall, so we have no read on whether v2.7 over-blocks legitimate non-English SMS and the per-language breakdown is exactly where that risk lives.

Yes to the multilingual ham control set. Even a couple hundred legit messages across the weaker languages would let us track per-language false-positive rate next to block rate. If you put it together I’ll wire it into run_eval as its own benchmark and report FP rates per language alongside the existing metrics.

On the loss config: plain / lr 1e-5 / 2 epochs was deliberate — class-weighted at 4 epochs overfit IMC25 and dropped block rate to ~65%. I’ll make REPORT.md and TIER1.md state that explicitly so it doesn’t get “corrected” back to the overfitting recipe.

Hi @ajamous . I Put the multilingual ham control set together and ran it through model 2.7 (47 legit
messages across en/es/it/pt — otp, bank, delivery, government, telecom, appointment,
plus personal/personal_money), using the existing csv loader.

Two findings, and the first is reassuring:

Conversational ham is clean in all four languages — personal and personal_money
pass 8/8. So no multilingual over-blocking of genuine personal ham.

Realistic transactional A2P gets classified as spam, and it's language-independent
(English too, not only non-English):

bank_alert 0/7, government 0/7, appointment 0/4 pass; delivery and
telecom 1/7 each
false-positive rate by language: en 50%, es 69%, it 77%, pt 62%
every error goes to spam, none to phishing
I checked whether this is a real FP or just a labeling difference: in the v2.4 training
set, ham already includes ~4,950 transactional rows ("Your order #X has been shipped",
"Your account balance is $X", "Your verification code is X"), so transactional A2P is
meant to be ham. The likely cause is that those ham examples are short and templated, so
the model doesn't generalize to realistic, branded A2P (real bank/delivery/gov texts) —
and the gap widens in non-English.

For a telecom firewall this is exactly the high-value traffic that must pass, so it
seemed worth surfacing. The set is in the fable5 format and ready to drop into
evals/datasets/ as a ham benchmark (block rate on it = false-positive rate). Happy to
extend the languages and split clean-ham vs transactional so the two signals stay
separate.

[Frhnfaya]

Following up with the data work — focused there since that's where the gap is.

1. Measured the current model (2.7). Built an 88-message held-out benchmark of
legitimate SMS across en/es/it/pt, weighted toward the categories with the thinnest
ham coverage in v2.4. Baseline false-positive rate:

• overall 67% (59/88) — every error to spam, none to phishing
• government 94%, delivery 92%, appointment/telecom 75%, bank_alert 67%, otp 50%
• personal/personal_money 0% — conversational ham is clean, so it's specifically a
  transactional-A2P problem, not multilingual (en 68% ≈ es/it 64%)

2. Root cause is in v2.4. Transactional ham is thin and over-templated: government
174 ham vs 2,208 spam, appointment only 11 ham rows, and ham-transactional examples
median ~34 chars ("Your order #X shipped"), so the model never generalised to realistic
branded messages.

3. Built a training set to close it. 486 rows: 282 realistic branded A2P ham + 204
matched hard-negative scam twins (labelled phishing, so it won't blind the model to real
attacks). Gap-weighted (government/delivery heaviest), four languages, and
auto-deduplicated against the benchmark — zero overlap, leakage-free.

Honest caveat: the benchmark and training set are disjoint but both constructed, so the
before/after proves the model can learn the A2P-is-ham boundary when given the data; a
sample of real operator A2P would be the gold-standard confirmation on top.

If useful, you could fine-tune on it (mixed with a rehearsal sample, same recipe as the
2.6/2.7 incremental run) and we re-run the same benchmark to measure the delta. I can
open a small PR adding the benchmark, training set, and a short README to evals/datasets/,
or send the files however you prefer.

[ajamous]

Saw the v2.10.0-rc.1 / model 2.7 release — solid. The "gentle recipe beats class-weighted because the heavier config overfit IMC25" result is a clean finding worth keeping in REPORT.md, and the explicit "treat 72.4% as validation, UCI/Mishra/adversarial as independent" framing resolves the circular-eval point well.
Correction on my earlier point #1: I pulled the public IMC25 set to check the scam_type values before assuming. It's banking / delivery / government / telecom / others / spam / wrong number / hey-mum-dad — no ham/legit class. So gold = "spam" if scam_type=="spam" else "phishing" is correct for this corpus; my "inflating phishing" concern doesn't hold. The only edge case is ~81 rows (0.2%) with an empty scam_type silently falling to phishing — a continue-and-count would just make that explicit. Minor.
The one methodology gap that does stand: IMC25 is all-attack, so the multilingual eval has no legitimate-message control. Block rate there measures attack recall, but there's no signal on non-English false positives — whether the model over-blocks legit non-English SMS, which is the weakest area in the per-language breakdown. A small multilingual ham control set alongside it would close that. Happy to put one together if useful.

@Frhnfaya Thanks for pulling the IMC25 set to verify rather than assuming, agreed, no ham/legit class in that corpus, so the non-spam→phishing mapping holds.
I’ll have the loader skip-and-count the ~81 empty scam_type rows so it’s explicit rather than silent. The false-positive point is the one that matters. IMC25 only measures attack recall, so we have no read on whether v2.7 over-blocks legitimate non-English SMS and the per-language breakdown is exactly where that risk lives.
Yes to the multilingual ham control set. Even a couple hundred legit messages across the weaker languages would let us track per-language false-positive rate next to block rate. If you put it together I’ll wire it into run_eval as its own benchmark and report FP rates per language alongside the existing metrics.
On the loss config: plain / lr 1e-5 / 2 epochs was deliberate — class-weighted at 4 epochs overfit IMC25 and dropped block rate to ~65%. I’ll make REPORT.md and TIER1.md state that explicitly so it doesn’t get “corrected” back to the overfitting recipe.

Hi @ajamous . I Put the multilingual ham control set together and ran it through model 2.7 (47 legit messages across en/es/it/pt — otp, bank, delivery, government, telecom, appointment, plus personal/personal_money), using the existing csv loader.

Two findings, and the first is reassuring:

Conversational ham is clean in all four languages — personal and personal_money pass 8/8. So no multilingual over-blocking of genuine personal ham.

Realistic transactional A2P gets classified as spam, and it's language-independent (English too, not only non-English):

bank_alert 0/7, government 0/7, appointment 0/4 pass; delivery and telecom 1/7 each false-positive rate by language: en 50%, es 69%, it 77%, pt 62% every error goes to spam, none to phishing I checked whether this is a real FP or just a labeling difference: in the v2.4 training set, ham already includes ~4,950 transactional rows ("Your order #X has been shipped", "Your account balance is $X", "Your verification code is X"), so transactional A2P is meant to be ham. The likely cause is that those ham examples are short and templated, so the model doesn't generalize to realistic, branded A2P (real bank/delivery/gov texts) — and the gap widens in non-English.

Could you please share the steps to reproduce this issue ? Thanks!

[ajamous]

Following up with the data work — focused there since that's where the gap is.

1. Measured the current model (2.7). Built an 88-message held-out benchmark of legitimate SMS across en/es/it/pt, weighted toward the categories with the thinnest ham coverage in v2.4. Baseline false-positive rate:

• overall 67% (59/88) — every error to spam, none to phishing
• government 94%, delivery 92%, appointment/telecom 75%, bank_alert 67%, otp 50%
• personal/personal_money 0% — conversational ham is clean, so it's specifically a
  transactional-A2P problem, not multilingual (en 68% ≈ es/it 64%)

2. Root cause is in v2.4. Transactional ham is thin and over-templated: government 174 ham vs 2,208 spam, appointment only 11 ham rows, and ham-transactional examples median ~34 chars ("Your order #X shipped"), so the model never generalised to realistic branded messages.

3. Built a training set to close it. 486 rows: 282 realistic branded A2P ham + 204 matched hard-negative scam twins (labelled phishing, so it won't blind the model to real attacks). Gap-weighted (government/delivery heaviest), four languages, and auto-deduplicated against the benchmark — zero overlap, leakage-free.

Honest caveat: the benchmark and training set are disjoint but both constructed, so the before/after proves the model can learn the A2P-is-ham boundary when given the data; a sample of real operator A2P would be the gold-standard confirmation on top.

If useful, you could fine-tune on it (mixed with a rehearsal sample, same recipe as the 2.6/2.7 incremental run) and we re-run the same benchmark to measure the delta. I can open a small PR adding the benchmark, training set, and a short README to evals/datasets/, or send the files however you prefer.

Hi @Frhnfaya Go ahead and open your PR, including steps to reproduce and test. I will take a look ASAP. Thanks again for your contribution.

[Frhnfaya]

Saw the v2.10.0-rc.1 / model 2.7 release — solid. The "gentle recipe beats class-weighted because the heavier config overfit IMC25" result is a clean finding worth keeping in REPORT.md, and the explicit "treat 72.4% as validation, UCI/Mishra/adversarial as independent" framing resolves the circular-eval point well.
Correction on my earlier point #1: I pulled the public IMC25 set to check the scam_type values before assuming. It's banking / delivery / government / telecom / others / spam / wrong number / hey-mum-dad — no ham/legit class. So gold = "spam" if scam_type=="spam" else "phishing" is correct for this corpus; my "inflating phishing" concern doesn't hold. The only edge case is ~81 rows (0.2%) with an empty scam_type silently falling to phishing — a continue-and-count would just make that explicit. Minor.
The one methodology gap that does stand: IMC25 is all-attack, so the multilingual eval has no legitimate-message control. Block rate there measures attack recall, but there's no signal on non-English false positives — whether the model over-blocks legit non-English SMS, which is the weakest area in the per-language breakdown. A small multilingual ham control set alongside it would close that. Happy to put one together if useful.

@Frhnfaya Thanks for pulling the IMC25 set to verify rather than assuming, agreed, no ham/legit class in that corpus, so the non-spam→phishing mapping holds.
I’ll have the loader skip-and-count the ~81 empty scam_type rows so it’s explicit rather than silent. The false-positive point is the one that matters. IMC25 only measures attack recall, so we have no read on whether v2.7 over-blocks legitimate non-English SMS and the per-language breakdown is exactly where that risk lives.
Yes to the multilingual ham control set. Even a couple hundred legit messages across the weaker languages would let us track per-language false-positive rate next to block rate. If you put it together I’ll wire it into run_eval as its own benchmark and report FP rates per language alongside the existing metrics.
On the loss config: plain / lr 1e-5 / 2 epochs was deliberate — class-weighted at 4 epochs overfit IMC25 and dropped block rate to ~65%. I’ll make REPORT.md and TIER1.md state that explicitly so it doesn’t get “corrected” back to the overfitting recipe.

Hi @ajamous . I Put the multilingual ham control set together and ran it through model 2.7 (47 legit messages across en/es/it/pt — otp, bank, delivery, government, telecom, appointment, plus personal/personal_money), using the existing csv loader.
Two findings, and the first is reassuring:
Conversational ham is clean in all four languages — personal and personal_money pass 8/8. So no multilingual over-blocking of genuine personal ham.
Realistic transactional A2P gets classified as spam, and it's language-independent (English too, not only non-English):
bank_alert 0/7, government 0/7, appointment 0/4 pass; delivery and telecom 1/7 each false-positive rate by language: en 50%, es 69%, it 77%, pt 62% every error goes to spam, none to phishing I checked whether this is a real FP or just a labeling difference: in the v2.4 training set, ham already includes ~4,950 transactional rows ("Your order #X has been shipped", "Your account balance is $X", "Your verification code is X"), so transactional A2P is meant to be ham. The likely cause is that those ham examples are short and templated, so the model doesn't generalize to realistic, branded A2P (real bank/delivery/gov texts) — and the gap widens in non-English.

Could you please share the steps to reproduce this issue ? Thanks!

@ajamous Reproduction steps and all files are in PR #209. The build scripts are
self-contained — running build_benchmark.py generates the CSV, then the
existing csv loader handles the eval run. Steps and expected output are
in the PR description.