[cristiangirlea]

Summary

PR #18 left three kinds of name visible, all written in lower case where the cased NER model cannot see them:

• a lower-case name in a sentence that also holds a capital (Regards, olumide), which the title-cased second reading never rewrites;
• a handle (@nadia);
• some lower-case first names side by side.

That PR named the deeper fix: a model that ignores case. This PR adds one beside the cased model rather than in its place, because the measurement showed the two fail on different sentences.

The model. dslim/bert-base-NER-uncased is the same author's model trained on the same data, CoNLL-2003 English news. It is used in the int8 ONNX conversion that Xenova published alongside the cased one: Xenova/bert-base-NER-uncased at 5e99d439.

• Its model_int8.onnx is 104.6 MiB, inside the fetcher's 128 MiB limit. It is pinned by revision and SHA-256 like the others, in its own directory, redact-uncased.
• Its tokenizer is byte-identical to the rerank model's bert-base-uncased tokenizer.
• Its label order is the cased model's, so NER_LABELS serves both.
• The other candidate the Hub search turned up with an int8 conversion was elastic/distilbert-base-uncased-finetuned-conll03-english (63.8 MiB, in onnx-community's conversion). It is cheaper, but its precision was lower (0.77 against 0.93 alone), and it orders its labels differently, so it would have needed a second label table.

How it was chosen. On the development set only; the published sample stayed unseen until CI measured it. Before measuring any candidate, I added 20 invented sentences to tests/fixtures/ner-dev.json:

• lower-case names among capitals;
• four news sentences, where ignoring case could cost what capitals tell;
• eight sentences of words that are also names (pay the bill, a rose, Will the courier…), where an uncased model could invent a person.

Configurations at 0.85, on the 51 development sentences:

Configuration	Exact recall	Masked	Precision, exact	Inference time vs main
cased + title-cased reading (main)	0.67	0.69	0.90	1.0x
uncased BERT alone	0.71	0.76	0.93	0.7x
uncased DistilBERT alone	0.73	0.85	0.77	0.3x
cased + uncased BERT	0.82	0.89	0.87	1.5x
cased + title-cased reading + uncased BERT	0.87	0.93	0.89	1.6x
cased + title-cased reading + uncased DistilBERT	0.84	0.91	0.84	1.5x

What decided it:

• The uncased model alone is not enough. It lost records, masking 0.54 against main's 0.77: K. Watanabe, OYELARAN, Bisi, Copperleaf Mining. Capitals carry real signal there.
• Together, the two models cover each other's misses. Mixed-case sentences such as Cheers, rahim go from 0.18 to 0.82 masked, and news stays at 1.00.
• No new false positives. The uncased BERT tagged none of the eight name-like sentences. The one false positive, Bill the client, is the cased model's, and main already has it.

What changed:

• scripts/fetch_model.py: the redact-uncased pin.
• worker/ai_bridge/tasks.py: _ner loads both models. The cased model reads the text, then the title-cased copy of each run of uncased sentences. The uncased model reads the whole text as written. Every candidate goes through the existing merge, which keeps the more confident of two readings of the same words. Results report Xenova/bert-base-NER:int8+Xenova/bert-base-NER-uncased:int8.
• The MCP server's model table and the ONNX redact description name both models.

Results

On the published sample, measured by the model workflow at e49e388 (run 36902794112, AMD EPYC 7763), against PR #18's run on the same CPU type. Threshold 0.85:

	Exact recall, #18 → now	Masked under any label, #18 → now	Exact precision, #18 → now
news	1.00 → 1.00	1.00 → 1.00	1.00 → 1.00
informal	0.47 → 0.60	0.63 → 0.80	0.74 → 0.72
records	0.92 → 0.94	0.97 → 1.00	0.94 → 0.94

At 0.5, every register is now fully masked; informal was 0.87 before.

What still leaks at 0.85:

• a name after a sign-off (Regards, olumide);
• a handle (@nadia);
• some lower-case places and first names (porto, galway, fatima);
• a capitalised company in an informal note (Halvern).

The uncased model also tags a bare domain name such as example.com as an organisation. Inside an email address the EMAIL rule's span covers it; elsewhere it is masked when ORG is requested. The protocol now says so.

What it costs. Each cold redact job loads and runs a second 105 MB model. The 2000-character benchmark row rose from 2.40 s to 3.40 s on the EPYC 7763; that run's rerank and embed rows were 5-8% slower than the earlier run's. Locally, on a Ryzen 9 9950X, the row takes 2.76 s. The worst case, 8192 lower-case characters that every reading covers, takes 5.8 s locally, inside the 30 s default deadline. The model cache grows from about 215 MB to about 330 MB.

The redact tool's description, the protocol, the README, the security notes and the testing guide name both models and quote these figures.

Review

A self-review at high effort produced 10 findings, all fixed, tests first where behaviour changed:

• An upgrade would have broken redaction silently. A cache fetched before this branch has no redact-uncased directory. Every ONNX redact job asking for an entity would then fail as task_failed, with nothing logged. Now an ONNX worker refuses to start while any model directory lacks its two files, and names them and the fetcher command. MODEL_DIRECTORIES lists what the backend loads, and a test holds it equal to the fetcher's directories. The README tells operators to fetch again after an upgrade that adds a model.
• The uncased model's label order was checked for PER only. The smoke sentence now holds a lower-case person, place and organisation that only the uncased model finds: I spoke with esperanza in lisbon, she works for brightline insurance. A conversion that ordered its labels like the DistilBERT candidate would fail it.
• Stale timings and figures. The redact timing in the tool description, docs/mcp.md and docs/testing.md was re-measured. The description and the testing guide quote this PR's table.
• The protocol named one model, and the README's licence notice omitted the uncased model. Both are fixed, along with the cache size, the model counts, the test counts and the fetcher's size-limit comment.
• Domain names tagged as organisations. Confirmed with a probe (Visit example.com for details. scored 0.94) and documented in the protocol.
• A test lost its exact span check when its expectations changed. It compares every span again.

Verification

Tests were written first and observed failing at each step: the uncased reading, the pin, both models' names, the start-up refusal, and the directory list. The smoke tests also failed against the old worker on the model name before the implementation, and passed after it, through both the PHP client and FrankenPHP.

• python-tests: 120 passed, up from 109.
• mcp-tests: 119 passed.
• php-tests: 101 contract checks passed.
• Locally, both model smoke tests passed with the three-label case. The worker refused to start against a directory without models.
• The model workflow passed at e49e388: run 36902794112, whose table is the one published.
• After PR ci: run on the lab's self-hosted runner scale set, with Compose pinned where the image lacks it #20 moved CI to the lab's self-hosted runner, main was merged into this branch (001a6ed). Its checks then reran there, one ephemeral lab-php-python-ai-bridge pod per job:
  • Tests, run 37150219679: every PHP version, the MCP server and deterministic passed.
  • The model workflow, run 37150219608, passed on an AMD Ryzen 9 9950X with the same masked shares at 0.85 as the published run: informal 0.80, records 1.00, news 1.00. Informal exact precision was 0.75 against 0.72, within cross-CPU noise.
  • The redact benchmark median there is 1.46 s, against 1.2 s for the single-model task on the same lab runner in PR ci: run on the lab's self-hosted runner scale set, with Compose pinned where the image lacks it #20's run. The published 2.8-3.4 s range comes from Docker Desktop on Windows and an EPYC 7763 runner.