[cristiangirlea]

Summary

PR #17 measured the redact task's NER pass. It found every entity in news-style sentences, about four in five in terse records, and about one in eight in informal lower-case messages. It also found a surname left visible after an initial. This PR closes most of those gaps, measured the same way.

How the changes were chosen. Tuning on the published sample would overfit its figures. So the rules were chosen on a separate, invented development set, tests/fixtures/ner-dev.json, with 31 sentences. Some of them hold no entity, and some are full of acronyms, so that false positives show. The published sample, tests/fixtures/ner-sample.json, stayed unseen until the final measurement. NER_SAMPLE points the measurement at either set.

What changed in the worker (worker/ai_bridge/tasks.py):

• A second, title-cased reading. A sentence written all in lower case or all in capitals is read again from a title-cased copy that keeps every offset. So is a surname in capitals before a comma and a given name, as in WIERZBICKI, Tomasz, the records convention.
  • Sentence-case and mixed-case sentences are left alone. Title-casing "the board approved" made "Board" an organisation at 0.99.
  • Capitals elsewhere are left alone too. Title-cased, NATO lost its confidence and CEO became an organisation.
• Pieces of one name are joined before the threshold. Pieces that touch, or meet at a hyphen, full stop or apostrophe, become one entity, scored by their most confident piece. So the initial of A. Okonkwo carries the surname the model was less sure of. My first version averaged the pieces instead. On the development set that dropped Jean-Luc Moreau below the threshold entirely, so it was changed.
• The merge keeps the more confident of two readings. Two model candidates with the same extent now keep the higher score. The as-written reading of WIERZBICKI was a place at 0.71; the title-cased one, a person at 0.998. A rule still outranks the model.

What changed in the measurement (scripts/ner_measure.py):

• A masked column. It reports the share of entities hidden under any label, which is what a redaction does.
• One job per threshold again. PR feat: measure the NER pass's precision and recall on an invented sample #17 filtered a single low-threshold job locally. That stopped being equivalent once two readings meet in the worker's merge: a lower threshold lets weaker spans in, where they can displace stronger ones.

Results

On the published sample, measured by the model workflow on GitHub's runner. The "before" column is PR #17's CI run on the same runner type. Threshold 0.85, exact boundaries:

	Recall before	Recall now	Masked now, any label
news	1.00	1.00	1.00
informal	0.13	0.47	0.63
records	0.78	0.92	0.97
PER	0.57	0.79	0.83

At 0.5, informal masking reaches 0.87 and records 1.00. On the development set at 0.85, recall rose from 0.18 to 0.64 in lower case, from 0.00 to 0.67 in all capitals and from 0.27 to 0.64 in records. Its sentences with no entity stayed clean.

What still leaks:

• a lower-case name in a sentence that also holds a capital, such as Regards, olumide;
• a handle, such as @nadia;
• some lower-case first names side by side, such as ines and pedro.

A title-cased reading can also take a filler word into a name (lol tomasz) or call a person an organisation, but the name stays hidden. The deeper fix is a case-insensitive NER model, which would need its own pinning and measurement. The plan records that.

The ONNX redact description and the protocol now describe the second reading, and quote the masked figures. docs/testing.md publishes the table and says how the rules were chosen.

Review

A self-review at high effort produced 9 findings. Eight are fixed, tests first:

• The second reading covered the whole text. Its spans in sentences it hadn't changed could replace the original's labels, and a long text with one lower-case line paid a full second pass. It now reads only the runs of rewritten sentences, with spans shifted back to the text's offsets. Both sets measure exactly as before, and it costs a window or two.
• The docs, the protocol and the tool description quoted PR feat: measure the NER pass's precision and recall on an invented sample #17's figures. They are refreshed from this PR's CI table. The protocol now says a joined name carries its most confident piece's score, and gives the new merge order.
• The measurement treated HTTP 429 as fatal. That happened to me while tuning. It now waits up to 400 s, since the worker holds finished jobs for 300 s.
• No test confined the second reading to its sentences. One does now.
• Both sets were written by the same author. The docs say so, and say the figures describe these sentences, not informal text in general.
• The casing rules work around a cased-only model. The docs and the plan now name a case-insensitive model as the deeper fix.

Skipped: the surname-first rule landed in 180ac82, whose message mentions only the merge change. The history is already pushed, so I recorded this in the next commit's message instead of rewriting it.

Verification

The tests were written first and observed failing at every step: the development-set setting, the second reading, the joining, the joining rule's change from mean to best piece, the surname rule, the merge tie-break, the per-threshold jobs, the masked column, the confinement and the capacity wait.

• python-tests: 109 passed, up from 91.
• mcp-tests: 119 passed with the new description.
• php-tests and php-integration are unaffected; this changes only the ONNX path.
• The model workflow passed on the measured head, run 36862817554, and its table is the one published. It passed again on the final head 6900249, run 36863750957, after the docs and the tool description changed.