feat: close most of the measured NER recall gaps

docker/compose.yaml

@@ -204,6 +204,7 @@ services:
     environment:
       <<: *python-env
       NER_TARGET: http://model-worker:8090
+      NER_SAMPLE: ${NER_SAMPLE:-}
       BENCH_COMMIT: ${BENCH_COMMIT:-unknown}
       BENCH_RUNNER: ${BENCH_RUNNER:-local, unspecified}
     command: [python, scripts/ner_measure.py]

docs/protocol.md

@@ -25,7 +25,7 @@ Successful reranking returns `{"model":"...","rankings":[{"index":0,"score":1.0}
 
 Submit `{"task":"embed","input":{"texts":["..."]}}` with 1–32 texts of 1–8192 characters each, totalling at most 200000 characters. A successful result is `{"model":"...","dimensions":384,"vectors":[[...]]}` with one unit-length vector per text in input order; components are rounded to six decimals, so the dot product of two vectors is their cosine similarity. The service returns vectors and stores nothing; the caller owns any index built from them. The deterministic backend identifies itself as `hashing-bow-not-a-model`: a signed feature hash of word tokens that is stable across processes and ignores word order. The ONNX backend is `sentence-transformers/all-MiniLM-L6-v2` with mean pooling; it truncates input beyond 256 word pieces and is trained on English text.
 
-Submit `{"task":"redact","input":{"text":"...","entities":["PER"],"min_score":0.85}}` with 1–8192 characters of valid Unicode text. `entities` is an optional list of distinct labels among `PER`, `ORG` and `LOC` (default `["PER"]`; `MISC` is never offered) and `min_score` an optional number from 0 to 1 (default 0.85); both shape only the model pass. A successful result is `{"model":"...","text":"...","spans":[{"start":0,"end":5,"label":"EMAIL","source":"rule:email","score":1.0}]}`: the text with each span replaced by `[LABEL]`, and the spans sorted and disjoint, with offsets counted in Unicode code points of the original text. Deterministic rules run on every backend and are tagged `rule:`: `CARD` (any 13–19 digit window of a digit run that passes Luhn, longest first), `IBAN` (mod 97, trimming trailing groups the pattern swallowed; a lower-case one also needs a country code from the IBAN registry, and mixed case is never taken), `EMAIL` (letters of any script in the local part and the domain, and a letter or punycode top-level label), `IPV4` and `IPV6` (any form Python's `ipaddress` parser accepts that holds a digit, not starting inside a word, an IPv4-mapped tail and a `%zone` identifier included, the colon of `address: message` excluded) at score 1.0, and `PHONE` at 0.8 because it is a pattern, not a proof: 7–15 digits with a leading plus or separators, or at least ten bare digits, never a date shape. Known pattern false positives remain: a dotted quadruple of small numbers such as a version string matches `IPV4`, hex words joined by colons that the parser accepts, such as `dead::beef`, and eight colon-separated groups of up to four digits, such as `10:20:30:40:50:60:70:80`, match `IPV6`, and separated digit groups such as amounts can match `PHONE`; filter by `source` and `score` where that matters. On the default backend, or when `entities` is empty, the model name is `rules-only-not-a-model` and no model runs. The ONNX backend adds spans tagged `model:PER`, `model:ORG` or `model:LOC` carrying the model's mean word confidence, from `Xenova/bert-base-NER:int8`, the int8 ONNX conversion of `dslim/bert-base-NER` (CoNLL-2003, English news). Overlapping candidates are merged into one span carrying the leftmost, then longest, then strongest label, so no part of any candidate stays visible. Redaction is assistive: rules cannot see unstructured personal data, and the model misses names unlike its news training data: on an invented sample it found every entity in news-style sentences but about one in eight in informal lower-case messages ([the measurement](testing.md#measured-ner-precision-and-recall-informational)), so it is not a compliance control.
+Submit `{"task":"redact","input":{"text":"...","entities":["PER"],"min_score":0.85}}` with 1–8192 characters of valid Unicode text. `entities` is an optional list of distinct labels among `PER`, `ORG` and `LOC` (default `["PER"]`; `MISC` is never offered) and `min_score` an optional number from 0 to 1 (default 0.85); both shape only the model pass. A successful result is `{"model":"...","text":"...","spans":[{"start":0,"end":5,"label":"EMAIL","source":"rule:email","score":1.0}]}`: the text with each span replaced by `[LABEL]`, and the spans sorted and disjoint, with offsets counted in Unicode code points of the original text. Deterministic rules run on every backend and are tagged `rule:`: `CARD` (any 13–19 digit window of a digit run that passes Luhn, longest first), `IBAN` (mod 97, trimming trailing groups the pattern swallowed; a lower-case one also needs a country code from the IBAN registry, and mixed case is never taken), `EMAIL` (letters of any script in the local part and the domain, and a letter or punycode top-level label), `IPV4` and `IPV6` (any form Python's `ipaddress` parser accepts that holds a digit, not starting inside a word, an IPv4-mapped tail and a `%zone` identifier included, the colon of `address: message` excluded) at score 1.0, and `PHONE` at 0.8 because it is a pattern, not a proof: 7–15 digits with a leading plus or separators, or at least ten bare digits, never a date shape. Known pattern false positives remain: a dotted quadruple of small numbers such as a version string matches `IPV4`, hex words joined by colons that the parser accepts, such as `dead::beef`, and eight colon-separated groups of up to four digits, such as `10:20:30:40:50:60:70:80`, match `IPV6`, and separated digit groups such as amounts can match `PHONE`; filter by `source` and `score` where that matters. On the default backend, or when `entities` is empty, the model name is `rules-only-not-a-model` and no model runs. The ONNX backend adds spans tagged `model:PER`, `model:ORG` or `model:LOC` carrying the model's mean word confidence, from `Xenova/bert-base-NER:int8`, the int8 ONNX conversion of `dslim/bert-base-NER` (CoNLL-2003, English news). The model was trained on cased text, so a sentence written all in lower case or all in capitals, and a surname in capitals before a comma and a given name, are read a second time from a title-cased copy with the same offsets. Pieces of one name that touch or meet at a hyphen, full stop or apostrophe are joined before the threshold and carry the score of their most confident piece; any other model span carries its mean word confidence. Overlapping candidates are merged into one span carrying the leftmost, then longest, then strongest label, then the more confident of two model readings, so no part of any candidate stays visible. Redaction is assistive: rules cannot see unstructured personal data, and the model misses names unlike its news training data: on an invented sample it masked every entity in news-style sentences and nearly every one in terse records, but about three in five in informal messages ([the measurement](testing.md#measured-ner-precision-and-recall-informational)), so it is not a compliance control.
 
 Failures use generic error objects with `code` and `message`. Task errors include `task_failed`, `worker_crashed`, `worker_start_failed` and `deadline_exceeded`. Internal exception text is not sent to clients. HTTP contract failures return 400, missing auth 401, missing routes/jobs 404, a connection idle for five seconds while its body is read 408 (a stall while reading headers closes the connection without a response), oversized bodies 413 and capacity exhaustion 429.
 

docs/testing.md

@@ -42,7 +42,7 @@ PHP branch support was verified locally on 2026-09-17 on PHP 8.2.33, 8.3.33, 8.4
 
 | Layer | Scope and counting |
 | --- | --- |
-| Python worker | 88 tests: lifecycle, validation and HTTP, process crashes, cancellation, timeouts, retention, allocation/exit races, redaction rules including IPv6, lower-case IBANs and email addresses in any script and their speed on hostile input, NER aggregation and batched cross-encoder scoring driven by stubs, the NER measurement's sample and scoring, progress bounds, and the model-name table every result reports from |
+| Python worker | 109 tests: lifecycle, validation and HTTP, process crashes, cancellation, timeouts, retention, allocation/exit races, redaction rules including IPv6, lower-case IBANs and email addresses in any script and their speed on hostile input, NER aggregation, name joining, the second title-cased reading and the merge driven by stubs, batched cross-encoder scoring, the NER measurement's sample and scoring, progress bounds, and the model-name table every result reports from |
 | MCP server | 119 tests, offline with hash-pinned wheels: the protocol client against an in-process worker, every tool through the SDK's in-memory client with test tasks enabled, the entry point as a subprocess, result validators on synthetic data, the index builder (chunking, the format and each corruption it refuses, building against the worker, caching, its command line) `bridge_search` (the model gate, path confinement, reloads, rerank on and off), the reachability of every answer in the LLM evaluation, that a default search sees only part of its corpus and that its two-step question takes two steps, that the demo worker reaches none of the semantic evaluation's answers, that a tool waits out a full worker, before each of a search's jobs and within one budget, and says so when it stays full, the evaluation runner's command, isolation checks, scoring and per-backend launcher and index check without starting `claude`, a check that the server's model names match the worker's, that every path parameter says where it resolves and every position in a result says what it counts, and a check that every timing claim in a tool description is a measured median |
 | PHP contracts + HTTP | 134 checks per PHP version; this **includes** the 101 contract checks, not 134 + 101 |
 | FrankenPHP | 12 jobs submitted by six concurrent test threads through a reused PHP worker, a 512-document `top_k` request, embedding and redaction round trips, and invalid-input, missing-job and multibyte boundary checks |
@@ -63,7 +63,7 @@ The original prototype was not entirely developed with TDD. Review corrections u
 - No full Symfony kernel or Laravel application/Octane integration test, including configuration caching and multi-user authorization.
 - No fresh consumer Composer-install smoke test or Packagist release yet; repository examples use a small local autoloader.
 - No sustained load/soak benchmark, throughput claim, p95/p99 latency, warm-model latency, measured memory-per-job limit or CPU cost per inference; the only timing figures are the cold-job medians below.
-- No representative ranking-quality evaluation (for example NDCG/MRR), multilingual accuracy evaluation, NER quality on real data or GPU benchmark; the NER figures below come from 48 invented sentences.
+- No representative ranking-quality evaluation (for example NDCG/MRR), multilingual accuracy evaluation, NER quality on real data or GPU benchmark; the NER figures below come from 48 invented sentences by the same author as the development set used to tune the redact task.
 - No repeated or semantic LLM evaluation. The one recorded run below is a single session per question per model on the demo worker, so it shows that the tools can be used, not a success rate or retrieval quality.
 - No verified native Windows/macOS service execution or ARM runtime; Docker tests target Linux amd64/Python 3.12.
 - No durable delivery, restart recovery, automatic retry, idempotency or multi-instance result routing. Jobs and results live in memory.
@@ -101,33 +101,36 @@ docker compose -f docker/compose.yaml --profile model down
 
 ## Measured NER precision and recall (informational)
 
-`scripts/ner_measure.py` runs every sentence of `tests/fixtures/ner-sample.json` through the ONNX worker's redact task with PER, ORG and LOC, at the task's default threshold and a lower one, and compares the model's spans with the sample's labels; rule spans are not counted. The model workflow prints a fresh table to its job summary on every run, and fails only if a job fails, never on a score. The table below is from that workflow, run 36856834447 on pull request #17 at branch head `f37290b`; the commit in its header is GitHub's merge ref. The sample is invented and small, so these figures describe this pinned model on these three registers, not a quality guarantee.
+`scripts/ner_measure.py` runs every sentence of `tests/fixtures/ner-sample.json` through the ONNX worker's redact task with PER, ORG and LOC, at the task's default threshold and a lower one, each its own job, and compares the model's spans with the sample's labels; rule spans are not counted. The model workflow prints a fresh table to its job summary on every run, and fails only if a job fails, never on a score. The table below is from that workflow, run 36862817554 on pull request #18 at branch head `2d14a59`; the commit in its header is GitHub's merge ref. The sample is invented and small, so these figures describe this pinned model on these three registers, not a quality guarantee.
 
-Measured 2026-10-01 UTC, commit `2d80a9f0a8c31ac78891dcde140f487a2259efd3`, runner: GitHub-hosted ubuntu-latest, CPU: AMD EPYC 7763 64-Core Processor.
-Model `Xenova/bert-base-NER:int8`; 48 invented sentences with 104 labelled entities; one redact job per sentence at threshold 0.5, entities PER, ORG and LOC, and a higher threshold keeps the spans whose score reaches it. Exact: same label and boundaries. Overlapping: same label and a shared character, one gold span per predicted span.
+Measured 2026-10-01 UTC, commit `0fe5726cbd3cf1a94825a4f3d196494d8a966e32`, runner: GitHub-hosted ubuntu-latest, CPU: AMD EPYC 7763 64-Core Processor.
+Model `Xenova/bert-base-NER:int8`; `tests/fixtures/ner-sample.json`, 48 invented sentences with 104 labelled entities; one redact job per sentence and threshold, entities PER, ORG and LOC. Exact: same label and boundaries. Overlapping: same label and a shared character, one gold span per predicted span. Masked: a shared character under any label, the share of entities a redaction hides.
 
-| Threshold | Register or label | Gold | Predicted | Precision, exact | Recall, exact | Precision, overlapping | Recall, overlapping |
-| --- | --- | --- | --- | --- | --- | --- | --- |
-| 0.85 | news | 38 | 38 | 1.00 | 1.00 | 1.00 | 1.00 |
-| 0.85 | informal | 30 | 5 | 0.80 | 0.13 | 1.00 | 0.17 |
-| 0.85 | records | 36 | 35 | 0.80 | 0.78 | 0.89 | 0.86 |
-| 0.85 | PER | 42 | 30 | 0.80 | 0.57 | 0.93 | 0.67 |
-| 0.85 | ORG | 24 | 18 | 0.94 | 0.71 | 0.94 | 0.71 |
-| 0.85 | LOC | 38 | 30 | 0.97 | 0.76 | 0.97 | 0.76 |
-| 0.5 | news | 38 | 38 | 1.00 | 1.00 | 1.00 | 1.00 |
-| 0.5 | informal | 30 | 7 | 0.57 | 0.13 | 0.71 | 0.17 |
-| 0.5 | records | 36 | 38 | 0.79 | 0.83 | 0.87 | 0.92 |
-| 0.5 | PER | 42 | 32 | 0.78 | 0.60 | 0.91 | 0.69 |
-| 0.5 | ORG | 24 | 20 | 0.90 | 0.75 | 0.90 | 0.75 |
-| 0.5 | LOC | 38 | 31 | 0.94 | 0.76 | 0.94 | 0.76 |
+| Threshold | Register or label | Gold | Predicted | Precision, exact | Recall, exact | Precision, overlapping | Recall, overlapping | Masked |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| 0.85 | news | 38 | 38 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
+| 0.85 | informal | 30 | 19 | 0.74 | 0.47 | 0.84 | 0.53 | 0.63 |
+| 0.85 | records | 36 | 35 | 0.94 | 0.92 | 0.94 | 0.92 | 0.97 |
+| 0.85 | PER | 42 | 35 | 0.94 | 0.79 | 1.00 | 0.83 | 0.83 |
+| 0.85 | ORG | 24 | 21 | 0.90 | 0.79 | 0.90 | 0.79 | 0.79 |
+| 0.85 | LOC | 38 | 36 | 0.92 | 0.87 | 0.92 | 0.87 | 0.87 |
+| 0.5 | news | 38 | 38 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
+| 0.5 | informal | 30 | 26 | 0.73 | 0.63 | 0.81 | 0.70 | 0.87 |
+| 0.5 | records | 36 | 36 | 0.94 | 0.94 | 0.94 | 0.94 | 1.00 |
+| 0.5 | PER | 42 | 38 | 0.95 | 0.86 | 1.00 | 0.90 | 0.90 |
+| 0.5 | ORG | 24 | 24 | 0.88 | 0.88 | 0.88 | 0.88 | 0.88 |
+| 0.5 | LOC | 38 | 38 | 0.89 | 0.89 | 0.89 | 0.89 | 0.89 |
 
-The model found every entity in news-style sentences, the register of its CoNLL-2003 training text. In informal messages it found only the capitalised names: every lower-case name, place and company was missed, and lowering the threshold to 0.5 added only wrong spans, because the model labels those words as no entity at all. In terse records it found about four in five, and 0.5 found a few more at a small cost in precision. In a local run the records it missed were a name written surname first in capitals (`WIERZBICKI, Tomasz`), names cut into pieces at an initial or a hyphen, `A. Okonkwo` with only its initial tagged, so the surname would stay visible, and a clinic and a city with each other's labels.
+Since pull request #18, the redact task reads a sentence written all in lower case or all in capitals a second time, from a title-cased copy with the same offsets, and likewise a surname in capitals before a comma and a given name (`WIERZBICKI, Tomasz`), the records convention. It joins pieces of one name that touch or meet at a hyphen, full stop or apostrophe before the threshold, scored by the most confident piece, so the initial of `A. Okonkwo` carries the surname; and of two readings of the same span it keeps the more confident. Those rules were chosen on a separate development set, `tests/fixtures/ner-dev.json`, so this sample stayed unseen until the final measurement; both are small and were written by the same author, so the figures describe these sentences, not informal text in general.
 
-int8 inference is not bit-identical across CPUs. A local run on an AMD Ryzen 9 9950X matched the news and informal rows at 0.85 but differed by a span or two in records and ORG, and at 0.5 it predicted two fewer wrong spans in informal messages, so read a difference of a span or two between runners as noise. The redact tool's description quotes these figures; re-measure and update both together. To reproduce locally after acquiring the models:
+Against the run before those changes on the same runner type, exact recall at 0.85 rose from 0.13 to 0.47 in informal messages and from 0.78 to 0.92 in records, and news stayed at 1.00; the share of entities masked under any label, which is what a redaction hides, is 0.63 in informal messages and 0.97 in records, and 0.87 and 1.00 at 0.5. What still leaks: a lower-case name in a sentence that also holds a capital (`Regards, olumide`), a handle (`@nadia`) and some lower-case first names side by side (`ines and pedro`). A title-cased reading also takes a filler word into a name (`lol tomasz`) or calls a person an organisation, which still hides the name. The deeper fix would be a case-insensitive NER model, which would need its own pinning and measurement.
+
+int8 inference is not bit-identical across CPUs: a local run on an AMD Ryzen 9 9950X has differed from GitHub's EPYC runners by a span or two in records and ORG, so read such differences as noise. The redact tool's description quotes these figures; re-measure and update both together. To reproduce locally after acquiring the models:
 
 ```sh
 docker compose -f docker/compose.yaml up -d model-worker
 docker compose -f docker/compose.yaml run --rm --no-deps -T model-ner
+NER_SAMPLE=tests/fixtures/ner-dev.json docker compose -f docker/compose.yaml run --rm --no-deps -T model-ner
 ```
 
 ## LLM evaluation (informational)

mcp_server/bridge_mcp/tools.py

@@ -59,8 +59,9 @@
         # The recall figures come from scripts/ner_measure.py's table in docs/testing.md; re-measure and update both.
         "redact": ("Xenova/bert-base-NER:int8", "checksum and pattern rules plus dslim/bert-base-NER (the "
                    "int8 ONNX conversion published as Xenova/bert-base-NER) for PER, ORG and LOC. Trained on "
-                   "English news: on an invented sample it found every entity in news-style sentences, about four in "
-                   "five in terse records and about one in eight in informal lower-case messages (docs/testing.md)."),
+                   "cased English news, so lower-case and all-capitals sentences are read again in title case: on an "
+                   "invented sample it masked every entity in news-style sentences, nearly every one in terse records "
+                   "and about three in five in informal messages (docs/testing.md)."),
     },
 }