diff --git a/.github/workflows/model-smoke.yml b/.github/workflows/model-smoke.yml index 34dd839..cadaec0 100644 --- a/.github/workflows/model-smoke.yml +++ b/.github/workflows/model-smoke.yml @@ -7,7 +7,7 @@ on: schedule: - cron: "17 6 * * 1" # Any change that can alter model acquisition or inference. Not a required check: it downloads - # about 215 MB and needs the network, which the deterministic gate deliberately does not. + # about 330 MB and needs the network, which the deterministic gate deliberately does not. pull_request: paths: - scripts/fetch_model.py diff --git a/README.md b/README.md index f0500b2..16e6a0d 100644 --- a/README.md +++ b/README.md @@ -9,12 +9,12 @@ Includes a FrankenPHP worker-mode example, deterministic fault tests, and an opt - PHP 8.2–8.5 client using cURL, with validated job objects and typed rerank, embedding and redaction results. Rerank accepts up to 512 candidate documents and an optional `top_k` so a large set narrows to a small result; the ONNX backend scores them 32 pairs per forward pass. - Python 3.12 service with an explicit task allowlist and a separate process per task. - Text embedding: up to 32 texts per job return unit-length 384-dimensional vectors for an index the caller owns. -- Redaction: checksum and pattern rules for card numbers, IBANs, emails, IPv4 addresses and phone numbers on every backend, plus named-entity masking with int8 `bert-base-NER` on the ONNX profile. Assistive, not a compliance control. +- Redaction: checksum and pattern rules for card numbers, IBANs, emails, IPv4 and IPv6 addresses and phone numbers on every backend, plus named-entity masking with the int8 cased and uncased `bert-base-NER` models on the ONNX profile. Assistive, not a compliance control. - Bounded concurrency, queue/result capacity, retention, request sizes and execution deadlines. - Cancellation, progress polling, crash detection and generic errors that omit internal exception text. - A FrankenPHP example that submits work and returns a job ID immediately. - A stdio MCP server with one tool per task, a second client of the same protocol beside the PHP client, plus an index builder over the protocol and a `bridge_search` tool that searches what it built, returning offsets and snippets rather than vectors, a compose service that puts the pinned ONNX models behind it, and a ten-question evaluation for LLM hosts with a runner for the Claude Code CLI; see [the MCP guide](docs/mcp.md). -- CPU inference with `cross-encoder/ms-marco-TinyBERT-L2-v2`, `sentence-transformers/all-MiniLM-L6-v2` and int8 `dslim/bert-base-NER` through ONNX Runtime, using pinned model files and wheel hashes. +- CPU inference with `cross-encoder/ms-marco-TinyBERT-L2-v2`, `sentence-transformers/all-MiniLM-L6-v2` and int8 `dslim/bert-base-NER` and `dslim/bert-base-NER-uncased` through ONNX Runtime, using pinned model files and wheel hashes. The default backends are **`lexical-demo-not-a-model`** for reranking and **`hashing-bow-not-a-model`** for embedding, deterministic word overlap and feature hashing for testing the integration, and **`rules-only-not-a-model`** for redaction, which runs the real checksum and pattern rules but no model. None of them is described as AI. Enable the optional model profile below for real inference. @@ -57,7 +57,7 @@ Acquire dependencies and model data in the bounded, network-enabled fetcher: docker compose -f docker/compose.yaml run --rm --no-deps fetcher ``` -This downloads wheels and about 215 MB of model and tokenizer data into ignored `.cache/`, one directory per task under `models/`; the fetcher refuses any single artifact over 128 MiB. A cache created before the embed task used a flat layout that the fetcher does not delete: remove `.cache/models` and fetch again after upgrading. Wheels are checked against `requirements-model.lock`; the model revision and SHA-256 hashes are fixed in `scripts/fetch_model.py`. On Linux, ensure `.cache/` exists and is writable by container UID 65532 before acquisition. For a disposable local cache, `mkdir -p .cache && chmod 777 .cache` is sufficient; never apply that permission to the repository or another directory. +This downloads wheels and about 330 MB of model and tokenizer data into ignored `.cache/`, one directory per model under `models/`; the fetcher refuses any single artifact over 128 MiB. Fetch again after an upgrade that adds a model: an ONNX worker refuses to start while a model directory lacks its files, and names it. A cache created before the embed task used a flat layout that the fetcher does not delete: remove `.cache/models` and fetch again after upgrading. Wheels are checked against `requirements-model.lock`; the model revision and SHA-256 hashes are fixed in `scripts/fetch_model.py`. On Linux, ensure `.cache/` exists and is writable by container UID 65532 before acquisition. For a disposable local cache, `mkdir -p .cache && chmod 777 .cache` is sufficient; never apply that permission to the repository or another directory. Then run **without internet access**: @@ -141,4 +141,4 @@ See [the integration guide](docs/integration.md), [test evidence and limits](doc ## License -Code: [MIT](LICENSE). The optional [TinyBERT reranker](https://huggingface.co/cross-encoder/ms-marco-TinyBERT-L2-v2) and [MiniLM embedding model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) are Apache-2.0 at their pinned revisions; the [BERT NER model](https://huggingface.co/dslim/bert-base-NER) is MIT and is fetched as the int8 ONNX conversion published by [Xenova](https://huggingface.co/Xenova/bert-base-NER), pinned by revision and hash. Model weights and dependency wheels are downloaded separately and retain their own licenses. +Code: [MIT](LICENSE). The optional [TinyBERT reranker](https://huggingface.co/cross-encoder/ms-marco-TinyBERT-L2-v2) and [MiniLM embedding model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) are Apache-2.0 at their pinned revisions; the cased and uncased BERT NER models, [dslim/bert-base-NER](https://huggingface.co/dslim/bert-base-NER) and [dslim/bert-base-NER-uncased](https://huggingface.co/dslim/bert-base-NER-uncased), are MIT and are fetched as the int8 ONNX conversions published by Xenova ([cased](https://huggingface.co/Xenova/bert-base-NER), [uncased](https://huggingface.co/Xenova/bert-base-NER-uncased)), pinned by revision and hash. Model weights and dependency wheels are downloaded separately and retain their own licenses. diff --git a/docs/mcp.md b/docs/mcp.md index 8fbafdc..a6faaef 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -19,7 +19,7 @@ Every description states what the backend actually is, in the worker's own words | `bridge_health` | none | `backend`, `protocol`, the models by task | instant | | `bridge_rerank` | `query`, then `documents` (≤64) or `documents_path` (≤512 in a file), `top_k` | `model`, `considered`, `results[]` of `index` (the document's zero-based position), `score`, `snippet` | 0.4 s on the demo backend; with the ONNX model 0.7-0.9 s for 32 documents and 1.0-1.2 s for 512 | | `bridge_embed_similarity` | `texts` (2–32) | `model`, `matrix`, `pairs[]` of `a`, `b`, `similarity` | 0.4 s on the demo backend; 1.1-1.3 s for 32 texts with the ONNX model | -| `bridge_redact` | `text`, `entities` (subset of PER, ORG, LOC), `min_score` | `model`, masked `text`, `spans[]` with `source` | 0.4 s on the demo backend; 1.6-2.4 s for 2000 characters with the ONNX model | +| `bridge_redact` | `text`, `entities` (subset of PER, ORG, LOC), `min_score` | `model`, masked `text`, `spans[]` with `source` | 0.4 s on the demo backend; 2.8-3.4 s for 2000 characters with the ONNX models | | `bridge_search` | `query`, `index_path` (an index directory under the root), `top_k` (1-50), `rerank` (default true) | `embed_model`, `rerank_model`, `chunks`, `considered`, `results[]` of `source`, `chunk`, `start`, `end`, `similarity`, `rerank_score`, `snippet` | one embed job, plus one rerank job unless `rerank` is false | The times are the worker's, from [the measured table](testing.md#measured-latency-informational), and the ranges are the difference between two runners' CPUs; the model loads afresh for every call, and the MCP server adds its polling interval and HTTP overhead on top. A `documents_path` file is either a JSON array of strings or one document per non-blank line; lines holding only whitespace are skipped and not counted, so a result's `index` is its position among the documents, not a line number. A file over 4 MiB is refused before it is parsed, but the limits that bind are the worker's: at most 512 documents, and the query plus all documents within 200000 characters (the encoded request within 262144 bytes), so a usable file is roughly 200 KB of text. Relative paths resolve under the root, and every path, relative or absolute, must resolve inside it after following symlinks. Snippets are the first 160 characters of a document with whitespace collapsed; they are content from the agent's own files and enter its context like any other file it reads. diff --git a/docs/protocol.md b/docs/protocol.md index c8590b0..ad87404 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -25,7 +25,7 @@ Successful reranking returns `{"model":"...","rankings":[{"index":0,"score":1.0} Submit `{"task":"embed","input":{"texts":["..."]}}` with 1–32 texts of 1–8192 characters each, totalling at most 200000 characters. A successful result is `{"model":"...","dimensions":384,"vectors":[[...]]}` with one unit-length vector per text in input order; components are rounded to six decimals, so the dot product of two vectors is their cosine similarity. The service returns vectors and stores nothing; the caller owns any index built from them. The deterministic backend identifies itself as `hashing-bow-not-a-model`: a signed feature hash of word tokens that is stable across processes and ignores word order. The ONNX backend is `sentence-transformers/all-MiniLM-L6-v2` with mean pooling; it truncates input beyond 256 word pieces and is trained on English text. -Submit `{"task":"redact","input":{"text":"...","entities":["PER"],"min_score":0.85}}` with 1–8192 characters of valid Unicode text. `entities` is an optional list of distinct labels among `PER`, `ORG` and `LOC` (default `["PER"]`; `MISC` is never offered) and `min_score` an optional number from 0 to 1 (default 0.85); both shape only the model pass. A successful result is `{"model":"...","text":"...","spans":[{"start":0,"end":5,"label":"EMAIL","source":"rule:email","score":1.0}]}`: the text with each span replaced by `[LABEL]`, and the spans sorted and disjoint, with offsets counted in Unicode code points of the original text. Deterministic rules run on every backend and are tagged `rule:`: `CARD` (any 13–19 digit window of a digit run that passes Luhn, longest first), `IBAN` (mod 97, trimming trailing groups the pattern swallowed; a lower-case one also needs a country code from the IBAN registry, and mixed case is never taken), `EMAIL` (letters of any script in the local part and the domain, and a letter or punycode top-level label), `IPV4` and `IPV6` (any form Python's `ipaddress` parser accepts that holds a digit, not starting inside a word, an IPv4-mapped tail and a `%zone` identifier included, the colon of `address: message` excluded) at score 1.0, and `PHONE` at 0.8 because it is a pattern, not a proof: 7–15 digits with a leading plus or separators, or at least ten bare digits, never a date shape. Known pattern false positives remain: a dotted quadruple of small numbers such as a version string matches `IPV4`, hex words joined by colons that the parser accepts, such as `dead::beef`, and eight colon-separated groups of up to four digits, such as `10:20:30:40:50:60:70:80`, match `IPV6`, and separated digit groups such as amounts can match `PHONE`; filter by `source` and `score` where that matters. On the default backend, or when `entities` is empty, the model name is `rules-only-not-a-model` and no model runs. The ONNX backend adds spans tagged `model:PER`, `model:ORG` or `model:LOC` carrying the model's mean word confidence, from `Xenova/bert-base-NER:int8`, the int8 ONNX conversion of `dslim/bert-base-NER` (CoNLL-2003, English news). The model was trained on cased text, so a sentence written all in lower case or all in capitals, and a surname in capitals before a comma and a given name, are read a second time from a title-cased copy with the same offsets. Pieces of one name that touch or meet at a hyphen, full stop or apostrophe are joined before the threshold and carry the score of their most confident piece; any other model span carries its mean word confidence. Overlapping candidates are merged into one span carrying the leftmost, then longest, then strongest label, then the more confident of two model readings, so no part of any candidate stays visible. Redaction is assistive: rules cannot see unstructured personal data, and the model misses names unlike its news training data: on an invented sample it masked every entity in news-style sentences and nearly every one in terse records, but about three in five in informal messages ([the measurement](testing.md#measured-ner-precision-and-recall-informational)), so it is not a compliance control. +Submit `{"task":"redact","input":{"text":"...","entities":["PER"],"min_score":0.85}}` with 1–8192 characters of valid Unicode text. `entities` is an optional list of distinct labels among `PER`, `ORG` and `LOC` (default `["PER"]`; `MISC` is never offered) and `min_score` an optional number from 0 to 1 (default 0.85); both shape only the model pass. A successful result is `{"model":"...","text":"...","spans":[{"start":0,"end":5,"label":"EMAIL","source":"rule:email","score":1.0}]}`: the text with each span replaced by `[LABEL]`, and the spans sorted and disjoint, with offsets counted in Unicode code points of the original text. Deterministic rules run on every backend and are tagged `rule:`: `CARD` (any 13–19 digit window of a digit run that passes Luhn, longest first), `IBAN` (mod 97, trimming trailing groups the pattern swallowed; a lower-case one also needs a country code from the IBAN registry, and mixed case is never taken), `EMAIL` (letters of any script in the local part and the domain, and a letter or punycode top-level label), `IPV4` and `IPV6` (any form Python's `ipaddress` parser accepts that holds a digit, not starting inside a word, an IPv4-mapped tail and a `%zone` identifier included, the colon of `address: message` excluded) at score 1.0, and `PHONE` at 0.8 because it is a pattern, not a proof: 7–15 digits with a leading plus or separators, or at least ten bare digits, never a date shape. Known pattern false positives remain: a dotted quadruple of small numbers such as a version string matches `IPV4`, hex words joined by colons that the parser accepts, such as `dead::beef`, and eight colon-separated groups of up to four digits, such as `10:20:30:40:50:60:70:80`, match `IPV6`, and separated digit groups such as amounts can match `PHONE`; filter by `source` and `score` where that matters. On the default backend, or when `entities` is empty, the model name is `rules-only-not-a-model` and no model runs. The ONNX backend adds spans tagged `model:PER`, `model:ORG` or `model:LOC` carrying the model's mean word confidence, from two models trained on CoNLL-2003 English news, in the int8 ONNX conversions `Xenova/bert-base-NER` and `Xenova/bert-base-NER-uncased`; its results name `Xenova/bert-base-NER:int8+Xenova/bert-base-NER-uncased:int8`. The cased `dslim/bert-base-NER` reads the text, and reads a sentence written all in lower case or all in capitals, and a surname in capitals before a comma and a given name, a second time from a title-cased copy with the same offsets. The uncased `dslim/bert-base-NER-uncased` then reads the whole text ignoring case, which finds lower-case names the cased model misses, such as one in a sentence that holds a capital; it also tags a domain name such as `example.com` as `ORG`, which inside an email address the `EMAIL` span covers. Pieces of one name that touch or meet at a hyphen, full stop or apostrophe are joined before the threshold and carry the score of their most confident piece; any other model span carries its mean word confidence. Overlapping candidates are merged into one span carrying the leftmost, then longest, then strongest label, then the more confident of two model readings, so no part of any candidate stays visible. Redaction is assistive: rules cannot see unstructured personal data, and the models miss names unlike their news training data: on an invented sample they masked every entity in news-style sentences and terse records, but four in five in informal messages ([the measurement](testing.md#measured-ner-precision-and-recall-informational)), so it is not a compliance control. Failures use generic error objects with `code` and `message`. Task errors include `task_failed`, `worker_crashed`, `worker_start_failed` and `deadline_exceeded`. Internal exception text is not sent to clients. HTTP contract failures return 400, missing auth 401, missing routes/jobs 404, a connection idle for five seconds while its body is read 408 (a stall while reading headers closes the connection without a response), oversized bodies 413 and capacity exhaustion 429. diff --git a/docs/security.md b/docs/security.md index ceae606..b1bd9fa 100644 --- a/docs/security.md +++ b/docs/security.md @@ -10,10 +10,10 @@ This experimental service is designed for trusted application-to-service traffic - CI may run on a self-hosted runner, which executes a workflow's code with that runner's network access, and the lab's runs a privileged Docker-in-Docker daemon. The workflows as written send a fork's pull request to GitHub's disposable runners, but a fork can edit them in its own pull request and name the self-hosted runner, and GitHub runs a pull request's workflows as the pull request has them. The boundary is therefore the repository's fork pull request approval policy: with a self-hosted runner configured, require approval for all outside contributors, not only first-time ones, and review a fork's workflow changes before approving its run. The runner installs Docker Compose from a pinned release checked by SHA-256 when its image lacks it. - Model files, native libraries and runtime dependencies still carry supply-chain risk. Review upgrades, regenerate locks deliberately and rerun tests before changing pins. - Requests and results may contain sensitive documents. They live in RAM, may be present in process/core dumps, and are not encrypted by this service. Minimize retention and use appropriate host/storage policies. Embedding vectors are derived from their input text and can reveal much of it; store and transmit them with the same care as the text. -- Redaction is assistive, not a compliance control. The rules find only the structured identifiers they recognise, and the optional named-entity model was trained on English news, so its recall on names from other domains or languages is lower and unmeasured here; review redacted output before treating it as free of personal data. The model files are a third-party ONNX conversion of the original weights, pinned by revision and hash but not signed by the original author. +- Redaction is assistive, not a compliance control. The rules find only the structured identifiers they recognise, and the optional named-entity models were trained on English news, so their recall on names from other domains or languages is lower and unmeasured here; review redacted output before treating it as free of personal data. The model files are third-party ONNX conversions of the original weights, pinned by revision and hash but not signed by the original author. - The MCP server is a client of this service for an AI agent over stdio, where there is no authentication: the host that launches it inherits its trust, and it holds the token from its environment and never returns it. Running it over HTTP or SSE would be a new authorisation surface that the bearer token does not cover; it is out of scope. An agent can be prompt-injected and can then fill a worker's capacity, so point the MCP server at a dedicated worker, never at one serving a production application; compose's `mcp` and `mcp-model` each have their own. It reads files only under `BRIDGE_MCP_ROOT` and writes none; snippets it returns are the agent's own file content and enter the agent's context like any file it reads. Index files built for `bridge_search` are derived data: `index.json` holds every chunk's text verbatim and `vectors.f32` holds vectors that can reveal it, so keep them with the same care as their sources. The builder runs with the service token like the PHP client and is the only component that writes, into the directory the operator names. - Access logs are disabled to avoid accidentally recording identifiers and inputs. Internal exceptions become generic task errors. This is not a full observability or compliance solution. - Cancel/timeout terminates the task process, but cannot roll back previously completed external effects. No task in the prototype modifies an external system. -- The default health endpoint proves HTTP service availability, not model integrity, ability to spawn processes or successful inference. Use a controlled model smoke test for readiness validation. +- The default health endpoint proves HTTP service availability, not model integrity, ability to spawn processes or successful inference. An ONNX worker checks before it starts that each model directory holds its tokenizer and model files and refuses to start otherwise; that proves presence, not integrity, which the fetcher's hashes establish. Use a controlled model smoke test for readiness validation. The FrankenPHP executable is copied from its pinned image into a task-only executable tmpfs. This removes upstream file capabilities without granting the container capabilities. Other temporary storage is non-executable; the optional Python dependency tmpfs must permit native library mappings. diff --git a/docs/testing.md b/docs/testing.md index 93381a4..557731b 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -48,18 +48,18 @@ PHP branch support was verified locally on 2026-09-17 on PHP 8.2.33, 8.3.33, 8.4 | Layer | Scope and counting | | --- | --- | -| Python worker | 117 tests: lifecycle, validation and HTTP, process crashes, cancellation, timeouts, retention, allocation/exit races, redaction rules including IPv6, lower-case IBANs and email addresses in any script and their speed on hostile input, NER aggregation, name joining, the second title-cased reading and the merge driven by stubs, batched cross-encoder scoring, the NER measurement's sample and scoring, the workflows' runner routing and Compose install, progress bounds, and the model-name table every result reports from | +| Python worker | 128 tests: lifecycle, validation and HTTP, process crashes, cancellation, timeouts, retention, allocation/exit races, redaction rules including IPv6, lower-case IBANs and email addresses in any script and their speed on hostile input, NER aggregation, name joining, the second title-cased reading, the uncased model's reading of the whole text and the merge driven by stubs, the fetcher's pins and the model directories an ONNX worker checks for before it starts, batched cross-encoder scoring, the NER measurement's sample and scoring, the workflows' runner routing and Compose install, progress bounds, and the model-name table every result reports from | | MCP server | 119 tests, offline with hash-pinned wheels: the protocol client against an in-process worker, every tool through the SDK's in-memory client with test tasks enabled, the entry point as a subprocess, result validators on synthetic data, the index builder (chunking, the format and each corruption it refuses, building against the worker, caching, its command line) `bridge_search` (the model gate, path confinement, reloads, rerank on and off), the reachability of every answer in the LLM evaluation, that a default search sees only part of its corpus and that its two-step question takes two steps, that the demo worker reaches none of the semantic evaluation's answers, that a tool waits out a full worker, before each of a search's jobs and within one budget, and says so when it stays full, the evaluation runner's command, isolation checks, scoring and per-backend launcher and index check without starting `claude`, a check that the server's model names match the worker's, that every path parameter says where it resolves and every position in a result says what it counts, and a check that every timing claim in a tool description is a measured median | | PHP contracts + HTTP | 134 checks per PHP version; this **includes** the 101 contract checks, not 134 + 101 | | FrankenPHP | 12 jobs submitted by six concurrent test threads through a reused PHP worker, a 512-document `top_k` request, embedding and redaction round trips, and invalid-input, missing-job and multibyte boundary checks | | PHP syntax | All 12 PHP source/example/test files | -| Optional ONNX | Through both the PHP CLI and FrankenPHP HTTP: 2 rerank cases, a 70-document rerank across three batches with its relevant document in the second, a parity check that three documents score within 1e-3 alone and inside that set, 1 embedding and 2 redaction cases, the second asserting that clean text produces no spans; and the ONNX-backed MCP server end to end, every tool through compose's `mcp-model` service over stdio, including a search over an index the embedding model built, with a query worded unlike the note it must find, and the calls for every question in the semantic evaluation, whose answers the models must all reach; smoke checks, not quality datasets | +| Optional ONNX | Through both the PHP CLI and FrankenPHP HTTP: 2 rerank cases, a 70-document rerank across three batches with its relevant document in the second, a parity check that three documents score within 1e-3 alone and inside that set, 1 embedding and 3 redaction cases: a name, an email address and a place; a sentence whose lower-case person, place and organisation only the uncased model finds, which also checks its label order; and clean text, which must produce no spans; and the ONNX-backed MCP server end to end, every tool through compose's `mcp-model` service over stdio, including a search over an index the embedding model built, with a query worded unlike the note it must find, and the calls for every question in the semantic evaluation, whose answers the models must all reach; smoke checks, not quality datasets | Test durations reported by the runners are suite durations, not inference latency or a reproducible performance benchmark. The optional real-model smoke tests run on the default PHP 8.5 image with cached dependencies and no internet access during inference. They were not rerun on every PHP minor. On 2026-09-17, three PHP documentation snippets passed syntax validation and the fenced JSON examples parsed successfully; that check has not been repeated since, and it is not a framework boot test. -The default backend is deterministic word overlap (`lexical-demo-not-a-model`). Passing its tests proves plumbing/lifecycle behavior, not AI quality. The embedding and redaction demos are `hashing-bow-not-a-model` and `rules-only-not-a-model`: the first proves the contract only, while the second runs the real checksum and pattern rules with no model. The optional ONNX checks use the three pinned, hash-verified models (the TinyBERT cross-encoder, MiniLM embeddings and the int8 BERT NER conversion) with local inference. The model workflow runs weekly, on pull requests that touch the model path, and on demand; it is not a required check. +The default backend is deterministic word overlap (`lexical-demo-not-a-model`). Passing its tests proves plumbing/lifecycle behavior, not AI quality. The embedding and redaction demos are `hashing-bow-not-a-model` and `rules-only-not-a-model`: the first proves the contract only, while the second runs the real checksum and pattern rules with no model. The optional ONNX checks use the four pinned, hash-verified models (the TinyBERT cross-encoder, MiniLM embeddings and the int8 conversions of the cased and uncased BERT NER models) with local inference. The model workflow runs weekly, on pull requests that touch the model path, and on demand; it is not a required check. The original prototype was not entirely developed with TDD. Review corrections used observed red-green regression tests. PHP 8.5 testing similarly reproduced a deprecated `curl_close()` call under the strict error handler before replacing it with object release. Passing tests alone do not prove a test-first history or complete correctness. @@ -93,10 +93,12 @@ Measured 2026-09-27 UTC by the model smoke workflow on pull request #8 at branch | lexical | embed 32 texts of ~40 words | 5 | 0.37 | 0.37 | 0.37 | cold process, no model | | lexical | redact 2000 characters, PER+ORG+LOC | 5 | 0.36 | 0.36 | 0.36 | cold process, no model | -The CPU a runner gets matters more than run-to-run noise. An earlier run on the same pull request, at `41393c0` on an AMD EPYC 7763, measured ONNX medians 15-50% higher: 0.85 s for 32 documents, 1.20 s for 512, 1.30 s for embedding and 2.40 s for redaction, and 0.41-0.43 s on the lexical rows. Its memory row used 24 documents of 8000 characters, not the worst case above. The ranges quoted in the MCP tool descriptions span both runs. +The CPU a runner gets matters more than run-to-run noise. An earlier run on the same pull request, at `41393c0` on an AMD EPYC 7763, measured ONNX medians 15-50% higher: 0.85 s for 32 documents, 1.20 s for 512, 1.30 s for embedding and 2.40 s for redaction, and 0.41-0.43 s on the lexical rows. Its memory row used 24 documents of 8000 characters, not the worst case above. The ranges quoted in the MCP tool descriptions span both runs, except redaction's, which the next paragraph replaces. Every job is cold by design: a fresh process imports its dependencies and, on the ONNX backend, loads the model before any work. The lexical rows are therefore the cost of that process alone, about 0.4 s, and most of each ONNX row is start-up and model loading rather than scoring: 512 documents take 0.26 s more than 32, and a full batch at the 512-token limit fits the model worker's memory limit. There is no measurement from before batched reranking, because the benchmark was added with it, so these figures do not state a speed-up. They are worker time only; the PHP client, FrankenPHP and the MCP server add their own polling and HTTP overhead. The benchmark asserts that every job succeeds with progress ending on the total, and a failure fails the smoke run; it asserts nothing about speed. +Since pull request #19 every redact job also loads and runs the uncased NER model. That pull request's model workflow (run 36902794112 at branch head `e49e388`, on an AMD EPYC 7763) measured a median of 3.40 s for the redact row, against 2.40 s in the earlier EPYC 7763 run above, while its rerank and embed rows came out 5-8% slower than that run's (0.90, 1.30 and 1.40 s): the second model costs about a second per cold job. A local run on an AMD Ryzen 9 9950X measured 2.76 s, and the range the redact tool quotes spans these two. 8192 lower-case characters, which every reading covers, took 5.8 s locally. + Every model smoke run prints a fresh table to its job summary. To reproduce locally after acquiring the models: ```sh @@ -107,31 +109,31 @@ docker compose -f docker/compose.yaml --profile model down ## Measured NER precision and recall (informational) -`scripts/ner_measure.py` runs every sentence of `tests/fixtures/ner-sample.json` through the ONNX worker's redact task with PER, ORG and LOC, at the task's default threshold and a lower one, each its own job, and compares the model's spans with the sample's labels; rule spans are not counted. The model workflow prints a fresh table to its job summary on every run, and fails only if a job fails, never on a score. The table below is from that workflow, run 36862817554 on pull request #18 at branch head `2d14a59`; the commit in its header is GitHub's merge ref. The sample is invented and small, so these figures describe this pinned model on these three registers, not a quality guarantee. +`scripts/ner_measure.py` runs every sentence of `tests/fixtures/ner-sample.json` through the ONNX worker's redact task with PER, ORG and LOC, at the task's default threshold and a lower one, each its own job, and compares the models' spans with the sample's labels; rule spans are not counted. The model workflow prints a fresh table to its job summary on every run, and fails only if a job fails, never on a score. The table below is from that workflow, run 36902794112 on pull request #19 at branch head `e49e388`; the commit in its header is GitHub's merge ref. The sample is invented and small, so these figures describe these pinned models on these three registers, not a quality guarantee. -Measured 2026-10-01 UTC, commit `0fe5726cbd3cf1a94825a4f3d196494d8a966e32`, runner: GitHub-hosted ubuntu-latest, CPU: AMD EPYC 7763 64-Core Processor. -Model `Xenova/bert-base-NER:int8`; `tests/fixtures/ner-sample.json`, 48 invented sentences with 104 labelled entities; one redact job per sentence and threshold, entities PER, ORG and LOC. Exact: same label and boundaries. Overlapping: same label and a shared character, one gold span per predicted span. Masked: a shared character under any label, the share of entities a redaction hides. +Measured 2026-10-01 UTC, commit `d5b3edc039b6370b51f5fe455785b88ef2aa5e29`, runner: GitHub-hosted ubuntu-latest, CPU: AMD EPYC 7763 64-Core Processor. +Model `Xenova/bert-base-NER:int8+Xenova/bert-base-NER-uncased:int8`; `tests/fixtures/ner-sample.json`, 48 invented sentences with 104 labelled entities; one redact job per sentence and threshold, entities PER, ORG and LOC. Exact: same label and boundaries. Overlapping: same label and a shared character, one gold span per predicted span. Masked: a shared character under any label, the share of entities a redaction hides. | Threshold | Register or label | Gold | Predicted | Precision, exact | Recall, exact | Precision, overlapping | Recall, overlapping | Masked | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0.85 | news | 38 | 38 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | -| 0.85 | informal | 30 | 19 | 0.74 | 0.47 | 0.84 | 0.53 | 0.63 | -| 0.85 | records | 36 | 35 | 0.94 | 0.92 | 0.94 | 0.92 | 0.97 | -| 0.85 | PER | 42 | 35 | 0.94 | 0.79 | 1.00 | 0.83 | 0.83 | -| 0.85 | ORG | 24 | 21 | 0.90 | 0.79 | 0.90 | 0.79 | 0.79 | -| 0.85 | LOC | 38 | 36 | 0.92 | 0.87 | 0.92 | 0.87 | 0.87 | +| 0.85 | informal | 30 | 25 | 0.72 | 0.60 | 0.84 | 0.70 | 0.80 | +| 0.85 | records | 36 | 36 | 0.94 | 0.94 | 0.94 | 0.94 | 1.00 | +| 0.85 | PER | 42 | 37 | 0.92 | 0.81 | 1.00 | 0.88 | 0.88 | +| 0.85 | ORG | 24 | 24 | 0.88 | 0.88 | 0.88 | 0.88 | 0.88 | +| 0.85 | LOC | 38 | 38 | 0.92 | 0.92 | 0.92 | 0.92 | 0.92 | | 0.5 | news | 38 | 38 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | -| 0.5 | informal | 30 | 26 | 0.73 | 0.63 | 0.81 | 0.70 | 0.87 | +| 0.5 | informal | 30 | 33 | 0.67 | 0.73 | 0.73 | 0.80 | 1.00 | | 0.5 | records | 36 | 36 | 0.94 | 0.94 | 0.94 | 0.94 | 1.00 | | 0.5 | PER | 42 | 38 | 0.95 | 0.86 | 1.00 | 0.90 | 0.90 | -| 0.5 | ORG | 24 | 24 | 0.88 | 0.88 | 0.88 | 0.88 | 0.88 | -| 0.5 | LOC | 38 | 38 | 0.89 | 0.89 | 0.89 | 0.89 | 0.89 | +| 0.5 | ORG | 24 | 30 | 0.73 | 0.92 | 0.73 | 0.92 | 0.92 | +| 0.5 | LOC | 38 | 39 | 0.92 | 0.95 | 0.92 | 0.95 | 0.95 | -Since pull request #18, the redact task reads a sentence written all in lower case or all in capitals a second time, from a title-cased copy with the same offsets, and likewise a surname in capitals before a comma and a given name (`WIERZBICKI, Tomasz`), the records convention. It joins pieces of one name that touch or meet at a hyphen, full stop or apostrophe before the threshold, scored by the most confident piece, so the initial of `A. Okonkwo` carries the surname; and of two readings of the same span it keeps the more confident. Those rules were chosen on a separate development set, `tests/fixtures/ner-dev.json`, so this sample stayed unseen until the final measurement; both are small and were written by the same author, so the figures describe these sentences, not informal text in general. +Since pull request #18, the redact task reads a sentence written all in lower case or all in capitals a second time, from a title-cased copy with the same offsets, and likewise a surname in capitals before a comma and a given name (`WIERZBICKI, Tomasz`), the records convention. It joins pieces of one name that touch or meet at a hyphen, full stop or apostrophe before the threshold, scored by the most confident piece, so the initial of `A. Okonkwo` carries the surname; and of two readings of the same span it keeps the more confident. Since pull request #19, the uncased `dslim/bert-base-NER-uncased` reads the whole text once more, ignoring case, beside the cased model, which keeps what capitals tell in news and records: alone, the uncased model masked only 0.54 of the development set's records against 0.77 for the cased readings. Those choices were made on a separate development set, `tests/fixtures/ner-dev.json`, which before the uncased model was measured gained lower-case names among capitals, news sentences and sentences of common words that are also names (`pay the bill`, `a rose`, `Will the courier`); the uncased model invented no name in any of them. This sample stayed unseen until each final measurement; both sets are small and were written by the same author, so the figures describe these sentences, not informal text in general. -Against the run before those changes on the same runner type, exact recall at 0.85 rose from 0.13 to 0.47 in informal messages and from 0.78 to 0.92 in records, and news stayed at 1.00; the share of entities masked under any label, which is what a redaction hides, is 0.63 in informal messages and 0.97 in records, and 0.87 and 1.00 at 0.5. What still leaks: a lower-case name in a sentence that also holds a capital (`Regards, olumide`), a handle (`@nadia`) and some lower-case first names side by side (`ines and pedro`). A title-cased reading also takes a filler word into a name (`lol tomasz`) or calls a person an organisation, which still hides the name. The deeper fix would be a case-insensitive NER model, which would need its own pinning and measurement. +Against pull request #18's run on the same CPU type, exact recall at 0.85 rose from 0.47 to 0.60 in informal messages and from 0.92 to 0.94 in records, and news stayed at 1.00; the share of entities masked under any label, which is what a redaction hides, rose from 0.63 to 0.80 in informal messages and from 0.97 to 1.00 in records, and at 0.5 it is 1.00 in all three. Exact precision held at 0.94 in records and moved from 0.74 to 0.72 in informal messages. What still leaks at 0.85: a name after a sign-off (`Regards, olumide`), a handle (`@nadia`), some lower-case places and first names (`porto`, `galway`, `fatima`) and a capitalised company in an informal note (`Halvern`). A reading can still take a filler word into a name (`lol tomasz`) or give a name the wrong label, which still hides it, and the uncased model tags a domain name such as `example.com` as an organisation. -int8 inference is not bit-identical across CPUs: a local run on an AMD Ryzen 9 9950X has differed from GitHub's EPYC runners by a span or two in records and ORG, so read such differences as noise. The redact tool's description quotes these figures; re-measure and update both together. To reproduce locally after acquiring the models: +int8 inference is not bit-identical across CPUs: a local run on an AMD Ryzen 9 9950X has differed from GitHub's EPYC runners by a span or two, so read such differences as noise. The redact tool's description quotes these figures; re-measure and update both together. To reproduce locally after acquiring the models: ```sh docker compose -f docker/compose.yaml up -d model-worker diff --git a/mcp_server/bridge_mcp/tools.py b/mcp_server/bridge_mcp/tools.py index e5dfe58..f996cbe 100644 --- a/mcp_server/bridge_mcp/tools.py +++ b/mcp_server/bridge_mcp/tools.py @@ -57,18 +57,22 @@ "embed": ("sentence-transformers/all-MiniLM-L6-v2", "through ONNX Runtime on CPU; English, truncated at " "256 word pieces; unit-length mean-pooled vectors."), # The recall figures come from scripts/ner_measure.py's table in docs/testing.md; re-measure and update both. - "redact": ("Xenova/bert-base-NER:int8", "checksum and pattern rules plus dslim/bert-base-NER (the " - "int8 ONNX conversion published as Xenova/bert-base-NER) for PER, ORG and LOC. Trained on " - "cased English news, so lower-case and all-capitals sentences are read again in title case: on an " - "invented sample it masked every entity in news-style sentences, nearly every one in terse records " - "and about three in five in informal messages (docs/testing.md)."), + "redact": ("Xenova/bert-base-NER:int8+Xenova/bert-base-NER-uncased:int8", "checksum and pattern rules plus " + "dslim/bert-base-NER and dslim/bert-base-NER-uncased (the int8 ONNX conversions published as " + "Xenova/bert-base-NER and Xenova/bert-base-NER-uncased) for PER, ORG and LOC, both trained on " + "English news. The cased model reads the text, and again in title case where a sentence is all " + "lower case or all capitals; the uncased model reads it once more ignoring case, which finds most " + "lower-case names the cased model misses. On an invented sample they masked every entity in " + "news-style sentences and terse records and four in five in informal messages (docs/testing.md)."), }, } # Worker time from submission to a terminal state: the p50 of five cold jobs on one CPU, as the range between # two GitHub-hosted runners measured 2026-09-27 (docs/testing.md's "Measured latency" table and the earlier run -# it quotes): the lexical rows, and on onnx the 32- and 512-document rerank rows, the embed row and the redact -# row. The same figures appear in docs/mcp.md's tool table; a re-measurement updates all three places. +# it quotes): the lexical rows, and on onnx the 32- and 512-document rerank rows and the embed row. Redaction runs +# two models since pull request #19, so its range is that pull request's run on an EPYC 7763 runner and a local +# Ryzen 9 9950X, measured 2026-10-01. The same figures appear in docs/mcp.md's tool table; a re-measurement updates +# all three places. _COLD = "median of five cold jobs on one CPU on two runners, the model loaded afresh for each; docs/testing.md" TIMING = { "lexical": {task: "about 0.4 s of worker time (median of five cold jobs on one CPU on two runners, almost all " @@ -76,7 +80,8 @@ "onnx": { "rerank": f"0.7-0.9 s of worker time for 32 documents and 1.0-1.2 s for 512 ({_COLD})", "embed": f"1.1-1.3 s of worker time for 32 texts ({_COLD})", - "redact": f"1.6-2.4 s of worker time for 2000 characters with all three entity kinds ({_COLD})", + "redact": "2.8-3.4 s of worker time for 2000 characters with all three entity kinds (median of five cold jobs " + "on one CPU on two machines, both models loaded afresh for each; docs/testing.md)", }, } diff --git a/scripts/fetch_model.py b/scripts/fetch_model.py index 42d7cba..457ff24 100644 --- a/scripts/fetch_model.py +++ b/scripts/fetch_model.py @@ -5,7 +5,8 @@ from pathlib import Path from urllib.request import urlopen -# One directory per task under /cache/models. Hashes are SHA-256 of the exact files at the revision. +# One directory per model under /cache/models, named for the task that loads it. Hashes are SHA-256 of the exact +# files at the revision. MODELS = { "rerank": { "repository": "cross-encoder/ms-marco-TinyBERT-L2-v2", @@ -39,8 +40,22 @@ "model.onnx": "7de0a4606c65b60da275a72f37b76a102c41e2b79c6463096a9d0cb800bf3f2c", }, }, + # The same author's uncased model, dslim/bert-base-NER-uncased, in the same converter's int8 conversion. The + # cased model finds almost no name written in lower case; this one reads "rahim" as it reads "Rahim". Both share + # dslim's label order, which NER_LABELS in the worker holds. + "redact-uncased": { + "repository": "Xenova/bert-base-NER-uncased", + "revision": "5e99d439d48710336764eea7b6f8ff87f2f40ae4", + "license": "MIT", + "files": {"tokenizer.json": "tokenizer.json", "onnx/model_int8.onnx": "model.onnx"}, + "hashes": { + "tokenizer.json": "d241a60d5e8f04cc1b2b3e9ef7a4921b27bf526d9f6050ab90f9267a1f9e5c66", + "model.onnx": "37103d2168679e618ebaaa672ad33bcb1167c1776f840e1506a83527cb48b5a7", + }, + }, } -# Raised from 100 MiB for the 103.5 MiB int8 NER model; the fp32 and 4-bit variants were rejected. +# Raised from 100 MiB for the int8 NER models, 103.5 MiB cased and 104.6 MiB uncased; the fp32 and 4-bit variants +# were rejected. LIMIT = 128 * 1024 * 1024 diff --git a/tests/fixtures/ner-dev.json b/tests/fixtures/ner-dev.json index cebb16c..481a8ce 100644 --- a/tests/fixtures/ner-dev.json +++ b/tests/fixtures/ner-dev.json @@ -1,5 +1,5 @@ { - "about": "A development set for tuning the redact task's NER pass, kept apart from ner-sample.json, whose figures are the ones published. Invented like the sample; some sentences hold no entity at all, so that false positives show, and the acronyms are there for the same reason.", + "about": "A development set for tuning the redact task's NER pass, kept apart from ner-sample.json, whose figures are the ones published. Invented like the sample; some sentences hold no entity at all, so that false positives show, and the acronyms and the common words that are also names (bill, rose, Will) are there for the same reason.", "items": [ {"register": "lower case", "text": "can you tell oskar that the meeting moved to tuesday", "entities": [["PER", "oskar"]]}, {"register": "lower case", "text": "landed in dublin, heading to the hotel now", "entities": [["LOC", "dublin"]]}, @@ -37,6 +37,27 @@ {"register": "news", "text": "Talia Brenner joined Greenfold Energy as chief engineer in Oslo.", "entities": [["PER", "Talia Brenner"], ["ORG", "Greenfold Energy"], ["LOC", "Oslo"]]}, {"register": "news", "text": "The museum in Seville reopened after repairs.", "entities": [["LOC", "Seville"]]}, - {"register": "mixed case", "text": "meeting w/ Anil tmrw at the Kestrel office in bristol", "entities": [["PER", "Anil"], ["ORG", "Kestrel"], ["LOC", "bristol"]]} + {"register": "mixed case", "text": "meeting w/ Anil tmrw at the Kestrel office in bristol", "entities": [["PER", "Anil"], ["ORG", "Kestrel"], ["LOC", "bristol"]]}, + + {"register": "name-like words", "text": "can you pay the bill and mark the invoice as done", "entities": []}, + {"register": "name-like words", "text": "may i bring a rose for the table", "entities": []}, + {"register": "name-like words", "text": "jack up the car before you change the tyre", "entities": []}, + {"register": "name-like words", "text": "the turkey needs another hour in the oven", "entities": []}, + {"register": "name-like words", "text": "i ate an apple and a handful of dates", "entities": []}, + {"register": "name-like words", "text": "Will the courier arrive before noon?", "entities": []}, + {"register": "name-like words", "text": "Bill the client for the extra hours.", "entities": []}, + {"register": "name-like words", "text": "Mark the boxes that need to go upstairs.", "entities": []}, + {"register": "mixed case", "text": "Cheers, rahim", "entities": [["PER", "rahim"]]}, + {"register": "mixed case", "text": "Hi team, soren is out today and back on Monday", "entities": [["PER", "soren"]]}, + {"register": "mixed case", "text": "I'll meet nkechi and mara at the café", "entities": [["PER", "nkechi"], ["PER", "mara"]]}, + {"register": "mixed case", "text": "Flight to tbilisi leaves at 9, gate 12, OK?", "entities": [["LOC", "tbilisi"]]}, + {"register": "mixed case", "text": "Thanks @lindiwe, the slides look great", "entities": [["PER", "lindiwe"]]}, + {"register": "mixed case", "text": "Ask haruto whether Quillon Labs sent the contract", "entities": [["PER", "haruto"], ["ORG", "Quillon Labs"]]}, + {"register": "news", "text": "Ministers from Chile and Peru met in Quito on Tuesday.", "entities": [["LOC", "Chile"], ["LOC", "Peru"], ["LOC", "Quito"]]}, + {"register": "news", "text": "Harbourline Logistics appointed Dana Okafor as its new chief executive.", "entities": [["ORG", "Harbourline Logistics"], ["PER", "Dana Okafor"]]}, + {"register": "news", "text": "Rain delayed the final in Auckland, said organiser Petra Novak.", "entities": [["LOC", "Auckland"], ["PER", "Petra Novak"]]}, + {"register": "news", "text": "Shares in Copperleaf Mining fell after the report from Toronto.", "entities": [["ORG", "Copperleaf Mining"], ["LOC", "Toronto"]]}, + {"register": "all capitals", "text": "MEETING WITH HELGA BRANDT MOVED TO ROOM 3", "entities": [["PER", "HELGA BRANDT"]]}, + {"register": "records", "text": "Account holder: Luis Santos, Faro.", "entities": [["PER", "Luis Santos"], ["LOC", "Faro"]]} ] } diff --git a/tests/mcp/test_tools.py b/tests/mcp/test_tools.py index d6f9ba2..cf81c4b 100644 --- a/tests/mcp/test_tools.py +++ b/tests/mcp/test_tools.py @@ -73,7 +73,8 @@ async def test_descriptions_state_the_real_backend_and_never_the_token(self): self.assertNotIn(TOKEN, text) onnx = build_server(self.bridge, backend="onnx", root=None, timeout_ms=10000) text = " ".join(tool.description or "" for tool in await self.tools(onnx)) - for model in ["cross-encoder/ms-marco-TinyBERT-L2-v2", "sentence-transformers/all-MiniLM-L6-v2", "bert-base-NER"]: + for model in ["cross-encoder/ms-marco-TinyBERT-L2-v2", "sentence-transformers/all-MiniLM-L6-v2", "bert-base-NER", + "bert-base-NER-uncased"]: self.assertIn(model, text) self.assertNotIn("not-a-model", text) self.assertNotIn(TOKEN, text) diff --git a/tests/model_cases.py b/tests/model_cases.py index dc78991..5f18a52 100644 --- a/tests/model_cases.py +++ b/tests/model_cases.py @@ -9,3 +9,8 @@ SIMILAR_TEXTS = ["A dog barks loudly.", "Puppies make barking noises.", "The stock market fell today."] # The leading dash is one code point of three bytes: byte-based model offsets would misplace every mask. REDACT_TEXT = "\u2014 John Smith wrote to john@example.com about Berlin." +# Lower-case names in a sentence that holds a capital: the cased model tags nothing here even at threshold 0 and the +# sentence is not rewritten, so only the uncased model can mask them, and only if it reads each of its three labels in +# the right order. +UNCASED_TEXT = "I spoke with esperanza in lisbon, she works for brightline insurance." +UNCASED_MASKED = "I spoke with [PER] in [LOC], she works for [ORG]." diff --git a/tests/model_integration.py b/tests/model_integration.py index c7c72b3..01456fc 100644 --- a/tests/model_integration.py +++ b/tests/model_integration.py @@ -1,7 +1,7 @@ """Small model smoke test, not a ranking-quality benchmark.""" from integration import finished, request, wait_ready -from model_cases import REDACT_TEXT, RERANK_CASES, SIMILAR_TEXTS +from model_cases import REDACT_TEXT, RERANK_CASES, SIMILAR_TEXTS, UNCASED_MASKED, UNCASED_TEXT FRANCE = ["Bananas are yellow fruit.", "Paris is the capital of France.", "A car has four wheels."] @@ -56,14 +56,19 @@ def main(): status, job, _ = request("/redact", {"text": REDACT_TEXT, "entities": ["PER", "LOC"]}) assert status == 202 result = finished(job["id"])["result"] - assert result["model"] == "Xenova/bert-base-NER:int8", result["model"] + assert result["model"] == "Xenova/bert-base-NER:int8+Xenova/bert-base-NER-uncased:int8", result["model"] assert result["text"] == "\u2014 [PER] wrote to [EMAIL] about [LOC].", result["text"] assert [s["source"] for s in result["spans"]] == ["model:PER", "rule:email", "model:LOC"], result["spans"] assert all(0 < s["score"] <= 1 for s in result["spans"]) + status, job, _ = request("/redact", {"text": UNCASED_TEXT, "entities": ["PER", "ORG", "LOC"], "min_score": 0.5}) + assert status == 202 + result = finished(job["id"])["result"] + assert result["text"] == UNCASED_MASKED, result + assert [s["source"] for s in result["spans"]] == ["model:PER", "model:LOC", "model:ORG"], result["spans"] # Precision guard: ordinary text must not grow spans. status, job, _ = request("/redact", {"text": "The weather is nice today and the meeting starts at noon."}) assert finished(job["id"])["result"]["spans"] == [] - print("PASS: 2 real ONNX ranking cases, a 70-document batched ranking with score parity, 1 embedding case and 2 redaction cases through HTTP -> FrankenPHP worker -> PHP client -> Python model") + print("PASS: 2 real ONNX ranking cases, a 70-document batched ranking with score parity, 1 embedding case and 3 redaction cases through HTTP -> FrankenPHP worker -> PHP client -> Python model") if __name__ == "__main__": diff --git a/tests/php/model.php b/tests/php/model.php index d38c304..e658250 100644 --- a/tests/php/model.php +++ b/tests/php/model.php @@ -87,14 +87,25 @@ $text = "\u{2014} John Smith wrote to john@example.com about Berlin."; $job = $client->submitRedact($text, ['PER', 'LOC'], timeoutMs: 60000); $redaction = RedactResult::fromJob($client->wait($job->id, 60000), $text); -if ($redaction->model !== 'Xenova/bert-base-NER:int8' || $redaction->text !== "\u{2014} [PER] wrote to [EMAIL] about [LOC]." +if ($redaction->model !== 'Xenova/bert-base-NER:int8+Xenova/bert-base-NER-uncased:int8' + || $redaction->text !== "\u{2014} [PER] wrote to [EMAIL] about [LOC]." || array_column($redaction->spans, 'source') !== ['model:PER', 'rule:email', 'model:LOC'] ) { throw new RuntimeException('Real NER smoke case failed: ' . json_encode($redaction->spans)); } +// Lower-case names in a sentence that holds a capital: only the uncased model masks them, and only if it reads each +// of its labels in the right order (tests/model_cases.py). +$text = 'I spoke with esperanza in lisbon, she works for brightline insurance.'; +$job = $client->submitRedact($text, ['PER', 'ORG', 'LOC'], 0.5, 60000); +$redaction = RedactResult::fromJob($client->wait($job->id, 60000), $text); +if ($redaction->text !== 'I spoke with [PER] in [LOC], she works for [ORG].' + || array_column($redaction->spans, 'source') !== ['model:PER', 'model:LOC', 'model:ORG'] +) { + throw new RuntimeException('Uncased NER smoke case failed: ' . json_encode($redaction->spans)); +} $clean = 'The weather is nice today and the meeting starts at noon.'; $job = $client->submitRedact($clean, timeoutMs: 60000); if (RedactResult::fromJob($client->wait($job->id, 60000), $clean)->spans !== []) { throw new RuntimeException('Real NER smoke case produced spans on clean text'); } -echo "PASS: 2 real ONNX ranking cases, a 70-document batched ranking with score parity, 1 embedding case and 2 redaction cases through the PHP client\n"; +echo "PASS: 2 real ONNX ranking cases, a 70-document batched ranking with score parity, 1 embedding case and 3 redaction cases through the PHP client\n"; diff --git a/tests/python/test_fetch_model.py b/tests/python/test_fetch_model.py new file mode 100644 index 0000000..ddc33e1 --- /dev/null +++ b/tests/python/test_fetch_model.py @@ -0,0 +1,61 @@ +"""scripts/fetch_model.py pins every model file the worker loads, by revision and SHA-256. + +The download itself needs the network and runs in the model workflow; these check the pins offline. +""" + +import importlib.util +import unittest +from pathlib import Path +from unittest.mock import patch + +from ai_bridge import tasks + +REPO = Path(__file__).resolve().parents[2] +spec = importlib.util.spec_from_file_location("fetch_model", REPO / "scripts" / "fetch_model.py") +fetch_model = importlib.util.module_from_spec(spec) +spec.loader.exec_module(fetch_model) + + +class PinTests(unittest.TestCase): + def test_every_model_is_pinned_by_revision_licence_and_hash(self): + for directory, pin in fetch_model.MODELS.items(): + with self.subTest(directory=directory): + self.assertRegex(pin["revision"], r"^[0-9a-f]{40}$") + self.assertTrue(pin["license"]) + self.assertEqual(sorted(pin["files"].values()), ["model.onnx", "tokenizer.json"]) + self.assertEqual(set(pin["hashes"]), set(pin["files"].values())) + for digest in pin["hashes"].values(): + self.assertRegex(digest, r"^[0-9a-f]{64}$") + + def test_the_uncased_ner_model_is_the_int8_conversion_of_the_same_author_s_uncased_weights(self): + pin = fetch_model.MODELS["redact-uncased"] + self.assertEqual(pin["repository"], "Xenova/bert-base-NER-uncased") + self.assertEqual(pin["files"]["onnx/model_int8.onnx"], "model.onnx") + self.assertEqual(pin["license"], fetch_model.MODELS["redact"]["license"]) + + def test_every_directory_the_ner_pass_loads_is_one_the_fetcher_fills(self): + loaded = [] + + class Encoding: + word_ids, offsets, overflowing = [], [], [] + + class Tokenizer: + @staticmethod + def encode(text): + return Encoding() + + def load(directory, max_length, stride=0): + loaded.append(Path(directory).name) + return Tokenizer(), None, set() + + with patch.object(tasks, "_onnx", side_effect=load), patch.object(tasks, "_probabilities", return_value=[]): + tasks._ner("ask priya", ["PER"], 0.85, "/models", lambda *_: None) + self.assertEqual(sorted(loaded), ["redact", "redact-uncased"]) + self.assertLessEqual(set(loaded), set(fetch_model.MODELS)) + + def test_the_worker_checks_at_start_for_exactly_the_directories_the_fetcher_fills(self): + self.assertEqual(sorted(tasks.MODEL_DIRECTORIES), sorted(fetch_model.MODELS)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/python/test_jobs.py b/tests/python/test_jobs.py index 50b42dc..7c4007b 100644 --- a/tests/python/test_jobs.py +++ b/tests/python/test_jobs.py @@ -524,8 +524,12 @@ def test_every_offset_is_kept_even_where_case_would_change_length(self): class NerPassesTests(unittest.TestCase): - """_ner runs the model on the text and, when a sentence is uncased, on its title-cased copy; spans from both - come back in the text's own offsets, the text's first. A stub model tags every capitalised word as a person.""" + """_ner runs the cased model on the text and, when a sentence is uncased, on its title-cased copy, then the + uncased model on the text as written; spans from every reading come back in the text's own offsets. The stub + cased model tags every capitalised word as a person; the stub uncased model tags the names it knows whatever + their case, as the real one reads "rahim" and "Rahim" alike.""" + + UNCASED_NAMES = {"priya", "nadia", "rahim"} class Encoding: def __init__(self, text): @@ -544,41 +548,80 @@ def encode(self, text): @staticmethod def probabilities(session, names, window): + # The session stands in as its model directory's name, so the stub knows which model it is. rows = [] for position, word_id in enumerate(window.word_ids): row = [0.0] * len(tasks.NER_LABELS) start, end = window.offsets[position] - capitalised = word_id is not None and window.text[start:end][:1].isupper() - row[tasks.NER_LABELS.index("B-PER" if capitalised else "O")] = 0.99 + word = window.text[start:end] + if session == "redact-uncased": + person = word_id is not None and word.lower() in NerPassesTests.UNCASED_NAMES + else: + person = word_id is not None and word[:1].isupper() + row[tasks.NER_LABELS.index("B-PER" if person else "O")] = 0.99 rows.append(row) return rows + def stubs(self): + tokenizers = {} + + def load(directory, max_length, stride=0): + name = Path(directory).name + tokenizers[name] = self.Tokenizer() + return tokenizers[name], name, set() + + return tokenizers, patch.object(tasks, "_onnx", side_effect=load), \ + patch.object(tasks, "_probabilities", self.probabilities) + def run_ner(self, text): - tokenizer, events = self.Tokenizer(), [] - with patch.object(tasks, "_onnx", return_value=(tokenizer, object(), set())), \ - patch.object(tasks, "_probabilities", self.probabilities): - spans = tasks._ner(text, ["PER"], 0.85, "/unused", lambda completed, total: events.append((completed, total))) - return [(span["start"], span["end"]) for span in spans], tokenizer.texts, events + events = [] + tokenizers, load, probabilities = self.stubs() + with load, probabilities: + spans = tasks._ner(text, ["PER"], 0.85, "/models", lambda completed, total: events.append((completed, total))) + texts = {name: tokenizer.texts for name, tokenizer in tokenizers.items()} + return [(span["start"], span["end"]) for span in spans], texts, events def test_an_uncased_sentence_is_read_again_in_title_case(self): spans, texts, events = self.run_ner("ask priya") - self.assertEqual(texts, ["ask priya", "Ask Priya"]) - self.assertEqual(spans, [(0, 3), (4, 9)]) # offsets of the original text - self.assertEqual(events, [(0, 2), (1, 2), (2, 2)]) + self.assertEqual(texts["redact"], ["ask priya", "Ask Priya"]) + # "Ask" and "Priya" from the title-cased copy, in offsets of the original text; "priya" again from the uncased model. + self.assertEqual(sorted(spans), [(0, 3), (4, 9), (4, 9)]) + self.assertEqual(events, [(0, 3), (1, 3), (2, 3), (3, 3)]) def test_only_the_rewritten_sentences_are_read_again(self): # The cased sentence is read once; the second reading covers the lower-case one alone, so it costs one more # window rather than the whole text again, and its spans land at the original offsets. spans, texts, events = self.run_ner("Ask Priya now. ask nadia") - self.assertEqual(texts, ["Ask Priya now. ask nadia", " Ask Nadia"]) + self.assertEqual(texts["redact"], ["Ask Priya now. ask nadia", " Ask Nadia"]) self.assertIn((15, 18), spans) # "ask" self.assertIn((19, 24), spans) # "nadia" - self.assertEqual(events[-1], (2, 2)) + self.assertEqual(events[-1], (3, 3)) - def test_cased_text_is_read_once(self): + def test_cased_text_is_read_once_by_each_model(self): spans, texts, events = self.run_ner("Ask Priya.") - self.assertEqual(texts, ["Ask Priya."]) - self.assertEqual(events, [(0, 1), (1, 1)]) + self.assertEqual(texts, {"redact": ["Ask Priya."], "redact-uncased": ["Ask Priya."]}) + self.assertEqual(events, [(0, 2), (1, 2), (2, 2)]) + + def test_the_uncased_model_reads_the_whole_text_as_written(self): + # Not the title-cased copy: case means nothing to it, and the cased sentences need it too. + spans, texts, events = self.run_ner("Ask Priya now. ask nadia") + self.assertEqual(texts["redact-uncased"], ["Ask Priya now. ask nadia"]) + + def test_a_lower_case_name_among_capitals_is_found_by_the_uncased_model(self): + # The sentence holds capitals, so it is not rewritten, and the cased model passes over "rahim". + spans, texts, events = self.run_ner("the 2PM call is with rahim") + self.assertEqual(texts["redact"], ["the 2PM call is with rahim"]) + self.assertEqual(spans, [(21, 26)]) + + def test_redaction_masks_what_only_the_uncased_model_finds_and_names_both_models(self): + tokenizers, load, probabilities = self.stubs() + with load, probabilities: + result = execute("redact", {"text": "the 2PM call is with rahim"}, "onnx", "/models", lambda *_: None) + self.assertEqual(result["text"], "the 2PM call is with [PER]") + self.assertEqual([span["source"] for span in result["spans"]], ["model:PER"]) + self.assertEqual(result["model"], tasks.MODEL_NAMES["onnx"]["redact"]) + self.assertIn("Xenova/bert-base-NER:int8", result["model"]) + self.assertIn("Xenova/bert-base-NER-uncased:int8", result["model"]) class ModelNameTests(unittest.TestCase): diff --git a/tests/python/test_server.py b/tests/python/test_server.py index 7e6dad5..3d27bb3 100644 --- a/tests/python/test_server.py +++ b/tests/python/test_server.py @@ -1,8 +1,13 @@ import http.client import json +import os +import sys +import tempfile import threading import unittest +from unittest.mock import patch +from ai_bridge import server, tasks from ai_bridge.jobs import Settings from ai_bridge.server import BridgeServer @@ -95,3 +100,50 @@ def test_weak_token_is_rejected(self): if __name__ == "__main__": unittest.main() + + +class StartupTests(unittest.TestCase): + """An ONNX worker refuses to start without its model files, naming what is missing, rather than failing every job + as task_failed: a cache fetched before a model was added holds no directory for it.""" + + def models(self, root, complete): + for name in complete: + os.makedirs(os.path.join(root, name)) + for file in ("tokenizer.json", "model.onnx"): + open(os.path.join(root, name, file), "w").close() + + def start(self, backend, root): + environment = {"BRIDGE_BACKEND": backend, "BRIDGE_MODEL_DIR": root, "BRIDGE_TOKEN": TOKEN} + with patch.dict(os.environ, environment), patch.object(sys, "argv", ["ai_bridge.server", "--port", "0"]), \ + patch.object(server, "BridgeServer") as bridge, patch.object(server.signal, "signal"): + server.main() + return bridge + + def test_missing_models_names_each_directory_without_both_files(self): + with tempfile.TemporaryDirectory() as root: + self.models(root, ["rerank", "embed", "redact"]) + os.makedirs(os.path.join(root, "redact-uncased")) + open(os.path.join(root, "redact-uncased", "tokenizer.json"), "w").close() + self.assertEqual(tasks.missing_models(root), ["redact-uncased"]) + self.assertEqual(tasks.missing_models(os.path.join(root, "absent")), list(tasks.MODEL_DIRECTORIES)) + + def test_an_onnx_worker_refuses_to_start_without_its_models(self): + with tempfile.TemporaryDirectory() as root: + self.models(root, ["rerank", "embed", "redact"]) + with self.assertRaises(SystemExit) as raised: + self.start("onnx", root) + message = str(raised.exception.code) + self.assertIn("redact-uncased", message) + self.assertIn("fetcher", message) + self.assertNotIn(TOKEN, message) + + def test_an_onnx_worker_with_every_model_starts(self): + with tempfile.TemporaryDirectory() as root: + self.models(root, tasks.MODEL_DIRECTORIES) + bridge = self.start("onnx", root) + bridge.assert_called_once() + + def test_a_lexical_worker_needs_no_models(self): + with tempfile.TemporaryDirectory() as root: + bridge = self.start("lexical", root) + bridge.assert_called_once() diff --git a/worker/ai_bridge/server.py b/worker/ai_bridge/server.py index 70a7575..7b44a46 100644 --- a/worker/ai_bridge/server.py +++ b/worker/ai_bridge/server.py @@ -11,6 +11,7 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from .jobs import CapacityError, JobStore, Settings +from .tasks import missing_models MAX_BODY = 262144 @@ -170,6 +171,12 @@ def main(): model_dir=os.environ.get("BRIDGE_MODEL_DIR", "/models"), test_tasks=os.environ.get("BRIDGE_TEST_TASKS") == "1", ) + # Without its files every job needing a model would fail as task_failed, which cannot say why: a cache fetched + # before a model was added has no directory for it. + missing = missing_models(settings.model_dir) if settings.backend == "onnx" else [] + if missing: + raise SystemExit(f"{settings.model_dir} lacks the model files for {', '.join(missing)}; acquire them with " + "the fetcher: docker compose -f docker/compose.yaml run --rm --no-deps fetcher") with BridgeServer((args.host, args.port), os.environ.get("BRIDGE_TOKEN", ""), settings) as server: def stop(*_): threading.Thread(target=server.shutdown, daemon=True).start() diff --git a/worker/ai_bridge/tasks.py b/worker/ai_bridge/tasks.py index 3ff1568..cd7b1b0 100644 --- a/worker/ai_bridge/tasks.py +++ b/worker/ai_bridge/tasks.py @@ -27,13 +27,17 @@ "lexical": {"rerank": "lexical-demo-not-a-model", "embed": "hashing-bow-not-a-model", "redact": "rules-only-not-a-model"}, "onnx": {"rerank": "cross-encoder/ms-marco-TinyBERT-L2-v2", "embed": "sentence-transformers/all-MiniLM-L6-v2", - "redact": "Xenova/bert-base-NER:int8"}, + "redact": "Xenova/bert-base-NER:int8+Xenova/bert-base-NER-uncased:int8"}, } # MISC (nationalities, events, products) is deliberately not offered: it is the noisiest class and rarely personal data. REDACT_ENTITIES = ("PER", "ORG", "LOC") REDACT_DEFAULTS = {"entities": ["PER"], "min_score": 0.85} -# Label order of dslim/bert-base-NER; the smoke test fails loudly if a re-pinned model changes it. +# Label order of dslim/bert-base-NER and dslim/bert-base-NER-uncased, which share it; the smoke tests fail loudly if a +# re-pinned model changes it. NER_LABELS = ("O", "B-MISC", "I-MISC", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC") +# What the ONNX backend loads from its model directory: one directory per model, each holding the tokenizer.json and +# model.onnx that scripts/fetch_model.py writes. tests/python/test_fetch_model.py keeps the two lists in step. +MODEL_DIRECTORIES = ("rerank", "embed", "redact", "redact-uncased") class InvalidInput(ValueError): @@ -198,6 +202,12 @@ def _onnx(directory: Path, max_length: int, stride: int = 0): return tokenizer, session, {item.name for item in session.get_inputs()} +def missing_models(model_dir: str) -> list: + """The model directories under model_dir that lack either file, in MODEL_DIRECTORIES order.""" + return [name for name in MODEL_DIRECTORIES + if not all((Path(model_dir) / name / file).is_file() for file in ("tokenizer.json", "model.onnx"))] + + def _signed_counts(words: list) -> list: # sha256 keeps the demo stable across processes; hash() is salted per process. vector = [0.0] * EMBED_DIMENSIONS @@ -494,19 +504,23 @@ def _probabilities(session, names, window) -> list: def _ner(text: str, entities: list, min_score: float, model_dir: str, progress) -> list: - # Overlapping 512 piece windows cover long text; offsets stay relative to the whole text. Each run of uncased - # sentences is read again from its title-cased copy, and only that run, so a second reading costs a window or two - # and cannot relabel a sentence that needed no help; where both readings find the same span, _merge keeps the - # more confident. - tokenizer, session, names = _onnx(Path(model_dir) / "redact", 512, stride=64) + # Overlapping 512 piece windows cover long text; offsets stay relative to the whole text. The cased model reads the + # text, and each run of uncased sentences again from its title-cased copy, and only that run, so a second reading + # costs a window or two and cannot relabel a sentence that needed no help. The uncased model then reads the whole + # text as written: case means nothing to it, so it finds a lower-case name in a sentence that holds a capital, + # which neither cased reading can, while the cased model keeps what capitals tell in news and records. Where + # readings find the same span, _merge keeps the more confident. + cased = _onnx(Path(model_dir) / "redact", 512, stride=64) + uncased = _onnx(Path(model_dir) / "redact-uncased", 512, stride=64) variant, ranges = _recase(text) + readings = [(cased, 0, text)] + [(cased, start, variant[start:end]) for start, end in ranges] + [(uncased, 0, text)] windows = [] - for offset, reading in [(0, text)] + [(start, variant[start:end]) for start, end in ranges]: + for (tokenizer, session, names), offset, reading in readings: encoding = tokenizer.encode(reading) - windows.extend((offset, reading, window) for window in [encoding, *encoding.overflowing]) + windows.extend((session, names, offset, reading, window) for window in [encoding, *encoding.overflowing]) progress(0, len(windows)) candidates = [] - for index, (offset, reading, window) in enumerate(windows): + for index, (session, names, offset, reading, window) in enumerate(windows): for span in _entities(window, _probabilities(session, names, window), entities, min_score, reading): candidates.append({**span, "start": span["start"] + offset, "end": span["end"] + offset}) progress(index + 1, len(windows)) @@ -514,9 +528,9 @@ def _ner(text: str, entities: list, min_score: float, model_dir: str, progress) def _merge(candidates: list) -> list: - # Leftmost first, then longest, then the stronger source, then the more confident: of the two readings of an - # uncased sentence, a sure "person" beats an unsure "place" for the same words. A candidate overlapping the kept - # span extends it instead of being dropped, so no part of either stays visible. + # Leftmost first, then longest, then the stronger source, then the more confident: of two model readings of the + # same words, a sure "person" beats an unsure "place". A candidate overlapping the kept span extends it instead of + # being dropped, so no part of either stays visible. spans = [] for span in sorted(candidates, key=lambda span: (span["start"], span["start"] - span["end"], span["priority"], -span["score"])):