Part of the AI training-data de-identification epic.
Goal
Turn Phileas from "redact one document" into "de-identify a large corpus efficiently."
Scope
- Build the policy → filter pipeline once and reuse it across millions of records; make it immutable + thread-safe so it parallelizes across all cores. (The per-request rebuild/allocation is the throughput killer, so this depends on reusing PhileasConfiguration and filter services across requests and eliminating per-request service construction.)
- A batch / streaming API that consumes iterables/streams instead of one string per call.
- Columnar / record-oriented I/O — JSONL, Parquet, Arrow — with field-level redaction (scrub the
text field, preserve structure; handle chat transcripts and code).
Dual-use: this is a general high-throughput win (AI proxy, Philter, streaming), not specific to the training-data bet — worth building regardless.
Depends on: reusing PhileasConfiguration and filter services across requests, and eliminating per-request service construction.
Part of the AI training-data de-identification epic.
Goal
Turn Phileas from "redact one document" into "de-identify a large corpus efficiently."
Scope
textfield, preserve structure; handle chat transcripts and code).Dual-use: this is a general high-throughput win (AI proxy, Philter, streaming), not specific to the training-data bet — worth building regardless.
Depends on: reusing PhileasConfiguration and filter services across requests, and eliminating per-request service construction.