Skip to content

fix: three asks a platform raised against 26.09.06 on the day it shipped — v26.09.07 - #169

Merged
ancongui merged 1 commit into
mainfrom
fix/wave6-asks
Sep 24, 2026
Merged

ancongui merged 1 commit into
mainfrom
fix/wave6-asks

Conversation

@ancongui

Copy link
Copy Markdown
Contributor

Three asks a platform raised against 26.09.06 on the day it shipped. Each was found by running a service on the framework, not by reading it.

An identifier in a log message survives redaction, and PHONE means a phone number

The built-in PHONE rule was guarded by digits only — (?<!\d) and (?!\d) — and a hex letter or a hyphen satisfies a digit-only guard, so it matched inside a uuid. Measured on this checkout over 200 000 generated ids per shape: 24.9% of uuid4 and 24.8% of uuid7 were mangled, producing lines such as run 01a0d4d8-bc85-<PHONE>-1ff4a6a3a450. A run id grepped out of an application therefore never matched its own log line — silently and only sometimes, which is worse than always.

The same rule missed compact E.164 (+34911234567, +442071838750, +8613800138000): eleven contiguous digits cannot be split by \d{3}\d{4}. It gave up searchability and privacy at once, so this is not a trade.

  • Identifier shapes — uuid of any version, W3C trace id and span id, ULID — are carved out of the message before any entity pattern runs; the entity rules only ever see the gaps between them, on the regex engine and the Presidio engine alike. This is the message-text half of the rule fix(logging): never redact trace/span correlation keys #156 applied to structlog correlation field keys.
  • The 16- and 32-character hex shapes require at least one hex letter, so a bare 16-digit card is still read as a card rather than hidden behind the guard. There is a test for exactly that.
  • PHONE is re-anchored on word boundaries and matches compact E.164, separated international, (212) 555-0143, 202-555-0143, 555-0143. Behaviour change: a national number written as seven bare digits with no separator is deliberately no longer matched.
  • New key pyfly.logging.redaction.preserve-patterns (named-regex map, symmetric with extra-patterns) for a consumer's own id shape; an invalid entry is warned about and ignored.

After: 0.00% of uuid4 and uuid7 mangled, no true positive lost, three gained.

A serving process boots the Postgres event bus with no right to create schema

PostgresEventBus.start() ran the outbox CREATE TABLE/INDEX IF NOT EXISTS in every process at every boot, including a pure publisher (publish() lazily starts the bus). PostgreSQL checks CREATE on the schema before IF NOT EXISTS, and CREATE INDEX IF NOT EXISTS on a table you do not own fails even when the index exists — a GRANT CREATE ON SCHEMA does not fix it. The framework was asking every deployment to make its serving role an owner of two framework-internal tables.

start() now probes with one to_regclass round trip (needs only USAGE on the schema) and issues DDL only when a table is missing. A first boot against an empty database still creates everything. pyfly.eda.postgres.auto-create-tables: false is the opt-out for a deployment whose schema belongs to its migrations. The consumer group's cursor row stays outside the skip: it is data, not schema.

The CQRS cache-invalidation bridge subscribes one handler per rule, not the wildcard

EdaCacheInvalidationBridge.subscribe() registered "*" the moment a CQRS context had an EDA bus, whether or not any rule was registered — a process subscribed to everything in order to discard it. On the Postgres bus that is not passive: _drain refuses to advance the group's cursor while no handler is registered, precisely so events published before a worker subscribed are not lost, and a wildcard handler that dispatches nothing defeated that guard. An API sharing a worker's consumer group drained the worker's queue into a no-op and its jobs stayed queued.

The bridge now subscribes an event type when a rule for it is registered, and nothing while it has no rule; registering before or after the framework attaches it both work. pyfly.cqrs.cache.invalidation.enabled: false refuses the bean without disabling CQRS.

Tests

File Tests
tests/logging/test_identifiers_survive_redaction.py 40 — table-driven over uuid v1/v4/v7, trace id, span id, ULID, ISO timestamp, sha256, base64 token, IPv4, E.164, national numbers, 16-digit card; a sweep over 3 000 generated uuids per version; a real phone number still leaving a LogRecord; allow-list and span-splicing edge cases
tests/eda/test_postgres_outbox_ddl_gate.py 6 — start() against a fake asyncpg, asserting on the statements sent
tests/integration/test_eda_postgres_least_privilege.py 3 — real PostgreSQL, a role with USAGE + SELECT/INSERT/UPDATE and nothing else; fails on the old adapter with permission denied for schema public
tests/cqrs/test_cache_bridge_subscribes_narrowly.py 11
tests/eda/test_cursor_guard_with_cache_bridge.py 2 — the regression that names the damage

tests/logging/test_redaction_processor.py::test_trace_ids_are_never_redacted_even_with_phone_pattern had a sanity line asserting that PHONE would corrupt a trace id if unguarded. That is no longer true, which is the point; the assertion is inverted and the field guard it protects is untouched.

Gates

  • uv run ruff format --check src/ tests/ and uv run ruff check src/ tests/ — clean
  • uv run mypy src/pyfly --strict — 723 files, no issues
  • CI's pytest selection on 3.12 and 3.13 — 5150 passed, 7 skipped, 59 deselected (baseline on main with the same selection: 5097 collected; +60 tests, none lost)
  • python -m mkdocs build --strict — clean
  • pytest -m integration tests/integration/test_eda_postgres_least_privilege.py against a real PostgreSQL — 3 passed

Docs updated in docs/modules/logging.md, docs/modules/events.md and docs/modules/cqrs.md; version bumped to 26.09.07 per the repository's release habit.

…ped — v26.09.07

Carve identifier shapes (uuid, W3C trace/span id, ULID) out of a log message before any PII pattern runs, and make PHONE mean a phone number: it was guarded by digits only, so it mangled about one uuid in four and never matched compact E.164, giving up searchability and privacy at once. Probe before running the Postgres outbox DDL, so a serving process needs no CREATE on the schema and no ownership of framework tables, with pyfly.eda.postgres.auto-create-tables as the opt-out. Subscribe the CQRS cache-invalidation bridge one event type per registered rule instead of the wildcard, so a process with no rule is not a consumer of the bus and the drain loop's cursor guard works as written.
@ancongui
ancongui merged commit 02b792f into main Sep 24, 2026
9 checks passed
@ancongui
ancongui deleted the fix/wave6-asks branch September 24, 2026 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant