Skip to content

ingestion safety: fail-closed pii masking + health visibility - #243

Merged
Polliog merged 18 commits into
developfrom
fix/ingestion-safety
Jun 10, 2026
Merged

ingestion safety: fail-closed pii masking + health visibility#243
Polliog merged 18 commits into
developfrom
fix/ingestion-safety

Conversation

@Polliog

@Polliog Polliog commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

What this does

Makes ingestion fail-closed and gives visibility to silent failures.

Security

  • Fail-closed PII masking: previously, if the masking step threw, the batch was stored unmasked with only a console.warn. maskLogBatch now reports per-record failures and the ingestion service rejects exactly those records before the reservoir write (the whole batch if rule compilation itself fails). Rejected records are reported back to the client in the new optional rejected: [{ index, reason: 'pii_masking_failed' }] response field and as partialSuccess.rejectedLogRecords on the OTLP logs endpoint. Every rejection increments the ingestion.pii_rejected counter. Unmasked data can no longer reach any storage engine.

Added

  • Ingestion health for admins: new GET /api/v1/admin/stats/ingestion-health endpoint aggregating the last 24h of the new counters (pii_rejected, detection_enqueue_failed, exception_enqueue_failed, identifier_failed) plus SIEM enrichment status (GeoLite2/IPsum), shown in an admin dashboard card that turns red on failures. Detection/exception job enqueues now retry once and, on a second failure, log loudly instead of failing silently.

Fixed

  • Silent failures made visible in the admin/monitoring UI (version check no longer claims "up to date" when GitHub is unreachable; the monitoring page shows error banners instead of empty catch {}; GeoLite2/IPsum warn once per outage).
  • getStats date-filter tests strengthened (exact assertions instead of toBeGreaterThanOrEqual(0)).
  • Email number formatting pinned to en-US locale (digests and alert emails no longer depend on the server locale).

@codecov

codecov Bot commented Jun 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.91983% with 31 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
packages/backend/src/modules/ingestion/service.ts 75.70% 26 Missing ⚠️
packages/backend/src/modules/admin/routes.ts 73.68% 5 Missing ⚠️

📢 Thoughts on this report? Let us know!

@Polliog
Polliog merged commit 98e71d7 into develop Jun 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant