Skip to content

perf: scan multi-prefix token detectors once - #4

Merged
ptukovar merged 1 commit into
ptukovar:mainfrom
tboser:perf/single-pass-multi-prefix-detectors
Aug 11, 2026
Merged

perf: scan multi-prefix token detectors once#4
ptukovar merged 1 commit into
ptukovar:mainfrom
tboser:perf/single-pass-multi-prefix-detectors

Conversation

@tboser

@tboser tboser commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • scan each same-kind prefix family in one pass instead of rescanning the full input once per prefix
  • skip prefix comparisons cheaply when the current byte cannot begin any configured prefix
  • cover every GitHub, Slack, Stripe, and OpenAI prefix variant with synthetic fixtures

Why

The four multi-prefix detectors currently account for 20 complete passes over a clean input. This is visible when a caller selects credential detectors without the PII set.

On an Intel Xeon 8488C, with a synthetic clean 8 MiB JSON body and the same 14-detector credential-only set before/after this commit:

Case Before After
full set, serial, one caller 17.6 MiB/s 93.4 MiB/s
full set, parallel, one caller 70.8 MiB/s 316.5 MiB/s
full set, parallel, four callers 172.4 MiB/s 552.2 MiB/s
GitHub detector 64.9 MiB/s 2,275.9 MiB/s
Slack detector 65.0 MiB/s 2,326.8 MiB/s
Stripe detector 63.6 MiB/s 2,237.4 MiB/s
OpenAI detector 189.7 MiB/s 2,339.3 MiB/s

Inputs and credentials used for tests and measurement are synthetic.

Validation

  • cargo fmt --all -- --check
  • cargo clippy --all-targets -- -D warnings
  • cargo test --all-features
  • cargo build --no-default-features
  • cargo test --release --test complexity -- --ignored

@ptukovar

Copy link
Copy Markdown
Owner

Thank you 😸

@ptukovar
ptukovar merged commit be544cb into ptukovar:main Aug 11, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants