Password Breach Checker

The purpose of this repository is a proof of concept to make a clone of the API found at: haveibeenpwned

The general guideline is to explore optimization techniques in Go and use the provided password hash file as much as possible to avoid using any database while at the same time having a high performant service.

Architecture

The database is split into 256 memory-mapped binary files (00.bin–FF.bin), each keyed by the first byte of the SHA1 hash. This reduces binary search from ~30 comparisons (single file) to ~21 comparisons per prefix file, with zero lock contention since all data is read-only.

Each 24-byte binary record contains:

• 20 bytes — full SHA1 hash (sorted ascending)
• 4 bytes — breach count (big-endian uint32)

An optional Bloom filter (bloom.bin) provides fast negative lookups, skipping the binary search entirely for hashes that are definitively not in the database.

Downloading

The cmd/download utility downloads all SHA1 hash ranges directly from the HIBP k-anonymity API, binarizes them on the fly, and writes 256 sorted .bin files — no intermediate text storage needed.

# Download all 256 prefix files to ./data (resume supported)
go run ./cmd/download/main.go -o data

# Options:
#   -o        Output directory (default: ".")
#   -c        Concurrent downloads per prefix (default: 16)
#   -m        Minimum breach count to include (default: 1)
#   -rate     Min delay between API requests (default: 5ms)
#   -n        Estimated entries for bloom filter sizing (default: 2B)
#   -fpr      Bloom filter false positive rate (default: 0.01 = 1%)

Resume support: existing .bin files are skipped, so interrupted downloads can be restarted.

Verifying & Bloom Filter Generation

The cmd/verify utility verifies that all 256 .bin files are sorted and generates a bloom.bin file from the existing binary files — no re-download needed.

go run ./cmd/verify/main.go data

This first stats all files to compute the exact total record count, then verifies sort order and populates the bloom filter in a single pass.

Server

The cmd/server provides an API to check passwords against the database by receiving a SHA1 hash and returning a breach count, where 0 means not found. It includes a minimal frontend embedded in the binary.

# Run with bloom filter (default)
go run ./cmd/server/main.go -db data -port 3000

# Run without bloom filter
go run ./cmd/server/main.go -db data -port 3000 -bloom=false

# Disable logging for load tests
go run ./cmd/server/main.go -db data -port 3000 -log=false

Flags:

Flag	Default	Description
-db	""	Directory containing 00.bin–FF.bin prefix files
-port	"3000"	Port to run server
-log	true	Enable/disable logging middleware
-bloom	true	Enable bloom filter for fast negative lookups (requires bloom.bin in db dir)

Check Utility

The cmd/check utility checks a single password against the database.

go run ./cmd/check/main.go -f data -p "password123" -bloom=false

K6 Load Testing

A K6 script is provided in load.js for load testing the API. It tracks 200 status checks and counts of found vs not-found responses.

k6 run --vus 30 --duration 60s load.js

Example results without bloom filter (30 VUs, 60s):

     http_reqs......................: 1720612 28675.85/s
     checks_total...................: 1720612
     checks_succeeded...............: 100.00%
     not_found......................: 1720612 28675.85/s
     http_req_duration..............: avg=956.03µs med=786µs p(95)=2.26ms
     iteration_duration.............: avg=1.04ms med=872.62µs p(95)=2.35ms

Example results with bloom filter (30 VUs, 60s):

     http_reqs......................: 4433287 73887.54/s
     checks_total...................: 4433287
     checks_succeeded...............: 100.00%
     not_found......................: 4433287 73887.54/s
     http_req_duration..............: avg=301.92µs med=246µs p(95)=759µs
     iteration_duration.............: avg=400.48µs med=335.54µs p(95)=898.12µs

The bloom filter provides a 2.6x throughput increase (28.6K → 73.9K req/s) and 3.2x latency reduction (avg 956µs → 302µs) for not-found queries.

Note: These are "happy path" results where all queries are not found (random SHA1 hashes). Real-world traffic with actual breached passwords would hit the binary search path more often, reducing the bloom filter's advantage.

Benchmarks

Database-level benchmarks on Apple M2 Max, 2,048,908,128 records across 256 prefix files:

Benchmark	Without Bloom	With Bloom
Single hash (cached)	57.70 ns/op	23.08 ns/op
Random hashes	311,174 ns/op	14,743 ns/op
Existing hashes (worst case)	315,789 ns/op	462,425 ns/op

• Single hash: Same hash searched repeatedly — benefits from CPU cache. Bloom is 2.5x faster.
• Random hashes: Different random hash each iteration — simulates not-found queries. Bloom is 21x faster (skips binary search entirely).
• Existing hashes: Searches for hashes known to be in the DB — bloom filter always passes through to binary search, adding ~150µs overhead. This is the worst case for the bloom filter.

goos: darwin
goarch: arm64
cpu: Apple M2 Max

# Without bloom filter
BenchmarkSearch-12            19917796    57.70 ns/op    0 B/op    0 allocs/op
BenchmarkSearchRandom-12          3825    311174 ns/op   0 B/op    0 allocs/op
BenchmarkSearchCompare-12         3505    315789 ns/op   0 B/op    0 allocs/op

# With bloom filter
BenchmarkSearchBloom-12       53722322    23.08 ns/op    0 B/op    0 allocs/op
BenchmarkSearchRandomBloom-12    69834    14743 ns/op    0 B/op    0 allocs/op
BenchmarkSearchCompareBloom-12    2199    462425 ns/op   0 B/op    0 allocs/op