feed-quality-gate

Fail-closed validation firewall for inbound transaction feeds: schema, manifest control totals, exact duplicate fingerprints, and robust outlier checks at 25k rows/s in a flat 257 MB, with machine-readable quarantine verdicts.

What this solves

• Bad feeds reach financial reporting silently: a truncated extract, a redelivered file under a new name, amounts that stopped parsing. This gate holds them at the door with evidence.
• Every rejection is a machine-readable verdict (per-check stats, sample rows, reasons), so the conversation with the sender starts from facts, and every acceptance is idempotent and remembered.
• Bounded by design and measured: peak memory plateaus at 257 MB from 1M to 3M rows, so the gate's capacity does not depend on how big a file upstream decides to send.

Why this exists

Every reporting pipeline in banking and payments ingests files it did not produce: core banking extracts, processor settlement feeds, scheme files. The failure pattern is always the same and always expensive: the file arrives, lands in the warehouse, dashboards refresh, and three days later someone notices revenue is short because the extract was truncated at row 2,970,000, or doubled because Tuesday's file was redelivered with Wednesday's name. By then the bad data has been joined, aggregated, and quoted.

feed-quality-gate sits between arrival and ingestion. Each feed must carry a manifest sidecar declaring what the sender intended (row count, control total, business date, source system); the gate verifies the file against the manifest, never the other way around. Four checks run in chunked passes: schema and parseability, control totals accumulated in integer cents (float accumulation drifts at millions of rows), SHA-256 row fingerprints for duplicates both within the file and against every previously accepted feed, and robust (median/MAD) amount outliers per transaction type. Verdicts are fail-closed: accepted feeds move on with their verdict document, anything else lands in quarantine with the reason. A ledger makes reprocessing idempotent: the identical file redelivered is recognized by hash before any check runs.

It runs identically against a local inbox directory or an S3-compatible landing zone (AWS or MinIO; compose file included), because that is where these feeds actually land.

Architecture

flowchart LR
    A[feed.csv + manifest sidecar\nlocal inbox or s3://bucket/inbox/] --> H{ledger:\nfile hash seen?}
    H -->|yes| Q[quarantine/\nduplicate_delivery verdict]
    H -->|no| C1[schema check]
    C1 -->|fail| Q2[quarantine/ + verdict JSON\nreasons + sample rows]
    C1 --> C2[control totals vs manifest\ninteger-cent accumulation]
    C2 --> C3[duplicate fingerprints\nSQLite staging, within + cross-file]
    C3 --> C4[outliers per txn_type\nmedian/MAD, warn or reject]
    C2 & C3 & C4 --> V{all reject-severity\nchecks passed?}
    V -->|no| Q2
    V -->|yes| OK[accepted/ + verdict JSON\nfingerprints committed to ledger]

Loading

Tech stack

Technology	Role in this project	Why chosen here
Python 3.10+	Gate, checks, CLI	Checks read like the control procedures they implement
pandas	Chunked CSV passes	dtype=str chunks keep parsing decisions explicit and memory bounded
SQLite (stdlib)	Ledger + duplicate staging	Set arithmetic on millions of fingerprints in SQL at flat memory (ADR-002); idempotency ledger in one file
hashlib SHA-256	File and row fingerprints	Exact evidence: every duplicate verdict names real transaction ids
boto3 + MinIO (optional)	S3 landing zone mode	Feeds land in object storage in practice; same pipeline, safe move semantics
pytest + moto	29 tests, 95% measured coverage	S3 path tested against a real S3 API mock, no daemon required

Quickstart

Prerequisites: Python 3.10+, pip, git.

git clone https://github.com/Vanithanallamothu/feed-quality-gate.git
cd feed-quality-gate
pip install -e ".[dev,s3]"

# generate one clean and two defective feeds into an inbox
mkdir -p /tmp/fq/inbox
python data/generator.py --rows 50000 --out /tmp/fq/inbox/feed_a.csv --seed 7
python data/generator.py --rows 30000 --out /tmp/fq/inbox/feed_b.csv --seed 8 \
    --seq-start 1000000 --defect truncated
python data/generator.py --rows 20000 --out /tmp/fq/inbox/feed_c.csv --seed 9 \
    --seq-start 2000000 --defect dup-rows

# run the gate
fqgate process --inbox /tmp/fq/inbox --accepted /tmp/fq/ok \
    --quarantine /tmp/fq/bad --ledger /tmp/fq/ledger.db

# inspect a rejection verdict
cat /tmp/fq/bad/feed_b.csv.verdict.json

# run the tests
pytest --cov=fqgate

S3 mode (start MinIO with docker compose up -d, then):

AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin \
fqgate process --s3-bucket feeds --s3-endpoint http://localhost:9000 \
    --ledger ledger.db

Performance under load

Methodology: benchmark/run_benchmark.py generates clean feeds at three sizes and runs the full gate (file hash, manifest, four chunked check passes, ledger writes, file moves, verdict writes) 3 times per size with a fresh ledger, on a 2-CPU Linux container, Python 3.11. Peak RSS is the gate process's own high-water mark, reported by the child at exit. Raw output in benchmark/results/throughput.json.

xychart-beta
    title "Gate wall clock and peak memory vs feed size"
    x-axis ["100k rows", "1M rows", "3M rows"]
    y-axis "seconds (bars) / MB (line)" 0 --> 300
    bar "p50 seconds" [2.8, 35.9, 121.7]
    line "peak RSS MB" [135.4, 255.9, 256.8]

Loading

Rows	File size	p50	p95	p99	Throughput	Gate peak RSS
100,000	7.0 MB	2.79 s	2.95 s	2.97 s	35,202 rows/s	135.4 MB
1,000,000	69.6 MB	35.92 s	35.95 s	35.95 s	28,239 rows/s	255.9 MB
3,000,000	208.7 MB	121.69 s	124.14 s	124.35 s	24,644 rows/s	256.8 MB

The flat memory line is the design goal and it cost something real: the earlier in-memory implementation ran 47k rows/s but peaked at 1,041 MB at 3M rows and grew linearly (both numbers measured, see Hardest problem solved). Where it degrades: throughput declines gently with size as SQLite staging and ledger index inserts grow; 25k rows/s still clears a 3M-row daily feed in about two minutes, and the four-pass chunked design means wall clock scales with file size, never memory.

Architecture decisions

Two ADRs in docs/adr/:

• ADR-001: Fail closed, quarantine by default. The cost asymmetry: a delayed feed is bounded, wrong numbers in reporting are not.
• ADR-002: Bounded memory over maximum throughput. Why SQLite staging beat both in-memory sets and bloom filters, with the measured before and after.

Intentionally out of scope

• No streaming or Kafka intake. Feeds arrive as daily files today; the trigger is any upstream moving to continuous delivery, at which point the checks stay and the chunk source changes.
• No schema inference or auto-mapping. The schema contract is explicit config; guessing column meanings in a controls tool is how wrong data gets in politely.
• No parallel multi-file processing. Feeds process sequentially so the cross-file duplicate ledger stays race-free; the trigger to revisit is a sustained multi-sender backlog, which would shard the ledger by source system.

Security and compliance

• No secrets in code or config. S3 credentials come exclusively from the standard AWS environment/profile chain; the compose file's minioadmin defaults are local-development only and labeled as such.
• Transaction data is sensitive: verdicts embed at most 10 sample rows as evidence, logs carry counts and file names, never amounts by account. Run the gate where the data already lives.
• The ledger stores row fingerprints (SHA-256) and file hashes, not transaction contents.
• CI runs lint and tests with the default GitHub token scope; the S3 path is tested against moto's mock, so CI needs no cloud credentials.

Failure modes

Failure	Detection	Behavior	Recovery
Feed without manifest	Sidecar lookup	Quarantined: if the sender cannot say what they sent, it cannot be verified	Sender supplies the manifest; redelivery is clean
Truncated or padded file	Control totals vs manifest (integer cents)	Rejected with both counts and both sums in the verdict	Sender re-extracts; ledger treats the fixed file as new
Same file redelivered under any name	File SHA-256 in ledger, checked before any parsing	Quarantined as duplicate_delivery with the prior verdict named	None needed; this is the gate working
Overlapping incremental extracts	Cross-file row fingerprints	Rejected, verdict names sample txn ids and the file that first delivered them	Sender fixes the extract window
Unreadable or binary file	Parser exception boundary	That feed quarantined with the parse error; the rest of the inbox continues	Investigate the sender's export job
Gate killed mid-feed	Ledger transactionality	Staging table is scratch; the feed was not recorded accepted, so reprocessing is safe	Rerun the inbox; idempotency holds
Disk fills during quarantine move	OS error surfaces, nonzero exit	Gate itself fails loudly (reserved for gate failures, not feed failures)	Free space, rerun; nothing was recorded accepted

Hardest problem solved

This one is two bugs, and the second hid the first. The benchmark showed gate memory growing linearly with feed size, peaking around 1.8 GB at 3 million rows, which defeats the entire point of chunked reading. I rewrote the two offending checks: duplicate detection had been holding every row fingerprint in Python sets and lists, so I moved the staging into SQLite and let SQL do the set arithmetic; the outlier check had been concatenating whole dataframes, so it now accumulates only float64 amounts per transaction type plus a bounded evidence heap.

Then I reran the benchmark and the number did not move. Same 1.8 GB, to the tenth of a megabyte, which is not how memory bugs behave. The measurement was the second bug: the benchmark read RUSAGE_CHILDREN, which reports the high-water mark across all child processes, and the data generator, which legitimately builds the whole feed in RAM, was a child of the same benchmark. I had been measuring the generator and attributing it to the gate. The fix has the gate process report its own RUSAGE_SELF peak at exit (commit d60fa83).

With honest instrumentation, the before and after became real: the pre-fix gate peaked at 1,041 MB at 3M rows (measured by checking out the pre-fix code and rerunning); the fixed gate plateaus at 257 MB from 1M to 3M rows, at a throughput cost of about 40 percent, a trade ADR-002 defends. The generalizable lesson: when a fix does not move the metric, suspect the metric. Measurement code deserves the same skepticism as the code it measures.

Future work

• Shard the ledger by source system to allow parallel multi-sender processing (the trigger named in the scope section).
• A fqgate report command aggregating verdict JSONs into weekly sender scorecards; the verdicts are machine-readable precisely for this.
• Vectorize row fingerprinting (currently the dominant CPU cost) if feed sizes grow past tens of millions of rows.
• Schema contract versioning per source system, so a sender's layout change is a reviewed config change, not a surprise rejection.
• First metric to watch in real use: rejection rate by sender and check, which tells you which upstream extract jobs are quietly unreliable.