Closes one finance-ops loop: given a merchant's order ledger, Razorpay's own payment records, and a bank statement, the agent explains where every rupee that landed in the bank came from — and honestly flags whatever it can't explain, instead of guessing.
Razorpay doesn't pay out order-by-order. It bundles several orders into one settlement, takes a fee plus GST on that fee, and pays the net total days later. One bank line can represent many orders; some orders won't have settled yet; and real months contain refunds landing in the wrong batch, duplicate charges, missing settlements, and stray unrelated credits. The job is telling those apart.
Layers, outermost first. Each depends only on the ones below it.
lekha/
cli.py one entry point: python -m lekha generate|reconcile|score
reporting.py findings -> text and JSON
scoring.py grades against the sealed answer key (the ONLY reader of it)
explaining.py the local model; phrasing only, never arithmetic
matching.py the reconciler; all arithmetic, fully deterministic
loading.py CSV -> domain objects, collecting unreadable rows
parsing.py tolerant amount / date / column parsing
domain.py the typed things the system reasons about
config.py settings as data, tagged [P] published or [A] assumed
generating.py synthetic months plus the sealed key
tests/ 127 tests, including Python-vs-JavaScript conformance
examples/ the teaching scripts the design was proved with
docs/ the dashboard (one self-contained HTML file)
Two boundaries carry the design:
matching never imports scoring. The reconciler cannot reach the
answer key, so its accuracy is measured rather than self-reported. A test
asserts this rather than trusting it.
explaining never computes. Every figure in a written explanation was
already settled by matching. Asked to do the matching instead, the model
scored 0/9 — that measurement is why the boundary exists.
The reconciliation rules exist twice: in Python, and in JavaScript inside the dashboard so a merchant can reconcile their own files with nothing uploaded and nothing installed. Two copies of the same rules can drift apart silently.
Rather than pretend otherwise, tests/test_conformance.py extracts the
JavaScript straight out of the shipped page, runs identical fixtures
through both engines under Node, and fails if any answer differs. The
shared thresholds are asserted equal too.
Two things taking turns, not one monolithic program:
- CODE (plain Python, deterministic): loads the three files, rolls Razorpay's payment records up by settlement, reads the settlement reference embedded in the bank's own narration, and checks the arithmetic to the paisa. Also detects any order charged twice, and decides pending vs genuinely overdue.
- AGENT (Llama 3.2 3B via Ollama — free, on-device): writes the plain-English reason for whatever CODE couldn't reconcile. It never does arithmetic.
Why that split: the design this was built from tagged combination-search as agent work. Testing showed that was wrong — the model scored 0/9 at guessing which orders sum to a bank line, even with retry and feedback, while plain code solved the same lines instantly by reading the settlement reference Razorpay already prints. The architecture reflects what was measured, not the original guess.
[P]published, real-world figure — Razorpay's 2% fee, 18% GST, T+2 business-day settlement.[A]assumed by us, no empirical basis — refund/duplicate/stray rates, batch size. These are swept (below); results are reported as a range, not a single flattering point.
Five months, 60 orders each, generated from fixed seeds. Month 1 was used for development; months 2–5 were sealed and opened once, at the end.
| Month | Settlements | Match rate | Duplicates | Stray credits | False alarms |
|---|---|---|---|---|---|
| 1 (built against) | 18 / 19 | 94.7% | 2 / 2 | 2 / 2 | 0 |
| 2 (held out) | 15 / 16 | 93.8% | 2 / 2 | 2 / 2 | 0 |
| 3 (held out) | 17 / 19 | 89.5% | 2 / 2 | 2 / 2 | 0 |
| 4 (held out) | 17 / 18 | 94.4% | 2 / 2 | 2 / 2 | 0 |
| 5 (held out) | 18 / 19 | 94.7% | 2 / 2 | 2 / 2 | 0 |
| Held-out mean | 67 / 72 | 93.1% ± 2.1pp | 8 / 8 | 8 / 8 | 0 |
Pooled across all five months: 85 / 91 settlements (93.4%), 0 false alarms, 10/10 duplicates, 10/10 stray credits.
A number that only holds at the setting we invented is not a result.
sweep.py varies each [A] parameter across a plausible range, 8 seeds
per point (seeds 9000+, never used for development or held-out testing),
248 runs total.
| Parameter | Range swept | Match rate across that range |
|---|---|---|
P_DUPLICATE |
0 → 0.12 | 97.9% → 76.3% |
N_ORDERS |
50 → 200 | 94.2% → 89.4% |
P_REFUND |
0 → 0.25 | 92.5% → 98.0% |
N_MISSING_SETTLEMENTS |
0 → 6 | 93.3% → 96.7% |
ROUNDING_DRIFT_CASES |
0 → 10 | 94.0% → 98.1% |
N_STRAY_CREDITS |
0 → 10 | 94.7% (no effect) |
Across the entire swept space the match rate runs 76.3% – 98.1%. So the honest headline is not a single number — it is: this metric is dominated by how often duplicates occur, a quantity we invented.
What did NOT move, in any of the 248 runs:
- false positives: 0 — it never once flagged a clean settlement
- duplicate recall: 100%
- stray-credit recall: 100%
Those are the claims that survive the sweep. The match rate is the fragile one.
A settlement carrying a duplicate charge still reconciles exactly — the double charge sits on both Razorpay's side and the bank's, so the settlement-level check correctly calls it clean and the scorer counts that as a miss. The separate duplicate detector catches every one (10/10). Two checks answering two different questions, reported separately rather than blended into one flattering number.
The written explanations were vague — that was an eyeball judgement, so
score_explanations.py turns it into a number: does the note actually cite
the figures it was handed?
| Fully grounded | Figure coverage | Length | |
|---|---|---|---|
| Original prompt (months 1, 3) | 10% / 0% | 25% / 11% | ~500 chars |
| Revised prompt (months 13, 14) | 78% / 60% | 83% / 70% | ~220 chars |
Evaluated on months never used for anything else, so this is not tuning against a known answer.
Two bugs found and fixed while measuring this, both of which had been inflating the result:
- The model echoed the prompt's FACTS block before answering, so a naive grounding check scored 100% while the actual sentences cited nothing. Only the model's own note is scored now.
- Explanations came back corrupted (
'NEFT-CLI / 'NEFT-CLIENT-DIRECT-1700) because the Ollama CLI streams through a terminal renderer even when piped. Now uses the HTTP API, which returns plain JSON.
Still weakest on the missing case (50% / 0% grounded). Not fixed.
test_real_formats.py checks the column mapper against the header names
real exports use — Razorpay payment/settlement exports, and HDFC / ICICI /
SBI / Axis statement layouts — plus amount formats (Rs. 1,200.00, ₹1200,
12,34,567.89, (500), 450 DR) and six date formats.
It found four real failures, now fixed: Deposit Amt., Deposit Amount (INR ), Tran Date and Fee (Rs) were all unrecognised. Matching is
deliberately guarded so a field can never bind to an excluded header — a
loose match on "amount" would otherwise read Withdrawal Amount as
income.
This is not the same as testing on real data. No live account was used and no real statement was downloaded. It checks that the shapes a real export presents are handled, so the first thing that happens on real data isn't a crash.
A separate realism bug was found in the generator itself: when a refund exceeded the settlement it landed in, the bank credit went negative — 6 impossible rows across the batches, including held-out ones. Real processors deduct what they can and carry the remainder forward, which the generator now does. All results above are measured after that fix; the previously reported 91.3% was partly measured on data that could not occur.
No paid API key, and nothing to install -- the Python side uses only the standard library.
ollama pull llama3.2:3b # one-time, for explanations
python -m lekha generate --seed 6
python -m lekha reconcile data/batch_seed6 --as-of 2026-08-31
python -m lekha score data/batch_seed6/report.json answer_keys/seed6_key.json
python -m pytest tests/ # 127 tests
python sweep.py # sensitivity analysis
python test_real_formats.py # real export column formatsAdd --no-ai to reconcile to skip explanations (much faster, and
required for sweeps).
python -m lekha reconcile myfolder --as-of 2026-08-31Only orders.csv and bank_statement.csv are required. Without a
processor export, fees are estimated and the report says so. If the bank
narration carries no settlement reference -- most don't -- it falls back
to matching on amounts and states which mode it used. Rows it cannot read
are listed with line numbers, never skipped silently.
The generator was not deterministic. Duplicate payments were emitted
while iterating a Python set of strings; string hashing is randomised
per process, so the iteration order -- and, because an id is drawn inside
that loop, the generated content -- differed between runs. The same seed
produced different files, which silently broke the premise that seeds are
reproducible.
Fixed by sorting before iterating. CI now regenerates seed 1 twice on every push and diffs the output, so the claim is checked rather than trusted. All published figures were re-measured afterwards and are unchanged.
- Never run against a live Razorpay account or a real bank export. Only the formats are checked, not real data. This is the biggest open gap and cannot be closed without an actual merchant account.
- The
missing-case explanations remain poorly grounded. - Fee rate is treated as a flat 2%; real Razorpay pricing varies by payment method (UPI and RuPay are often cheaper), so a real merchant's fees would not match a single rate.
- No handling for partial settlements, chargebacks, or international payments.