llm-spend-attributor

Turns LLM request traces into an owned bill: who spent it, on which feature and prompt version, what a cache would really save, and whether anyone is on course to blow their budget. On the committed 123,417-request corpus it finds a prompt version that tripled one team's input tokens, two days after it shipped.

What this solves

• "Which team is this bill?" has no answer, so nobody owns the number. This tool attributes every request through an explicit chain and reports the mix: on the committed corpus 90.2% of spend carries an owner's own tag, 6.1% is matched by service account, 2.5% by an endpoint rule, and 1.2% has no owner at all. The unattributed slice is reported, never spread across teams.
• A prompt version can triple a team's spend and look like normal growth for weeks. A robust z-score over daily spend flags the incident in this corpus at 10.17 against a trailing median, decomposes the day, and names fraud-narrative prompt version v2.3 as 98% of the rise. A conventional mean-and-sigma z-score puts the same day at 2.90, so a 3.5-sigma rule would have missed it.
• Cache savings estimates are usually one optimistic number. This one produces three: $7,018 from 16,263 byte-identical prompts inside the TTL with no correctness risk, $1,159 more from 3,649 near-duplicate hits found with MinHash LSH, and a measured 7.2% false-hit rate on those near hits, meaning 264 of them would have returned an answer the model did not actually give for that prompt. Combined, $8,176, or 9.7% of spend.

Executive summary

Inference spend arrives as one invoice and a pile of request logs, and the gap between them is where FinOps arguments live. The invoice says the number; the logs know which team, which feature, and which prompt version caused it, but only if someone joins them up, prices each request at the rate in force when it was made, and is honest about the requests nobody can identify.

spendlens does that join. It loads request traces into an embedded DuckDB warehouse in one SQL statement (123,417 rows in 1.57 s, about 78,687 rows per second), attributes each request through a rules chain that records which rule claimed it, and costs it with an as-of join against a versioned price book, so a June request is priced at June's rate even though the model got cheaper on 1 July. On top of that it answers the four questions a platform team is actually asked: where is the money going, is any of it unowned, is anything anomalous, and what would a cache save.

Across the 60-day window it accounts for $83,924 at organisation scale, of which 1.2% has no owner. For July it reports $40,860 month to date with an unattributed share of 1.3%, forecasts $42,384 at month end from a weekday-aware run rate, and fails its own gate because risk is projected at $4,352 against a $3,400 budget, which is 128%. The cause is not a mystery in the output: the same run names fraud-narrative version v2.3, shipped 2026-07-12, whose average input tokens went from 2,510 to 7,785, a 3.1x jump. Every number in this document is regenerated and checked by CI.

Watch it work (30 seconds)

Every line of terminal text is the real stdout of the command shown with it, captured by tools/record_demo.py, with each segment paced by that command's measured wall time. It is a replay of a captured session rather than a live screen recording, and docs/video/manifest.json lists each command with its exit code and duration. Higher-quality MP4: docs/video/spendlens-demo.mp4.

The dashboard

One self-contained HTML file, no JavaScript and no external assets, including the charts, which are inline SVG. That constraint exists because this file gets uploaded as a CI artifact and opened months later from a shared drive, and both of those paths break a page that fetches a chart library from a CDN.

Full-page screenshot, including the attribution bar, the anomaly cards with their causes, and the per-feature cache table.

How much of this is a tag, and how much is a guess

This is the first question worth asking of any cost report, and most of them cannot answer it.

source	share of spend	what it means
explicit tag	90.2% ($75,668)	the calling team set a tag, so ownership is theirs, not inferred
service account	6.1% ($5,157)	no tag, but the calling account is mapped to a team in pricing/attribution_rules.yaml
endpoint rule	2.5% ($2,084)	no tag and no known account, but the endpoint path implies a team
unattributed	1.2% ($1,015)	nothing identified an owner

Unattributed spend is never spread proportionally across teams. Spreading it makes a chargeback table look tidy while guaranteeing nobody fixes their tagging, and it quietly bills the platform team for someone else's batch job. It gets a tile of its own at the top of the dashboard, a gate threshold of its own (5%, a common FinOps target), and its own line in every summary.

Catching the incident

Three things make this output actionable rather than merely alarming:

1. Robust statistics. The score uses the trailing median and median absolute deviation, not the mean and standard deviation. A spend series contains exactly the spikes being hunted, and a spike inflates a standard deviation enough to hide itself: the worst day here scores 10.17 robust against 2.90 classical. The threshold is 3.5, so the conventional score would have said nothing.
2. Decomposition. Each flagged day is broken down by feature and prompt version against the trailing week, and shares are taken over the sum of the positive movements, so a riser can never be reported as more than 100% of the rise.
3. A named suspect with a date. fraud-narrative v2.3, first seen 2026-07-12, average input tokens 2,510 on the previous version against 7,785 on the new one, a 3.1x jump. That is a pull request someone can revert.

One detail in that output repays a second look. The worst day is 2026-07-14 and its decomposition reports input tokens of 7,867 before against 7,915 on the day, which looks like nothing until you notice both figures are v2.3: the comparison is against the trailing week for the same feature and version, so two days after the incident the trailing window is already contaminated by it. That is the correct comparison for "what changed today", and it is why the version-level view (2,510 to 7,785) is reported beside it rather than instead of it. A tool that only offered one of those two framings would mislead on one of the two questions.

Architecture

the diagram source, and why this is an image

GitHub renders mermaid fences itself, and when it works the source is the picture. It does not always work: this diagram parses and renders with mermaid 10 and 11 locally, and GitHub showed Unable to render rich display: Cannot read properties of undefined (reading 'render'), which is a failure inside their renderer rather than a syntax error here. So the picture is generated once by tools/render_diagrams.py, committed, and embedded, which renders identically on GitHub, in an editor preview, in a PDF and offline. The source below is in a plain fence so nothing tries to render it, and regenerating the image after editing it is one command.

flowchart LR
  subgraph inputs[Committed inputs]
    T[traces.jsonl.gz<br/>123,417 requests, 60 days]
    P[price_book.yaml<br/>rates with effective dates]
    R[attribution_rules.yaml<br/>the ownership chain]
    B[budgets.yaml<br/>monthly limits per team]
  end

  subgraph warehouse[DuckDB, one embedded file]
    I[read_json<br/>one statement, schema pinned]
    J[reject missing fields<br/>keep them in their own table]
    D[drop repeated request ids<br/>keep retries, they really billed]
    V[costed view<br/>ASOF JOIN on effective_from<br/>+ attribution_source column]
  end

  subgraph answers[What gets asked of it]
    A1[attribution mix<br/>tag vs guess vs nothing]
    A2[robust-z anomalies<br/>+ cause by prompt version]
    A3[cache simulation<br/>MinHash LSH + false-hit rate]
    A4[weekday-aware forecast<br/>vs budget]
  end

  T --> I --> J --> D --> V
  P --> V
  R --> V
  V --> A1 & A2 & A3 & A4
  B --> A4
  A1 & A2 & A3 & A4 --> O[self-contained HTML dashboard<br/>+ markdown summary<br/>+ exit code]

Four boundaries refuse rather than guess: a trace line missing a model or a token count is rejected into its own table instead of being costed as zero, a model with no price in the book stops the run rather than being costed as free, an attribution chain that does not end in the unattributed link is rejected at load, and a budget file with a non-positive number is rejected before any arithmetic.

Tech stack

Technology	Role here	Why chosen
DuckDB (embedded)	The warehouse: ingest, costing, aggregation	The interesting questions are joins and window functions, which belong in SQL. An embedded file means no service container in CI and nothing to run in a scheduled job. Its ASOF JOIN expresses "the rate in force at this moment" directly
SQL ASOF JOIN	Pricing each request at its own rate	A plain equi-join on model multiplies every request by the number of rate changes; joining on the newest rate prices June traffic at July's price. There is 1 rate change inside this window, and it is applied correctly by construction
MinHash LSH, in numpy	Finding near-duplicate prompts	All-pairs comparison inside a feature is quadratic, about 7.6 billion pairs here. Banded MinHash surfaces 215,602 candidate pairs in one pass, and the exact Jaccard is computed only for those
Python statistics	Robust z-scores and weekday medians	The rolling MAD is the median of deviations from the same window's median, which cannot be written as a plain window function without nesting aggregates. SQL turns 123k rows into a few hundred team-days; the arithmetic then happens where it can be unit-tested
PyYAML	Price book, attribution rules, budgets	Every policy that a finance or platform person might need to change is a reviewable file, not a code change
pytest, pytest-cov	74 tests, 90% line coverage	Price arithmetic is asserted against numbers computed outside the implementation, and the SQL and Python costing paths are compared row by row
Inline SVG, hand-generated	Dashboard charts	The output has to survive being an offline artifact, so it cannot fetch a charting library
Playwright, ffmpeg	Screenshots and the demo video	Every image here is rendered from the tool's real output, so the documentation cannot drift from the behaviour

Quickstart

Prerequisites: Python 3.10 or newer and git. No database server, no cloud account, no API key.

git clone https://github.com/sivananda1995/llm-spend-attributor.git
cd llm-spend-attributor
python -m venv .venv && source .venv/bin/activate    # Windows: .venv\Scripts\activate
pip install -e ".[dev]"

spendlens ingest         # 123,417 traces into the warehouse, about 1.57 s
spendlens attribution    # how much of spend is a tag and how much is a guess
spendlens anomalies      # the spike, and the prompt version behind it
spendlens cache          # what a cache saves, and what it would get wrong
spendlens gate           # forecast month end, exit 1 on a breach
spendlens report --html reports/spend.html    # the dashboard

make verify              # lint, 74 tests, and every readme number re-measured

make help lists every target. The corpus regenerates deterministically with make data (--seed 20260814 reproduces the committed file), and so do the benchmarks, the screenshots, and the video.

To point it at your own traces, keep the JSONL shape in data/generate_traces.py (the required fields are request_id, timestamp, model, input_tokens, output_tokens), then edit the three policy files: pricing/price_book.yaml, pricing/attribution_rules.yaml, and pricing/budgets.yaml.

A note on scale, stated up front

The committed traces are a 1-in-100 sample, and every dollar figure in this repository is multiplied by a sampling factor of 100 that lives in spendlens.yaml and is printed on every report. A full month of real request telemetry is hundreds of megabytes and does not belong in a git repository; a 9.4 MB gzipped sample does.

Two properties of that factor are asserted by tests: it scales every absolute figure linearly, and it changes no ratio, so every share, percentage, and ranking in this README is what the raw sample says. If you prefer to read the sample's own numbers, set the factor to 1 and every dollar figure divides by 100.

Performance

Method: benchmark/bench_ingest.py. Container with 2 vCPU and 7 GB RAM, Python 3.11.15, DuckDB 1.5.5. Query latency is measured over 15 repeats of the aggregates the dashboard actually runs, against the full 123,417-row corpus.

what	measured
ingest, 123,417 rows from gzipped JSONL	1.57 s, about 78,687 rows per second
the same work row by row in Python, 10,000 rows	62.572 s, a 264.4x difference
heaviest dashboard aggregate, p50	53.61 ms
cache simulation over 123,417 prompts	about 8 s, dominated by MinHash signature construction
cache result on that corpus	$8,176 saved, 9.7% of spend, from 16,263 exact and 3,649 near hits

The row-by-row version is not a straw man; it is what this repository did first, and it had not finished the full corpus after two minutes when it was killed. The benchmark keeps both strategies so the ratio is a measurement rather than a memory. Full results: benchmark/results/ingest_and_queries.md.

Where it stops being enough: at these query latencies an embedded warehouse is comfortable to roughly 10 million requests per month on one machine. Past that, the honest move is the same SQL against a real warehouse, because nothing here depends on DuckDB except the connection.

Tests, coverage, and receipts

74 tests, 90% line coverage, measured with pytest --cov=spendlens. CI enforces an 88% floor by parsing coverage.xml.

The tests worth reading first:

• test_prices.py asserts the cost arithmetic against numbers computed on paper, including the boundary case that a rate effective from a date applies on that date, and that provider-cached tokens are not billed twice.
• test_warehouse.py costs every row twice, once in SQL and once in Python, and asserts they agree. Two costing implementations that disagree are worse than one.
• test_cache.py runs the tokeniser in three subprocesses with different PYTHONHASHSEED values and asserts identical output, which is the regression test for the bug described below.
• test_anomaly_budget.py checks the robust z-score against hand arithmetic and asserts that a contaminated history produces a classical score too low to fire, which is the argument for the design.

Every number here is checked by CI

make receipts    # python tools/collect_metrics.py && python tools/check_readme_numbers.py

tools/collect_metrics.py re-runs the pipeline, the gate, the anomaly scan, the cache simulation, the benchmarks, and the test suite, then writes all 52 values to docs/metrics.json with the exact command that produced each one. tools/check_readme_numbers.py asserts this README still contains each current value and fails with a list of the stale ones. Both run in CI, so a number that moves breaks the build like a failing test.

Architecture Decision Records

• ADR-001: prices are data with effective dates, joined as-of
• ADR-002: unattributed spend is a bucket, never an allocation
• ADR-003: robust z-scores over a classical z-score, and where that fails
• ADR-004: DuckDB, an embedded file, over a warehouse or a dataframe

Intentionally out of scope

• Live provider APIs. Nothing here calls a model provider. The tool reads traces, which is the right boundary anyway: the billing question is answered from what was already logged, and a cost tool that needs production credentials is a cost tool nobody will install.
• Chargeback enforcement. This produces showback: who spent what, with the confidence attached. Turning that into an internal invoice is a finance workflow with approval steps, and the unattributed slice has to reach zero before it would be fair.
• Semantic caching for real. The cache module simulates and measures; it does not implement a cache. The false-hit rate it produces is the input to that decision, and the 7.2% measured here is why the decision needs one.
• Right-sizing recommendations. "Move this feature to the small model" needs a quality measurement to be safe, which is a different tool. ticket-router spends on a small model already; the honest version of this recommendation needs an evaluation harness beside it.
• Multi-currency. The price book carries a currency field and everything downstream assumes one. Adding a second means a rate table with its own effective dates, which is exactly the ADR-001 problem again.

Security and compliance

• Prompt text never reaches a log line. Log records carry counts, costs, ids, and timings only. The cache module reads prompt text and reduces it to token sets in memory; nothing it touches is logged. This matters because trace corpora contain whatever users typed, which in support traffic routinely includes names, account numbers, and internal hostnames.
• Prompt text is separable. The warehouse keeps prompt_text because the cache simulation needs it, and prompt_sha256 is stored alongside. A deployment that cannot retain prompt text can drop the column, keep the hash, and lose only the near-duplicate half of the cache analysis.
• No credentials of any kind. The tool reads files and writes files. There is nothing to leak from a CI log, and the workflow declares permissions: contents: read.
• Policy is reviewable. Prices, attribution rules, and budgets are committed YAML, so changing who owns which spend is a pull request with a diff, not an untracked click.
• The committed corpus is synthetic, generated from a seeded model of an organisation. No customer text and no licence question.

Failure modes

Failure	Detection	Behaviour	Recovery
Trace line missing a model or token count	Checked in SQL during ingest	Row lands in the rejected table with a reason, and the count is reported. Never costed as zero	Fix the upstream logger; the rejected table answers "why is our total below the invoice"
A model in the traces has no price	Left join against the price book after load	The run stops with the model named	Add the rate with its effective_from. Costing an unpriced model as free understates the bill silently
Log shipper delivers a line twice	Repeated request_id dropped by a window function at ingest	Counted and reported as duplicates_dropped	None needed. Retries are deliberately kept, because a retry really did bill
A rate change is missed	Not detectable from inside, so the report footnotes every change it applied	Costs use the rate in force per request	Compare the footnote against the provider's price history
Attribution rules leave requests unclaimed	Every request gets an attribution_source	Falls into unattributed, which has its own gate threshold at 5%	Add a tag at the caller, which is the only real fix, or a rule if the caller cannot be changed
Cache similarity threshold set too low	The false-hit rate is measured and reported next to the savings	Savings rise and so does the false-hit rate, visibly	Raise the threshold. spendlens cache --threshold 0.95 shows the trade in one command
Not enough history to forecast	Forecast method is returned with the number	Falls back to a linear projection and labels itself linear_fallback	Wait for four weeks of history, or read the label and discount accordingly
Budget missing for a team	Compared against the budget file	Reported as NO_BUDGET, never blocks a build	Add a budget. Blocking on a missing one teaches people to delete theirs
Anomaly history is perfectly flat	MAD of zero detected explicitly	Flagged when the day rises 20% or more, and marked flat_history	None needed. This case was silently skipped in an early version, which is the worst possible behaviour for a stable service that suddenly triples

Hardest problem solved

The savings figure moved between two runs of identical code on identical data.

I noticed it while writing the dashboard: the cache simulation had reported $7,996 earlier and $8,225 the second time, on the same committed corpus, from the same commit. For a repository whose entire claim is that its numbers are measured, a number that moves on its own is the worst possible defect, and it is worse than being wrong in a fixed direction, because it means nothing downstream can be trusted either.

The first thing to establish was whether it was really nondeterminism or a change I had forgotten making. Running the same command three times under different PYTHONHASHSEED values settled it: 3,367 near hits and $8,074.62, then 3,047 and $7,978.32, then 3,873 and $8,268.57. A 3.6% swing in a savings estimate, controlled by an environment variable that has nothing to do with cost.

The trail from there was short, because the seed pointed straight at hashing. MinHash needs each token mapped to an integer, and my _signatures function built that mapping by walking the documents and assigning ids in first-seen order. The documents were frozenset objects, iteration order of a set depends on the hash values of its members, and Python salts string hashes per process. Different ids meant different minima, which meant different LSH bands, which meant different candidate pairs, different clusters, and a different number of cache hits. Nothing in the cost arithmetic was wrong; the input to it was quietly reshuffling.

The fix was to remove the ordering from the equation rather than to sort around it: token ids now come from a blake2b content hash, so a token has the same id in every process forever. Three runs under different hash seeds now produce byte-identical output, and tests/test_cache.py asserts exactly that by launching subprocesses with different seeds and comparing. The lesson I would repeat: a per-process hash seed is a fine default for dictionaries and a landmine anywhere its value escapes into a computed result, and the tell is a number that changes when nothing did.

Future work

• Per-feature budgets, not only per-team. The team table is the conversation finance wants; the feature table is the one an engineering manager acts on, and the gate currently only understands teams.
• A tagging-coverage trend. The unattributed share is measured per run but not tracked over time, and the useful version of that number is its direction. It is the one metric I would put on a wall.
• Cache threshold sweep as a first-class command. --threshold exists, so the savings-against-false-hits curve is four runs and a chart away. That curve is the actual decision artifact.
• Before real production use: replace the synthetic corpus with a week of real traces, confirm the price book against the provider's invoice for a closed month (the invoice is the only ground truth that matters), and set budgets from three months of history rather than from a round number.
• First metric to watch after adoption: the unattributed share. If it is not falling, the tool is producing a report nobody is acting on, and that is measurable rather than a matter of opinion.