feat(python): LiteLLM callback posts usage to /tokens - #237
Merged
Conversation
Land the unpublished sanction-sdk package with a duck-typed CustomLogger so Python agents can meter LiteLLM completions without importing litellm. This is post-call reporting; the fail-closed budget wall stays the LLM gateway. Co-authored-by: Cursor <cursoragent@cursor.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| This is meter/report after the provider call. Fail-closed spend is still | ||
| ``/api/gateway/<provider>`` with ``x-sanction-key``. Callback exceptions never | ||
| propagate into the completion. | ||
| """ |
There was a problem hiding this comment.
Duck-typed logger may never fire
High Severity
SanctionLiteLLMLogger is documented as a drop-in for litellm.callbacks, but it only duck-types CustomLogger and never subclasses it. LiteLLM’s public callback contract is CustomLogger instances or callables; a plain object is not registered into success hooks, so the documented one-liner can silently never post to /tokens.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit c296cc6. Configure here.
| tokens_in, tokens_out = _usage_from(kwargs) | ||
|
|
||
| cost_usd = _as_float(kwargs.get("response_cost")) | ||
| return model, tokens_in, tokens_out, cost_usd |
There was a problem hiding this comment.
Streaming usage is not collected
Medium Severity
completion_usage reads tokens only from response_obj.usage or a top-level kwargs["usage"], then _payload drops events when tokens and cost are all zero. LiteLLM streaming success events put the assembled usage on complete_streaming_response (and cost sometimes only on standard_logging_object), so streamed completions are skipped or posted with zero tokens.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit c296cc6. Configure here.
Hero becomes copy + wallet object (card, verified mandate, orbit); new dark the MCP terminal panel. Styling scoped in brand.css, responsive at 900/640, reduced-motion guarded. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Incoming webhooks still deep-link. Channel archive URLs plus a Slack app post Approve/Deny via chat.postMessage; the click is HMAC-verified and settles the same pending approval as the dashboard. Co-authored-by: Cursor <cursoragent@cursor.com>
Interactive Approve/Deny no longer needs a pasted channel URL and a platform bot token. The remaining pickup is standing up the Slack app and setting the OAuth env vars. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the monthly-zero report with group and agent cards, a three-item rail, and add-an-agent on the group. Old destinations stay under Vault. Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(mcp): hosted Streamable HTTP wallet endpoint at /mcp Give agents a URL to paste. Same ten cooperative tools as stdio, API-key auth, stateless JSON responses so it runs on Vercel. Broker intercept of tools/call stays Next. Co-authored-by: Cursor <cursoragent@cursor.com> * test(mcp): cover shared wallet tool handlers The hosted URL extracts tools into lib/mcpServer; without calling them the coverage ratchet falls below 90%. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
Humans who open the published wallet URL have no agent key. MCP clients still fail closed with JSON. Co-authored-by: Cursor <cursoragent@cursor.com>
… review (#244) * fix(site): reconcile the agent-wallet story with the services-led home page #239 put the platform pitch in slot #2 of a home page that #232 had just repositioned services-led, leaving two pitches on one page instead of one argument. - Move the #agent-wallet deep dive below "What we've built" so it joins the proof chapter as one continuous dark run, with a hairline marking the seam between the two dark tones. The exec buyer now reads why they called before meeting `npx`. - Stop repeating the wallet headline verbatim in the Sanction Platform card; it names the job instead ("Answer for what your agents spend and do"). - Write the bridge the page was missing: the engagement installs the agents, the platform governs what they spend and do. - Give the wallet section a services-side CTA so book-a-call and install-a-package stop being two funnels that never meet. - Date the MIA ribbon comment and back it with a backlog entry, so a launch-window element does not become permanent by default. The nav needed no new entry — "What we've built" now sits directly above the section it previously had no way to reach. The backlog also captures the reposition doc drift (README and AGENTS.md still describe a product-only identity) as its own docs-only pass, kept deliberately separate from this layout change. Verified: tsc clean, eslint 0 errors, 1083 tests passing, and the page rendered at 1440px and 390px with 0px horizontal overflow. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Eb6zTfLyE42dYS8tgWfvn * docs(readme): structure the README around who it's for and how it's used Ingests an external positioning analysis (via /INPUT) — the structural findings applied; feature ideas routed to the backlog; suggestions that contradicted confirmed decisions or live code pushed back. - "Who runs Sanction" up front, leading with the confirmed primary case: organizations governing their own internal AI usage and spend. - A concrete decision example under "What it does" using the real response contract — status, stable decision_code, verbatim remediation string — instead of leaving "checked and debited atomically" abstract. - API section regrouped by workflow (ask permission / escalate / carry credentials / prove / shape policy / run the fleet / standards); same endpoints and descriptions, no longer one wall. - Distribution gains a shortest-path matrix; policy packs surfaced by name; "changing policy in production" and "when to use the credential vault" sections added. - Corrected a stale claim while in the file: @sanction/sdk is not on the npm registry (verified against the registry) — install instructions now point at sdk/ with publish marked pending; the wider docs sweep is queued. Pricing left exactly as it stands — free-or-agreement is a confirmed decision, not an omission. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Eb6zTfLyE42dYS8tgWfvn * feat(console): month runway charts + seat health — the collapsed sprint arc One arc collapsing the three ingest sprints (external analysis, verified piece by piece against the live code): Console — surface what's already measured: - Month runway card on /dashboard/spend: cumulative token cost and authorized spend drawn day by day against the monthly caps, with the linear projection and "budget hit ~<date>" made visible instead of numeric. Pure SVG; series math (bucketByDay, cumulative) factored into lib/burn.ts beside the existing pace helpers, unit-tested. - Seat health card: flags seats whose denial rate is hot (≥25% over a real sample) or climbing (last 7 days ≥1.5× the month baseline), with each seat's most-hit denial code via the same decisionCode mapping agents see. Drift logic is pure (lib/seatHealth.ts), unit-tested; a healthy fleet renders as good news, not an empty table. - Fix (adjacent, render-check caught it): the 14-day token trend bars rendered 0px — items-end kept the bar columns at content height, so the bars' percentage heights resolved against auto. Columns stretch again. Engine — design before code: - docs/plans/policy-inheritance.md: budgets cascade today, rules don't; proposes evaluation-time ancestor-first overlay if inheritance is wanted. - docs/plans/context-conditional-rules.md: closed-vocabulary `when` predicates over a richer context snapshot; rejects free-form expressions. Both stamped proposed — the product call stays with the owner. Docs: - FRAMEWORK-ADAPTERS.md: stop instructing `npm install @sanction/sdk` — the package is not on the npm registry; points at sdk/ until publish. - AGENTS.md: dated note recording the 2026-08-10 site reposition so future sessions don't "fix" the repo/site identity mismatch unprompted. Verified: tsc clean, eslint 0 errors, 1092 tests passing (18 new), and the spend page rendered against a seeded Meridian demo org — runway projections, cap lines, exhaust dates, seat flag, and the revived trend bars all confirmed by screenshot. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Eb6zTfLyE42dYS8tgWfvn * chore(backlog): check off the two console items the sprint arc shipped Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Eb6zTfLyE42dYS8tgWfvn * chore(release): v0.8.0 — a wallet you can paste, hand over, and verify Cuts the 52-commit pack since v0.7.0 (2026-07-12) and realigns the three package versions that had drifted apart. - package.json 0.7.0 → 0.8.0; packages/sanction-mcp 0.7.0 → 0.8.0; sdk 0.6.0 → 0.8.0. All three move together again. - Changelog gains the v0.8.0 release header summarizing the pack: the hosted Streamable HTTP wallet URL and MCP 2026-07-28 conformance, the public Wallet Card + mandate verify, team membership and roles, Slack OAuth install with interactive Approve/Deny, the roster console with month runway and seat health, providers connected once, and the gateway metering fix that stops unpriced models billing $0. - Roadmap: adds the shipped roster-console/team-roles item to Now, and restates the SDK item against verified reality — it is 0.8.0, builds, typechecks, and passes 55 tests, with the npm scope named as the single remaining precondition rather than a vague "once the org is wired". Claim verification (the "ten tools, not nine" rule) turned up a live error this pass: the README said nine policy packs; lib/policyPacks.ts registers eleven — payment-agent-mandate and no-egress were below the fold when that line was written. Corrected. MCP tool count re-counted from lib/mcpServer.ts: ten, matching the claim. Boundaries kept honest in the notes: MCP stays cooperative (broker interception of tools/call is Next, not claimed today), and the SDK is described as publish-pending rather than installable from npm. Verified: tsc clean, eslint 0 errors, 1092 tests passing, changelog and roadmap both parse and render their new entries. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Eb6zTfLyE42dYS8tgWfvn * fix(enforce): roll back cascade denials; stop callers opting out of metering Two defects on the money path, found by an adversarial pre-release review and verified against the code before fixing. Both were fail-open or state-corrupting; the release should not carry either. 1. Phantom spend on denied cascade decisions (app/api/v1/authorize/route.ts). reserveCascadeDailySpend walks capped ancestors root->leaf, incrementing each counter, and throws CascadeBudgetExceeded on the first cap breach. lib/cascadeBudget.ts documents the contract: the throw must roll the surrounding transaction back. The spend route caught it INSIDE db.$transaction and returned normally, which commits — so every ancestor already incremented kept the spend of a request that was denied. The reconcile is GREATEST(spentCents, rolled), so it only ever heals upward: the phantom total was permanent and compounding, and a parent pool would start denying legitimate spend on money nobody spent. Requires >=2 capped ancestors, which no test built. Now unguarded inside the transaction and handled outside it, matching the grant path and the AuthZEN PDP, which were already correct. 2. Token metering was opt-in for the party being governed (app/api/gateway/[provider]/[...path]/route.ts, lib/gateway.ts). OpenAI-compatible providers emit stream usage only when the caller sets stream_options.include_usage — and the gateway forwarded the request body verbatim while docs/GATEWAY.md instructed the caller to set it. An agent streaming without it produced no usage block, so no tokenLog row was written, no budget moved, and the pre-call wall (which reads those rows) never saw the spend. forceStreamUsage() now sets the flag on the way out; a stream that still reports nothing is logged rather than passing in silence. Anthropic and Gemini stream usage by default and are untouched. Unparseable bodies pass through unchanged — a proxy must not corrupt a request it does not understand. Also fixed here, same review: - Subtree-cap denials persisted no policyRevision/decisionContextJson, so GET /authorize/{id}/evidence returned nulls for a denial class that is itself appealable. The denial now carries the evidence captured before the reservation attempt. - MCP_SERVER_VERSION was left at 0.7.0 by the version bump, so every MCP host would report 0.7.0 for a 0.8.0 release. - Three different coverage numbers across README (88%), CONTRIBUTING (80/80/85) and TRACEABILITY (80/80/85), none matching vitest.config.ts (90/90/94/83). All three now state the enforced gate, with the config named as the single source of truth. Test count corrected from "700+" and "500+" to 1,100+/1,000+. 5 new regression tests pin the metering fix, including the exact bypass. Gate: tsc clean, eslint 0 errors, 1097 passing. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Eb6zTfLyE42dYS8tgWfvn --------- Co-authored-by: Claude <noreply@anthropic.com>
A truthsync pass over the five truth surfaces. Docs + spec only; no behavior changes. Every claim written here was verified against the code first. Actively false, now corrected: - docs/DOMAIN.md claimed Sanction sits "inline as the enforcement point" for MCP. That is the exact interception claim the Wallet Card, lib/roadmap.ts, and the changelog all explicitly refuse to make. Replaced with the cooperative-MCP contract: the gateway intercepts, MCP does not, broker mode is Next. - docs/DOMAIN.md said "No wallet switcher yet"; it shipped 2026-07-17 and TRACEABILITY already marked that gap closed — the two files contradicted each other in the same repo. - docs/DOMAIN.md's arc line listed human approval and Sanction Local as Next; both shipped. - TRACEABILITY ORG-VIS asserted "the resolve action stays wallet-scoped". Resolution now authorizes against subtreeWalletIds. A false row in the security registry is the worst kind, since the file's whole promise is that claims map to enforcing code. - README said the dashboard leads with budget runway (the roster is the home now) and described Slack as deep-link-only (interactive Approve/Deny and Add to Slack OAuth are the v0.8.0 flagship). Honest gap rather than a confident claim: the subtree-resolution change has no test that builds a real two-level org — the only covering test mocks subtreeWalletIds to a single wallet. Cited that way in ORG-VIS and added to Gaps + the backlog rather than dressed up as proven. Registry integrity (the file's own maintenance rule): - New rows TRACE-1 (W3C trace-context validation before host values become outbound headers), PROV-1 (provider keys vaulted + injected at the gateway), SIM-2 (sequential replay), each citing a test confirmed to exist. - GATEWAY row now carries both metering invariants: unpriced models meter at the fallback rate, and stream usage reporting is forced rather than left to the caller. - SIM-1's "sequential re-fold is slice 2" note was stale. - Test-suite map gained traceContext, providers, roster, seatHealth, format, funnel, simulation-sequential. Changelog backfill — four shipped capabilities had no entry of their own: provider connections, team membership + roles + wallet switcher, MCP wallet derivation, and org-owner subtree resolution. Inserted in date order. Doc bugs that break working code: - STARTER-KIT polled d.grant?.id; the API returns a flat grant_id, so the documented escalate->redeem loop silently never fired. That is the one path that makes Sanction Sanction. - lib/openapi.ts declared a REQUIRED wallet_id query param that the handler never reads (it derives the wallet from the request row), so every generated client — Bedrock included — demanded a value that does nothing. - sdk/README.md and docs/PAY-PER-CRAWL.md instructed an npm install that 404s; FRAMEWORK-ADAPTERS said "install from the repo path" without a command. All three now give a working path install. README also gained the capabilities it never described (roster + team roles, observe mode, outcomes/freeze/reallocation) and the eight endpoints missing from the API section. DOMAIN gained PolicyRevision, SlackInstall, Outcome, and BudgetReallocation — all verified present in prisma/schema.prisma. Gate: tsc clean, 1092 tests passing. Changelog parses and is date-ordered. Claude-Session: https://claude.ai/code/session_013Eb6zTfLyE42dYS8tgWfvn Co-authored-by: Claude <noreply@anthropic.com>
…hedge (#247) npm view sanction-sdk -> 0.8.0, FSL-1.1-MIT, published by the same automation path as sanction-mcp. The hedges written while the publish was blocked now come out: - README: both SDK rows become `npm install sanction-sdk`. - FRAMEWORK-ADAPTERS: the repo-path install workaround collapses to the real install command. - PAY-PER-CRAWL + sdk/README: the "Not on npm yet" blockquotes collapse to the install line. - roadmap: the SDK item's TypeScript half is Shipped; the Python adapters (LiteLLM callback, LangChain/LangGraph + CrewAI) stay Next. - changelog: a dated entry for the ship itself, and the v0.8.0 release entry drops its "until the publish workflow runs" clause. - backlog: dated ops reminder that the new granular npm token EXPIRES — when it does, both publish workflows start failing with no other warning. Expiry date to be filled in from npmjs.com; the old select-packages token gets deleted once confirmed unused. Claude-Session: https://claude.ai/code/session_013Eb6zTfLyE42dYS8tgWfvn Co-authored-by: Claude <noreply@anthropic.com>
FRAMEWORK-ADAPTERS keeps main's sanction-sdk rename and gains the Python package line (same sanction-sdk name on PyPI, publish pending). TRACEABILITY keeps main's WALLET-1/MCP-REMOTE-1/SLACK-1 and adds the PR's PY-1 row + suite map line. The changelog keeps every main entry and inserts the PR's Python entry in date order (2026-08-13). The roadmap keeps main's Next list, merges the SDK item's TS-shipped + Python-in-repo state, and fixes a doubled sentence the earlier edit left behind. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Eb6zTfLyE42dYS8tgWfvn
Owner
Author
|
Refreshed against today's main and re-verified — ready to merge. This PR predates six merges of truth-surface churn (the
Verification on the merged tree:
No force-push — the refresh is a merge commit on top of the original, so the branch history is intact. Generated by Claude Code |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.
There are 5 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 79fb4e9. Configure here.
| }, | ||
| }) | ||
| return NextResponse.json(await withAppeal(decisionResponse(rec, agent.name)), { status: httpFor(rec) }) | ||
| } |
There was a problem hiding this comment.
Subtree-cap denials skip exhaustion alerts
Medium Severity
Moving CascadeBudgetExceeded handling out of the transaction means subtree-cap denials return from catch and never reach the budget.exhausted delivery that still runs only after a successful $transaction. Operators lose the exhaustion ping for this denial class even though the denial row is still persisted.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 79fb4e9. Configure here.
| }, | ||
| }) | ||
| return NextResponse.json(await withAppeal(decisionResponse(rec, agent.name)), { status: httpFor(rec) }) | ||
| } |
There was a problem hiding this comment.
Cascade denial ignores idempotency races
Medium Severity
The CascadeBudgetExceeded handler writes a denial with the same idempotencyKey as base, but a unique-constraint failure from that create is not caught. Concurrent retries that already committed a winner (or another denial) now 500 instead of replaying the existing row.
Reviewed by Cursor Bugbot for commit 79fb4e9. Configure here.
| value: approvalId, | ||
| }, | ||
| ) | ||
| } |
There was a problem hiding this comment.
Slack buttons omit tool capability IDs
Medium Severity
Interactive Approve/Deny buttons are added only when the event payload includes approval_id. Spend and provision events send that field; tool and capability approval.created payloads send only request_id. Slack installs therefore get in-channel buttons for spend escalations but not for tool or capability ones.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 79fb4e9. Configure here.
ericlovold
pushed a commit
that referenced
this pull request
Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
packages/sanction-python(httpx-only runtime) withlog_tokensand a duck-typedSanctionLiteLLMLoggerfor LiteLLM callbacks./api/gateway/<provider>. The SDK never importslitellm.Test plan
cd packages/sanction-python && uv run ruff check src tests && uv run mypy src tests && uv run pytest --cov=sanction_sdk --cov-fail-under=90— 61 passed, 93% coveragetest_architecture.pystill fails the build if anyone addsimport litellmMade with Cursor