An autopilot agent that can't lie about what its tools did.
Global AI Hackathon Series with Qwen Cloud — Track 4: Autopilot Agent
A qwen3.7-max agent resolves real support tickets end-to-end (refunds, warranty
replacements). Watch what happens without a guard: it confidently reports "the
Standing Desk has 8 units in stock, so I shipped a free replacement and emailed
the tracking number" — when it never checked inventory, never shipped anything,
and the order is 40 days past the warranty window. The tool was never called; the
agent fabricated the result. This is the #1 blocker to trusting autonomous
agents in production, and every recall/RAG approach misses it: they check
whether the agent retrieved the right fact, not what it claims its tools did.
ReceiptGuard makes fabrication structurally impossible. Every tool call routes through a gateway that returns an unforgeable HMAC-signed receipt; every claim in the agent's answer is typed by epistemic source and cross-checked against those receipts. A claim no real tool produced is caught and the agent is forced to re-ground before the action commits — and when policy forbids auto-acting (e.g. outside the refund window) the agent escalates to a human instead. Every decision lands in a tamper-evident, hash-chained audit log.
A guardrail scores how confident the model sounds and filters on a threshold. ReceiptGuard demands cryptographic proof that each tool actually ran: claims are typed by epistemic source and checked against an HMAC-signed receipt the agent cannot forge — and unbacked claims aren't suppressed, they trigger a budgeted agent loop that re-runs the real tools and re-grounds the answer before it ships. Proof-of-execution, not vibe-checking.
Proven: 100% fabrication-detection at 0% false-positives, sub-millisecond — while a real qwen-flash judge with no receipts catches 0% of the same fabrications (and is ~1000× slower). An ablation shows the receipt value-check is load-bearing; claim-typing fails safe (100% tool_derived recall). Runs live on Qwen3.7-Max + Qwen-Flash via Alibaba Model Studio. (Full table below.)
vs the field: ZeroClaw / Fetch.ai AEVS prove a tool ran but never check the agent's prose against the receipt (ZeroClaw's own docs: receipts "don't constrain text output"). ReceiptGuard does the part they skip — types every claim, cross-checks each against its receipt, recovers or escalates, then resolves the ticket.
ticket ─▶ Autopilot agent (qwen3.7-max) ──tools──▶ ToolGateway ──▶ HMAC-signed receipts
│ │
▼ draft answer │
Claim extractor (qwen-flash) ── atomic claims, typed ──┐ │
▼ ▼
Cross-check claims vs receipts
│
┌───────────────────────┤
▼ all backed ▼ unbacked / contradicted
PROCEED Tiered recovery (GSAR): regenerate / replan
(qwen3.7-max thinking adjudicates,
reasoning_content = audit reason)
└──────────┬────────────┘
▼
SHA-256 hash-chained append-only audit ledger
- Receipts — every tool call routes through
ToolGateway, which emitsHMAC(secret, {tool, args_hash, output_hash, snippet, ts}). The agent never sees the secret, so it cannot forge a receipt for a tool it never called. - Typed claims —
qwen-flashsplits the answer into atomic claims, each taggedtool_derived | inference | absence | opinion(NabaOS pramana taxonomy, arXiv 2603.10060). - Cross-check — each tool-derived claim is matched to the receipt ledger: missing receipt → unbacked; value/count mismatch → contradicted; "nothing found" with a non-empty receipt → false absence.
- Tiered recovery — groundedness score →
proceed / regenerate / replanunder an explicit compute budget (GSAR, arXiv 2604.23366).qwen3.7-maxthinking-mode adjudicates contested claims; itsreasoning_contentis stored as the human-readable justification. - Audit — every decision is appended to a SHA-256 hash-chained ledger; tampering with any row breaks the chain (EU AI Act Art. 12 record-keeping).
The HMAC tool-receipt is not our invention and we don't claim it — it's a 2026 pattern shipped by ZeroClaw and Fetch.ai AEVS, and formalized in NabaOS and GSAR. Each stops at a different layer and stops there:
- ZeroClaw / AEVS prove what a tool executed — a signed receipt you can look up — but never check the agent's prose against it. ZeroClaw's docs: receipts "don't constrain text output"; AEVS verifies "what an agent executed, not why" and is "purely a post-execution recording system."
- MemTrust (arXiv 2601.07004) secures the memory, not the truthfulness of the answer.
- NabaOS / GSAR do claim-typing and graduated recovery — as a framework and a benchmark, not a running autopilot that takes an action.
ReceiptGuard's contribution is the closed loop, deployed: unforgeable receipt (floor) → qwen-flash epistemic claim-typing cross-checked against those receipts → GSAR budgeted regenerate/replan on unbacked claims → a policy gate that auto-resolves the refund/warranty or escalates to a human → SHA-256 hash-chained ledger → exposed as an MCP server → running on Qwen + Alibaba Function Compute, with an on-the-record ablation showing the verifier fails safe. We win on the composition and the deployment, not the primitive — and we say so out loud.
"Isn't this just a guardrail?" A guardrail scores how confident the model sounds and filters on a threshold — vibe-checking. ReceiptGuard demands cryptographic proof each tool actually ran: claims are checked against a receipt the agent cannot forge, and unbacked claims aren't suppressed — they trigger a real tool re-run before the action commits. Proof-of-execution, not prompt-based filtering.
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# runs fully OFFLINE in deterministic MOCK mode (no API key needed)
PYTHONPATH=src python -m receiptguard.cli run # baseline vs guarded (refund)
PYTHONPATH=src python -m receiptguard.cli run warranty_replacement # 2nd scenario: escalates to human
PYTHONPATH=src python -m receiptguard.cli run refund_damaged --adversarial # inject a lie -> watch it get caught
PYTHONPATH=src python -m receiptguard.cli serve # demo UI at http://localhost:8000
PYTHONPATH=src python -m receiptguard.cli bench 200 # detection benchmark + ablation + chart
PYTHONPATH=src python tests/test_receiptguard.py # test suite (20 tests)Copy .env.example → .env, set DASHSCOPE_API_KEY (Alibaba Model Studio).
With a key, the agent (qwen3.7-max, thinking mode), claim typer (qwen-flash),
and the LLM-judge baseline call real Qwen via the OpenAI-compatible DashScope
endpoint. No key → mock mode, everything still runs.
- Alibaba-usage proof file:
src/receiptguard/llm/qwen.py - MCP server (Function Compute, SSE):
python -m receiptguard.mcp.server(needspip install mcp)
| Criterion | How |
|---|---|
| Innovation 30% (tiebreaker) | the composition, not the primitive: receipts + epistemic claim-typing + GSAR tiered recovery + auto-act/escalate policy gate + audit ledger, served over MCP; reasoning_content as audit artifact; two-speed qwen-flash/qwen3.7-max fleet under a thinking_budget |
| Technical Depth 30% | deterministic verifier (no LLM in the hot path), modular adapters, hash-chained ledger, reproducible benchmark + ablations |
| Problem Value 25% | fabricated tool results are the #1 blocker to autonomous agents — one hallucinated "refund issued / replacement shipped" is a direct cash loss + a compliance event, multiplied across support volume; ReceiptGuard is a drop-in MCP gateway any Qwen agent installs |
| Presentation 15% | live split-screen demo: watch the agent get caught lying and self-correct, on the record |
receiptguard bench builds an adversarial set (fabricated reference, value
mismatch, false absence) and compares ReceiptGuard against an ablation and a
receipt-free judge. The judge row below is a real qwen-flash call on
Alibaba Model Studio (run with DASHSCOPE_API_KEY):
| System | Detection | False-positive | Overhead |
|---|---|---|---|
| ReceiptGuard | 100% | 0% | 0.007 ms/claim |
| ReceiptGuard (no value-check) — ablation | 66.7% | 0% | 0.004 ms/claim |
qwen-flash judge (no receipts) |
0.0% | 0% | ~1040 ms/claim |
Reading it: the ablation (cross-check tool presence but skip numeric matching)
misses every value/count mismatch — proving the receipt value-check is
load-bearing, not decoration. The real qwen-flash judge catches 0% of the
fabrications — because they are plausible, and a model with no receipts
structurally cannot tell a real tool result from an invented one — while costing
~1000× more latency per claim. ReceiptGuard hits 100%/0% deterministically at
sub-millisecond cost, because it checks cryptographic receipts, not plausibility.
That contrast (real LLM judge at 0%) is the whole thesis. Methodology +
the offline heuristic fallback: eval/BENCHMARK_METHODOLOGY.md.
Honesty note: the judge row is a real
qwen-flashrun (n=24); the set is self-authored and the point it proves is structural — a receipt-free checker cannot verify a plausible fabrication, so it lands at 0% while ReceiptGuard's deterministic receipt check lands at 100%. Re-run at larger n withpython eval/benchmark.py <N>.
Extraction-typing eval (receiptguard bench covers the verifier; this covers the
claim typer): PYTHONPATH=src python eval/extraction_eval.py scores extract_claims
on 15 labeled drafts. Offline: 80% overall typing, 100% tool_derived recall — and
every error is conservative (inference→tool_derived), so a mis-type means the
verifier checks more claims, never fewer. With the fail-closed gateway, no
fabrication slips through a typing error.
src/receiptguard/ gateway/(receipts,tools) claims/ verify/ recovery/ audit/ agent/ mcp/ api/ llm/
eval/ adversarial benchmark + chart
tests/ pytest invariants
static/ demo UI
License: MIT.