A transactional outbox for usage, so a charge is never lost or doubled - #27
Merged
Merged
Conversation
…ubled The gateway knows what a request cost. Something downstream has to be told, and "write to the database, then call the billing service" has two failure modes that both cost real money: the process dies between the two and the charge is lost, or the call is retried and the customer is billed twice. The outbox makes the effect and its message one transaction. migrations/004_outbox.sql adds usage_ledger and outbox in the gateway's own database; sealing a usage window writes both in a single commit, so either the window closed and the message exists, or neither happened. Delivery is at-least-once and the consumer side makes it exactly-once-in-effect: MigrateSink creates an inbox table keyed by message id in the billing database, and the relay inserts the charge and the inbox row together. A redelivery finds the inbox row already there and does nothing. That is the only way the guarantee holds without a distributed transaction between two databases neither of which can roll the other back. The suite needs two real databases, because "the effect and the inbox row commit together" is a claim about a transaction and there is nothing to test without one. It skips unless both are pointed at.
This is the gateway's half of the outbox: a goroutine that closes the current usage window and queues its charge in one transaction, so the window and the message it produces cannot disagree. USAGE_SEAL_INTERVAL defaults to 0, which keeps the sealer off. A deployment with nothing downstream to bill should not accumulate messages nobody will ever relay, and a background writer that starts itself is exactly the kind of thing that is discovered later as a table that grew forever. reqctx's "unauthenticated" tenant label becomes a named constant, because the sealer has to exclude it. Health probes, unmatched paths and rejected keys all land in that bucket; it is a metrics label and not an account, so anything that bills a tenant must not bill it. Delivering the charge is a separate process on purpose. Sealing and relaying being two processes is what lets either be killed at any point without losing or double-billing a window.
cmd/tollgate-outbox is the delivery half: `relay` drains the outbox into a sink, `sink` is the consumer that writes a charge and its inbox row in one transaction. They are separate from the gateway because the whole guarantee rests on the two sides being able to fail independently. scripts/outbox-crash.sh is the evidence. Nothing in it mocks a failure: every crash is SIGKILL to the process under test, so no deferred function runs, no buffer is flushed and no connection is closed politely. What survives is only what Postgres had already committed, which is exactly the claim being made. It kills the relay between the commit and the delivery, kills the sink mid-transaction, and then counts what the consumer database ended up with - once, not zero and not twice. It stays out of CI: it drives its databases with `docker exec` into a container it expects by name, and it kills processes it started, which is not a shape a workflow step can take. The Go suite next to it covers the same guarantees short of the SIGKILL.
internal/outbox's whole subject is two databases that cannot roll each other back: the usage window and its message commit together in the gateway's database, and a redelivery is absorbed by an inbox row in the consumer's. With one database the suite has nothing to assert, so it skips itself - which meant it ran nowhere, on any machine, ever. The ledger job already has a Postgres service. This creates the consumer database inside it and points the suite at both, so seven real tests run instead of being reported green while skipping. -count=1 for the same reason the two jobs above it use it. Verified before landing: against a local postgres:16-alpine with both databases, `go test ./internal/outbox/ -count=1 -race` is 7 passed, 0 failed, and a second run against the same databases passes too, so the migration and the sink schema are re-appliable the way this job will re-apply them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
C13. The gateway knew what a request cost and nothing downstream was
ever told. "Write the row, then call billing" has two failure modes
that both cost real money: die between the two and the charge is lost;
retry the call and the customer is billed twice.
internal/outbox+migrations/004_outbox.sql. Sealing a usagewindow writes
usage_ledgerandoutboxin one commit, so either thewindow closed and the message exists or neither happened. Delivery is
at-least-once; the consumer makes it exactly-once-in-effect by
inserting the charge and an inbox row keyed by message id in one
transaction, so a redelivery finds the row already there and does
nothing. That is the only way the guarantee holds between two
databases neither of which can roll the other back.
runUsageSealeron an interval,USAGE_SEAL_INTERVAL, defaulting to 0 so a deployment with nothing tobill does not accumulate messages nobody will relay. reqctx's
"unauthenticated"label becomes a named constant because the sealermust exclude it: it is a metrics bucket, not an account.
cmd/tollgate-outbox(relayandsink) andscripts/outbox-crash.sh, which SIGKILLs the process under testbetween the commit and the delivery and counts what the consumer ended
up with. Nothing in it mocks a failure.
The wiring files were merged, not copied.
wt-c13was cut beforethe abuse-limit and budget work existed, so its
config.go,gateway.goandreqctx.gocarry a base without them and copyingthose three would have silently deleted +117/-11 of landed wiring. This
branch uses the three-way merged versions instead. The result is
+24/-2 across those files, and
middleware.RequestSize,middleware.Concurrencyandmiddleware.Budgetare all still in thechain.
The migration is
004_outbox.sql, not003_.003is already thebudget ledger on main, and
internal/store/migrate.goglobs and appliesin lexical order, so two files at one ordinal is a real defect rather
than a cosmetic one. The test that reads the file by name was updated
with it.
One commit is mine rather than the original work: the last one gives
the outbox suite the second database it needs so it runs in CI. Its
whole subject is two databases, so with one it skips itself, which
meant it ran nowhere on any machine. Verified before landing against a
local postgres:16-alpine with both databases:
go test ./internal/outbox/ -count=1 -raceis 7 passed, 0 failed, and a second run against thesame databases passes too, so the migration and sink schema re-apply the
way this job will re-apply them.
Verification, at the tip and at each of the four commits:
gofmt -l .empty,
go build ./...andgo vet ./...exit 0,go test ./...14 packages ok, 0 FAIL (13 before this branch).
🤖 Generated with Claude Code