Skip to content

Replace compute-billing push with the pull API#893

Merged
benben merged 2 commits into
mainfrom
ben/billing-pull-buckets
Jul 6, 2026
Merged

Replace compute-billing push with the pull API#893
benben merged 2 commits into
mainfrom
ben/billing-pull-buckets

Conversation

@benben

@benben benben commented Jul 6, 2026

Copy link
Copy Markdown
Member

Implements docs/design/billing-pull-api.md (#860): billing pulls compute usage from duckgres and acks a watermark; duckgres deletes acked buckets. The push path (leader drain → PostHog ingestion capture events) is removed. Per-connection metering (wall-clock × provisioned worker size, 60s buckets) is unchanged.

What changed

Bucket key + cursor (migration 000015)

  • duckgres_org_compute_usage key widens to (org_id, team_id, query_source, cpu, mem_gib, bucket_start). team_id is backfilled from the org's default_team_id for pre-existing rows; worker size is stored as exact NUMERIC decimals (vCPU / GiB — canonical decimal text computed with integer math, no floats), so fractional sizes group exactly.
  • New duckgres_compute_billing_cursor (single global ack cursor). The per-org push-drain state table is dropped.

Meter

  • Records team_id (org's default_team_id from the config snapshot) and query_source (the duckgres.query_source session GUC via server.ConnectionBilling, default standard; a mid-connection change bills the whole connection under the final value) alongside the provisioned worker size. Distinct keys accumulate — and bill — separately.
  • Metering is now always on for the remote backend (no enable knob); still strictly best-effort, nil-safe, never on the query hot path.

Pull API (internal-secret bearer → admin, registered inside the audited /api/v1 group)

  • GET /api/v1/billing/usage — aggregates all closed buckets in (cursor, watermark_high] into one row per key per UTC day; watermark_high = newest closed minute (now − 60s − 30s grace, grace > flush interval so every CP's contribution has landed). Response size is bounded by active keys × days, so billing downtime can't make it explode.
  • POST /api/v1/billing/ack {watermark_high} — advances the cursor monotonically + deletes buckets ≤ watermark_high in one transaction. Idempotent; an ack beyond the latest closed bucket is rejected (400) so unserved buckets can never be deleted.
  • Safety GC (leader-only, under the janitor lease): hard-deletes buckets older than 30 days, WARN with drop count (nonzero = billing stopped pulling — alertable).

Push removal

  • Deleted compute_drain.go / compute_capture.go (+tests); removed --billing-ingest-url / --billing-ingest-token and DUCKGRES_BILLING_INGEST_URL/_TOKEN everywhere (flags, env resolution, README, CLAUDE.md). Chart-side secret wiring cleanup happens in the charts repo separately.
  • Cutover is loss-free: buckets pending at deploy were not yet pushed (push deleted only after ship), and the pull API serves everything buffered since the (initially absent) cursor.

Tests

  • compute_meter_test.go: key separation by query_source and worker size, team resolution, unknown-team tolerance, remainder carry with size.
  • compute_billing_api_test.go (new): watermark window math, never-acked serves everything, empty-window short-circuit, ack advance/bounds/idempotency, store errors → 500, GC loop.
  • tests/configstore/migrations_postgres_test.go: v15, new columns, cursor table present, drain-state table absent.
  • e2e harness.sh: compute_usage_metering_wiredcompute_usage_pull_api — two real connections (one standard, one with SET duckgres.query_source='endpoints') must surface as separate usage rows carrying the org's provisioned default_team_id and a positive worker size; then ack advances the cursor (next GET's watermark_low equals it), deletes ≥1 bucket, re-ack is a no-op, and a future watermark is rejected with 400.

For the billing team (consumer contract)

Pull loop: GET /api/v1/billing/usage → process → POST /api/v1/billing/ack with the exact watermark_high from the GET. Cross-check watermark_low against your recorded last ack (mismatch = desync, alert). Auth: Authorization: Bearer <internal secret> over the existing posthog → mw duckgres ingress.

🤖 Generated with Claude Code

Billing now pulls compute usage from duckgres instead of duckgres
pushing capture events to PostHog ingestion
(docs/design/billing-pull-api.md). The per-connection metering is
unchanged; what changes is the bucket key and how data leaves:

- Migration 000015 widens the buffer key to (org_id, team_id,
  query_source, cpu, mem_gib, bucket_start) — team_id backfilled from
  the org's default_team_id, worker size as exact NUMERIC decimals —
  adds the single global ack cursor and drops the per-org push-drain
  state table.
- The meter records the org's default team (config-snapshot lookup) and
  the session's duckgres.query_source GUC (via server.ConnectionBilling)
  alongside the provisioned worker size; distinct keys accumulate and
  bill separately. Metering is now always on for the remote backend.
- GET /api/v1/billing/usage aggregates all closed buckets since the last
  ack into one row per key per UTC day, with watermark_low (cursor) and
  watermark_high (newest closed minute: now − 60s − 30s grace).
  POST /api/v1/billing/ack advances the cursor monotonically and deletes
  buckets ≤ watermark_high in one transaction; idempotent, and an ack
  beyond the latest closed bucket is rejected so unserved buckets can
  never be deleted. Both routes are internal-secret/admin-gated inside
  the audited /api/v1 group.
- A leader-only safety GC hard-deletes buckets older than 30 days and
  WARNs with the dropped count (billing not keeping up = alertable).
- The push path is gone: leader drain + capture client, and the
  --billing-ingest-url/--billing-ingest-token flags with their
  DUCKGRES_BILLING_INGEST_URL/_TOKEN envs.

e2e: compute_usage_metering_wired becomes compute_usage_pull_api — two
real connections (standard + endpoints GUC) must surface as separate
usage rows carrying the provisioned default_team_id and worker size,
then ack must advance the cursor, delete the acked buckets, stay
idempotent on re-ack, and reject a future watermark.
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area Added Changed Deleted
Test files 1 4 2
E2E/journey files 0 1 0
Workflow files 0 0 0

Signals

  • Test cases: +8 / -7
  • Assertions: +45 / -35
  • Skips or known failures added: 0
  • Workflow continue-on-error added: 0
  • Workflow path filters added: 0
  • Test commands removed from justfile: 0
  • E2E/journey retry lines added: 0

Coverage risk: likely reduced

Warnings

  • Test files deleted (likely reduced)
    • controlplane/compute_capture_test.go
    • controlplane/compute_drain_test.go
  • E2E or journey files changed (needs review)
    • tests/e2e-mw-dev/harness.sh

Its only production caller is kubernetes-tagged multitenant.go, so the
plain build's linter flagged it unused. The new test exercises the real
registration and locks in that both billing routes sit behind the
admin-gating middleware.
@benben benben merged commit e8bbab2 into main Jul 6, 2026
28 checks passed
@benben benben deleted the ben/billing-pull-buckets branch July 6, 2026 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant