[ElanHasson]

Part of MARSOHS-1479 (e2e canary runner DB tuning). Follow-up to #11, which declared tests_run_id_idx and runs_package_enqueued_at_idx.

Appends a migration that drops three indexes that the new ones supersede. The migration is DROP INDEX IF EXISTS so that it is a no-op in production, where the indexes are dropped by hand first with DROP INDEX CONCURRENTLY (see below); on fresh databases it removes the indexes that initial creates. down recreates them.

Do not merge until

The three hand-run DROP INDEX CONCURRENTLY statements below have been executed on the production cluster. If this merges first, tern runs a plain DROP INDEX inside a transaction at app startup, which takes an ACCESS EXCLUSIVE lock on runs/tests — brief, but it blocks inserts on tests (and the concurrent web/worker replicas) while it runs.

Merge order

• After db: declare tests(run_id) and runs(package, enqueued_at) indexes (MARSOHS-1642) #11 (merged 2026-09-21 19:06Z) — this branch is based on it and its migration is appended after add tests run_id index and runs package covering index.
• After the autovacuum-tuning PR (branch autovacuum-reloptions) if it lands first with a migration entry — rebase this branch so the pgMigrations slice order stays stable (tern tracks migrations by position/name; entries must never be reordered once applied).

Evidence (production, PG 16, pg_stat_user_indexes.idx_scan)

Observation window 2026-09-21 19:09:21Z → 19:33:18Z (24 min) while the app was active: tests_run_id_idx +127,683, runs_finished_at_idx +693, runs_package_enqueued_at_idx +13,324 over the same window.

Index	Definition	Size	idx_scan 19:09:21Z	19:12:46Z	19:18:30Z	19:33:18Z	Δ (24 min)	Δ excluding my EXPLAIN ANALYZE	What used to hit it
runs_package_idx	runs (package)	323 MB	27,895	27,895	27,895	27,895	0	0	ScheduleRun / ListRunsForPackage (WHERE package = $1) before the covering index existed. It is a strict prefix of runs_package_enqueued_at_idx, so nothing can need it.
runs_enqueued_at_started_at_idx	runs (enqueued_at, started_at)	414 MB	12	12	12	12	0	0	Old ListRunsForPackage plan (backward scan + package filter). ClaimRun uses runs_finished_at_idx.
tests_package_idx	tests (package)	423 MB	104	104	115	116	+12	+3	Package page ListTestsForPackageInRange — but only as a mis-plan, see below. No production caller filters tests by package alone (ListTestsForPackage is unused outside tests).

Total reclaimed: ~1,160 MB (737 MB from the two runs indexes alone).

Plan checks (EXPLAIN (ANALYZE, BUFFERS), all 15 packages, after today's VACUUM (ANALYZE) runs)

• ScheduleRun (SELECT count(*) FILTER (WHERE finished_at IS NULL), max(enqueued_at) FROM runs WHERE package = $1): 15/15 packages → Index Only Scan using runs_package_enqueued_at_idx, 1.7–40 ms.
• ListRunsForPackage (… WHERE package = $1 ORDER BY enqueued_at DESC LIMIT 5): 15/15 → Index Scan using runs_package_enqueued_at_idx, <0.1 ms.
• ClaimRun (WHERE started_at IS NULL AND finished_at IS NULL AND package = ANY($1) AND NOT package = ANY($2) ORDER BY enqueued_at LIMIT 1, several include/exclude shapes): all → Index Scan using runs_finished_at_idx + top-N sort, ~0.1 ms. Never touches a package index.
• ListTestsForPackageInRange (WHERE package = $1 AND result->'started_at' BETWEEN $2 AND $3 ORDER BY result->'started_at', 7-day window):
  • 6/15 packages (app, app-compatibility, droplet, kubernetes, registry, volume): Index Scan using tests_expr_idx + filter → ~10 ms.
  • 9/15 packages (account, agents, dbaas, dbaas-mongodb, domain, functions, loadbalancer, spaces, team): BitmapAnd(tests_expr_idx, tests_package_idx) → 80–1,020 ms. The tests_package_idx bitmap side reads 100k–590k TIDs per query (e.g. dbaas: 575,430 rows, 971 ms) to intersect with the 9,257 rows the 7-day tests_expr_idx range produces. The planner over-estimates the tests_expr_idx range (63k est vs 9k actual) and under-estimates the package bitmap cost.
  • Verified the fallback the planner takes without the bitmap path (SET LOCAL enable_bitmapscan = off in a rolled-back transaction) for dbaas, spaces, account, functions: Index Scan using tests_expr_idx, 9.5–10.9 ms — i.e. the same plan the fast 6 packages already get.

So tests_package_idx is not "unused" in the strict sense — the 3 non-me scans in the window are package-page views on one of the 9 affected packages — but every one of those scans made the request 8–100× slower than it would be without the index. Dropping it removes a mis-plan rather than a needed access path. (A composite tests (package, (result->'started_at')) would be the optimal index for that page and could be a follow-up; it is not needed for this PR to be safe.)

Hand-run production steps (before merging)

Run each statement on its own, outside a transaction (CONCURRENTLY cannot run inside one). Each waits for in-flight transactions on the table to finish, does not block reads/writes, and is safe to re-run if interrupted (an interrupted DROP INDEX CONCURRENTLY leaves the index marked INVALID; just run it again).

DROP INDEX CONCURRENTLY runs_package_idx;
DROP INDEX CONCURRENTLY runs_enqueued_at_started_at_idx;
DROP INDEX CONCURRENTLY tests_package_idx;

Verify afterwards:

SELECT indexrelname, pg_size_pretty(pg_relation_size(indexrelid))
FROM pg_stat_user_indexes WHERE schemaname = 'public' ORDER BY 1;

Expected: none of the three names present; tests_run_id_idx, runs_package_enqueued_at_idx, runs_finished_at_idx, runs_started_at_finished_at_idx, tests_expr_idx, runs_pkey, tests_pkey remain.

Rollback if needed (also outside a transaction):

CREATE INDEX CONCURRENTLY runs_package_idx ON runs (package);
CREATE INDEX CONCURRENTLY runs_enqueued_at_started_at_idx ON runs (enqueued_at, started_at);
CREATE INDEX CONCURRENTLY tests_package_idx ON tests (package);

Testing

• TestPG_Init_Indexes now also asserts the three indexes are absent after Init on a fresh database (they are created by initial, so this exercises the DROP). Fails without the migration, passes with it.
• gofmt -l . clean, go vet ./... clean, PG_DSN=… go test -race ./db/... green against a throwaway postgres:16.