[ElanHasson]

Jira: MARSOHS-1649

Problem

ListRunSummariesInRange selected runs.error and the full tests.result jsonb for every test row and unmarshalled it into tester.T just to read .Name and .State. Measured in prod for the dashboard's 30-day call: 94,127 rows, 39 MB of result jsonb transferred and unmarshalled, three times per page load (1 h / 1 d / 30 d). Since #10 runs.error is also up to ~32 KB of stdout/stderr per errored run, multiplied by that run's test rows.

Separately, the query required finished_at IS NOT NULL and bucketed by started_at only, so an in-flight run was invisible until it finished (~15 min for agents), and a 15-minute run painted one 5-minute cell followed by two blank ones. Short suites re-enqueued every 5 min fill every cell, which made long suites look broken by contrast.

Changes

Payload (db/pg.go)

• Select projected columns only: runs.package, runs.id, runs.started_at, runs.finished_at, (runs.error IS NOT NULL), tests.id, tests.result->>'name', tests.result->>'state'. Scanned into string/bool/NullUUID, no tester.T unmarshal.
• FROM runs LEFT JOIN tests ON tests.run_id = runs.id AND runs.error IS NULL. Two behaviour changes:
  • Errored runs with zero tests now appear in ErrorRunIDs. Previously the inner join from tests dropped them entirely, so a killed run / TestMain failure showed as a blank slot. They now render solid red.
  • Tests of errored runs are not fetched at all (they were fetched and then continued over).
• Row width on a synthetic dataset with 300-byte result padding (prod results are larger): 246 MB → 35 MB for ~500k rows; per row ≈ 520 B → 74 B. Extrapolating to the prod measurement, the 39 MB call becomes ~6–7 MB.
• Bucket index is floor-divided (bucketIndex) and clamped. Previously a run with started_at == end indexed one past the slice (<= end + exact-multiple range → panic). Range is now half-open [begin, end).
• The begin/end used for bucket arithmetic are truncated to microseconds to match timestamptz/pgx; the old code only got bucket 0 right for a run starting exactly at begin because int() truncates toward zero.

In-progress and spanning runs

• finished_at IS NOT NULL filter dropped. PackageSummary.RunningRunIDs added (subset of RunIDs with finished_at IS NULL); NumRunningRuns() on both RunSummary and PackageSummary (plus NumRuns/NumErrorRuns on PackageSummary so both types satisfy one interface).
• A run is attributed to every bucket in [started_at, coalesce(finished_at, now())), clamped to the range. Its tests are only counted in the bucket it started in, so nothing is double-counted; spanned buckets carry the run id only. Tests already submitted by a running run are counted as usual (simplest; the cell shows results and the tooltip says "1 running").
• To attribute runs that started before begin but were still executing at begin, the query looks back runSummaryLookback = 24h before begin. That keeps the query a bounded range scan (the scheduler resets/fails anything past the run timeout, so 24h is a very generous bound) and makes /run_summary for a spanned cell consistent with the strip.
• No SpanningRunIDs: colouring only needs "has results / has running / has errored / has runs", all derivable from the existing fields.

Templates

• summaryState helper in http/templates.go: results → existing pass/skip/fail bar (unchanged semantics); errored → solid bg-danger; running → animated grey stripes (.summary-running); active (a finished run was executing through this cell; its results are in the cell it started in) → static grey stripes (.summary-active); empty → bg-light.
• Tooltips show "(N running)"; /run_summary shows running/erred counts and lists running and erred runs with links to /runs/<id>.

db.DB interface unchanged; db/db_mock.go is unchanged after go generate.

EXPLAIN

Local Postgres 16, 103k runs / 1.03M tests over 60 days, default planner settings, 2-day window (≈ prod's 30-day selectivity against the multi-year table):

Sort (Sort Key: runs.started_at)
  -> Nested Loop Left Join  (Join Filter: runs.error IS NULL)
       -> Bitmap Heap Scan on runs
            -> BitmapOr
                 -> Bitmap Index Scan on runs_finished_at_idx        (Index Cond: finished_at IS NULL)   -- 3 in-flight runs
                 -> Bitmap Index Scan on runs_started_at_finished_at_idx
                      (Index Cond: started_at >= begin-24h AND started_at < end AND finished_at >= begin)
       -> Index Scan using tests_run_id_idx on tests  (Index Cond: run_id = runs.id)

The plan drives from runs_started_at_finished_at_idx → tests_run_id_idx as before; the finished_at IS NULL OR-branch is served by the existing runs_finished_at_idx and costs a handful of index tuples. The 1-hour call runs the same shape in ~1.7 ms.

Verification

• gofmt -l ., go vet ./..., go generate ./... && git diff --exit-code db/db_mock.go, go test -race ./... (with PG_DSN set) all clean.
• TestPG_ListRunSummariesInRange extended: errored run with zero tests appears in ErrorRunIDs; errored run's tests are not counted; running run spans buckets with RunningRunIDs set and its test counted once; finished run spans the buckets it executed in with half-open end; run started before the range is attributed but its tests are not; run at end excluded. Plus TestBucketIndex.
• New http/ui_test.go: summaryState unit test and a render test through NewUIHandler with the mock for / and /run_summary (running and errored) asserting 200 and the expected markup.
• Built the binary, seeded a local DB with a passed run, an errored zero-test run, a 14-minute finished run and an in-flight run, and curled /, /packages, /packages/{db,http}, /run_summary for running, errored, and spanned cells: all 200 with the expected states.

README has no UI/dashboard section, so no doc change.

[fumblehool]

lgtm 👍
note for future: The suggested fix was to use GROUP BY with count(*) FILTER (WHERE result->>'state' = 'passed'). The PR instead extracts only the needed fields but still iterates row-by-row. Both approaches dramatically reduce data transfer, but the GROUP BY approach would push more computation to Postgres.