CollectiveX explorer backed by a lazy-ingest Neon database - #497
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
The DB is a durable cache of Actions artifacts, but discovery still ran on every request: the routes are force-dynamic and call ensure* per request, and the module shared only the in-flight promise, so even a fully warm page spent a `list workflow runs` call just to conclude "already known". Cost scaled with traffic instead of with how often a sweep actually lands, adding GitHub latency to every response and burning rate limit for nothing. Give the two walking paths a 60s cooldown. ensureCollectiveXRun is left alone: it returns on a DB hit before touching GitHub, and throttling it would keep answering 404 for a run that had just appeared. Failures are remembered too, for a shorter 10s window, and rethrown rather than swallowed. Dropping them would let an outage read as "discovery fine, nothing found", which turns the routes' 502/503 into a 404 whenever the DB has no fallback row — that branch keys entirely off whether ensure* threw. The cooldown is module state that outlives a test case, and the suite does not reset modules, so resetCollectiveXDiscoveryCooldown() is exported and called in beforeEach; without it every test after the first would be served from the cooldown, issue no requests, and pass vacuously. Adds cover for warm-skip, expiry, and the remembered-failure identity, and folds the repeated run-listing fixture into one helper that documents why each call needs a fresh Response.
collectivex-latest.json carries one dataset payload — 3 series of 10 points — but pretty-printed that is 3,384 lines for 60 KB of content, and it dominated every diff that touched it (30% of this branch's insertions). Minifying both API fixtures drops 3,405 lines and 51 KB with byte-identical parsed content. These are machine-shaped payloads, replaced wholesale rather than hand-edited, so the readability that indentation buys is not worth the diff weight. Formatting has to be told to leave them alone: oxfmt reformats JSON as well as TS, and the lefthook pre-commit job re-stages what it fixes, so the next commit touching them would silently expand them again. oxfmt reads .prettierignore by default, so the exclusion lives there (and would carry over if Prettier is ever reintroduced).
Two red checks on this branch, unrelated to each other.
components/collectivex/types.ts re-exported the shared db types with
`export * from`, and Next's SWC loader rejects a star re-export anywhere in a
page's module graph ("Using `export * from '...'` in a page is disallowed").
The /collectivex page reaches this module through CollectiveXDisplay, so every
Cypress component spec failed at module build. Re-export the 18 types and 3
values explicitly.
.agents/skills/neon/SKILL.md was not oxfmt-clean, which fails `oxfmt --check`
repo-wide (it covers markdown, not just TS). Formatted, no content change.
Three findings from review, all the same shape: a caller-supplied value used without validation in a context that gives it meaning beyond a string. - fetchRunMeta interpolated repo and runId straight into an execSync command. Some callers validate, but the helper is exported and cannot rely on that, so it now rejects anything that is not `owner/name` and a digits-only run id. - ingest-collectivex validated the repo slug but took INGEST_RUN_ID verbatim in the env-var path; only --download parsed a numeric id. Validation now sits where both modes converge, before any `gh api` call. - selectShardArtifactNames interpolated runId into a RegExp, so a value with metacharacters changed what the pattern matched (CodeQL: regular expression injection) and could make the neighbouring `.+` backtrack badly. Escaped. Reported by CodeQL (alert 22) and Copilot on #497.
CollectiveX responses were tagged with a bare `collectivex`, while the main database tag is namespaced (`db:<CACHE_NAMESPACE>`). Deployments that share a Vercel project share its cache tags, so purging `collectivex` in one environment dropped every other environment's cached CollectiveX responses. Add collectiveXCacheTag(), namespaced the same way, and use it for both the response tag and the revalidateTag purge. COLLECTIVEX_CACHE_SCOPE stays bare on purpose: it is the public identifier accepted by /api/v1/invalidate?scope=, not a cache key. The route tests mock @/lib/api-cache wholesale, so they gain the new export — without it the handler threw and the route answered 500. Reported by Copilot on #497.
Migration 002 added run_attempt, but cx_run_docs stayed indexed on run_id alone while every doc read filters `run_id = ... and run_attempt = ...` and orders by id. A run holds one document per shard — 120 for a full 9-SKU sweep — so the attempt filter and the ordering were resolved per row. Index (run_id, run_attempt, id) so one index serves the filter and the sort. Added as 003 rather than an edit to 002: the runner skips already-applied files, so amending 002 would silently do nothing where it has already run. Reported by Copilot on #497.
…gests
The CLI and the lazy ingest each carried their own copy of the attempt-selection
rule, and the copies had already drifted: the lazy one broke same-attempt ties on
the greater artifact id, the CLI one kept whichever it saw first. GitHub permits a
repeated artifact name within a run, and the reader keeps the first shard per
case_id, so the same run could resolve to different documents depending on which
path ingested it.
Generalise the db helper to selectShardArtifacts<T extends {name, id?}>, keeping
the id tie-break where callers have ids and first-match where they do not, and
have selectShardArtifactNames wrap it. The lazy ingest now calls that helper and
matrixArtifactName instead of re-deriving both, which also drops its private
MATRIX_PREFIX/SHARD_PREFIX copies and the unescaped run-id interpolation that
mirrored the pattern CodeQL flagged in the other copy.
Also fix a latent mislabel next door: measuredPoints ran Math.max() over an empty
row set for a success shard with no measurements, giving -Infinity, so every
ladder point compared "beyond the largest measured" and the case was reported as
a backend token-capacity wall that was never observed. Nothing measured now reads
as pending/not-measured.
Tests: same-attempt id tie-break, first-match without ids, a run id containing
regex metacharacters, the empty-row shard, and a scope test pinning the delete
statement to one run and the two CollectiveX tables.
# Conflicts: # package.json # packages/app/scripts/capture-cypress-fixtures.ts # packages/db/package.json # pnpm-lock.yaml
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2095ad6. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the
/collectivextab's just-in-time GitHub artifact serving with a dedicated Neon database that acts as a durable, lazily-populated cache of GitHub Actions. Sweep runs now outlive their 14-day artifact retention once viewed, the run picker lists up to 50 persisted runs (was 8, rebuilt per request), and runs are deletable from the dashboard.Architecture
/api/v1/collectivex/{latest,runs,runs/[runId]}check the DB first; on a miss they discover completedCollectiveX Sweepruns (any harness branch — workflow identity only), download thecxsweep-matrix-*/cxshard-*artifact ZIPs in-request, and persist the raw documents. The shared reader (packages/db/src/collectivex/) assembles datasets at read time, so the evolving sweep JSON contract only ever touches the reader and applies retroactively to stored runs. No ingest workflow, cross-repo dispatch, or GitHub secrets.ON CONFLICT DO NOTHING(concurrent first-viewers race safely; a partial run can never become visible); aFOR UPDATE-guarded refresh replaces a run's contents when GitHub reports a newerrun_attempt(re-run of failed shards).cx_runs.deleted_at, docs freed) so re-discovery never resurrects a deleted run; the CLI (pnpm admin:db:ingest:collectivex <run>) pre-warms runs before artifact expiry, backfills, and un-deletes.collectivexCDN tag with a 60s freshness window;POST /api/v1/invalidate?scope=collectivexand run deletion purge only that scope.DELETE /api/v1/collectivex/runs/[runId]uses a dedicatedCOLLECTIVEX_ADMIN_SECRETBearer token (browser-held, independently rotatable — deliberately notINVALIDATE_SECRET), prompted once in the UI and remembered in localStorage.Design rationale and invariants:
docs/collectivex.md.Deploy prerequisites (already provisioned)
Neon marketplace store
inferencex-collectivexon this project; env varsDATABASE_COLLECTIVEX_READONLY_URL(readonly role @ pooled),DATABASE_COLLECTIVEX_WRITE_URL(owner @ direct),COLLECTIVEX_ADMIN_SECRETset for production + preview;migrations-collectivex/001applied; 8 real sweep runs already ingested.Validation
collectivex.cy.tsspecs (fixtures mode), including the delete flow (token prompt, 401 → token cleared, declined confirm).Authorizationheaders returning 401.pnpm typecheck && pnpm lint && pnpm fmtclean; production build succeeds.Note: unofficial-run overlay support does not apply — CollectiveX is not inference/evaluation chart data.
Note
High Risk
Introduces a new production data path (lazy GitHub ingest, separate DB writes, tombstoned deletes) and browser-stored admin auth; misconfiguration or ingest races could affect CollectiveX availability independent of the main dashboard.
Overview
CollectiveX stops serving sweep data directly from ephemeral GitHub artifacts and instead uses a separate Neon database as a durable cache filled on read by new
/api/v1/collectivex/latest,/runs, and/runs/[runId]routes (raw docs stored; sharedpackages/dbreader assembles responses). Runs survive 14-day artifact expiry, list progressively withdiscovery_completepolling, and fall back to stored rows when GitHub is down.The
/collectivexdashboard gains a multi-run table (visibility checkboxes, dash patterns in charts/legend), D3 explorer controls, matrix inventory, and admin DELETE viaCOLLECTIVEX_ADMIN_SECRET(Bearer, remembered inlocalStorage).POST /api/v1/invalidate?scope=collectivexand delete purge only the CollectiveX CDN tag; main blob cache is untouched.Supporting changes:
docs/collectivex.md, CollectiveX env vars in.env.example,admin:db:migrate:collectivex/admin:db:ingest:collectivex, Cypress fixtures +collectivex.cy.ts, tab nav/sitemap//zh/collectivex, and a Neon agent skill.Reviewed by Cursor Bugbot for commit 93fe1af. Bugbot is set up for automated code reviews on this repo. Configure here.