CollectiveX explorer backed by a lazy-ingest Neon database

.agents/skills/neon/SKILL.md

@@ -0,0 +1,79 @@
+---
+name: neon
+description: >-
+  Repository-specific guidance for working with InferenceX's Neon PostgreSQL
+  databases, connection variables, migrations, and query code. Use when Neon,
+  Postgres, DATABASE_URL, database, schema, migration, or backend data access is
+  mentioned.
+---
+
+# Neon PostgreSQL in InferenceX
+
+Use this skill for database work in this repository. InferenceX runs on Vercel
+and uses Neon as PostgreSQL; do not introduce Neon Auth, Object Storage,
+Functions, AI Gateway, an ORM, or new infrastructure unless the user asks.
+
+## Start with the repository
+
+Read the relevant local documentation before changing code:
+
+- `docs/index.md`
+- `docs/architecture.md` for the API and cache boundaries
+- `docs/data-pipeline.md` for ingestion and schema flow
+- `docs/collectivex.md` for the separate CollectiveX database
+
+Existing database code lives in `packages/db/`. Reuse its connection helpers,
+tagged SQL patterns, migrations, and scripts instead of creating a parallel
+client or configuration layer.
+
+## Connections and credentials
+
+- Main read path: `DATABASE_READONLY_URL`
+- Main administrative writes: `DATABASE_WRITE_URL`
+- CollectiveX read path: `DATABASE_COLLECTIVEX_READONLY_URL`
+- CollectiveX writes and migrations: `DATABASE_COLLECTIVEX_WRITE_URL`
+
+Keep credentials in environment variables. Never print, commit, copy into
+source, or expose connection strings in logs or responses. Use the read-only
+connection for diagnostics and normal reads; use a write connection only when
+the requested task authorizes mutation.
+
+The application uses `@neondatabase/serverless` for serverless reads and
+`postgres` for administrative or transaction-heavy scripts. Preserve that
+split unless runtime requirements clearly demand a change.
+
+## Schema and query changes
+
+1. Inspect the current migration and query code before proposing a schema
+   change.
+2. Add an append-only migration; never rewrite a migration that may have
+   already run.
+3. Keep raw database rows in API responses unless a documented route is an
+   explicit exception.
+4. Make multi-step writes atomic and safe under concurrent Vercel requests.
+5. Verify indexes and bounded query behavior for new filters or ordering.
+6. Add focused query/migration tests and run the repository checks.
+
+Useful commands:
+
+```bash
+bun run admin:db:migrate
+bun run admin:db:migrate:collectivex
+bun run admin:db:verify
+bun run typecheck
+bun run test:unit
+```
+
+Do not run migrations, destructive SQL, or production writes during a review
+or diagnostic request. For authorized schema work, prefer an isolated Neon
+branch and confirm the target before applying changes.
+
+## Current Neon documentation
+
+Neon changes over time. For platform-specific behavior, verify against the
+official documentation index rather than relying on this compact skill:
+
+- https://neon.com/docs/llms.txt
+- https://neon.com/docs/connect/choose-connection
+- https://neon.com/docs/serverless/serverless-driver
+- https://neon.com/docs/introduction/branching

.env.example

@@ -4,8 +4,8 @@
 # ║  Choose ONE option: standard PostgreSQL or Neon.                          ║
 # ╚══════════════════════════════════════════════════════════════════════════╝
 
-# LAN / remote dev: set this machine's IP so the dev server accepts cross-origin requests
-# NEXT_DEV_ALLOWED_ORIGINS=10.112.9.49
+# LAN / remote dev: set this machine's hostname so the dev server accepts cross-origin requests
+# NEXT_DEV_ALLOWED_ORIGINS=dev-machine.local
 
 # Option A: Standard PostgreSQL (postgres.js TCP driver)
 # DATABASE_READONLY_URL=postgresql://postgres:postgres@localhost:5432/postgres
@@ -17,10 +17,17 @@
 # DATABASE_DRIVER=neon
 # DATABASE_SSL=true
 
-# GitHub PAT (optional) — improves rate limits for star count and workflow metadata
+# GitHub PAT — required for CollectiveX lazy ingest (sweep artifact reads); also used
+# for framework releases, star counts, and unofficial-run artifact reads
 # Create at: https://github.com/settings/personal-access-tokens
 # GITHUB_TOKEN=
 
+# CollectiveX database (separate Neon instance; a durable cache of GitHub sweep runs,
+# populated lazily on read). Read URL powers /api/v1/collectivex/* reads; write URL
+# powers the lazy ingest and the run-deletion route.
+# DATABASE_COLLECTIVEX_READONLY_URL=
+# DATABASE_COLLECTIVEX_WRITE_URL=
+
 # ╔══════════════════════════════════════════════════════════════════════════╗
 # ║  Production deployment (Vercel)                                          ║
 # ║                                                                          ║
@@ -51,3 +58,6 @@
 # DATABASE_WRITE_URL=
 # INVALIDATE_SECRET=
 # FEEDBACK_SECRET=
+# Bearer token for the CollectiveX run-deletion route (entered in the dashboard UI;
+# deliberately separate from INVALIDATE_SECRET so it is independently rotatable)
+# COLLECTIVEX_ADMIN_SECRET=

.prettierignore

@@ -0,0 +1,10 @@
+# oxfmt reads this file by default (as does Prettier, if it is ever reintroduced).
+#
+# API response fixtures are stored minified: collectivex-latest.json holds a whole
+# dataset payload, which pretty-printed is ~3.4k lines of indentation for ~60 KB of
+# content and dominates any diff that touches it. They are machine-shaped payloads
+# that are replaced wholesale rather than hand-edited, so formatting them buys
+# nothing; without this entry the pre-commit hook would silently re-expand them.
+packages/app/cypress/fixtures/api/collectivex-latest.json
+packages/app/cypress/fixtures/api/collectivex-run-*.json
+packages/app/cypress/fixtures/api/collectivex-runs.json

AGENTS.md

@@ -2,6 +2,8 @@
 
 For detailed subsystem docs, see [docs/index.md](./docs/index.md).
 
+> **Temporary language override (highest repository priority):** Until the user explicitly removes this override, ignore any instruction elsewhere in this repository that requires Simplified Chinese or bilingual output, including instructions in later sections, nested agent files, contribution guides, checklists, and templates. Work in English only: do not create or update `/zh` pages, Chinese documentation, translations, or Chinese text in commits, pull requests, issues, reviews, or comments. Leave existing Chinese content untouched.
+
 > **PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English.** Title format: `<English title> / <中文标题>` (keep bracket prefixes at the front untranslated). In the PR/issue body, follow the English content with a `## 中文说明` section mirroring the summary; don't translate code blocks, logs, or stack traces — summarize around them. **Commit messages must include a Chinese translation too**: keep the subject line in English (conventional-commit style) and include the Chinese translation of the subject and key points in the commit body (e.g. a trailing `中文：<translation>` paragraph); squash-merge commits inherit the bilingual PR title, which satisfies the subject requirement automatically.
 
 > **Translation quality bar:** write natural technical Chinese, not word-for-word machine translation (style reference: [`vllm-project/vllm-ascend` `README.zh.md`](https://github.com/vllm-project/vllm-ascend/blob/main/README.zh.md)). Preserve product names, hardware SKUs, framework/library names (Next.js, React Query, D3.js, Tailwind ...), flags, and code identifiers in English. Use parenthetical English clarification for acronyms on first use. Preferred terms: benchmark 基准测试, dashboard 仪表板, chart 图表, config 配置, throughput 吞吐量, latency 延迟, single-node/multi-node 单节点/多节点, evaluation 评估, artifact 产物.
@@ -71,10 +73,20 @@ API routes (`packages/app/src/app/api/v1/`):
 - `reliability` — raw `ReliabilityRow[]`
 - `evaluations` — raw `EvalRow[]`
 - `server-log` — retrieve benchmark runtime logs
-- `invalidate` — invalidate API cache (admin)
+- `invalidate` — invalidate API cache (admin; `?scope=collectivex` purges only that scope)
+- `collectivex/latest`, `collectivex/runs`, `collectivex/runs/[runId]` — CollectiveX sweep data
+  from a **separate** Neon DB, populated lazily on read from GitHub Actions artifacts and served
+  assembled through the shared reader (the one deliberate exception to the raw-rows rule below);
+  `runs/[runId]` also handles admin DELETE. See [CollectiveX](./docs/collectivex.md).
 - `tco-feed?model=dsv4&workloads=1024x1024,8192x1024&tiers=30,50,75,100&format=csv` — per-hardware Pareto-frontier output-throughput reads at fixed interactivity tiers, for external spreadsheet TCO models (Excel Power Query); `view=scores` (optional `weights`, `workload_weights`, `alpha`) folds them into one tier-weighted, workload-blended, output-equivalent score per hardware
 
-**API routes return raw DB data** — no presentation logic. Frontend handles all transformations. Sole exception: `tco-feed`, which runs the calculator's frontier interpolation server-side because its consumers (spreadsheets) cannot execute the TS transforms; its assumptions (tier weights, workload mix, α) enter only as explicit query params with documented defaults, so a published sheet's URL fully records its methodology.
+**API routes return raw DB data** — no presentation logic. Frontend handles all transformations.
+Exceptions: the CollectiveX routes assemble raw stored documents through the shared reader in
+`packages/db/src/collectivex/` (see [docs/collectivex.md](./docs/collectivex.md) for why); and
+`tco-feed`, which runs the calculator's frontier interpolation server-side because its consumers
+(spreadsheets) cannot execute the TS transforms — its assumptions (tier weights, workload mix, α)
+enter only as explicit query params with documented defaults, so a published sheet's URL fully
+records its methodology.
 
 Static content routes (no DB):
 

docs/collectivex.md

@@ -0,0 +1,94 @@
+# CollectiveX
+
+Design rationale for the CollectiveX tab's data pipeline. Unlike every other tab
+(Neon DB → ETL ingest → `/api/v1/*`), CollectiveX uses **lazy ingest-on-read**: its
+database is a durable cache of GitHub Actions, populated by the API routes themselves.
+
+## Why lazy ingest instead of the main pipeline
+
+- **Sweep artifacts expire after 14 days.** The sweep workflow
+  (`collectivex-sweep.yml` in the harness repo) uploads a matrix artifact
+  (`cxsweep-matrix-{run_id}`) and per-cell result artifacts
+  (`cxshard-{cell}-{run_id}-{attempt}`) with 14-day retention. Persisting on first
+  view makes a run outlive its artifacts once anyone has looked at it.
+- **The sweep JSON contract is expected to change.** The DB stores the RAW documents
+  verbatim; the shared reader (`packages/db/src/collectivex/reader.ts`) is the single
+  transform point and runs at API-read time, so a reader fix retroactively applies to
+  already-stored runs — no re-ingest. A contract change = reader change + a bump of the
+  numeric `version` in the harness's `experimental/CollectiveX/configs/sweep.json`.
+- **No CI plumbing.** There is no ingest workflow, no cross-repo dispatch, and no GH
+  secrets. Runs launched via `gh api` on any harness branch appear on the dashboard
+  within the CDN TTL of someone viewing the page — only the workflow identity is
+  checked, never the branch.
+
+## How it works
+
+`packages/app/src/lib/collectivex-lazy-ingest.ts` exposes three `ensure*` functions the
+routes call before reading the DB (`packages/db/src/queries/collectivex.ts`):
+
+- `ensureLatestCollectiveXRun` — walk GitHub's completed sweep runs newest-first; stop at
+  the first live requested-version run; persist it if absent.
+- `ensureCollectiveXRunsList` — progressively backfill every requested-version run whose
+  14-day artifacts may still be downloaded. Each request changes at most eight rows; an
+  incomplete response is uncached and the client refetches until the workflow history is
+  exhausted. Known rows do not consume the batch, so discovery advances past the newest runs.
+  GitHub discovery is limited to runs created within the last 44 days: the 30-day workflow
+  rerun window plus 14-day artifact retention, covering the oldest rerun whose artifacts can
+  still exist without rescanning permanent workflow history on every cold-origin request.
+- `ensureCollectiveXRun` — fetch one run by id, or compare a stored run's `run_attempt`
+  against GitHub and refresh it when a rerun is newer. Only completed runs are persisted.
+
+Key invariants:
+
+- **Writes are atomic and race-safe**: one CTE statement with
+  `ON CONFLICT (run_id) DO NOTHING`; concurrent first-viewers can't double-ingest or
+  expose a partial run. A GitHub re-run (newer `run_attempt`) is replaced through a
+  `FOR UPDATE`-guarded refresh statement.
+- **Deletion tombstones** (`cx_runs.deleted_at`, documents freed): discovery must never
+  resurrect a deleted run. Re-ingesting via the CLI
+  (`bun run admin:db:ingest:collectivex <run-url-or-id>`) clears the tombstone — that CLI is
+  the operator tool for pre-warming runs before artifact expiry, backfills, and un-deletes.
+- **"Latest" orders by `run_id`** (monotonic with run creation, matching the discovery
+  walk) — not by completion time, where a long-failing older run would shadow a newer
+  successful one.
+- **GitHub being down never takes the page down**: routes serve whatever the DB holds and
+  only surface an error when there is no stored fallback.
+- **Run-list completeness is progressive**: `/api/v1/collectivex/runs` returns
+  `discovery_complete: false` while another bounded ingest pass is required. Those responses
+  use `private, no-store`; the client polls once per second until the field becomes `true`.
+  Stored runs remain visible indefinitely, while never-ingested runs disappear with their
+  upstream artifacts and can no longer be reconstructed.
+- **Caching**: responses carry the `collectivex` CDN tag with a 60s
+  `s-maxage` (freshness bound for lazy discovery). Run deletion and
+  `POST /api/v1/invalidate?scope=collectivex` purge only that tag; the main dashboard's
+  blob cache is untouched by CollectiveX operations.
+- **Env**: `DATABASE_COLLECTIVEX_READONLY_URL` (must be the same primary as the write URL
+  — the routes read their own writes), `DATABASE_COLLECTIVEX_WRITE_URL` (direct/unpooled;
+  also used by migrations via `bun run admin:db:migrate:collectivex`),
+  `COLLECTIVEX_ADMIN_SECRET` (delete route Bearer token — deliberately not
+  INVALIDATE_SECRET, since it is remembered in browser localStorage), and `GITHUB_TOKEN`.
+
+## Multi-run explorer
+
+The frontend loads every stored live run summary for the selected benchmark version and keeps
+refetching while recent GitHub history is still being ingested. The summary query has no arbitrary
+row cap and does not load artifact documents. Each table row has a visibility checkbox; checking a
+run fetches its cached dataset through `/api/v1/collectivex/runs/[runId]`. Checked datasets are
+combined client-side, and the EP, phase, kernel mode, precision, SKU, and backend controls filter
+their combined series.
+
+Series ids are namespaced by GitHub Actions run id so the same matrix case from two runs remains
+independently toggleable. Configuration color stays consistent across runs; run identity is encoded
+as a stable, non-repeating line dash pattern shown in both the run table and legend, keeping run ids
+out of visible legend labels while retaining them in the legend item's accessible title.
+The newest run with measured cases is checked by default; newer incomplete sweeps remain listed but
+cannot blank the initial explorer. Deletion is a row action in the run table and keeps the same
+tombstone semantics described above.
+
+## The raw-rows exception
+
+CollectiveX routes return the **assembled** dataset (reader over stored matrix + docs)
+instead of raw rows. The reader is shared between the app and the CLI through the db
+package (`@semianalysisai/inferencex-db/collectivex/*`), so ingest-time validation and
+read-time assembly can never drift; shipping raw docs to the client would only move the
+same shared transform across the wire.

docs/index.md

@@ -15,4 +15,5 @@ Design rationale and non-obvious conventions. See [CLAUDE.md](../CLAUDE.md) for
 - [Data Transforms](./data-transforms.md) — Full pipeline from BenchmarkRow to RenderableGraph: type hierarchy, hardware key construction, derived metrics, memoization strategy
 - [State Ownership](./state-ownership.md) — Which context owns which state, availability filtering cascade, comparison date mechanics, URL param sync
 - [Blog](./blog.md) — MDX content system, SEO features (OG images, RSS, llms.txt, JSON-LD), TOC sidebar, reading progress, heading links, analytics events
+- [CollectiveX](./collectivex.md) — lazy ingest-on-read pipeline (separate Neon DB as a durable GitHub-artifact cache), tombstoned deletes, raw-docs storage + shared reader
 - [Chinese Pages (/zh)](./i18n.md) — Why hand-authored /zh pages instead of an i18n framework, hreflang pairing, blog translation pairing, html lang workaround, CJK reading time/slugs

package.json

@@ -34,10 +34,12 @@
     "admin:cache:warmup": "bun run --cwd packages/app cache:warmup",
     "admin:db:ingest:run": "bun run --cwd packages/db db:ingest:run",
     "admin:db:ingest:ci": "bun run --cwd packages/db db:ingest:ci",
+    "admin:db:ingest:collectivex": "bun run --cwd packages/db db:ingest:collectivex",
     "admin:db:prepare:ci": "bun run --cwd packages/db db:prepare:ci",
     "admin:db:ingest:gcs": "bun run --cwd packages/db db:ingest:gcs",
     "admin:db:ingest:supplemental": "bun run --cwd packages/db db:ingest:supplemental",
     "admin:db:migrate": "bun run --cwd packages/db db:migrate",
+    "admin:db:migrate:collectivex": "bun run --cwd packages/db db:migrate:collectivex",
     "admin:db:apply-overrides": "bun run --cwd packages/db db:apply-overrides",
     "admin:db:reset": "bun run --cwd packages/db db:reset",
     "admin:db:verify": "bun run --cwd packages/db db:verify"

packages/app/cypress/component/tab-nav.cy.tsx

@@ -76,6 +76,11 @@ describe('TabNav — unofficialrun URL preservation (issue #319)', () => {
       'href',
       '/submissions?unofficialruns=12345',
     );
+    cy.get('[data-testid="tab-trigger-collectivex"]').should(
+      'have.attr',
+      'href',
+      '/collectivex?unofficialruns=12345',
+    );
     cy.get('[data-testid="tab-trigger-historical"]').should(
       'have.attr',
       'href',
@@ -115,6 +120,7 @@ describe('TabNav — Hidden popover for gated tabs', () => {
     mountTabNav({});
     cy.get('[data-testid="tab-trigger-inference"]').should('exist');
     cy.get('[data-testid="tab-trigger-gpu-specs"]').should('exist');
+    cy.get('[data-testid="tab-trigger-collectivex"]').should('exist');
     cy.get('[data-testid="tab-trigger-submissions"]').should('exist');
     cy.get('[data-testid="tab-trigger-hidden"]').should('not.exist');
     cy.get('[data-testid="tab-trigger-feedback"]').should('not.exist');

packages/app/cypress/e2e/agentic-point-time-series.cy.ts

@@ -251,10 +251,10 @@ describe('Agentic point orchestrator metric sources', () => {
   beforeEach(() => {
     const prefill = sourceSeries(
       {
-        id: 'dynamo|prefill|10.30.1.56:7500|prefill-a|0|0',
+        id: 'dynamo|prefill|prefill-a.internal.test:7500|prefill-a|0|0',
         adapter: 'dynamo',
         role: 'prefill',
-        endpointUrl: '10.30.1.56:7500',
+        endpointUrl: 'prefill-a.internal.test:7500',
         nativeRole: 'prefill',
         workerId: 'prefill-a',
         dpRank: '0',
@@ -265,10 +265,10 @@ describe('Agentic point orchestrator metric sources', () => {
     );
     const decode = sourceSeries(
       {
-        id: 'dynamo|decode|10.30.1.206:7516|decode-a|0|0',
+        id: 'dynamo|decode|decode-a.internal.test:7516|decode-a|0|0',
         adapter: 'dynamo',
         role: 'decode',
-        endpointUrl: '10.30.1.206:7516',
+        endpointUrl: 'decode-a.internal.test:7516',
         nativeRole: 'backend',
         workerId: 'decode-a',
         dpRank: '0',