Add more datasets for CockroachDB and decrease test flakiness - #5
Merged
Conversation
Ship 14 additional aa8y/cockroach-dataset tags with build transforms for pgFoundry Latin-1 dumps and sportsdb domains, plus README updates for ClickHouse, DuckDB, Druid, and Pinot in Future Work. Co-authored-by: Cursor <cursoragent@cursor.com>
The CockroachDB stackexchange transform was copied from the Postgres one and kept its `CREATE INDEX ... USING hash (...)` declarations. CockroachDB has no hash access method (its `USING HASH` is a differently-positioned hash-sharded index for a different purpose), so the init script failed with "unimplemented: this syntax", aborting container init -- which surfaced as the readiness timeout. Drop `USING hash` so those become plain B-tree indexes, which serve the equality lookups fine, and document the divergence from the Postgres hook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Loading the larger StackExchange sites (chess) through INSERT statements ran every row down CockroachDB's SQL write path and blew past the 300s test readiness budget -- chess took ~800s locally. Have the transform emit a per-table CSV plus an IMPORT INTO ... CSV DATA over nodelocal://, CockroachDB's bulk path (it writes SSTs directly). The final image pins the node's external-IO dir to /csv (--external-io-dir) where the build copies the CSVs. NULLs use the \N sentinel matched by WITH nullif, which keeps empty-string TEXT distinct from NULL (verified against v25.4.11). chess now loads in ~15s end-to-end. Update the stackexchange structure tests to assert IMPORT INTO rather than INSERT INTO. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pgFoundry dumps (usda, dellstore, world, ...) wrap their load in BEGIN/COMMIT and end with a table-less ANALYZE. CockroachDB rejects a bare ANALYZE as a syntax error, which aborts the whole init script (`cockroach sql -f` exits non-zero, the entrypoint stops, and the container never reports ready -- surfacing as a misleading "did not become ready in time"). Extend the shared pgfoundry transform to drop these transaction-control and maintenance statements (CockroachDB gathers statistics and reclaims space automatically). The line filter is now COPY-aware so the broadened pattern only matches top-level statements, never a data row that happens to begin with one of these words. Verified: usda (72s) and dellstore (21s) integration tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Yugabyte sportsdb dump is ~79k single-row INSERT statements. On CockroachDB each implicit-transaction INSERT pays a full Raft commit (an fsync), so loading them one at a time runs for many minutes and overran the 300s test readiness budget on the slower GHA runners. Extend the sportsdb transform to coalesce consecutive INSERTs that share an INSERT INTO <table> (<cols>) prefix into batched multi-row INSERTs (BATCH=500), collapsing those ~79k commits into a few hundred -- the same speed-up the pgfoundry hook applies to COPY blocks. The CREATE DOMAIN drop is unchanged. Verified: 79,138 rows preserved exactly; integration test passes in well under budget (107 tables, matching counts). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
chinook ships its data as ~15.6k single-row INSERTs (the same anti-pattern as
sportsdb's ~79k) -- slow on CockroachDB, where each implicit-transaction INSERT
costs a Raft commit. Generalize sportsdb's coalescing transform to also handle
quoted/schema-qualified identifiers ("Track", public.categories) and promote it
to a shared scripts/yugabyte hook; sportsdb and chinook both symlink to it
(mirroring the stackexchange/pgfoundry shared hooks). The CREATE DOMAIN drop is
a no-op for dumps without one.
chinook gains extraPrereqs: python3 for the transform. northwind and
pgexercises already ship multi-row INSERTs, so they don't symlink here.
Verified: chinook (15.6k rows -> 352 statements, 8s) and sportsdb (79k rows,
107 tables) integration tests pass with matching counts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MoMA is the repo's largest dataset (~176k rows) and was baked into the init script as multi-row INSERTs. Normalize its messy CSV exports (variable column counts, embedded newlines) into clean per-table CSVs at build time and load them with IMPORT INTO instead -- CockroachDB's bulk path, the same as the stackexchange hook -- cutting the load to a few seconds. Every schema column is text and an empty cell means NULL, so the IMPORT uses WITH nullif = '' (unquoted empty field -> NULL), preserving the previous \copy/INSERT semantics. The normalized CSVs ship in the image's external-IO dir (/csv) that --external-io-dir already pins. Verified: moma integration passes in ~3s, 2 tables with matching counts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.