Add portable SQLite datasets with build-time dialect translation

README.md

@@ -29,18 +29,18 @@ Each cell is the image tag to pull for that dataset on that engine; **—** mean
 | [AdventureWorks](https://github.com/lorint/AdventureWorks-for-Postgres) | `adventureworks` | — | — | — |
 | [Airlines](https://postgrespro.com/education/demodb) | `airlines` | — | — | — |
 | Chinook | [`yugabyte-chinook`](https://github.com/yugabyte/yugabyte-db/tree/master/sample) | [`chinook`](https://github.com/lerocha/chinook-database) | [`chinook`](https://github.com/yugabyte/yugabyte-db/tree/master/sample) | [`chinook`](https://github.com/lerocha/chinook-database) |
-| [Dell DVD Store](https://www.postgresql.org/ftp/projects/pgFoundry/dbsamples/) | `dellstore` | `dellstore` | — | — |
-| [French Towns](https://www.postgresql.org/ftp/projects/pgFoundry/dbsamples/) | `frenchtowns` | `frenchtowns` | — | — |
-| [ISO 3166](https://www.postgresql.org/ftp/projects/pgFoundry/dbsamples/) | `iso3166` | `iso3166` | — | — |
-| [MoMA](https://github.com/MuseumofModernArt/collection) | `moma` | `moma` | — | — |
+| [Dell DVD Store](https://www.postgresql.org/ftp/projects/pgFoundry/dbsamples/) | `dellstore` | `dellstore` | — | `dellstore` |
+| [French Towns](https://www.postgresql.org/ftp/projects/pgFoundry/dbsamples/) | `frenchtowns` | `frenchtowns` | — | `frenchtowns` |
+| [ISO 3166](https://www.postgresql.org/ftp/projects/pgFoundry/dbsamples/) | `iso3166` | `iso3166` | — | `iso3166` |
+| [MoMA](https://github.com/MuseumofModernArt/collection) | `moma` | `moma` | — | `moma` |
 | Northwind | [`yugabyte-northwind`](https://github.com/yugabyte/yugabyte-db/tree/master/sample) | [`northwind`](https://github.com/dalers/mywind) | [`northwind`](https://github.com/yugabyte/yugabyte-db/tree/master/sample) | [`northwind`](https://github.com/jpwhite3/northwind-SQLite3) |
 | [OMDb](https://github.com/df7cb/omdb-postgresql) | `omdb` | — | — | — |
-| [PGExercises](https://github.com/yugabyte/yugabyte-db/tree/master/sample) | `yugabyte-pgexercises` | `pgexercises` | — | — |
-| Sakila / Pagila | [`pagila`](https://github.com/devrimgunduz/pagila) | [`sakila`](https://dev.mysql.com/doc/sakila/en/) | — | — |
-| [SportsDB](https://github.com/yugabyte/yugabyte-db/tree/master/sample) | `sportsdb`, `yugabyte-sportsdb` | `sportsdb` | — | — |
-| [Stack Exchange](https://archive.org/details/stackexchange)¹ | `stackexchange-<site>` | `stackexchange-<site>` | — | — |
-| [USDA](https://www.postgresql.org/ftp/projects/pgFoundry/dbsamples/) | `usda` | `usda` | — | — |
-| World | [`world`](https://www.postgresql.org/ftp/projects/pgFoundry/dbsamples/) | [`world`](https://dev.mysql.com/doc/world-setup/en/) | — | — |
+| [PGExercises](https://github.com/yugabyte/yugabyte-db/tree/master/sample) | `yugabyte-pgexercises` | `pgexercises` | — | `pgexercises` |
+| Sakila / Pagila | [`pagila`](https://github.com/devrimgunduz/pagila) | [`sakila`](https://dev.mysql.com/doc/sakila/en/) | — | [`sakila`](https://github.com/bradleygrant/sakila-sqlite3) |
+| [SportsDB](https://github.com/yugabyte/yugabyte-db/tree/master/sample) | `sportsdb`, `yugabyte-sportsdb` | `sportsdb` | — | `sportsdb` |
+| [Stack Exchange](https://archive.org/details/stackexchange)¹ | `stackexchange-<site>` | `stackexchange-<site>` | — | `stackexchange-<site>` |
+| [USDA](https://www.postgresql.org/ftp/projects/pgFoundry/dbsamples/) | `usda` | `usda` | — | `usda` |
+| World | [`world`](https://www.postgresql.org/ftp/projects/pgFoundry/dbsamples/) | [`world`](https://dev.mysql.com/doc/world-setup/en/) | — | [`world`](https://www.postgresql.org/ftp/projects/pgFoundry/dbsamples/) |
 
 ¹ `<site>` is one of `beer`, `coffee`, `poker`, `woodworking`, `chess`, `cooking` (e.g. `stackexchange-chess`).
 
@@ -149,6 +149,20 @@ Sources are in the [matrix](#dataset-support-matrix); the notes below are SQLite
 
 * `chinook`: built at image-build time from the vendor's native `Chinook_Sqlite.sql` script (release `v1.4.5`); CamelCase identifiers (`Track`, `InvoiceLine`), with row counts matching the other `chinook` tags exactly.
 * `northwind`: the prebuilt jpwhite3/northwind-SQLite3 database shipped as-is — the port's *expanded* edition, whose `Orders` and especially `"Order Details"` tables carry far more rows than the classic sample, so this image is heavier than the others.
+* `world`: the pgFoundry PostgreSQL `world` dump hand-translated at build time through the shared `sqlite/scripts/pgsql` transform hook (`COPY` → batched `INSERT`s, Postgres-only noise stripped); three tables (`city`, `country`, `countrylanguage`) with row counts matching the other `world` tags exactly.
+* `iso3166`, `frenchtowns`, `usda`, `pgexercises`, `dellstore`, `sportsdb`: same shared `sqlite/scripts/pgsql` hook as `world` — plain PostgreSQL DDL + data dumps rewritten for SQLite at build time. SQLite cannot add constraints via `ALTER TABLE`, so PK/FK/unique constraints from the dump are dropped; tables and row counts still load faithfully (matching the MySQL tags for these datasets).
+* `sakila`: the bradleygrant/sakila-sqlite3 port's prebuilt `sakila_master.db` shipped as-is — stands in for PostgreSQL's `pagila` (16 base tables, MySQL-compatible row counts).
+* `moma`: schema authored in-repo (`sqlite/scripts/moma/schema.sql`, every column `text`); CSVs bulk-loaded at build time with the sqlite3 CLI's `.import` dot-command. Counts drift as MoMA refreshes its exports (recorded as floors).
+* `stackexchange-<site>`: per-table XML converted at build time by a shared hook (`sqlite/scripts/stackexchange`) to `CREATE TABLE` + batched `INSERT`s + indexes (double-quoted CamelCase identifiers). `cooking` is the largest; counts are recorded as floors.
+
+### Datasets not ported to SQLite
+
+The remaining datasets are either sourced from PostgreSQL-only upstreams or rely on PostgreSQL-specific features that can't be hand-translated without diverging from the upstream dataset. Plain DDL + data dumps are instead hand-translated (see the group above); these are the ones that remain PostgreSQL-only:
+
+* `pagila`: not omitted but *replaced* — `pagila` is a port of Sakila to PostgreSQL, and SQLite uses a native Sakila port directly (tag `sakila`, above).
+* `adventureworks`: the only maintained open port targets PostgreSQL; AdventureWorks is a Microsoft SQL Server sample with no comparable, maintained SQLite port, and its build relies on a Python reformat plus multiple schemas and materialized views — too much PostgreSQL-specific machinery to hand-translate faithfully.
+* `airlines`: the [postgrespro demo](https://postgrespro.com/education/demodb) is distributed as a binary-ish PostgreSQL `pg_dump` and leans on PostgreSQL features (`jsonb`, several million inlined rows); it is PostgreSQL-only.
+* `omdb`: [df7cb/omdb-postgresql](https://github.com/df7cb/omdb-postgresql) is PostgreSQL-specific — its views rely on the `tsm_system_rows` extension (no SQLite equivalent), so a port would have to drop them and would no longer be the upstream dataset.
 
 ## Usage
 
@@ -279,6 +293,5 @@ integration tests).
 
 * More MySQL datasets: port additional PostgreSQL datasets where a MySQL-native source exists or the upstream is format-neutral enough to hand-translate faithfully (see [Datasets not ported to MySQL](#datasets-not-ported-to-mysql)).
 * More CockroachDB datasets: most plain DDL + data PostgreSQL dumps should load with little or no change.
-* More SQLite datasets: from native SQLite sources or by building from format-neutral SQL.
 * Images for other popular databases.
 * Find and add more free data sources.

manifest.yml

@@ -566,6 +566,152 @@ contexts:
         structureTest:
           configs:
             - test/config/sqlite/northwind.yaml
+      # Tier 1 hand-translated PostgreSQL datasets. These ship only as
+      # PostgreSQL dumps, but are plain DDL + data (no PL/pgSQL, no extensions),
+      # so the shared scripts/pgsql transform hook rewrites the dialect at build
+      # time (COPY -> INSERT, strip Postgres-only noise). SQLite cannot add
+      # constraints via ALTER TABLE, so PK/FK/unique from the dump are dropped;
+      # tables and row counts still load faithfully. Each dataset's
+      # scripts/<dataset> dir is a symlink to scripts/pgsql.
+      world: &sqlite-world
+        extractUrl: https://ftp.postgresql.org/pub/projects/pgFoundry/dbsamples/world/world-1.0/world-1.0.tar.gz
+        sqlFiles: dbsamples-0.1/world/world.sql
+        extraPrereqs: python3
+        structureTest:
+          configs:
+            - test/config/sqlite/world.yaml
+      iso3166:
+        extractUrl: https://ftp.postgresql.org/pub/projects/pgFoundry/dbsamples/iso-3166/iso-3166-1.0/iso-3166-1.0.tar.gz
+        sqlFiles: iso-3166/iso-3166.sql
+        extraPrereqs: python3
+        structureTest:
+          configs:
+            - test/config/sqlite/iso3166.yaml
+      frenchtowns:
+        extractUrl: https://ftp.postgresql.org/pub/projects/pgFoundry/dbsamples/french-towns-communes-francais/french-towns-communes-francaises-1.0/french-towns-communes-francaises-1.0.tar.gz
+        sqlFiles: french-towns-communes-francaises.sql
+        extraPrereqs: python3
+        structureTest:
+          configs:
+            - test/config/sqlite/frenchtowns.yaml
+      usda:
+        extractUrl: https://ftp.postgresql.org/pub/projects/pgFoundry/dbsamples/usda/usda-r18-1.0/usda-r18-1.0.tar.gz
+        sqlFiles: usda-r18-1.0/usda.sql
+        extraPrereqs: python3
+        structureTest:
+          configs:
+            - test/config/sqlite/usda.yaml
+      pgexercises:
+        extractUrl: >-
+          https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/sample/clubdata_ddl.sql
+          https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/sample/clubdata_data.sql
+        sqlFiles: clubdata_ddl.sql clubdata_data.sql
+        extraPrereqs: python3
+        structureTest:
+          configs:
+            - test/config/sqlite/pgexercises.yaml
+      # Tier 2 hand-translated PostgreSQL datasets. Larger / slightly more
+      # involved than tier 1 but still handled by the shared scripts/pgsql hook:
+      # sportsdb is INSERT-based across five files with sequences and constraints;
+      # dellstore2 is COPY-based plus a PL/pgSQL stored function, which the hook
+      # drops (no mechanical SQLite translation -- the schema and all data still
+      # load faithfully).
+      dellstore:
+        extractUrl: https://ftp.postgresql.org/pub/projects/pgFoundry/dbsamples/dellstore2/dellstore2-normal-1.0/dellstore2-normal-1.0.tar.gz
+        sqlFiles: dellstore2-normal-1.0/dellstore2-normal-1.0.sql
+        extraPrereqs: python3
+        structureTest:
+          configs:
+            - test/config/sqlite/dellstore.yaml
+      sportsdb:
+        extractUrl: >-
+          https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/sample/sportsdb_tables.sql
+          https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/sample/sportsdb_inserts.sql
+          https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/sample/sportsdb_indexes.sql
+          https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/sample/sportsdb_constraints.sql
+          https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/sample/sportsdb_fks.sql
+        sqlFiles: sportsdb_tables.sql sportsdb_inserts.sql sportsdb_indexes.sql sportsdb_constraints.sql sportsdb_fks.sql
+        extraPrereqs: python3
+        structureTest:
+          configs:
+            - test/config/sqlite/sportsdb.yaml
+      # Sakila: MySQL's own sample database (the original of pagila). The
+      # bradleygrant/sakila-sqlite3 port publishes a prebuilt database file,
+      # shipped as-is (mirrors the northwind dbFile pattern).
+      sakila:
+        extractUrl: https://raw.githubusercontent.com/bradleygrant/sakila-sqlite3/main/sakila_master.db
+        dbFile: sakila_master.db
+        structureTest:
+          configs:
+            - test/config/sqlite/sakila.yaml
+      # MoMA research collection. Published only as CSV, so the schema is
+      # authored in-repo (sqlite/scripts/moma) and the CSVs are bulk-loaded at
+      # build time with the sqlite3 CLI's .import dot-command. Counts drift
+      # between builds (the CSVs are refreshed upstream), so the smoke test
+      # records floors.
+      moma:
+        extractUrl: >-
+          https://media.githubusercontent.com/media/MuseumofModernArt/collection/main/Artworks.csv
+          https://media.githubusercontent.com/media/MuseumofModernArt/collection/main/Artists.csv
+        sqlFiles: moma.sql
+        structureTest:
+          configs:
+            - test/config/sqlite/moma.yaml
+      # StackExchange sites. Each ships as per-table XML in a 7z archive (no
+      # SQL), so -- exactly like the postgres/mysql side -- every site builds
+      # through the shared scripts/stackexchange hooks (each site's scripts/<site>
+      # dir is a symlink to it): the extract hook un-7z's the dump and the
+      # transform hook converts the XML to a SQLite init script (CREATE TABLE +
+      # batched INSERTs + indexes). Adding a site is just a tag here + a
+      # structure-test config + an expected/sqlite/<site>.json floor file.
+      stackexchange-beer:
+        dataset: beer
+        extractUrl: https://archive.org/download/stackexchange/beer.stackexchange.com.7z
+        sqlFiles: beer.sql
+        extraPrereqs: p7zip python3 py3-lxml
+        structureTest:
+          configs:
+            - test/config/sqlite/stackexchange-beer.yaml
+      stackexchange-coffee:
+        dataset: coffee
+        extractUrl: https://archive.org/download/stackexchange/coffee.stackexchange.com.7z
+        sqlFiles: coffee.sql
+        extraPrereqs: p7zip python3 py3-lxml
+        structureTest:
+          configs:
+            - test/config/sqlite/stackexchange-coffee.yaml
+      stackexchange-poker:
+        dataset: poker
+        extractUrl: https://archive.org/download/stackexchange/poker.stackexchange.com.7z
+        sqlFiles: poker.sql
+        extraPrereqs: p7zip python3 py3-lxml
+        structureTest:
+          configs:
+            - test/config/sqlite/stackexchange-poker.yaml
+      stackexchange-woodworking:
+        dataset: woodworking
+        extractUrl: https://archive.org/download/stackexchange/woodworking.stackexchange.com.7z
+        sqlFiles: woodworking.sql
+        extraPrereqs: p7zip python3 py3-lxml
+        structureTest:
+          configs:
+            - test/config/sqlite/stackexchange-woodworking.yaml
+      stackexchange-chess:
+        dataset: chess
+        extractUrl: https://archive.org/download/stackexchange/chess.stackexchange.com.7z
+        sqlFiles: chess.sql
+        extraPrereqs: p7zip python3 py3-lxml
+        structureTest:
+          configs:
+            - test/config/sqlite/stackexchange-chess.yaml
+      stackexchange-cooking:
+        dataset: cooking
+        extractUrl: https://archive.org/download/stackexchange/cooking.stackexchange.com.7z
+        sqlFiles: cooking.sql
+        extraPrereqs: p7zip python3 py3-lxml
+        structureTest:
+          configs:
+            - test/config/sqlite/stackexchange-cooking.yaml
       # `latest` tracks a single representative dataset, mirroring postgres/mysql.
       latest:
         <<: *sqlite-chinook

sqlite/scripts/beer

@@ -0,0 +1 @@
+stackexchange

sqlite/scripts/chess

@@ -0,0 +1 @@
+stackexchange

sqlite/scripts/coffee

@@ -0,0 +1 @@
+stackexchange

sqlite/scripts/cooking

@@ -0,0 +1 @@
+stackexchange

sqlite/scripts/dellstore

@@ -0,0 +1 @@
+pgsql

sqlite/scripts/frenchtowns

@@ -0,0 +1 @@
+pgsql

sqlite/scripts/iso3166

@@ -0,0 +1 @@
+pgsql

sqlite/scripts/moma/schema.sql

@@ -0,0 +1,54 @@
+-- The Museum of Modern Art (MoMA) research collection dataset, SQLite flavour.
+-- https://github.com/MuseumofModernArt/collection
+--
+-- MoMA publishes only CSV/JSON (no SQL), so this schema is authored here and
+-- staged into the build dir by the moma transform hook. Every column is text:
+-- the published CSVs carry free-form values (approximate dates like "c. 1950",
+-- blank measurements, multi-valued ConstituentID) that don't map onto stricter
+-- types. Data is bulk-loaded by the sqlite3 CLI's .import dot-command; the CSVs
+-- are fetched alongside this script by EXTRACT_URL.
+
+CREATE TABLE artists (
+  constituent_id text,
+  display_name   text,
+  artist_bio     text,
+  nationality    text,
+  gender         text,
+  begin_date     text,
+  end_date       text,
+  wiki_qid       text,
+  ulan           text
+);
+
+CREATE TABLE artworks (
+  title            text,
+  artist           text,
+  constituent_id   text,
+  artist_bio       text,
+  nationality      text,
+  begin_date       text,
+  end_date         text,
+  gender           text,
+  date             text,
+  medium           text,
+  dimensions       text,
+  credit_line      text,
+  accession_number text,
+  classification   text,
+  department       text,
+  date_acquired    text,
+  cataloged        text,
+  object_id        text,
+  url              text,
+  image_url        text,
+  on_view          text,
+  circumference_cm text,
+  depth_cm         text,
+  diameter_cm      text,
+  height_cm        text,
+  length_cm        text,
+  weight_kg        text,
+  width_cm         text,
+  seat_height_cm   text,
+  duration_sec     text
+);

sqlite/scripts/moma/transform

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+# MoMA publishes no SQL, so stage the authored SQLite schema (shipped alongside
+# these build scripts) into the build dir as moma.sql and append sqlite3 CLI
+# dot-commands to bulk-import the CSVs fetched by EXTRACT_URL.
+set -e
+cp /usr/local/lib/scripts/moma/schema.sql moma.sql
+cat >>moma.sql <<'EOF'
+
+.mode csv
+.import --skip 1 Artists.csv artists
+.import --skip 1 Artworks.csv artworks
+EOF

sqlite/scripts/pgexercises

@@ -0,0 +1 @@
+pgsql