diff --git a/CLAUDE.md b/CLAUDE.md index 951fc07..17b1c0f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -44,7 +44,7 @@ This is a Maven-based Java 25 project. Run `./mvnw` from the repository root to - `TableFiller`: Handles filling individual tables with generated data 2. **Database Support (`io.bloviate.ext`)**: - - Database-specific implementations: `PostgresSupport`, `MySQLSupport`, `CockroachDBSupport`, `DefaultSupport` + - Database-specific implementations: `PostgresSupport`, `MySQLSupport`, `MariaDBSupport`, `CockroachDBSupport`, `H2Support`, `SQLiteSupport`, `BigQuerySupport`, `DefaultSupport` - Each provides database-specific SQL generation and data type mapping 3. **Data Generators (`io.bloviate.gen`)**: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 00d9b40..182b1cc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -132,8 +132,40 @@ Docker must be running. ``` Test schemas live under `bloviate-core/src/test/resources/` (TPCC, AuctionMark, Wikipedia, and -others). `BaseDatabaseTestCase` provides the shared `DataSource` plumbing and the -fidelity assertions used by the TPC-C tests. +others). `BaseDatabaseTestCase` provides the shared `DataSource` plumbing, the classpath script +runner, and the fidelity assertions used by the TPC-C tests. + +### BigQuery + +`BigQueryFillerTest` is the one test Docker cannot cover. BigQuery has no usable emulator — the +tbc-bq-jdbc driver deliberately removed its emulator tier because the emulator diverged far enough +from the service to hide real defects — so the test needs a live Google Cloud project, and it is +**skipped by default**. + +It is gated twice, and both gates must pass: + +1. `BLOVIATE_BQ_PROJECT` is set (with Application Default Credentials available), and +2. the driver is on the classpath, which only happens under `-Pbigquery`. + +The second gate exists so that setting the env var without the profile skips cleanly instead of +failing with "No suitable driver". `vc.tbc:tbc-bq-jdbc` is not on Maven Central yet, which is why it +is declared in an opt-in profile rather than as an ordinary test dependency — a default build must +stay resolvable for everyone. Bump `tbc-bq-jdbc.version` in the root POM by hand; Dependabot and +`versions:display-dependency-updates` cannot resolve that coordinate. + +```bash +# once, in a clone of https://github.com/Two-Bear-Capital/tbc-bq-jdbc +./mvnw clean install + +gcloud auth application-default login +export BLOVIATE_BQ_PROJECT=my-gcp-project + +./mvnw verify -Pbigquery -pl bloviate-core -Dtest=BigQueryFillerTest +``` + +Each run creates its own dataset and drops it afterwards (with a one-day default table expiration as +a backstop), because `DatabaseFiller` fills **every** table it finds in the connection's schema. +Running it writes real data and runs real jobs, both of which cost money. ## Databases for Testing diff --git a/README.md b/README.md index c4d5511..b9b15d7 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ by seed, and runs inside your JUnit/Testcontainers pipeline. **Full documentation, guides, and examples live at [bloviate.io](https://bloviate.io).** - [Quick Start](https://bloviate.io/guides/quickstart/) — install and fill a database or flat file -- [Database Support](https://bloviate.io/guides/database-support/) — PostgreSQL, MySQL, MariaDB, CockroachDB, H2, SQLite +- [Database Support](https://bloviate.io/guides/database-support/) — PostgreSQL, MySQL, MariaDB, CockroachDB, H2, SQLite, BigQuery - [Configuration](https://bloviate.io/guides/configuration/) — per-table/column control, distributions, seeds, parallelism - [Generators](https://bloviate.io/guides/generators/) — registry, realistic data, composite keys, TPC-C - [Testing Integrations](https://bloviate.io/guides/integrations/) — JUnit Jupiter and Testcontainers @@ -31,7 +31,7 @@ by seed, and runs inside your JUnit/Testcontainers pipeline. - **Deterministic by seed** — same seed + schema ⇒ byte-identical data, even under parallel fills - **Per-column control** and **pluggable generators**; realistic semantic values via Datafaker - **Parallel & partitioned fills** for large datasets, with referential integrity preserved -- **PostgreSQL, MySQL, MariaDB, CockroachDB, H2, SQLite**, plus CSV/TSV/pipe **flat-file** output +- **PostgreSQL, MySQL, MariaDB, CockroachDB, H2, SQLite, BigQuery**, plus CSV/TSV/pipe **flat-file** output - First-class **JUnit Jupiter** (JUnit 5/6) and **Testcontainers** integrations See the [full feature tour and guides on bloviate.io](https://bloviate.io). diff --git a/bloviate-core/pom.xml b/bloviate-core/pom.xml index 77bcd24..a5dcadf 100644 --- a/bloviate-core/pom.xml +++ b/bloviate-core/pom.xml @@ -168,4 +168,30 @@ + + + + bigquery + + + vc.tbc + tbc-bq-jdbc + test + + + + + diff --git a/bloviate-core/src/main/java/io/bloviate/db/DatabaseFiller.java b/bloviate-core/src/main/java/io/bloviate/db/DatabaseFiller.java index 44aa8b2..0164e1e 100644 --- a/bloviate-core/src/main/java/io/bloviate/db/DatabaseFiller.java +++ b/bloviate-core/src/main/java/io/bloviate/db/DatabaseFiller.java @@ -162,6 +162,8 @@ public void fill() throws SQLException { visualizeGraph(reversedGraph, database.catalog()); + warnIfEngineManagedCommitDiscouraged(); + // recommend the driver batch-rewrite URL parameter once per fill if it is missing if (connection != null) { warnIfBatchRewriteMissing(connection); @@ -594,6 +596,23 @@ private void warnIfBulkIgnored() { } } + /** + * Warns once per fill when an explicit commit strategy is configured against a support that + * would rather the engine stayed out of transaction management. The caller's choice is still + * honored — this only surfaces the cost, which is otherwise invisible (on BigQuery, an engine- + * managed transaction opens a session and silently disables the driver's load-job path). + */ + private void warnIfEngineManagedCommitDiscouraged() { + if (configuration.databaseSupport().prefersConnectionDefaultCommit() + && configuration.commitStrategy().managesTransaction()) { + logger.warn("{} recommends leaving transaction management to the connection, but commit " + + "strategy [{}] was configured; the engine will manage transactions as asked, " + + "which may be slower and can disable driver bulk-load paths", + configuration.databaseSupport().getClass().getSimpleName(), + configuration.commitStrategy().mode()); + } + } + /** * The commit strategy used by parallel workers. A pooled worker connection must not be left on * the connection's autocommit (that would commit per batch and lose the engine-managed @@ -603,12 +622,22 @@ private void warnIfBulkIgnored() { * large partition open in one server-side transaction (unbounded WAL/undo growth and lock * accumulation), which is the scale failure the parallel/bulk path most needs to avoid. Any * explicitly configured strategy (including {@link CommitStrategy#perTable()}) is honored as-is. + * + *

A {@link io.bloviate.ext.DatabaseSupport#prefersConnectionDefaultCommit() support that + * prefers the connection's own commit behavior} suppresses the upgrade, so + * {@code CONNECTION_DEFAULT} stays as configured. That is for engines where an engine-managed + * transaction is pure cost rather than protection — see the hook's documentation. + * + *

Package-private so the mapping can be unit-tested without a database. */ - private CommitStrategy effectiveParallelCommitStrategy() { + CommitStrategy effectiveParallelCommitStrategy() { CommitStrategy configured = configuration.commitStrategy(); - return configured.mode() == CommitStrategy.Mode.CONNECTION_DEFAULT - ? CommitStrategy.everyNBatches(DEFAULT_PARALLEL_COMMIT_BATCHES) - : configured; + if (configured.mode() != CommitStrategy.Mode.CONNECTION_DEFAULT) { + return configured; + } + return configuration.databaseSupport().prefersConnectionDefaultCommit() + ? configured + : CommitStrategy.everyNBatches(DEFAULT_PARALLEL_COMMIT_BATCHES); } /** diff --git a/bloviate-core/src/main/java/io/bloviate/ext/BigQuerySupport.java b/bloviate-core/src/main/java/io/bloviate/ext/BigQuerySupport.java new file mode 100644 index 0000000..ee7a2e3 --- /dev/null +++ b/bloviate-core/src/main/java/io/bloviate/ext/BigQuerySupport.java @@ -0,0 +1,310 @@ +/* + * Copyright (c) 2021 Tim Veil + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.bloviate.ext; + +import io.bloviate.db.Column; +import io.bloviate.db.Database; +import io.bloviate.gen.BigDecimalGenerator; +import io.bloviate.gen.ByteGenerator; +import io.bloviate.gen.SimpleStringGenerator; + +import java.sql.Connection; +import java.sql.JDBCType; +import java.util.Locale; +import java.util.Map; + +/** + * Google BigQuery-specific {@link DatabaseSupport}, written against the + * tbc-bq-jdbc driver + * (vc.tbc:tbc-bq-jdbc, 4.3.0 or later). + * + *

BigQuery is an analytical engine, and it diverges from the OLTP databases Bloviate + * otherwise targets in three ways this class has to account for: + * + *

+ * + *

Supported types: {@code STRING}, {@code BYTES}, {@code INT64}, + * {@code FLOAT64}, {@code NUMERIC}, {@code BIGNUMERIC} (within {@code NUMERIC} range), + * {@code BOOL}, {@code DATE}, {@code TIME}, {@code TIMESTAMP}. + * + *

Unsupported types: {@code DATETIME}, {@code JSON}, {@code GEOGRAPHY}, + * {@code INTERVAL}, {@code RANGE}, {@code ARRAY}, {@code STRUCT}. Supply a per-column generator + * through {@code ColumnConfiguration} or + * {@link GeneratorRegistry.Builder#registerColumnNamePattern} to fill them; the driver implements + * {@link Connection#createArrayOf} and {@link Connection#createStruct}, so a hand-written generator + * can write composites today. + * + *

Required driver settings: {@code includeStructFields=false} (the default) and + * {@code metadataLazyLoad=false} (the default). With struct fields spliced in, {@code getColumns} + * emits dotted sub-field rows alongside their parent and the generated {@code INSERT} is + * structurally invalid; with lazy metadata, an unfiltered {@code getTables}/{@code getColumns} + * returns nothing and the fill silently does no work. + * + * @since 3.1.0 + * @see AbstractDatabaseSupport + * @see DatabaseSupport + */ +public class BigQuerySupport extends AbstractDatabaseSupport { + + /** + * Cap on generated {@code STRING} length. A bare {@code STRING} column reports BigQuery's type + * maximum of 2,097,152; a declared {@code STRING(n)} below this cap is honored exactly. + * + *

{@link SimpleStringGenerator} independently caps itself at 2000 characters, so this is an + * additional ~8x reduction on top of that, not the only thing standing between a fill and + * two-megabyte cells. + */ + public static final int MAX_STRING_LENGTH = 256; + + /** + * Cap on generated {@code BYTES} length. A bare {@code BYTES} column reports BigQuery's type + * maximum of 10,485,760; a declared {@code BYTES(n)} below this cap is honored exactly. + * + *

{@link ByteGenerator} independently caps itself at 25 bytes, which is below this value, so + * today this clamp only bites for a declared {@code BYTES(n)} where {@code n} is between 25 and + * 128 — where it changes nothing either. It is kept as a stated bound so the intent + * survives a change to the generator's own cap. + */ + public static final int MAX_BYTES_LENGTH = 128; + + /** Maximum precision of BigQuery's {@code NUMERIC} type. */ + public static final int MAX_NUMERIC_PRECISION = 38; + + /** + * Maximum scale of BigQuery's {@code NUMERIC} type. The driver binds every {@code BigDecimal} + * as {@code NUMERIC}, so this caps {@code BIGNUMERIC} columns too: a {@code BIGNUMERIC} reports + * scale 38, and a value carrying more than 9 fractional digits is rejected as an out-of-range + * {@code NUMERIC} parameter regardless of what the destination column could hold. + */ + public static final int MAX_NUMERIC_SCALE = 9; + + /** Creates the BigQuery support with its default configuration. */ + public BigQuerySupport() { + } + + @Override + protected void configure(Map registry) { + + // STRING, and the three types the driver reads back as VARCHAR but cannot bind. + registry.put(JDBCType.VARCHAR, (column, random) -> switch (baseTypeName(column)) { + case "STRING" -> new SimpleStringGenerator.Builder(random) + .size(clamp(column.maxSize(), MAX_STRING_LENGTH)) + .build(); + case "JSON" -> throw unsupported(column, + "the driver binds strings as STRING and BigQuery will not coerce STRING to JSON"); + case "GEOGRAPHY" -> throw unsupported(column, + "the driver binds strings as STRING and BigQuery will not coerce STRING to GEOGRAPHY"); + case "INTERVAL" -> throw unsupported(column, + "the driver binds strings as STRING and BigQuery will not coerce STRING to INTERVAL"); + default -> throw unsupported(column, "no generator is registered for this type"); + }); + + registry.put(JDBCType.VARBINARY, (column, random) -> { + if (!"BYTES".equals(baseTypeName(column))) { + throw unsupported(column, "no generator is registered for this type"); + } + return new ByteGenerator.Builder(random) + .size(clamp(column.maxSize(), MAX_BYTES_LENGTH)) + .build(); + }); + + // NUMERIC and BIGNUMERIC both arrive as JDBC NUMERIC. Clamping BIGNUMERIC's reported + // (76, 38) down to NUMERIC's (38, 9) is always safe: a NUMERIC-range value is assignable + // to a BIGNUMERIC column, and the driver has no BIGNUMERIC parameter binding anyway. + GeneratorFactory bigDecimal = (column, random) -> { + int precision = clamp(column.maxSize(), MAX_NUMERIC_PRECISION); + Integer reportedScale = column.maxDigits(); + int scale = reportedScale == null || reportedScale < 0 + ? 0 + : Math.min(reportedScale, MAX_NUMERIC_SCALE); + return new BigDecimalGenerator.Builder(random) + .precision(precision) + .digits(Math.min(scale, precision)) + .build(); + }; + registry.put(JDBCType.NUMERIC, bigDecimal); + registry.put(JDBCType.DECIMAL, bigDecimal); + + // TIMESTAMP and DATETIME both arrive as JDBC TIMESTAMP; only TIMESTAMP can be bound. + GeneratorFactory inheritedTimestamp = registry.get(JDBCType.TIMESTAMP); + registry.put(JDBCType.TIMESTAMP, (column, random) -> { + if ("DATETIME".equals(baseTypeName(column))) { + throw unsupported(column, "the driver has no DATETIME parameter type and BigQuery " + + "will not coerce TIMESTAMP to DATETIME"); + } + return inheritedTimestamp.create(column, random); + }); + + // RANGE<...> is the only type the driver maps to OTHER. + registry.put(JDBCType.OTHER, (column, random) -> { + throw unsupported(column, "RANGE values have no JDBC parameter binding"); + }); + + registry.put(JDBCType.ARRAY, (column, random) -> { + throw unsupported(column, "array element generation is not yet supported"); + }); + + // Replace rather than remove the inherited SqlStructGenerator: it cannot know a BigQuery + // struct's shape and would silently generate garbage, and removing it would surface the + // generic "JDBCType [STRUCT] not supported" instead of the actionable message below. + registry.put(JDBCType.STRUCT, (column, random) -> { + throw unsupported(column, "a struct's field shape is not derivable from JDBC metadata"); + }); + } + + /** + * Enabled because BigQuery's {@code PRIMARY KEY}/{@code FOREIGN KEY} constraints are always + * {@code NOT ENFORCED}: there is no enforcement to suspend, so an unordered bulk fill needs no + * preparation and carries no correctness risk. Referential consistency does not depend on + * insert order either — a foreign-key column is seeded from its parent primary-key + * column's seed and replays the same value sequence whenever it is filled. + * + * @return always {@code true} + */ + @Override + public boolean supportsBulkLoad() { + return true; + } + + /** + * A no-op: BigQuery never enforces key constraints, so there is nothing to disable. This is + * deliberate, not a stub — it exists so {@link #supportsBulkLoad()} can return + * {@code true} and unlock the unordered fill path. The connection is not touched. + * + * @param connection ignored + * @param database ignored + * @return a handle recording that nothing was disabled + */ + @Override + public BulkLoadHandle disableConstraints(Connection connection, Database database) { + return BulkLoadHandle.of("no-op: BigQuery PRIMARY KEY/FOREIGN KEY are NOT ENFORCED"); + } + + /** + * A no-op, mirroring {@link #disableConstraints}. The connection is not touched. + * + * @param connection ignored + * @param database ignored + * @param handle ignored + */ + @Override + public void enableConstraints(Connection connection, Database database, BulkLoadHandle handle) { + // nothing was disabled; see disableConstraints + } + + /** + * Enabled because an engine-managed transaction is pure cost on BigQuery: each + * {@code executeBatch} is already a single atomic query job, while {@code setAutoCommit(false)} + * lazily starts a BigQuery session (per-connection overhead and quota) and disables the + * driver's NDJSON load-job path, which is the fastest way to bulk-insert. + * + * @return always {@code true} + */ + @Override + public boolean prefersConnectionDefaultCommit() { + return true; + } + + /** + * Returns {@code null}: rewriting a batch into a multi-row {@code INSERT} is unconditional in + * tbc-bq-jdbc, so there is no URL parameter to recommend and the fill engine should stay quiet + * rather than warn about a missing one. + * + *

This overrides nothing behaviorally — the interface default is already {@code null} + * — but it is stated explicitly so the reasoning is recorded where someone would look for + * it. In particular, load-job tuning ({@code batchLoadThreshold}) does not belong here: + * this hook's contract is a batch-rewrite toggle, and that property takes an integer, so + * advertising it would make the engine emit wrong advice. + * + * @return always {@code null} + */ + @Override + public String batchRewriteUrlParameter() { + return null; + } + + /** + * Reduces a BigQuery {@code TYPE_NAME} to its base type. The driver reports the raw + * {@code INFORMATION_SCHEMA.COLUMNS.data_type} text, so names arrive parameterized + * ({@code STRING(20)}, {@code NUMERIC(10, 2)}) or with type arguments ({@code ARRAY}, + * {@code STRUCT}, {@code RANGE}). + * + * @param column the column whose type name to normalize + * @return the upper-cased base type name, or an empty string when unreported + */ + private static String baseTypeName(Column column) { + String typeName = column.typeName(); + if (typeName == null || typeName.isBlank()) { + return ""; + } + String upper = typeName.trim().toUpperCase(Locale.ROOT); + int cut = upper.length(); + int paren = upper.indexOf('('); + if (paren >= 0) { + cut = paren; + } + int angle = upper.indexOf('<'); + if (angle >= 0 && angle < cut) { + cut = angle; + } + return upper.substring(0, cut).trim(); + } + + /** + * Clamps a reported size downward to a cap. Clamping only ever shortens, so a generated value + * always remains valid for the declared column; an unreported or non-positive size falls back + * to the cap. + * + * @param reported the size reported by {@code getColumns}, may be null + * @param cap the maximum value to allow + * @return the clamped size + */ + private static int clamp(Integer reported, int cap) { + if (reported == null || reported <= 0) { + return cap; + } + return Math.min(reported, cap); + } + + private static UnsupportedOperationException unsupported(Column column, String reason) { + return new UnsupportedOperationException(String.format( + "BigQuery type [%s] on column [%s.%s] cannot be filled by a plain parameter binding: %s. " + + "Supply a per-column generator via ColumnConfiguration (or " + + "GeneratorRegistry.registerColumnNamePattern) — see docs/DATABASE_SUPPORT.md#bigquery.", + column.typeName(), column.tableName(), column.name(), reason)); + } +} diff --git a/bloviate-core/src/main/java/io/bloviate/ext/DatabaseSupport.java b/bloviate-core/src/main/java/io/bloviate/ext/DatabaseSupport.java index 5847a80..481fae4 100644 --- a/bloviate-core/src/main/java/io/bloviate/ext/DatabaseSupport.java +++ b/bloviate-core/src/main/java/io/bloviate/ext/DatabaseSupport.java @@ -98,6 +98,33 @@ default Map readConstraints(Connection connection, Str return Map.of(); } + /** + * Whether the fill engine should leave transaction management to the connection rather than + * imposing one of its own. The default is {@code false}: on a parallel or unordered fill, + * {@link io.bloviate.db.DatabaseFiller} upgrades a + * {@link io.bloviate.db.CommitStrategy.Mode#CONNECTION_DEFAULT} strategy to a periodic commit so + * a long-running partition never sits in one unbounded transaction. A support that returns + * {@code true} suppresses that upgrade, so {@code CONNECTION_DEFAULT} stays + * {@code CONNECTION_DEFAULT} on every path and the engine never touches auto-commit. + * + *

This exists for engines where an engine-managed transaction is pure cost. On BigQuery each + * {@code executeBatch} is already a single atomic query job, while {@code setAutoCommit(false)} + * lazily starts a session (per-connection overhead and quota) and disables the driver's fastest + * bulk-insert path. Analytical engines without real transaction support are the other case. + * + *

An explicitly configured commit strategy always wins: this hook only affects the + * engine's own default, never a choice the caller made. It is also only effective if the pool + * leaves auto-commit on — a {@link javax.sql.DataSource} configured with + * {@code autoCommit=false} still hands out connections in a transaction, which Bloviate cannot + * override. + * + * @return whether the engine should defer to the connection's own commit behavior + * @since 3.1.0 + */ + default boolean prefersConnectionDefaultCommit() { + return false; + } + /** * Whether this support can disable and re-enable foreign-key enforcement for an * {@code UNORDERED_BULK} fill (see {@link io.bloviate.db.BulkLoadStrategy}). The default is @@ -164,9 +191,9 @@ default void enableConstraints(Connection connection, Database database, BulkLoa *

Matching is case-insensitive and substring-based: names containing * {@code "cockroach"} map to {@link CockroachDBSupport}, {@code "mariadb"} to * {@link MariaDBSupport}, {@code "mysql"} to {@link MySQLSupport}, {@code "postgres"} to - * {@link PostgresSupport}, {@code "h2"} to {@link H2Support}, and {@code "sqlite"} to - * {@link SQLiteSupport}. Anything else (including {@code null}) falls back to - * {@link DefaultSupport}. + * {@link PostgresSupport}, {@code "h2"} to {@link H2Support}, {@code "sqlite"} to + * {@link SQLiteSupport}, and {@code "bigquery"} to {@link BigQuerySupport}. Anything else + * (including {@code null}) falls back to {@link DefaultSupport}. * *

CockroachDB note: CockroachDB is typically reached through the * PostgreSQL JDBC driver, which reports its product name as {@code "PostgreSQL"}, so such @@ -182,6 +209,12 @@ default void enableConstraints(Connection connection, Database database, BulkLoa * {@link MariaDBSupport} extends {@link MySQLSupport} and the inherited type handling * works against MariaDB. * + *

BigQuery note: {@link BigQuerySupport} is written against the + * tbc-bq-jdbc driver, which reports {@code "BigQuery (TBC Driver)"}. The substring also matches + * Simba's {@code "Google BigQuery"}, which is usually what you want — but that driver + * reports type names differently, and {@link BigQuerySupport} discriminates on the raw + * {@code INFORMATION_SCHEMA} type text, so some columns may not resolve. + * * @param productName the database product name, may be null * @return the matching support, or {@link DefaultSupport} if none matches */ @@ -206,6 +239,9 @@ static DatabaseSupport forProduct(String productName) { if (name.contains("h2")) { return new H2Support(); } + if (name.contains("bigquery")) { + return new BigQuerySupport(); + } } return new DefaultSupport(); } diff --git a/bloviate-core/src/test/java/io/bloviate/db/BaseDatabaseTestCase.java b/bloviate-core/src/test/java/io/bloviate/db/BaseDatabaseTestCase.java index a2fe27f..be6e13d 100644 --- a/bloviate-core/src/test/java/io/bloviate/db/BaseDatabaseTestCase.java +++ b/bloviate-core/src/test/java/io/bloviate/db/BaseDatabaseTestCase.java @@ -22,10 +22,17 @@ import org.testcontainers.containers.JdbcDatabaseContainer; import javax.sql.DataSource; +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; import java.sql.Connection; import java.sql.ResultSet; import java.sql.SQLException; import java.sql.Statement; +import java.util.Map; import static org.junit.jupiter.api.Assertions.assertEquals; @@ -49,6 +56,56 @@ protected interface Verifier { void verify(Connection connection) throws SQLException; } + /** + * Executes a semicolon-delimited SQL script loaded from the test classpath. Line comments + * ({@code --}) are stripped; statements are split on {@code ;} at end of line. + */ + protected static void runScript(Connection connection, String resource) throws SQLException { + runScript(connection, resource, Map.of()); + } + + /** + * As {@link #runScript(Connection, String)}, but first replaces {@code ${name}} tokens in the + * script with the supplied values. Databases without transactional DDL or throwaway instances + * (BigQuery) need per-run table names so repeated or concurrent runs cannot collide. + */ + protected static void runScript(Connection connection, String resource, Map tokens) throws SQLException { + String sql = readResource(resource); + for (Map.Entry token : tokens.entrySet()) { + sql = sql.replace("${" + token.getKey() + "}", token.getValue()); + } + try (Statement statement = connection.createStatement()) { + for (String stmt : sql.split(";\\s*\\n")) { + String trimmed = stmt.strip(); + if (!trimmed.isEmpty()) { + statement.execute(trimmed); + } + } + } + } + + private static String readResource(String resource) { + try (InputStream in = BaseDatabaseTestCase.class.getClassLoader().getResourceAsStream(resource)) { + if (in == null) { + throw new IllegalArgumentException("init script not found on classpath: " + resource); + } + StringBuilder builder = new StringBuilder(); + try (BufferedReader reader = new BufferedReader(new InputStreamReader(in, StandardCharsets.UTF_8))) { + String line; + while ((line = reader.readLine()) != null) { + int comment = line.indexOf("--"); + if (comment >= 0) { + line = line.substring(0, comment); + } + builder.append(line).append('\n'); + } + } + return builder.toString(); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + protected static void assertRowCount(Connection connection, String table, long expected) throws SQLException { assertCount(connection, "select count(*) from " + table, expected); } diff --git a/bloviate-core/src/test/java/io/bloviate/db/BaseEmbeddedTest.java b/bloviate-core/src/test/java/io/bloviate/db/BaseEmbeddedTest.java index 0bec12b..56df6c4 100644 --- a/bloviate-core/src/test/java/io/bloviate/db/BaseEmbeddedTest.java +++ b/bloviate-core/src/test/java/io/bloviate/db/BaseEmbeddedTest.java @@ -16,16 +16,9 @@ package io.bloviate.db; -import java.io.BufferedReader; -import java.io.IOException; -import java.io.InputStream; -import java.io.InputStreamReader; -import java.io.UncheckedIOException; -import java.nio.charset.StandardCharsets; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; -import java.sql.Statement; /** * Base class for embedded, in-process database tests (H2, SQLite) that need no Docker. Subclasses @@ -55,41 +48,4 @@ protected void fillDatabase(String jdbcUrl, String user, String password, String } } - /** - * Executes a semicolon-delimited SQL script loaded from the test classpath. Line comments - * ({@code --}) are stripped; statements are split on {@code ;} at end of line. - */ - protected static void runScript(Connection connection, String resource) throws SQLException { - String sql = readResource(resource); - try (Statement statement = connection.createStatement()) { - for (String stmt : sql.split(";\\s*\\n")) { - String trimmed = stmt.strip(); - if (!trimmed.isEmpty()) { - statement.execute(trimmed); - } - } - } - } - - private static String readResource(String resource) { - try (InputStream in = BaseEmbeddedTest.class.getClassLoader().getResourceAsStream(resource)) { - if (in == null) { - throw new IllegalArgumentException("init script not found on classpath: " + resource); - } - StringBuilder builder = new StringBuilder(); - try (BufferedReader reader = new BufferedReader(new InputStreamReader(in, StandardCharsets.UTF_8))) { - String line; - while ((line = reader.readLine()) != null) { - int comment = line.indexOf("--"); - if (comment >= 0) { - line = line.substring(0, comment); - } - builder.append(line).append('\n'); - } - } - return builder.toString(); - } catch (IOException e) { - throw new UncheckedIOException(e); - } - } } diff --git a/bloviate-core/src/test/java/io/bloviate/db/BigQueryFillerTest.java b/bloviate-core/src/test/java/io/bloviate/db/BigQueryFillerTest.java new file mode 100644 index 0000000..472d2b2 --- /dev/null +++ b/bloviate-core/src/test/java/io/bloviate/db/BigQueryFillerTest.java @@ -0,0 +1,168 @@ +/* + * Copyright (c) 2021 Tim Veil + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.bloviate.db; + +import io.bloviate.ext.BigQuerySupport; +import io.bloviate.util.DatabaseUtils; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIf; + +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.Locale; +import java.util.Map; +import java.util.UUID; + +import static org.junit.jupiter.api.Assertions.assertFalse; + +/** + * Fills a real BigQuery dataset. There is no emulator — the tbc-bq-jdbc driver deliberately dropped + * its emulator tier because the emulator's semantics diverged far enough to hide real defects — so + * this test needs a live Google Cloud project and is skipped by default. + * + *

It is gated twice, and both gates must pass for it to run: + *

    + *
  1. {@code BLOVIATE_BQ_PROJECT} is set (and Application Default Credentials are available); and
  2. + *
  3. the driver is on the classpath, which only happens under {@code -Pbigquery}.
  4. + *
+ * The second gate matters on its own: with the env var set but the profile off, the test skips + * rather than failing with "No suitable driver". Nothing here imports a driver class, so it + * compiles in every build. + * + *
+ * ./mvnw clean install                        # once, in the tbc-bq-jdbc repo
+ * gcloud auth application-default login
+ * export BLOVIATE_BQ_PROJECT=my-gcp-project
+ * ./mvnw verify -Pbigquery -pl bloviate-core -Dtest=BigQueryFillerTest
+ * 
+ * + *

Each run creates its own dataset and drops it afterwards, because {@link DatabaseFiller} fills + * every table it finds in the connection's schema. The dataset also carries a one-day + * default table expiration so an aborted run cannot leave billable tables behind. + */ +@EnabledIf("bigQueryAvailable") +class BigQueryFillerTest extends BaseDatabaseTestCase { + + private static final String PROJECT_ENV = "BLOVIATE_BQ_PROJECT"; + private static final String DRIVER_CLASS = "vc.tbc.bq.jdbc.BQDriver"; + + private static final int BATCH_SIZE = 500; + private static final long ROW_COUNT = 25; + + @SuppressWarnings("unused") // referenced by @EnabledIf + static boolean bigQueryAvailable() { + if (System.getenv(PROJECT_ENV) == null) { + return false; + } + try { + Class.forName(DRIVER_CLASS); + return true; + } catch (ClassNotFoundException e) { + return false; + } + } + + @Test + void fillsScalarTypesAndFollowsUnenforcedForeignKeys() throws SQLException { + String project = System.getenv(PROJECT_ENV); + String runId = UUID.randomUUID().toString().replace("-", "").substring(0, 8).toLowerCase(Locale.ROOT); + String dataset = "bloviate_it_" + runId; + String url = String.format("jdbc:bigquery:%s/%s?authType=ADC", project, dataset); + + DatabaseConfiguration configuration = new DatabaseConfiguration.Builder( + BATCH_SIZE, ROW_COUNT, new BigQuerySupport()).build(); + + try (Connection connection = DriverManager.getConnection(url)) { + createDataset(connection, project, dataset); + try { + runScript(connection, "create_tables.bigquery.sql", + Map.of("dataset", dataset, "suffix", runId)); + + assertUnenforcedForeignKeysAreVisible(connection, runId); + + new DatabaseFiller.Builder(connection, configuration).build().fill(); + + verify(connection, dataset, runId); + } finally { + dropDataset(connection, project, dataset); + } + } + } + + private static void createDataset(Connection connection, String project, String dataset) throws SQLException { + try (Statement statement = connection.createStatement()) { + statement.execute(String.format( + "CREATE SCHEMA IF NOT EXISTS `%s.%s` OPTIONS(default_table_expiration_days = 1)", + project, dataset)); + } + } + + private static void dropDataset(Connection connection, String project, String dataset) throws SQLException { + try (Statement statement = connection.createStatement()) { + statement.execute(String.format("DROP SCHEMA IF EXISTS `%s.%s` CASCADE", project, dataset)); + } + } + + /** + * The single highest-value assertion here: BigQuery's keys are always {@code NOT ENFORCED}, and + * this proves the driver still surfaces them through {@code getImportedKeys} so Bloviate's + * dependency graph and foreign-key value alignment have something to work with. Asserted before + * the fill so a metadata regression is not mistaken for a data problem. + */ + private static void assertUnenforcedForeignKeysAreVisible(Connection connection, String runId) throws SQLException { + Database database = DatabaseUtils.getMetadata(connection); + Table nation = database.getTable("nation_" + runId); + + assertFalse(nation.foreignKeys().isEmpty(), + "expected a NOT ENFORCED foreign key on nation_" + runId + " to be visible via getImportedKeys"); + } + + private static void verify(Connection connection, String dataset, String runId) throws SQLException { + String region = String.format("`%s.region_%s`", dataset, runId); + String nation = String.format("`%s.nation_%s`", dataset, runId); + String standard = String.format("`%s.standard_types_%s`", dataset, runId); + String events = String.format("`%s.events_%s`", dataset, runId); + + assertRowCount(connection, region, ROW_COUNT); + assertRowCount(connection, nation, ROW_COUNT); + assertRowCount(connection, standard, ROW_COUNT); + assertRowCount(connection, events, ROW_COUNT); + + // foreign-key columns are seeded from their parent primary-key column, so every child value + // must resolve even though BigQuery never enforces the constraint + assertCount(connection, String.format( + "select count(*) from %s n left join %s r on n.n_regionkey = r.r_regionkey where r.r_regionkey is null", + nation, region), 0); + + // the clamps held against the real service: a bare STRING column reports a 2 MB maximum and + // a bare NUMERIC reports scale 9, neither of which should reach the wire unmodified + assertCount(connection, String.format( + "select count(*) from %s where length(c_string) > %d", + standard, BigQuerySupport.MAX_STRING_LENGTH), 0); + assertCount(connection, String.format( + "select count(*) from %s where length(c_string_sized) > 20", standard), 0); + assertCount(connection, String.format( + "select count(*) from %s where length(c_bytes) > %d", + standard, BigQuerySupport.MAX_BYTES_LENGTH), 0); + + // BIGNUMERIC is bound as NUMERIC, so its values must sit inside NUMERIC's range + assertCount(connection, String.format( + "select count(*) from %s where abs(c_bignumeric) >= 1e29", standard), 0); + } +} diff --git a/bloviate-core/src/test/java/io/bloviate/db/DatabaseFillerCommitStrategyTest.java b/bloviate-core/src/test/java/io/bloviate/db/DatabaseFillerCommitStrategyTest.java new file mode 100644 index 0000000..3e0fc86 --- /dev/null +++ b/bloviate-core/src/test/java/io/bloviate/db/DatabaseFillerCommitStrategyTest.java @@ -0,0 +1,131 @@ +/* + * Copyright (c) 2021 Tim Veil + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.bloviate.db; + +import io.bloviate.ext.BigQuerySupport; +import io.bloviate.ext.DatabaseSupport; +import io.bloviate.ext.DefaultSupport; +import org.junit.jupiter.api.Test; + +import javax.sql.DataSource; +import java.io.PrintWriter; +import java.sql.Connection; +import java.sql.SQLException; +import java.util.logging.Logger; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +/** + * Docker-free coverage of the commit strategy the parallel and unordered fill paths actually use. + * No connection is ever opened — only the strategy mapping is exercised. + */ +class DatabaseFillerCommitStrategyTest { + + /** Matches {@code DatabaseFiller.DEFAULT_PARALLEL_COMMIT_BATCHES}. */ + private static final int DEFAULT_PARALLEL_COMMIT_BATCHES = 64; + + private static CommitStrategy effectiveStrategy(DatabaseSupport support, CommitStrategy configured) { + DatabaseConfiguration configuration = new DatabaseConfiguration.Builder(128, 10, support) + .commitStrategy(configured) + .build(); + + return new DatabaseFiller.Builder(new UnusableDataSource(), configuration) + .threads(4) + .build() + .effectiveParallelCommitStrategy(); + } + + @Test + void upgradesConnectionDefaultToABoundedCommitCadence() { + // a pooled worker must not be left on the connection's autocommit, and a single per-table + // commit would hold a whole partition open in one server-side transaction + assertEquals(CommitStrategy.everyNBatches(DEFAULT_PARALLEL_COMMIT_BATCHES), + effectiveStrategy(new DefaultSupport(), CommitStrategy.connectionDefault())); + } + + @Test + void leavesConnectionDefaultAloneWhenTheSupportPrefersIt() { + // BigQuery: each executeBatch is already one atomic query job, while setAutoCommit(false) + // opens a session and disables the driver's load-job path + assertEquals(CommitStrategy.connectionDefault(), + effectiveStrategy(new BigQuerySupport(), CommitStrategy.connectionDefault())); + } + + @Test + void honorsAnExplicitStrategyEvenWhenTheSupportPrefersOtherwise() { + // the hook only suppresses the engine's own default; a caller's explicit choice still wins + assertEquals(CommitStrategy.perTable(), + effectiveStrategy(new BigQuerySupport(), CommitStrategy.perTable())); + assertEquals(CommitStrategy.everyNBatches(8), + effectiveStrategy(new BigQuerySupport(), CommitStrategy.everyNBatches(8))); + } + + @Test + void honorsAnExplicitStrategyForOrdinarySupports() { + assertEquals(CommitStrategy.perTable(), + effectiveStrategy(new DefaultSupport(), CommitStrategy.perTable())); + } + + /** A {@link DataSource} that fails loudly if anything actually tries to connect. */ + private static final class UnusableDataSource implements DataSource { + + @Override + public Connection getConnection() { + throw new UnsupportedOperationException("this test must not open a connection"); + } + + @Override + public Connection getConnection(String username, String password) { + return getConnection(); + } + + @Override + public PrintWriter getLogWriter() { + return null; + } + + @Override + public void setLogWriter(PrintWriter out) { + // no-op + } + + @Override + public void setLoginTimeout(int seconds) { + // no-op + } + + @Override + public int getLoginTimeout() { + return 0; + } + + @Override + public Logger getParentLogger() { + return Logger.getGlobal(); + } + + @Override + public T unwrap(Class iface) throws SQLException { + throw new SQLException("not a wrapper"); + } + + @Override + public boolean isWrapperFor(Class iface) { + return false; + } + } +} diff --git a/bloviate-core/src/test/java/io/bloviate/ext/BatchRewriteParameterTest.java b/bloviate-core/src/test/java/io/bloviate/ext/BatchRewriteParameterTest.java index daa41a4..957a43b 100644 --- a/bloviate-core/src/test/java/io/bloviate/ext/BatchRewriteParameterTest.java +++ b/bloviate-core/src/test/java/io/bloviate/ext/BatchRewriteParameterTest.java @@ -43,4 +43,12 @@ void cockroachDbHasNoBatchRewriteParameter() { void defaultSupportHasNoBatchRewriteParameter() { assertNull(new DefaultSupport().batchRewriteUrlParameter()); } + + @Test + void bigQueryHasNoBatchRewriteParameter() { + // tbc-bq-jdbc rewrites batches into multi-row INSERTs unconditionally, so there is no URL + // parameter to recommend -- advertising batchLoadThreshold here would emit wrong advice, + // since it takes an integer rather than acting as a toggle + assertNull(new BigQuerySupport().batchRewriteUrlParameter()); + } } diff --git a/bloviate-core/src/test/java/io/bloviate/ext/BigQuerySupportTest.java b/bloviate-core/src/test/java/io/bloviate/ext/BigQuerySupportTest.java new file mode 100644 index 0000000..2a59e70 --- /dev/null +++ b/bloviate-core/src/test/java/io/bloviate/ext/BigQuerySupportTest.java @@ -0,0 +1,284 @@ +/* + * Copyright (c) 2021 Tim Veil + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.bloviate.ext; + +import io.bloviate.db.Column; +import io.bloviate.gen.BigDecimalGenerator; +import io.bloviate.gen.BooleanGenerator; +import io.bloviate.gen.ByteGenerator; +import io.bloviate.gen.DataGenerator; +import io.bloviate.gen.DoubleGenerator; +import io.bloviate.gen.LongGenerator; +import io.bloviate.gen.SimpleStringGenerator; +import io.bloviate.gen.SqlDateGenerator; +import io.bloviate.gen.SqlTimeGenerator; +import io.bloviate.gen.SqlTimestampGenerator; +import org.junit.jupiter.api.Test; + +import java.math.BigDecimal; +import java.sql.JDBCType; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Random; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Docker-free coverage of {@link BigQuerySupport}. The BigQuery integration test needs a live + * project, so these synthetic-{@link Column} tests are the only ones that run in CI — they carry + * the type mapping, the size/precision clamps, and the rejection messages. + */ +class BigQuerySupportTest { + + /** How many draws to take before trusting a bound; generators are random per call. */ + private static final int DRAWS = 50; + + private static final Random RANDOM = new Random(1); + + private final BigQuerySupport support = new BigQuerySupport(); + + private static Column column(JDBCType type, Integer maxSize, Integer maxDigits, String typeName) { + return new Column("c", "t", null, null, type, maxSize, maxDigits, typeName, false, true, null, 1); + } + + private DataGenerator generatorFor(JDBCType type, Integer maxSize, Integer maxDigits, String typeName) { + return support.getDataGenerator(column(type, maxSize, maxDigits, typeName), RANDOM); + } + + private DataGenerator generatorFor(JDBCType type, Integer maxSize, String typeName) { + return generatorFor(type, maxSize, null, typeName); + } + + private UnsupportedOperationException rejectionFor(JDBCType type, String typeName) { + return assertThrows(UnsupportedOperationException.class, () -> generatorFor(type, null, typeName)); + } + + // ---------------------------------------------------------------- natively bindable types + + @Test + void mapsNativelyBindableTypesToTheInheritedDefaults() { + // these ride on AbstractDatabaseSupport's defaults; asserting them here means a future + // change to registerDefaults that breaks BigQuery fails this test rather than a live fill + assertInstanceOf(LongGenerator.class, generatorFor(JDBCType.BIGINT, null, "INT64")); + assertInstanceOf(DoubleGenerator.class, generatorFor(JDBCType.DOUBLE, null, "FLOAT64")); + assertInstanceOf(BooleanGenerator.class, generatorFor(JDBCType.BOOLEAN, null, "BOOL")); + assertInstanceOf(SqlDateGenerator.class, generatorFor(JDBCType.DATE, null, "DATE")); + assertInstanceOf(SqlTimeGenerator.class, generatorFor(JDBCType.TIME, null, "TIME")); + assertInstanceOf(SqlTimestampGenerator.class, generatorFor(JDBCType.TIMESTAMP, null, "TIMESTAMP")); + } + + // ---------------------------------------------------------------------------- STRING/BYTES + + @Test + void honorsADeclaredStringWidth() { + DataGenerator generator = generatorFor(JDBCType.VARCHAR, 20, "STRING(20)"); + assertInstanceOf(SimpleStringGenerator.class, generator); + + for (int i = 0; i < DRAWS; i++) { + assertTrue(((String) generator.generate()).length() <= 20); + } + } + + @Test + void clampsABareStringDownFromTheBigQueryTypeMaximum() { + // a bare STRING column reports 2,097,152 — the type limit, not a declared width + DataGenerator generator = generatorFor(JDBCType.VARCHAR, 2_097_152, "STRING"); + + for (int i = 0; i < DRAWS; i++) { + assertTrue(((String) generator.generate()).length() <= BigQuerySupport.MAX_STRING_LENGTH); + } + } + + @Test + void clampsBytesDownFromTheBigQueryTypeMaximum() { + DataGenerator generator = generatorFor(JDBCType.VARBINARY, 10_485_760, "BYTES"); + assertInstanceOf(ByteGenerator.class, generator); + + for (int i = 0; i < DRAWS; i++) { + assertTrue(((Byte[]) generator.generate()).length <= BigQuerySupport.MAX_BYTES_LENGTH); + } + } + + @Test + void honorsADeclaredBytesWidth() { + DataGenerator generator = generatorFor(JDBCType.VARBINARY, 16, "BYTES(16)"); + + for (int i = 0; i < DRAWS; i++) { + assertTrue(((Byte[]) generator.generate()).length <= 16); + } + } + + @Test + void fallsBackToTheCapWhenSizeIsUnreported() { + // COLUMN_SIZE is nullable metadata; neither branch may unbox it + DataGenerator string = generatorFor(JDBCType.VARCHAR, null, "STRING"); + DataGenerator bytes = generatorFor(JDBCType.VARBINARY, null, "BYTES"); + + assertTrue(((String) string.generate()).length() <= BigQuerySupport.MAX_STRING_LENGTH); + assertTrue(((Byte[]) bytes.generate()).length <= BigQuerySupport.MAX_BYTES_LENGTH); + } + + // -------------------------------------------------------------------- NUMERIC / BIGNUMERIC + + @Test + void generatesAValidNumeric() { + DataGenerator generator = generatorFor(JDBCType.NUMERIC, 38, 9, "NUMERIC"); + assertInstanceOf(BigDecimalGenerator.class, generator); + + for (int i = 0; i < DRAWS; i++) { + BigDecimal value = (BigDecimal) generator.generate(); + assertTrue(value.scale() <= BigQuerySupport.MAX_NUMERIC_SCALE); + assertTrue(value.precision() <= BigQuerySupport.MAX_NUMERIC_PRECISION); + } + } + + @Test + void clampsBigNumericIntoNumericRange() { + // BIGNUMERIC reports (76, 38). The driver binds every BigDecimal as NUMERIC, whose maximum + // scale is 9, so an unclamped draw carries 25 fractional digits and BigQuery rejects it as + // an out-of-range NUMERIC parameter — regardless of what the destination column could hold. + DataGenerator generator = generatorFor(JDBCType.NUMERIC, 76, 38, "BIGNUMERIC"); + + for (int i = 0; i < DRAWS; i++) { + BigDecimal value = (BigDecimal) generator.generate(); + assertTrue(value.scale() <= BigQuerySupport.MAX_NUMERIC_SCALE, + "scale was " + value.scale() + " for " + value); + assertTrue(value.precision() <= BigQuerySupport.MAX_NUMERIC_PRECISION, + "precision was " + value.precision() + " for " + value); + } + } + + @Test + void honorsADeclaredNumericScale() { + DataGenerator generator = generatorFor(JDBCType.NUMERIC, 10, 2, "NUMERIC(10, 2)"); + + for (int i = 0; i < DRAWS; i++) { + BigDecimal value = (BigDecimal) generator.generate(); + assertTrue(value.scale() <= 2, "scale was " + value.scale() + " for " + value); + assertTrue(value.precision() <= 10, "precision was " + value.precision() + " for " + value); + } + } + + @Test + void toleratesUnreportedNumericPrecisionAndScale() { + DataGenerator generator = generatorFor(JDBCType.NUMERIC, null, null, "NUMERIC"); + BigDecimal value = (BigDecimal) generator.generate(); + + assertEquals(0, value.scale()); + assertTrue(value.precision() <= BigQuerySupport.MAX_NUMERIC_PRECISION); + } + + // ------------------------------------------------------------------------ rejected types + + @Test + void rejectsDatetimeWhichIsIndistinguishableFromTimestampByJdbcTypeAlone() { + UnsupportedOperationException e = rejectionFor(JDBCType.TIMESTAMP, "DATETIME"); + + assertTrue(e.getMessage().contains("DATETIME")); + assertTrue(e.getMessage().contains("ColumnConfiguration")); + } + + @Test + void rejectsTypesWithNoParameterBinding() { + Map rejected = new LinkedHashMap<>(); + rejected.put(JDBCType.OTHER, "RANGE"); + rejected.put(JDBCType.ARRAY, "ARRAY"); + rejected.put(JDBCType.STRUCT, "STRUCT"); + + for (String typeName : new String[]{"JSON", "GEOGRAPHY", "INTERVAL"}) { + assertRejected(JDBCType.VARCHAR, typeName); + } + rejected.forEach(this::assertRejected); + } + + private void assertRejected(JDBCType jdbcType, String typeName) { + UnsupportedOperationException e = rejectionFor(jdbcType, typeName); + + // the raw type name is echoed so the message names the column's actual declared type + assertTrue(e.getMessage().contains(typeName), e.getMessage()); + // and points at the escape hatch, not at registerTypeName — which cannot match a + // parameterized BigQuery type name like "numeric(10, 2)" + assertTrue(e.getMessage().contains("ColumnConfiguration"), e.getMessage()); + assertFalse(e.getMessage().contains("registerTypeName"), e.getMessage()); + } + + @Test + void rejectsStructRatherThanInheritingTheStructGenerator() { + // AbstractDatabaseSupport maps STRUCT to SqlStructGenerator by default, which cannot know a + // BigQuery struct's field shape and would silently generate garbage + assertThrows(UnsupportedOperationException.class, + () -> generatorFor(JDBCType.STRUCT, null, "STRUCT")); + } + + // -------------------------------------------------------------------- type-name normalizer + + @Test + void normalizesTypeNameCaseWhitespaceAndParameters() { + assertInstanceOf(SimpleStringGenerator.class, generatorFor(JDBCType.VARCHAR, 20, "string(20)")); + assertInstanceOf(SimpleStringGenerator.class, generatorFor(JDBCType.VARCHAR, null, " STRING ")); + assertInstanceOf(SimpleStringGenerator.class, generatorFor(JDBCType.VARCHAR, null, "String")); + } + + @Test + void normalizesTypeArgumentsWhenRejecting() { + assertThrows(UnsupportedOperationException.class, + () -> generatorFor(JDBCType.STRUCT, null, "Struct")); + } + + @Test + void toleratesAnUnreportedTypeName() { + // TYPE_NAME should always be present, but a null must surface as the normal rejection + // rather than a NullPointerException from the normalizer + assertThrows(UnsupportedOperationException.class, () -> generatorFor(JDBCType.VARCHAR, 10, null)); + assertThrows(UnsupportedOperationException.class, () -> generatorFor(JDBCType.VARBINARY, 10, "")); + } + + // ------------------------------------------------------------------------------ capabilities + + @Test + void enablesBulkLoadWithNoOpConstraintHandling() throws Exception { + assertTrue(support.supportsBulkLoad()); + + // BigQuery keys are always NOT ENFORCED, so there is nothing to disable — and nothing that + // needs a connection. Passing null proves the implementation never touches one. + BulkLoadHandle handle = support.disableConstraints(null, null); + assertTrue(handle.description().contains("NOT ENFORCED")); + support.enableConstraints(null, null, handle); + } + + @Test + void prefersTheConnectionsOwnCommitBehavior() { + assertTrue(support.prefersConnectionDefaultCommit()); + } + + @Test + void recommendsNoBatchRewriteParameter() { + // batch rewrite is unconditional in tbc-bq-jdbc, so there is no URL parameter to suggest + assertNull(support.batchRewriteUrlParameter()); + } + + @Test + void readsNoConstraints() { + // BigQuery has no CHECK constraints or enum/domain types + assertEquals(Map.of(), support.readConstraints(null, null, null)); + } +} diff --git a/bloviate-core/src/test/java/io/bloviate/ext/DatabaseSupportSelectionTest.java b/bloviate-core/src/test/java/io/bloviate/ext/DatabaseSupportSelectionTest.java index 4f8284d..ac1fdec 100644 --- a/bloviate-core/src/test/java/io/bloviate/ext/DatabaseSupportSelectionTest.java +++ b/bloviate-core/src/test/java/io/bloviate/ext/DatabaseSupportSelectionTest.java @@ -59,12 +59,26 @@ void selectsSqliteByProductName() { assertInstanceOf(SQLiteSupport.class, DatabaseSupport.forProduct("SQLite")); } + @Test + void selectsBigQueryByProductName() { + // the tbc-bq-jdbc driver reports this exact string + assertInstanceOf(BigQuerySupport.class, DatabaseSupport.forProduct("BigQuery (TBC Driver)")); + } + + @Test + void selectsBigQueryForOtherBigQueryDrivers() { + // Simba's driver reports "Google BigQuery"; the substring match catches it, which is + // usually what you want -- see the documented caveat about its differing type names. + assertInstanceOf(BigQuerySupport.class, DatabaseSupport.forProduct("Google BigQuery")); + } + @Test void matchingIsCaseInsensitive() { assertInstanceOf(MySQLSupport.class, DatabaseSupport.forProduct("mysql")); assertInstanceOf(PostgresSupport.class, DatabaseSupport.forProduct("POSTGRESQL")); assertInstanceOf(MariaDBSupport.class, DatabaseSupport.forProduct("mariadb")); assertInstanceOf(SQLiteSupport.class, DatabaseSupport.forProduct("SQLITE")); + assertInstanceOf(BigQuerySupport.class, DatabaseSupport.forProduct("BIGQUERY")); } @Test diff --git a/bloviate-core/src/test/resources/create_tables.bigquery.sql b/bloviate-core/src/test/resources/create_tables.bigquery.sql new file mode 100644 index 0000000..e22d8b9 --- /dev/null +++ b/bloviate-core/src/test/resources/create_tables.bigquery.sql @@ -0,0 +1,60 @@ +-- BigQuery test schema for BigQueryFillerTest. +-- +-- Substituted tokens: ${dataset} (the target dataset) and ${suffix} (a per-run id, so repeated or +-- concurrent runs never collide). Every table carries a 2-hour expiration so a failed run cannot +-- leave billable tables behind. +-- +-- Only natively bindable types appear here. DatabaseFiller fills every table in the dataset, so a +-- single DATETIME/JSON/GEOGRAPHY/ARRAY/STRUCT column would fail the whole fill; rejection of those +-- is covered by BigQuerySupportTest instead. +-- +-- Keys are declared NOT ENFORCED, which is the only form BigQuery accepts. They are never enforced +-- at write time, but the driver surfaces them through getPrimaryKeys/getImportedKeys, which is what +-- drives Bloviate's dependency graph and foreign-key value alignment. + +CREATE TABLE `${dataset}.region_${suffix}` ( + r_regionkey INT64 NOT NULL, + r_name STRING(25), + r_comment STRING(152), + PRIMARY KEY (r_regionkey) NOT ENFORCED +) OPTIONS(expiration_timestamp = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 2 HOUR)); + +CREATE TABLE `${dataset}.nation_${suffix}` ( + n_nationkey INT64 NOT NULL, + n_regionkey INT64 NOT NULL, + n_name STRING(25), + n_comment STRING(152), + PRIMARY KEY (n_nationkey) NOT ENFORCED, + FOREIGN KEY (n_regionkey) REFERENCES `${dataset}.region_${suffix}`(r_regionkey) NOT ENFORCED +) OPTIONS(expiration_timestamp = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 2 HOUR)); + +-- Every natively bindable type, in both its bare and parameterized form where one exists, so the +-- size and precision clamps are exercised against the real service. +CREATE TABLE `${dataset}.standard_types_${suffix}` ( + c_int64 INT64, + c_float64 FLOAT64, + c_numeric NUMERIC, + c_numeric_scaled NUMERIC(10, 2), + c_bignumeric BIGNUMERIC, + c_bool BOOL, + c_string STRING, + c_string_sized STRING(20), + c_bytes BYTES, + c_bytes_sized BYTES(16), + c_date DATE, + c_time TIME, + c_timestamp TIMESTAMP +) OPTIONS(expiration_timestamp = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 2 HOUR)); + +-- Any real BigQuery table is partitioned and clustered; this proves the batch/DML path works +-- against such a destination rather than only against a plain heap table. +CREATE TABLE `${dataset}.events_${suffix}` ( + e_id INT64 NOT NULL, + e_name STRING(50), + e_amount NUMERIC(12, 2), + e_ts TIMESTAMP NOT NULL, + PRIMARY KEY (e_id) NOT ENFORCED +) +PARTITION BY DATE(e_ts) +CLUSTER BY e_id +OPTIONS(expiration_timestamp = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 2 HOUR)); diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 69b906a..5cedb69 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -263,6 +263,15 @@ applies there too. > correctly-parameterized URL if you construct the `DataSource` yourself. CockroachDB ignores the > parameter, so no warning is emitted there. +> **Tip — BigQuery.** Rewriting is unconditional in tbc-bq-jdbc, so there is no parameter to set and +> no warning to emit. Instead, size `batchSize` against BigQuery's limit of 10,000 query parameters +> per query: the effective rows per job is `10_000 / columnCount`, so the default 128 leaves most of +> a job unused. Start from `batchSize = 5000`, and set the driver's `batchLoadThreshold` to match to +> move large batches onto its NDJSON load-job path. Leave `CommitStrategy` at its default — +> `BigQuerySupport` opts out of engine-managed transactions because `setAutoCommit(false)` starts a +> BigQuery session and silently disables that load path, and each `executeBatch` is already one +> atomic job. Bloviate logs a warning if a commit strategy is configured anyway. + ## Bulk load (unordered fill) The parallel path normally barriers between topological levels, so a **deep, narrow foreign-key diff --git a/docs/DATABASE_SUPPORT.md b/docs/DATABASE_SUPPORT.md index 3df22cf..9f6e74b 100644 --- a/docs/DATABASE_SUPPORT.md +++ b/docs/DATABASE_SUPPORT.md @@ -12,6 +12,7 @@ explicitly, or let Bloviate detect it from the connection. | MariaDB | `MariaDBSupport` | Extends `MySQLSupport` (MariaDB speaks the MySQL wire protocol) | | H2 | `H2Support` | Standard JDBC types **plus** `UUID` and `JSON` (embedded; no Docker) | | SQLite | `SQLiteSupport` | Standard JDBC types via type affinity (embedded; no Docker) | +| BigQuery | `BigQuerySupport` | Scalar types only — see [BigQuery](#bigquery) for what is excluded | | Generic JDBC | `DefaultSupport` | Standard JDBC types only | All of them resolve the cross-database defaults for the common JDBC types (integers, decimals, @@ -44,6 +45,95 @@ per SQLite convention, and every value round-trips through affinity rules. Forei default (`PRAGMA foreign_keys = ON` enables them), but Bloviate orders fills by the foreign-key graph regardless. +## BigQuery + +Requires the [tbc-bq-jdbc](https://github.com/Two-Bear-Capital/tbc-bq-jdbc) driver +(`vc.tbc:tbc-bq-jdbc`, 4.3.0 or later), which is not yet on Maven Central — install it locally with +`./mvnw clean install` in that repo, or use its GitHub Releases jar. + +```java +String url = "jdbc:bigquery:my-project/my_dataset?authType=ADC"; +``` + +**Supported:** `STRING`, `BYTES`, `INT64`, `FLOAT64`, `NUMERIC`, `BIGNUMERIC`, `BOOL`, `DATE`, +`TIME`, `TIMESTAMP`. + +**Not supported:** `DATETIME`, `JSON`, `GEOGRAPHY`, `INTERVAL`, `RANGE`, `ARRAY`, `STRUCT`. These +fail fast with a message naming the column, before any rows are written. + +The reason is that BigQuery's *write* surface is narrower than its read surface. The driver reads +`JSON`/`GEOGRAPHY`/`INTERVAL` back as `VARCHAR` and `DATETIME` as `TIMESTAMP`, but has no parameter +binding that produces those types, and BigQuery will not implicitly coerce a `STRING` or `TIMESTAMP` +parameter into them. Filling them requires SQL-side construction (`PARSE_JSON(?)`, +`ST_GEOGFROMTEXT(?)`, `CAST(? AS DATETIME)`), which the fill engine does not yet emit. + +To fill one anyway, supply a per-column generator through `ColumnConfiguration` or +`GeneratorRegistry.registerColumnNamePattern`. The driver does implement `Connection.createArrayOf` +and `Connection.createStruct`, so a hand-written generator can write composites today. + +> `GeneratorRegistry.registerTypeName` is a poor fit here: it matches type names exactly, and +> BigQuery reports the raw `INFORMATION_SCHEMA` text (`string(20)`, `numeric(10, 2)`, +> `array`), so only unparameterized names ever match. + +> Generators registered by column-name pattern — including everything `bloviate-datafaker` +> contributes — rank **above** the support's own type mapping. A `GEOGRAPHY` column whose name +> matches such a pattern will get that generator instead of the error above, and fail at insert +> time. Exclude those columns explicitly if you use pattern-based generators. + +### Generated value sizes + +BigQuery reports `COLUMN_SIZE` as the *type* maximum, not a declared width: a bare `STRING` reports +2,097,152 and a bare `BYTES` reports 10,485,760. Sizes are therefore clamped downward — to 256 +characters and 128 bytes — while a declared `STRING(20)` is honored exactly. + +Those caps are ceilings, not target lengths, and the generators impose their own limits underneath: +`SimpleStringGenerator` never exceeds 2000 characters and `ByteGenerator` never exceeds 25 bytes. So +the string clamp is a further ~8x reduction that you will observe, whereas the `BYTES` clamp sits +above the generator's own limit and does not currently change any generated value. It is stated as a +bound so the intent survives a change to that generator. + +`BIGNUMERIC` is clamped for a different reason: it reports precision 76 and scale 38, but the driver +binds every `BigDecimal` as `NUMERIC`, so generated values are clamped to `NUMERIC`'s (38, 9). The +binding is what forces this, not the destination column — a `NUMERIC`-range value is always valid in +a `BIGNUMERIC` column. + +### Required driver settings + +Both are the driver's defaults; overriding either breaks the fill. + +- **`includeStructFields=false`** — with struct fields spliced in, `getColumns` reports dotted + sub-field rows alongside their parent and the generated `INSERT` is structurally invalid. +- **`metadataLazyLoad=false`** — with lazy metadata, an unfiltered `getTables`/`getColumns` returns + nothing and the fill silently does no work. + +### Keys and fill order + +BigQuery accepts `PRIMARY KEY`/`FOREIGN KEY` only as `NOT ENFORCED`, but the driver still surfaces +them through `getPrimaryKeys`/`getImportedKeys`, so Bloviate orders fills by the foreign-key graph +and aligns child values with their parents exactly as it does elsewhere. Declare keys in your DDL if +you want referentially consistent data; without them the graph has no edges and foreign-key columns +get independent random values. + +Because nothing is ever enforced, `UNORDERED_BULK` is free here — there is no enforcement to suspend +and no ordering requirement, so `BigQuerySupport` enables it with no-op constraint handling. + +### Performance and cost + +Every statement is a BigQuery job, so per-row inserts are prohibitively slow and batching is +mandatory. The driver collapses a JDBC batch into a single multi-row `INSERT` automatically (no URL +parameter needed), chunked to stay under BigQuery's 10,000-parameters-per-query limit — so the +effective rows per job is `10_000 / columnCount`, and the default `batchSize` of 128 leaves most of +that on the table. Start from `batchSize = 5000`. + +Setting `batchLoadThreshold` to a matching value moves large batches onto the driver's NDJSON +load-job path, which avoids DML quotas and per-job query cost entirely. That path requires +auto-commit to stay on, which is why `BigQuerySupport` opts out of engine-managed transactions: +`setAutoCommit(false)` starts a BigQuery session and silently disables it. **Leave +`CommitStrategy` at its default** — each `executeBatch` is already one atomic job, so an explicit +strategy buys nothing and costs the load path. Bloviate logs a warning if you set one anyway. + +> Filling a BigQuery dataset writes real data to real storage and runs real jobs. Both cost money. + > **PostgreSQL connection requirement:** the vendor types above are bound as their text > representations, and PostgreSQL won't implicitly cast `varchar` to `uuid`/`jsonb`/`bit`/etc. > Open the connection with `stringtype=unspecified` so the server infers each column's type: @@ -66,3 +156,8 @@ DatabaseSupport support = DatabaseSupport.forConnection(connection); > `MariaDBSupport`. The legacy MySQL Connector/J driver reports `MySQL` even against a MariaDB > server and resolves to `MySQLSupport`; since `MariaDBSupport` adds no divergent behavior, the two > are equivalent. + +> **BigQuery note:** `BigQuerySupport` matches any product name containing `bigquery`, which covers +> both tbc-bq-jdbc (`BigQuery (TBC Driver)`) and Simba (`Google BigQuery`). It is written against +> tbc-bq-jdbc's metadata, though, and discriminates on the raw `INFORMATION_SCHEMA` type text, so +> some columns may not resolve under Simba's driver. diff --git a/pom.xml b/pom.xml index 4b6ced1..fd22d21 100644 --- a/pom.xml +++ b/pom.xml @@ -86,6 +86,14 @@ 3.5.9 2.4.240 3.53.2.0 + + 4.3.0 1.37 @@ -258,6 +266,11 @@ sqlite-jdbc ${sqlite.version} + + vc.tbc + tbc-bq-jdbc + ${tbc-bq-jdbc.version} +