From cca965d8f9af6cb468f78e7bea321faa3a2b2181 Mon Sep 17 00:00:00 2001 From: Tim Veil <3260845+timveil@users.noreply.github.com> Date: Mon, 3 Aug 2026 14:58:14 -0400 Subject: [PATCH 1/2] feat(ext): add Google BigQuery database support (#569) Adds BigQuerySupport, driven by tbc-bq-jdbc 4.3.0. Covers the natively bindable scalar types: STRING, BYTES, INT64, FLOAT64, NUMERIC, BIGNUMERIC, BOOL, DATE, TIME and TIMESTAMP. BigQuery is the first analytical engine Bloviate targets, and the gaps it exposes are general rather than vendor quirks, so they are addressed as DatabaseSupport capabilities rather than special cases: - prefersConnectionDefaultCommit() lets a support keep the engine out of transaction management. On BigQuery each executeBatch is already one atomic query job, while setAutoCommit(false) starts a session and disables the driver's NDJSON load-job path. Consulted only in effectiveParallelCommitStrategy(); an explicitly configured strategy still wins, with a warning. - supportsBulkLoad() is enabled with no-op constraint handling. BigQuery keys are always NOT ENFORCED, so there is nothing to suspend, and foreign-key values are seeded from the parent primary-key column rather than depending on insert order. Type handling notes: - TYPE_NAME arrives as raw INFORMATION_SCHEMA text (STRING(20), ARRAY, RANGE), so it is normalized before dispatch. - COLUMN_SIZE is the type maximum, not a declared width; sizes are clamped downward, leaving STRING(20) honest while taming bare STRING. - BIGNUMERIC reports (76, 38) but the driver binds every BigDecimal as NUMERIC, so values are clamped to NUMERIC's (38, 9). Without this the generated value carries 25 fractional digits and is rejected as an out-of-range NUMERIC parameter. - DATETIME, JSON, GEOGRAPHY, INTERVAL, RANGE, ARRAY and STRUCT have no parameter binding and BigQuery will not coerce into them, so they throw at generator resolution, before any rows are written, naming the column and the ColumnConfiguration escape hatch. STRUCT explicitly replaces the inherited SqlStructGenerator, which cannot know a struct's shape. Testing: BigQuery has no usable emulator, so BigQueryFillerTest needs a live project and is skipped by default. It is gated on both BLOVIATE_BQ_PROJECT and the driver being on the classpath, so setting the env var without -Pbigquery skips cleanly instead of failing with "No suitable driver". The driver is not on Maven Central yet, so it is declared only inside that opt-in profile and a default build stays resolvable. Nothing in the test imports a driver class, so it always compiles. Docker-free unit tests carry the coverage floors. The classpath script runner moves from BaseEmbeddedTest up to BaseDatabaseTestCase and gains token substitution, so the BigQuery schema can use per-run table names without duplicating the reader. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 2 +- CONTRIBUTING.md | 36 ++- README.md | 4 +- bloviate-core/pom.xml | 26 ++ .../java/io/bloviate/db/DatabaseFiller.java | 37 ++- .../java/io/bloviate/ext/BigQuerySupport.java | 305 ++++++++++++++++++ .../java/io/bloviate/ext/DatabaseSupport.java | 42 ++- .../io/bloviate/db/BaseDatabaseTestCase.java | 57 ++++ .../java/io/bloviate/db/BaseEmbeddedTest.java | 44 --- .../io/bloviate/db/BigQueryFillerTest.java | 168 ++++++++++ .../db/DatabaseFillerCommitStrategyTest.java | 131 ++++++++ .../ext/BatchRewriteParameterTest.java | 8 + .../io/bloviate/ext/BigQuerySupportTest.java | 284 ++++++++++++++++ .../ext/DatabaseSupportSelectionTest.java | 14 + .../test/resources/create_tables.bigquery.sql | 60 ++++ docs/CONFIGURATION.md | 9 + docs/DATABASE_SUPPORT.md | 87 +++++ pom.xml | 13 + 18 files changed, 1271 insertions(+), 56 deletions(-) create mode 100644 bloviate-core/src/main/java/io/bloviate/ext/BigQuerySupport.java create mode 100644 bloviate-core/src/test/java/io/bloviate/db/BigQueryFillerTest.java create mode 100644 bloviate-core/src/test/java/io/bloviate/db/DatabaseFillerCommitStrategyTest.java create mode 100644 bloviate-core/src/test/java/io/bloviate/ext/BigQuerySupportTest.java create mode 100644 bloviate-core/src/test/resources/create_tables.bigquery.sql diff --git a/CLAUDE.md b/CLAUDE.md index 951fc07..17b1c0f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -44,7 +44,7 @@ This is a Maven-based Java 25 project. Run `./mvnw` from the repository root to - `TableFiller`: Handles filling individual tables with generated data 2. **Database Support (`io.bloviate.ext`)**: - - Database-specific implementations: `PostgresSupport`, `MySQLSupport`, `CockroachDBSupport`, `DefaultSupport` + - Database-specific implementations: `PostgresSupport`, `MySQLSupport`, `MariaDBSupport`, `CockroachDBSupport`, `H2Support`, `SQLiteSupport`, `BigQuerySupport`, `DefaultSupport` - Each provides database-specific SQL generation and data type mapping 3. **Data Generators (`io.bloviate.gen`)**: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 00d9b40..182b1cc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -132,8 +132,40 @@ Docker must be running. ``` Test schemas live under `bloviate-core/src/test/resources/` (TPCC, AuctionMark, Wikipedia, and -others). `BaseDatabaseTestCase` provides the shared `DataSource` plumbing and the -fidelity assertions used by the TPC-C tests. +others). `BaseDatabaseTestCase` provides the shared `DataSource` plumbing, the classpath script +runner, and the fidelity assertions used by the TPC-C tests. + +### BigQuery + +`BigQueryFillerTest` is the one test Docker cannot cover. BigQuery has no usable emulator — the +tbc-bq-jdbc driver deliberately removed its emulator tier because the emulator diverged far enough +from the service to hide real defects — so the test needs a live Google Cloud project, and it is +**skipped by default**. + +It is gated twice, and both gates must pass: + +1. `BLOVIATE_BQ_PROJECT` is set (with Application Default Credentials available), and +2. the driver is on the classpath, which only happens under `-Pbigquery`. + +The second gate exists so that setting the env var without the profile skips cleanly instead of +failing with "No suitable driver". `vc.tbc:tbc-bq-jdbc` is not on Maven Central yet, which is why it +is declared in an opt-in profile rather than as an ordinary test dependency — a default build must +stay resolvable for everyone. Bump `tbc-bq-jdbc.version` in the root POM by hand; Dependabot and +`versions:display-dependency-updates` cannot resolve that coordinate. + +```bash +# once, in a clone of https://github.com/Two-Bear-Capital/tbc-bq-jdbc +./mvnw clean install + +gcloud auth application-default login +export BLOVIATE_BQ_PROJECT=my-gcp-project + +./mvnw verify -Pbigquery -pl bloviate-core -Dtest=BigQueryFillerTest +``` + +Each run creates its own dataset and drops it afterwards (with a one-day default table expiration as +a backstop), because `DatabaseFiller` fills **every** table it finds in the connection's schema. +Running it writes real data and runs real jobs, both of which cost money. ## Databases for Testing diff --git a/README.md b/README.md index c4d5511..b9b15d7 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ by seed, and runs inside your JUnit/Testcontainers pipeline. **Full documentation, guides, and examples live at [bloviate.io](https://bloviate.io).** - [Quick Start](https://bloviate.io/guides/quickstart/) — install and fill a database or flat file -- [Database Support](https://bloviate.io/guides/database-support/) — PostgreSQL, MySQL, MariaDB, CockroachDB, H2, SQLite +- [Database Support](https://bloviate.io/guides/database-support/) — PostgreSQL, MySQL, MariaDB, CockroachDB, H2, SQLite, BigQuery - [Configuration](https://bloviate.io/guides/configuration/) — per-table/column control, distributions, seeds, parallelism - [Generators](https://bloviate.io/guides/generators/) — registry, realistic data, composite keys, TPC-C - [Testing Integrations](https://bloviate.io/guides/integrations/) — JUnit Jupiter and Testcontainers @@ -31,7 +31,7 @@ by seed, and runs inside your JUnit/Testcontainers pipeline. - **Deterministic by seed** — same seed + schema ⇒ byte-identical data, even under parallel fills - **Per-column control** and **pluggable generators**; realistic semantic values via Datafaker - **Parallel & partitioned fills** for large datasets, with referential integrity preserved -- **PostgreSQL, MySQL, MariaDB, CockroachDB, H2, SQLite**, plus CSV/TSV/pipe **flat-file** output +- **PostgreSQL, MySQL, MariaDB, CockroachDB, H2, SQLite, BigQuery**, plus CSV/TSV/pipe **flat-file** output - First-class **JUnit Jupiter** (JUnit 5/6) and **Testcontainers** integrations See the [full feature tour and guides on bloviate.io](https://bloviate.io). diff --git a/bloviate-core/pom.xml b/bloviate-core/pom.xml index 77bcd24..a5dcadf 100644 --- a/bloviate-core/pom.xml +++ b/bloviate-core/pom.xml @@ -168,4 +168,30 @@ + + + + bigquery + + + vc.tbc + tbc-bq-jdbc + test + + + + + diff --git a/bloviate-core/src/main/java/io/bloviate/db/DatabaseFiller.java b/bloviate-core/src/main/java/io/bloviate/db/DatabaseFiller.java index 44aa8b2..51d2e8f 100644 --- a/bloviate-core/src/main/java/io/bloviate/db/DatabaseFiller.java +++ b/bloviate-core/src/main/java/io/bloviate/db/DatabaseFiller.java @@ -162,6 +162,8 @@ public void fill() throws SQLException { visualizeGraph(reversedGraph, database.catalog()); + warnIfEngineManagedCommitDiscouraged(); + // recommend the driver batch-rewrite URL parameter once per fill if it is missing if (connection != null) { warnIfBatchRewriteMissing(connection); @@ -587,6 +589,23 @@ private void warnIfPartitionsIgnored() { * Bulk loading needs per-worker session control, so it only applies to the {@code threads > 1} * {@link DataSource} path; elsewhere the engine fills in dependency order. */ + /** + * Warns once per fill when an explicit commit strategy is configured against a support that + * would rather the engine stayed out of transaction management. The caller's choice is still + * honored — this only surfaces the cost, which is otherwise invisible (on BigQuery, an engine- + * managed transaction opens a session and silently disables the driver's load-job path). + */ + private void warnIfEngineManagedCommitDiscouraged() { + if (configuration.databaseSupport().prefersConnectionDefaultCommit() + && configuration.commitStrategy().managesTransaction()) { + logger.warn("{} recommends leaving transaction management to the connection, but commit " + + "strategy [{}] was configured; the engine will manage transactions as asked, " + + "which may be slower and can disable driver bulk-load paths", + configuration.databaseSupport().getClass().getSimpleName(), + configuration.commitStrategy().mode()); + } + } + private void warnIfBulkIgnored() { if (configuration.bulkLoadStrategy().isUnordered()) { logger.warn("UNORDERED_BULK is ignored on the sequential fill path; use the DataSource " @@ -603,12 +622,22 @@ private void warnIfBulkIgnored() { * large partition open in one server-side transaction (unbounded WAL/undo growth and lock * accumulation), which is the scale failure the parallel/bulk path most needs to avoid. Any * explicitly configured strategy (including {@link CommitStrategy#perTable()}) is honored as-is. + * + *

A {@link io.bloviate.ext.DatabaseSupport#prefersConnectionDefaultCommit() support that + * prefers the connection's own commit behavior} suppresses the upgrade, so + * {@code CONNECTION_DEFAULT} stays as configured. That is for engines where an engine-managed + * transaction is pure cost rather than protection — see the hook's documentation. + * + *

Package-private so the mapping can be unit-tested without a database. */ - private CommitStrategy effectiveParallelCommitStrategy() { + CommitStrategy effectiveParallelCommitStrategy() { CommitStrategy configured = configuration.commitStrategy(); - return configured.mode() == CommitStrategy.Mode.CONNECTION_DEFAULT - ? CommitStrategy.everyNBatches(DEFAULT_PARALLEL_COMMIT_BATCHES) - : configured; + if (configured.mode() != CommitStrategy.Mode.CONNECTION_DEFAULT) { + return configured; + } + return configuration.databaseSupport().prefersConnectionDefaultCommit() + ? configured + : CommitStrategy.everyNBatches(DEFAULT_PARALLEL_COMMIT_BATCHES); } /** diff --git a/bloviate-core/src/main/java/io/bloviate/ext/BigQuerySupport.java b/bloviate-core/src/main/java/io/bloviate/ext/BigQuerySupport.java new file mode 100644 index 0000000..8a82447 --- /dev/null +++ b/bloviate-core/src/main/java/io/bloviate/ext/BigQuerySupport.java @@ -0,0 +1,305 @@ +/* + * Copyright (c) 2021 Tim Veil + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.bloviate.ext; + +import io.bloviate.db.Column; +import io.bloviate.db.Database; +import io.bloviate.gen.BigDecimalGenerator; +import io.bloviate.gen.ByteGenerator; +import io.bloviate.gen.SimpleStringGenerator; + +import java.sql.Connection; +import java.sql.JDBCType; +import java.util.Locale; +import java.util.Map; + +/** + * Google BigQuery-specific {@link DatabaseSupport}, written against the + * tbc-bq-jdbc driver + * (vc.tbc:tbc-bq-jdbc, 4.3.0 or later). + * + *

BigQuery is an analytical engine, and it diverges from the OLTP databases Bloviate + * otherwise targets in three ways this class has to account for: + * + *

    + *
  • {@code COLUMN_SIZE} is a type maximum, not a declared width. A bare + * {@code STRING} column reports 2,097,152 and a bare {@code BYTES} column 10,485,760 — + * the type limits, not anything the schema asked for. Sizes are clamped downward to + * {@link #MAX_STRING_LENGTH}/{@link #MAX_BYTES_LENGTH}, which leaves a declared + * {@code STRING(20)} honest while taming the bare form. This matters because a batch buffers + * every value as a query parameter before flushing, and the whole chunk has to fit inside one + * {@code jobs.insert} request.
    • + *
  • The write side is narrower than the read side. The driver reads + * {@code JSON}, {@code GEOGRAPHY} and {@code INTERVAL} back as {@link JDBCType#VARCHAR} and + * {@code DATETIME} as {@link JDBCType#TIMESTAMP}, but has no parameter binding that produces + * those types, and BigQuery will not implicitly coerce a {@code STRING}/{@code TIMESTAMP} + * parameter into them. Filling those columns needs SQL-side construction + * ({@code PARSE_JSON(?)}, {@code ST_GEOGFROMTEXT(?)}, {@code CAST(? AS DATETIME)}), which is + * not yet wired through the fill engine, so they are rejected here with an actionable + * message rather than generated and rejected by the server mid-batch.
    • + *
  • Keys are always {@code NOT ENFORCED}. BigQuery accepts declarative + * {@code PRIMARY KEY}/{@code FOREIGN KEY} constraints but never enforces them, and the driver + * surfaces them through {@code getPrimaryKeys}/{@code getImportedKeys}. Bloviate's FK-aware + * ordering therefore works, and {@link #supportsBulkLoad()} is safe to enable.
    • + *
+ * + *

Supported types: {@code STRING}, {@code BYTES}, {@code INT64}, + * {@code FLOAT64}, {@code NUMERIC}, {@code BIGNUMERIC} (within {@code NUMERIC} range), + * {@code BOOL}, {@code DATE}, {@code TIME}, {@code TIMESTAMP}. + * + *

Unsupported types: {@code DATETIME}, {@code JSON}, {@code GEOGRAPHY}, + * {@code INTERVAL}, {@code RANGE}, {@code ARRAY}, {@code STRUCT}. Supply a per-column generator + * through {@code ColumnConfiguration} or + * {@link GeneratorRegistry.Builder#registerColumnNamePattern} to fill them; the driver implements + * {@link Connection#createArrayOf} and {@link Connection#createStruct}, so a hand-written generator + * can write composites today. + * + *

Required driver settings: {@code includeStructFields=false} (the default) and + * {@code metadataLazyLoad=false} (the default). With struct fields spliced in, {@code getColumns} + * emits dotted sub-field rows alongside their parent and the generated {@code INSERT} is + * structurally invalid; with lazy metadata, an unfiltered {@code getTables}/{@code getColumns} + * returns nothing and the fill silently does no work. + * + * @since 3.1.0 + * @see AbstractDatabaseSupport + * @see DatabaseSupport + */ +public class BigQuerySupport extends AbstractDatabaseSupport { + + /** + * Cap on generated {@code STRING} length. A bare {@code STRING} column reports BigQuery's type + * maximum of 2,097,152; a declared {@code STRING(n)} below this cap is honored exactly. + * + *

{@link SimpleStringGenerator} independently caps itself at 2000 characters, so this is an + * additional ~8x reduction on top of that, not the only thing standing between a fill and + * two-megabyte cells. + */ + public static final int MAX_STRING_LENGTH = 256; + + /** + * Cap on generated {@code BYTES} length. A bare {@code BYTES} column reports BigQuery's type + * maximum of 10,485,760; a declared {@code BYTES(n)} below this cap is honored exactly. + * + *

{@link ByteGenerator} independently caps itself at 25 bytes, which is below this value, so + * today this clamp only bites for a declared {@code BYTES(n)} where {@code n} is between 25 and + * 128 — where it changes nothing either. It is kept as a stated bound so the intent + * survives a change to the generator's own cap. + */ + public static final int MAX_BYTES_LENGTH = 128; + + /** Maximum precision of BigQuery's {@code NUMERIC} type. */ + public static final int MAX_NUMERIC_PRECISION = 38; + + /** + * Maximum scale of BigQuery's {@code NUMERIC} type. The driver binds every {@code BigDecimal} + * as {@code NUMERIC}, so this caps {@code BIGNUMERIC} columns too: a {@code BIGNUMERIC} reports + * scale 38, and a value carrying more than 9 fractional digits is rejected as an out-of-range + * {@code NUMERIC} parameter regardless of what the destination column could hold. + */ + public static final int MAX_NUMERIC_SCALE = 9; + + /** Creates the BigQuery support with its default configuration. */ + public BigQuerySupport() { + } + + @Override + protected void configure(Map registry) { + + // STRING, and the three types the driver reads back as VARCHAR but cannot bind. + registry.put(JDBCType.VARCHAR, (column, random) -> switch (baseTypeName(column)) { + case "STRING" -> new SimpleStringGenerator.Builder(random) + .size(clamp(column.maxSize(), MAX_STRING_LENGTH)) + .build(); + case "JSON" -> throw unsupported(column, + "the driver binds strings as STRING and BigQuery will not coerce STRING to JSON"); + case "GEOGRAPHY" -> throw unsupported(column, + "the driver binds strings as STRING and BigQuery will not coerce STRING to GEOGRAPHY"); + case "INTERVAL" -> throw unsupported(column, + "the driver binds strings as STRING and BigQuery will not coerce STRING to INTERVAL"); + default -> throw unsupported(column, "no generator is registered for this type"); + }); + + registry.put(JDBCType.VARBINARY, (column, random) -> { + if (!"BYTES".equals(baseTypeName(column))) { + throw unsupported(column, "no generator is registered for this type"); + } + return new ByteGenerator.Builder(random) + .size(clamp(column.maxSize(), MAX_BYTES_LENGTH)) + .build(); + }); + + // NUMERIC and BIGNUMERIC both arrive as JDBC NUMERIC. Clamping BIGNUMERIC's reported + // (76, 38) down to NUMERIC's (38, 9) is always safe: a NUMERIC-range value is assignable + // to a BIGNUMERIC column, and the driver has no BIGNUMERIC parameter binding anyway. + GeneratorFactory bigDecimal = (column, random) -> { + int precision = clamp(column.maxSize(), MAX_NUMERIC_PRECISION); + Integer reportedScale = column.maxDigits(); + int scale = reportedScale == null || reportedScale < 0 + ? 0 + : Math.min(reportedScale, MAX_NUMERIC_SCALE); + return new BigDecimalGenerator.Builder(random) + .precision(precision) + .digits(Math.min(scale, precision)) + .build(); + }; + registry.put(JDBCType.NUMERIC, bigDecimal); + registry.put(JDBCType.DECIMAL, bigDecimal); + + // TIMESTAMP and DATETIME both arrive as JDBC TIMESTAMP; only TIMESTAMP can be bound. + GeneratorFactory inheritedTimestamp = registry.get(JDBCType.TIMESTAMP); + registry.put(JDBCType.TIMESTAMP, (column, random) -> { + if ("DATETIME".equals(baseTypeName(column))) { + throw unsupported(column, "the driver has no DATETIME parameter type and BigQuery " + + "will not coerce TIMESTAMP to DATETIME"); + } + return inheritedTimestamp.create(column, random); + }); + + // RANGE<...> is the only type the driver maps to OTHER. + registry.put(JDBCType.OTHER, (column, random) -> { + throw unsupported(column, "RANGE values have no JDBC parameter binding"); + }); + + registry.put(JDBCType.ARRAY, (column, random) -> { + throw unsupported(column, "array element generation is not yet supported"); + }); + + // Replace rather than remove the inherited SqlStructGenerator: it cannot know a BigQuery + // struct's shape and would silently generate garbage, and removing it would surface the + // generic "JDBCType [STRUCT] not supported" instead of the actionable message below. + registry.put(JDBCType.STRUCT, (column, random) -> { + throw unsupported(column, "a struct's field shape is not derivable from JDBC metadata"); + }); + } + + /** + * Enabled because BigQuery's {@code PRIMARY KEY}/{@code FOREIGN KEY} constraints are always + * {@code NOT ENFORCED}: there is no enforcement to suspend, so an unordered bulk fill needs no + * preparation and carries no correctness risk. Referential consistency does not depend on + * insert order either — a foreign-key column is seeded from its parent primary-key + * column's seed and replays the same value sequence whenever it is filled. + * + * @return always {@code true} + */ + @Override + public boolean supportsBulkLoad() { + return true; + } + + /** + * A no-op: BigQuery never enforces key constraints, so there is nothing to disable. This is + * deliberate, not a stub — it exists so {@link #supportsBulkLoad()} can return + * {@code true} and unlock the unordered fill path. The connection is not touched. + * + * @param connection ignored + * @param database ignored + * @return a handle recording that nothing was disabled + */ + @Override + public BulkLoadHandle disableConstraints(Connection connection, Database database) { + return BulkLoadHandle.of("no-op: BigQuery PRIMARY KEY/FOREIGN KEY are NOT ENFORCED"); + } + + /** + * A no-op, mirroring {@link #disableConstraints}. The connection is not touched. + * + * @param connection ignored + * @param database ignored + * @param handle ignored + */ + @Override + public void enableConstraints(Connection connection, Database database, BulkLoadHandle handle) { + // nothing was disabled; see disableConstraints + } + + /** + * Enabled because an engine-managed transaction is pure cost on BigQuery: each + * {@code executeBatch} is already a single atomic query job, while {@code setAutoCommit(false)} + * lazily starts a BigQuery session (per-connection overhead and quota) and disables the + * driver's NDJSON load-job path, which is the fastest way to bulk-insert. + * + * @return always {@code true} + */ + @Override + public boolean prefersConnectionDefaultCommit() { + return true; + } + + /** + * Not overridden deliberately: rewriting a batch into a multi-row {@code INSERT} is + * unconditional in tbc-bq-jdbc, so there is no URL parameter to recommend. Load-job tuning + * ({@code batchLoadThreshold}) is documented rather than advertised here, because this hook's + * contract is a batch-rewrite toggle. + * + * @return always {@code null} + */ + @Override + public String batchRewriteUrlParameter() { + return null; + } + + /** + * Reduces a BigQuery {@code TYPE_NAME} to its base type. The driver reports the raw + * {@code INFORMATION_SCHEMA.COLUMNS.data_type} text, so names arrive parameterized + * ({@code STRING(20)}, {@code NUMERIC(10, 2)}) or with type arguments ({@code ARRAY}, + * {@code STRUCT}, {@code RANGE}). + * + * @param column the column whose type name to normalize + * @return the upper-cased base type name, or an empty string when unreported + */ + private static String baseTypeName(Column column) { + String typeName = column.typeName(); + if (typeName == null || typeName.isBlank()) { + return ""; + } + String upper = typeName.trim().toUpperCase(Locale.ROOT); + int cut = upper.length(); + int paren = upper.indexOf('('); + if (paren >= 0) { + cut = paren; + } + int angle = upper.indexOf('<'); + if (angle >= 0 && angle < cut) { + cut = angle; + } + return upper.substring(0, cut).trim(); + } + + /** + * Clamps a reported size downward to a cap. Clamping only ever shortens, so a generated value + * always remains valid for the declared column; an unreported or non-positive size falls back + * to the cap. + * + * @param reported the size reported by {@code getColumns}, may be null + * @param cap the maximum value to allow + * @return the clamped size + */ + private static int clamp(Integer reported, int cap) { + if (reported == null || reported <= 0) { + return cap; + } + return Math.min(reported, cap); + } + + private static UnsupportedOperationException unsupported(Column column, String reason) { + return new UnsupportedOperationException(String.format( + "BigQuery type [%s] on column [%s.%s] cannot be filled by a plain parameter binding: %s. " + + "Supply a per-column generator via ColumnConfiguration (or " + + "GeneratorRegistry.registerColumnNamePattern) — see docs/DATABASE_SUPPORT.md#bigquery.", + column.typeName(), column.tableName(), column.name(), reason)); + } +} diff --git a/bloviate-core/src/main/java/io/bloviate/ext/DatabaseSupport.java b/bloviate-core/src/main/java/io/bloviate/ext/DatabaseSupport.java index 5847a80..481fae4 100644 --- a/bloviate-core/src/main/java/io/bloviate/ext/DatabaseSupport.java +++ b/bloviate-core/src/main/java/io/bloviate/ext/DatabaseSupport.java @@ -98,6 +98,33 @@ default Map readConstraints(Connection connection, Str return Map.of(); } + /** + * Whether the fill engine should leave transaction management to the connection rather than + * imposing one of its own. The default is {@code false}: on a parallel or unordered fill, + * {@link io.bloviate.db.DatabaseFiller} upgrades a + * {@link io.bloviate.db.CommitStrategy.Mode#CONNECTION_DEFAULT} strategy to a periodic commit so + * a long-running partition never sits in one unbounded transaction. A support that returns + * {@code true} suppresses that upgrade, so {@code CONNECTION_DEFAULT} stays + * {@code CONNECTION_DEFAULT} on every path and the engine never touches auto-commit. + * + *

This exists for engines where an engine-managed transaction is pure cost. On BigQuery each + * {@code executeBatch} is already a single atomic query job, while {@code setAutoCommit(false)} + * lazily starts a session (per-connection overhead and quota) and disables the driver's fastest + * bulk-insert path. Analytical engines without real transaction support are the other case. + * + *

An explicitly configured commit strategy always wins: this hook only affects the + * engine's own default, never a choice the caller made. It is also only effective if the pool + * leaves auto-commit on — a {@link javax.sql.DataSource} configured with + * {@code autoCommit=false} still hands out connections in a transaction, which Bloviate cannot + * override. + * + * @return whether the engine should defer to the connection's own commit behavior + * @since 3.1.0 + */ + default boolean prefersConnectionDefaultCommit() { + return false; + } + /** * Whether this support can disable and re-enable foreign-key enforcement for an * {@code UNORDERED_BULK} fill (see {@link io.bloviate.db.BulkLoadStrategy}). The default is @@ -164,9 +191,9 @@ default void enableConstraints(Connection connection, Database database, BulkLoa *

Matching is case-insensitive and substring-based: names containing * {@code "cockroach"} map to {@link CockroachDBSupport}, {@code "mariadb"} to * {@link MariaDBSupport}, {@code "mysql"} to {@link MySQLSupport}, {@code "postgres"} to - * {@link PostgresSupport}, {@code "h2"} to {@link H2Support}, and {@code "sqlite"} to - * {@link SQLiteSupport}. Anything else (including {@code null}) falls back to - * {@link DefaultSupport}. + * {@link PostgresSupport}, {@code "h2"} to {@link H2Support}, {@code "sqlite"} to + * {@link SQLiteSupport}, and {@code "bigquery"} to {@link BigQuerySupport}. Anything else + * (including {@code null}) falls back to {@link DefaultSupport}. * *

CockroachDB note: CockroachDB is typically reached through the * PostgreSQL JDBC driver, which reports its product name as {@code "PostgreSQL"}, so such @@ -182,6 +209,12 @@ default void enableConstraints(Connection connection, Database database, BulkLoa * {@link MariaDBSupport} extends {@link MySQLSupport} and the inherited type handling * works against MariaDB. * + *

BigQuery note: {@link BigQuerySupport} is written against the + * tbc-bq-jdbc driver, which reports {@code "BigQuery (TBC Driver)"}. The substring also matches + * Simba's {@code "Google BigQuery"}, which is usually what you want — but that driver + * reports type names differently, and {@link BigQuerySupport} discriminates on the raw + * {@code INFORMATION_SCHEMA} type text, so some columns may not resolve. + * * @param productName the database product name, may be null * @return the matching support, or {@link DefaultSupport} if none matches */ @@ -206,6 +239,9 @@ static DatabaseSupport forProduct(String productName) { if (name.contains("h2")) { return new H2Support(); } + if (name.contains("bigquery")) { + return new BigQuerySupport(); + } } return new DefaultSupport(); } diff --git a/bloviate-core/src/test/java/io/bloviate/db/BaseDatabaseTestCase.java b/bloviate-core/src/test/java/io/bloviate/db/BaseDatabaseTestCase.java index a2fe27f..be6e13d 100644 --- a/bloviate-core/src/test/java/io/bloviate/db/BaseDatabaseTestCase.java +++ b/bloviate-core/src/test/java/io/bloviate/db/BaseDatabaseTestCase.java @@ -22,10 +22,17 @@ import org.testcontainers.containers.JdbcDatabaseContainer; import javax.sql.DataSource; +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; import java.sql.Connection; import java.sql.ResultSet; import java.sql.SQLException; import java.sql.Statement; +import java.util.Map; import static org.junit.jupiter.api.Assertions.assertEquals; @@ -49,6 +56,56 @@ protected interface Verifier { void verify(Connection connection) throws SQLException; } + /** + * Executes a semicolon-delimited SQL script loaded from the test classpath. Line comments + * ({@code --}) are stripped; statements are split on {@code ;} at end of line. + */ + protected static void runScript(Connection connection, String resource) throws SQLException { + runScript(connection, resource, Map.of()); + } + + /** + * As {@link #runScript(Connection, String)}, but first replaces {@code ${name}} tokens in the + * script with the supplied values. Databases without transactional DDL or throwaway instances + * (BigQuery) need per-run table names so repeated or concurrent runs cannot collide. + */ + protected static void runScript(Connection connection, String resource, Map tokens) throws SQLException { + String sql = readResource(resource); + for (Map.Entry token : tokens.entrySet()) { + sql = sql.replace("${" + token.getKey() + "}", token.getValue()); + } + try (Statement statement = connection.createStatement()) { + for (String stmt : sql.split(";\\s*\\n")) { + String trimmed = stmt.strip(); + if (!trimmed.isEmpty()) { + statement.execute(trimmed); + } + } + } + } + + private static String readResource(String resource) { + try (InputStream in = BaseDatabaseTestCase.class.getClassLoader().getResourceAsStream(resource)) { + if (in == null) { + throw new IllegalArgumentException("init script not found on classpath: " + resource); + } + StringBuilder builder = new StringBuilder(); + try (BufferedReader reader = new BufferedReader(new InputStreamReader(in, StandardCharsets.UTF_8))) { + String line; + while ((line = reader.readLine()) != null) { + int comment = line.indexOf("--"); + if (comment >= 0) { + line = line.substring(0, comment); + } + builder.append(line).append('\n'); + } + } + return builder.toString(); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + protected static void assertRowCount(Connection connection, String table, long expected) throws SQLException { assertCount(connection, "select count(*) from " + table, expected); } diff --git a/bloviate-core/src/test/java/io/bloviate/db/BaseEmbeddedTest.java b/bloviate-core/src/test/java/io/bloviate/db/BaseEmbeddedTest.java index 0bec12b..56df6c4 100644 --- a/bloviate-core/src/test/java/io/bloviate/db/BaseEmbeddedTest.java +++ b/bloviate-core/src/test/java/io/bloviate/db/BaseEmbeddedTest.java @@ -16,16 +16,9 @@ package io.bloviate.db; -import java.io.BufferedReader; -import java.io.IOException; -import java.io.InputStream; -import java.io.InputStreamReader; -import java.io.UncheckedIOException; -import java.nio.charset.StandardCharsets; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; -import java.sql.Statement; /** * Base class for embedded, in-process database tests (H2, SQLite) that need no Docker. Subclasses @@ -55,41 +48,4 @@ protected void fillDatabase(String jdbcUrl, String user, String password, String } } - /** - * Executes a semicolon-delimited SQL script loaded from the test classpath. Line comments - * ({@code --}) are stripped; statements are split on {@code ;} at end of line. - */ - protected static void runScript(Connection connection, String resource) throws SQLException { - String sql = readResource(resource); - try (Statement statement = connection.createStatement()) { - for (String stmt : sql.split(";\\s*\\n")) { - String trimmed = stmt.strip(); - if (!trimmed.isEmpty()) { - statement.execute(trimmed); - } - } - } - } - - private static String readResource(String resource) { - try (InputStream in = BaseEmbeddedTest.class.getClassLoader().getResourceAsStream(resource)) { - if (in == null) { - throw new IllegalArgumentException("init script not found on classpath: " + resource); - } - StringBuilder builder = new StringBuilder(); - try (BufferedReader reader = new BufferedReader(new InputStreamReader(in, StandardCharsets.UTF_8))) { - String line; - while ((line = reader.readLine()) != null) { - int comment = line.indexOf("--"); - if (comment >= 0) { - line = line.substring(0, comment); - } - builder.append(line).append('\n'); - } - } - return builder.toString(); - } catch (IOException e) { - throw new UncheckedIOException(e); - } - } } diff --git a/bloviate-core/src/test/java/io/bloviate/db/BigQueryFillerTest.java b/bloviate-core/src/test/java/io/bloviate/db/BigQueryFillerTest.java new file mode 100644 index 0000000..472d2b2 --- /dev/null +++ b/bloviate-core/src/test/java/io/bloviate/db/BigQueryFillerTest.java @@ -0,0 +1,168 @@ +/* + * Copyright (c) 2021 Tim Veil + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.bloviate.db; + +import io.bloviate.ext.BigQuerySupport; +import io.bloviate.util.DatabaseUtils; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIf; + +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.Locale; +import java.util.Map; +import java.util.UUID; + +import static org.junit.jupiter.api.Assertions.assertFalse; + +/** + * Fills a real BigQuery dataset. There is no emulator — the tbc-bq-jdbc driver deliberately dropped + * its emulator tier because the emulator's semantics diverged far enough to hide real defects — so + * this test needs a live Google Cloud project and is skipped by default. + * + *

It is gated twice, and both gates must pass for it to run: + *

    + *
  1. {@code BLOVIATE_BQ_PROJECT} is set (and Application Default Credentials are available); and
    2. + *
  2. the driver is on the classpath, which only happens under {@code -Pbigquery}.
    3. + *
+ * The second gate matters on its own: with the env var set but the profile off, the test skips + * rather than failing with "No suitable driver". Nothing here imports a driver class, so it + * compiles in every build. + * + * 
+ * ./mvnw clean install                        # once, in the tbc-bq-jdbc repo
+ * gcloud auth application-default login
+ * export BLOVIATE_BQ_PROJECT=my-gcp-project
+ * ./mvnw verify -Pbigquery -pl bloviate-core -Dtest=BigQueryFillerTest
+ *
+ * + *

Each run creates its own dataset and drops it afterwards, because {@link DatabaseFiller} fills + * every table it finds in the connection's schema. The dataset also carries a one-day + * default table expiration so an aborted run cannot leave billable tables behind. + */ +@EnabledIf("bigQueryAvailable") +class BigQueryFillerTest extends BaseDatabaseTestCase { + + private static final String PROJECT_ENV = "BLOVIATE_BQ_PROJECT"; + private static final String DRIVER_CLASS = "vc.tbc.bq.jdbc.BQDriver"; + + private static final int BATCH_SIZE = 500; + private static final long ROW_COUNT = 25; + + @SuppressWarnings("unused") // referenced by @EnabledIf + static boolean bigQueryAvailable() { + if (System.getenv(PROJECT_ENV) == null) { + return false; + } + try { + Class.forName(DRIVER_CLASS); + return true; + } catch (ClassNotFoundException e) { + return false; + } + } + + @Test + void fillsScalarTypesAndFollowsUnenforcedForeignKeys() throws SQLException { + String project = System.getenv(PROJECT_ENV); + String runId = UUID.randomUUID().toString().replace("-", "").substring(0, 8).toLowerCase(Locale.ROOT); + String dataset = "bloviate_it_" + runId; + String url = String.format("jdbc:bigquery:%s/%s?authType=ADC", project, dataset); + + DatabaseConfiguration configuration = new DatabaseConfiguration.Builder( + BATCH_SIZE, ROW_COUNT, new BigQuerySupport()).build(); + + try (Connection connection = DriverManager.getConnection(url)) { + createDataset(connection, project, dataset); + try { + runScript(connection, "create_tables.bigquery.sql", + Map.of("dataset", dataset, "suffix", runId)); + + assertUnenforcedForeignKeysAreVisible(connection, runId); + + new DatabaseFiller.Builder(connection, configuration).build().fill(); + + verify(connection, dataset, runId); + } finally { + dropDataset(connection, project, dataset); + } + } + } + + private static void createDataset(Connection connection, String project, String dataset) throws SQLException { + try (Statement statement = connection.createStatement()) { + statement.execute(String.format( + "CREATE SCHEMA IF NOT EXISTS `%s.%s` OPTIONS(default_table_expiration_days = 1)", + project, dataset)); + } + } + + private static void dropDataset(Connection connection, String project, String dataset) throws SQLException { + try (Statement statement = connection.createStatement()) { + statement.execute(String.format("DROP SCHEMA IF EXISTS `%s.%s` CASCADE", project, dataset)); + } + } + + /** + * The single highest-value assertion here: BigQuery's keys are always {@code NOT ENFORCED}, and + * this proves the driver still surfaces them through {@code getImportedKeys} so Bloviate's + * dependency graph and foreign-key value alignment have something to work with. Asserted before + * the fill so a metadata regression is not mistaken for a data problem. + */ + private static void assertUnenforcedForeignKeysAreVisible(Connection connection, String runId) throws SQLException { + Database database = DatabaseUtils.getMetadata(connection); + Table nation = database.getTable("nation_" + runId); + + assertFalse(nation.foreignKeys().isEmpty(), + "expected a NOT ENFORCED foreign key on nation_" + runId + " to be visible via getImportedKeys"); + } + + private static void verify(Connection connection, String dataset, String runId) throws SQLException { + String region = String.format("`%s.region_%s`", dataset, runId); + String nation = String.format("`%s.nation_%s`", dataset, runId); + String standard = String.format("`%s.standard_types_%s`", dataset, runId); + String events = String.format("`%s.events_%s`", dataset, runId); + + assertRowCount(connection, region, ROW_COUNT); + assertRowCount(connection, nation, ROW_COUNT); + assertRowCount(connection, standard, ROW_COUNT); + assertRowCount(connection, events, ROW_COUNT); + + // foreign-key columns are seeded from their parent primary-key column, so every child value + // must resolve even though BigQuery never enforces the constraint + assertCount(connection, String.format( + "select count(*) from %s n left join %s r on n.n_regionkey = r.r_regionkey where r.r_regionkey is null", + nation, region), 0); + + // the clamps held against the real service: a bare STRING column reports a 2 MB maximum and + // a bare NUMERIC reports scale 9, neither of which should reach the wire unmodified + assertCount(connection, String.format( + "select count(*) from %s where length(c_string) > %d", + standard, BigQuerySupport.MAX_STRING_LENGTH), 0); + assertCount(connection, String.format( + "select count(*) from %s where length(c_string_sized) > 20", standard), 0); + assertCount(connection, String.format( + "select count(*) from %s where length(c_bytes) > %d", + standard, BigQuerySupport.MAX_BYTES_LENGTH), 0); + + // BIGNUMERIC is bound as NUMERIC, so its values must sit inside NUMERIC's range + assertCount(connection, String.format( + "select count(*) from %s where abs(c_bignumeric) >= 1e29", standard), 0); + } +} diff --git a/bloviate-core/src/test/java/io/bloviate/db/DatabaseFillerCommitStrategyTest.java b/bloviate-core/src/test/java/io/bloviate/db/DatabaseFillerCommitStrategyTest.java new file mode 100644 index 0000000..3e0fc86 --- /dev/null +++ b/bloviate-core/src/test/java/io/bloviate/db/DatabaseFillerCommitStrategyTest.java @@ -0,0 +1,131 @@ +/* + * Copyright (c) 2021 Tim Veil + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.bloviate.db; + +import io.bloviate.ext.BigQuerySupport; +import io.bloviate.ext.DatabaseSupport; +import io.bloviate.ext.DefaultSupport; +import org.junit.jupiter.api.Test; + +import javax.sql.DataSource; +import java.io.PrintWriter; +import java.sql.Connection; +import java.sql.SQLException; +import java.util.logging.Logger; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +/** + * Docker-free coverage of the commit strategy the parallel and unordered fill paths actually use. + * No connection is ever opened — only the strategy mapping is exercised. + */ +class DatabaseFillerCommitStrategyTest { + + /** Matches {@code DatabaseFiller.DEFAULT_PARALLEL_COMMIT_BATCHES}. */ + private static final int DEFAULT_PARALLEL_COMMIT_BATCHES = 64; + + private static CommitStrategy effectiveStrategy(DatabaseSupport support, CommitStrategy configured) { + DatabaseConfiguration configuration = new DatabaseConfiguration.Builder(128, 10, support) + .commitStrategy(configured) + .build(); + + return new DatabaseFiller.Builder(new UnusableDataSource(), configuration) + .threads(4) + .build() + .effectiveParallelCommitStrategy(); + } + + @Test + void upgradesConnectionDefaultToABoundedCommitCadence() { + // a pooled worker must not be left on the connection's autocommit, and a single per-table + // commit would hold a whole partition open in one server-side transaction + assertEquals(CommitStrategy.everyNBatches(DEFAULT_PARALLEL_COMMIT_BATCHES), + effectiveStrategy(new DefaultSupport(), CommitStrategy.connectionDefault())); + } + + @Test + void leavesConnectionDefaultAloneWhenTheSupportPrefersIt() { + // BigQuery: each executeBatch is already one atomic query job, while setAutoCommit(false) + // opens a session and disables the driver's load-job path + assertEquals(CommitStrategy.connectionDefault(), + effectiveStrategy(new BigQuerySupport(), CommitStrategy.connectionDefault())); + } + + @Test + void honorsAnExplicitStrategyEvenWhenTheSupportPrefersOtherwise() { + // the hook only suppresses the engine's own default; a caller's explicit choice still wins + assertEquals(CommitStrategy.perTable(), + effectiveStrategy(new BigQuerySupport(), CommitStrategy.perTable())); + assertEquals(CommitStrategy.everyNBatches(8), + effectiveStrategy(new BigQuerySupport(), CommitStrategy.everyNBatches(8))); + } + + @Test + void honorsAnExplicitStrategyForOrdinarySupports() { + assertEquals(CommitStrategy.perTable(), + effectiveStrategy(new DefaultSupport(), CommitStrategy.perTable())); + } + + /** A {@link DataSource} that fails loudly if anything actually tries to connect. */ + private static final class UnusableDataSource implements DataSource { + + @Override + public Connection getConnection() { + throw new UnsupportedOperationException("this test must not open a connection"); + } + + @Override + public Connection getConnection(String username, String password) { + return getConnection(); + } + + @Override + public PrintWriter getLogWriter() { + return null; + } + + @Override + public void setLogWriter(PrintWriter out) { + // no-op + } + + @Override + public void setLoginTimeout(int seconds) { + // no-op + } + + @Override + public int getLoginTimeout() { + return 0; + } + + @Override + public Logger getParentLogger() { + return Logger.getGlobal(); + } + + @Override + public T unwrap(Class iface) throws SQLException { + throw new SQLException("not a wrapper"); + } + + @Override + public boolean isWrapperFor(Class iface) { + return false; + } + } +} diff --git a/bloviate-core/src/test/java/io/bloviate/ext/BatchRewriteParameterTest.java b/bloviate-core/src/test/java/io/bloviate/ext/BatchRewriteParameterTest.java index daa41a4..957a43b 100644 --- a/bloviate-core/src/test/java/io/bloviate/ext/BatchRewriteParameterTest.java +++ b/bloviate-core/src/test/java/io/bloviate/ext/BatchRewriteParameterTest.java @@ -43,4 +43,12 @@ void cockroachDbHasNoBatchRewriteParameter() { void defaultSupportHasNoBatchRewriteParameter() { assertNull(new DefaultSupport().batchRewriteUrlParameter()); } + + @Test + void bigQueryHasNoBatchRewriteParameter() { + // tbc-bq-jdbc rewrites batches into multi-row INSERTs unconditionally, so there is no URL + // parameter to recommend -- advertising batchLoadThreshold here would emit wrong advice, + // since it takes an integer rather than acting as a toggle + assertNull(new BigQuerySupport().batchRewriteUrlParameter()); + } } diff --git a/bloviate-core/src/test/java/io/bloviate/ext/BigQuerySupportTest.java b/bloviate-core/src/test/java/io/bloviate/ext/BigQuerySupportTest.java new file mode 100644 index 0000000..2a59e70 --- /dev/null +++ b/bloviate-core/src/test/java/io/bloviate/ext/BigQuerySupportTest.java @@ -0,0 +1,284 @@ +/* + * Copyright (c) 2021 Tim Veil + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.bloviate.ext; + +import io.bloviate.db.Column; +import io.bloviate.gen.BigDecimalGenerator; +import io.bloviate.gen.BooleanGenerator; +import io.bloviate.gen.ByteGenerator; +import io.bloviate.gen.DataGenerator; +import io.bloviate.gen.DoubleGenerator; +import io.bloviate.gen.LongGenerator; +import io.bloviate.gen.SimpleStringGenerator; +import io.bloviate.gen.SqlDateGenerator; +import io.bloviate.gen.SqlTimeGenerator; +import io.bloviate.gen.SqlTimestampGenerator; +import org.junit.jupiter.api.Test; + +import java.math.BigDecimal; +import java.sql.JDBCType; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Random; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Docker-free coverage of {@link BigQuerySupport}. The BigQuery integration test needs a live + * project, so these synthetic-{@link Column} tests are the only ones that run in CI — they carry + * the type mapping, the size/precision clamps, and the rejection messages. + */ +class BigQuerySupportTest { + + /** How many draws to take before trusting a bound; generators are random per call. */ + private static final int DRAWS = 50; + + private static final Random RANDOM = new Random(1); + + private final BigQuerySupport support = new BigQuerySupport(); + + private static Column column(JDBCType type, Integer maxSize, Integer maxDigits, String typeName) { + return new Column("c", "t", null, null, type, maxSize, maxDigits, typeName, false, true, null, 1); + } + + private DataGenerator generatorFor(JDBCType type, Integer maxSize, Integer maxDigits, String typeName) { + return support.getDataGenerator(column(type, maxSize, maxDigits, typeName), RANDOM); + } + + private DataGenerator generatorFor(JDBCType type, Integer maxSize, String typeName) { + return generatorFor(type, maxSize, null, typeName); + } + + private UnsupportedOperationException rejectionFor(JDBCType type, String typeName) { + return assertThrows(UnsupportedOperationException.class, () -> generatorFor(type, null, typeName)); + } + + // ---------------------------------------------------------------- natively bindable types + + @Test + void mapsNativelyBindableTypesToTheInheritedDefaults() { + // these ride on AbstractDatabaseSupport's defaults; asserting them here means a future + // change to registerDefaults that breaks BigQuery fails this test rather than a live fill + assertInstanceOf(LongGenerator.class, generatorFor(JDBCType.BIGINT, null, "INT64")); + assertInstanceOf(DoubleGenerator.class, generatorFor(JDBCType.DOUBLE, null, "FLOAT64")); + assertInstanceOf(BooleanGenerator.class, generatorFor(JDBCType.BOOLEAN, null, "BOOL")); + assertInstanceOf(SqlDateGenerator.class, generatorFor(JDBCType.DATE, null, "DATE")); + assertInstanceOf(SqlTimeGenerator.class, generatorFor(JDBCType.TIME, null, "TIME")); + assertInstanceOf(SqlTimestampGenerator.class, generatorFor(JDBCType.TIMESTAMP, null, "TIMESTAMP")); + } + + // ---------------------------------------------------------------------------- STRING/BYTES + + @Test + void honorsADeclaredStringWidth() { + DataGenerator generator = generatorFor(JDBCType.VARCHAR, 20, "STRING(20)"); + assertInstanceOf(SimpleStringGenerator.class, generator); + + for (int i = 0; i < DRAWS; i++) { + assertTrue(((String) generator.generate()).length() <= 20); + } + } + + @Test + void clampsABareStringDownFromTheBigQueryTypeMaximum() { + // a bare STRING column reports 2,097,152 — the type limit, not a declared width + DataGenerator generator = generatorFor(JDBCType.VARCHAR, 2_097_152, "STRING"); + + for (int i = 0; i < DRAWS; i++) { + assertTrue(((String) generator.generate()).length() <= BigQuerySupport.MAX_STRING_LENGTH); + } + } + + @Test + void clampsBytesDownFromTheBigQueryTypeMaximum() { + DataGenerator generator = generatorFor(JDBCType.VARBINARY, 10_485_760, "BYTES"); + assertInstanceOf(ByteGenerator.class, generator); + + for (int i = 0; i < DRAWS; i++) { + assertTrue(((Byte[]) generator.generate()).length <= BigQuerySupport.MAX_BYTES_LENGTH); + } + } + + @Test + void honorsADeclaredBytesWidth() { + DataGenerator generator = generatorFor(JDBCType.VARBINARY, 16, "BYTES(16)"); + + for (int i = 0; i < DRAWS; i++) { + assertTrue(((Byte[]) generator.generate()).length <= 16); + } + } + + @Test + void fallsBackToTheCapWhenSizeIsUnreported() { + // COLUMN_SIZE is nullable metadata; neither branch may unbox it + DataGenerator string = generatorFor(JDBCType.VARCHAR, null, "STRING"); + DataGenerator bytes = generatorFor(JDBCType.VARBINARY, null, "BYTES"); + + assertTrue(((String) string.generate()).length() <= BigQuerySupport.MAX_STRING_LENGTH); + assertTrue(((Byte[]) bytes.generate()).length <= BigQuerySupport.MAX_BYTES_LENGTH); + } + + // -------------------------------------------------------------------- NUMERIC / BIGNUMERIC + + @Test + void generatesAValidNumeric() { + DataGenerator generator = generatorFor(JDBCType.NUMERIC, 38, 9, "NUMERIC"); + assertInstanceOf(BigDecimalGenerator.class, generator); + + for (int i = 0; i < DRAWS; i++) { + BigDecimal value = (BigDecimal) generator.generate(); + assertTrue(value.scale() <= BigQuerySupport.MAX_NUMERIC_SCALE); + assertTrue(value.precision() <= BigQuerySupport.MAX_NUMERIC_PRECISION); + } + } + + @Test + void clampsBigNumericIntoNumericRange() { + // BIGNUMERIC reports (76, 38). The driver binds every BigDecimal as NUMERIC, whose maximum + // scale is 9, so an unclamped draw carries 25 fractional digits and BigQuery rejects it as + // an out-of-range NUMERIC parameter — regardless of what the destination column could hold. + DataGenerator generator = generatorFor(JDBCType.NUMERIC, 76, 38, "BIGNUMERIC"); + + for (int i = 0; i < DRAWS; i++) { + BigDecimal value = (BigDecimal) generator.generate(); + assertTrue(value.scale() <= BigQuerySupport.MAX_NUMERIC_SCALE, + "scale was " + value.scale() + " for " + value); + assertTrue(value.precision() <= BigQuerySupport.MAX_NUMERIC_PRECISION, + "precision was " + value.precision() + " for " + value); + } + } + + @Test + void honorsADeclaredNumericScale() { + DataGenerator generator = generatorFor(JDBCType.NUMERIC, 10, 2, "NUMERIC(10, 2)"); + + for (int i = 0; i < DRAWS; i++) { + BigDecimal value = (BigDecimal) generator.generate(); + assertTrue(value.scale() <= 2, "scale was " + value.scale() + " for " + value); + assertTrue(value.precision() <= 10, "precision was " + value.precision() + " for " + value); + } + } + + @Test + void toleratesUnreportedNumericPrecisionAndScale() { + DataGenerator generator = generatorFor(JDBCType.NUMERIC, null, null, "NUMERIC"); + BigDecimal value = (BigDecimal) generator.generate(); + + assertEquals(0, value.scale()); + assertTrue(value.precision() <= BigQuerySupport.MAX_NUMERIC_PRECISION); + } + + // ------------------------------------------------------------------------ rejected types + + @Test + void rejectsDatetimeWhichIsIndistinguishableFromTimestampByJdbcTypeAlone() { + UnsupportedOperationException e = rejectionFor(JDBCType.TIMESTAMP, "DATETIME"); + + assertTrue(e.getMessage().contains("DATETIME")); + assertTrue(e.getMessage().contains("ColumnConfiguration")); + } + + @Test + void rejectsTypesWithNoParameterBinding() { + Map rejected = new LinkedHashMap<>(); + rejected.put(JDBCType.OTHER, "RANGE"); + rejected.put(JDBCType.ARRAY, "ARRAY"); + rejected.put(JDBCType.STRUCT, "STRUCT"); + + for (String typeName : new String[]{"JSON", "GEOGRAPHY", "INTERVAL"}) { + assertRejected(JDBCType.VARCHAR, typeName); + } + rejected.forEach(this::assertRejected); + } + + private void assertRejected(JDBCType jdbcType, String typeName) { + UnsupportedOperationException e = rejectionFor(jdbcType, typeName); + + // the raw type name is echoed so the message names the column's actual declared type + assertTrue(e.getMessage().contains(typeName), e.getMessage()); + // and points at the escape hatch, not at registerTypeName — which cannot match a + // parameterized BigQuery type name like "numeric(10, 2)" + assertTrue(e.getMessage().contains("ColumnConfiguration"), e.getMessage()); + assertFalse(e.getMessage().contains("registerTypeName"), e.getMessage()); + } + + @Test + void rejectsStructRatherThanInheritingTheStructGenerator() { + // AbstractDatabaseSupport maps STRUCT to SqlStructGenerator by default, which cannot know a + // BigQuery struct's field shape and would silently generate garbage + assertThrows(UnsupportedOperationException.class, + () -> generatorFor(JDBCType.STRUCT, null, "STRUCT")); + } + + // -------------------------------------------------------------------- type-name normalizer + + @Test + void normalizesTypeNameCaseWhitespaceAndParameters() { + assertInstanceOf(SimpleStringGenerator.class, generatorFor(JDBCType.VARCHAR, 20, "string(20)")); + assertInstanceOf(SimpleStringGenerator.class, generatorFor(JDBCType.VARCHAR, null, " STRING ")); + assertInstanceOf(SimpleStringGenerator.class, generatorFor(JDBCType.VARCHAR, null, "String")); + } + + @Test + void normalizesTypeArgumentsWhenRejecting() { + assertThrows(UnsupportedOperationException.class, + () -> generatorFor(JDBCType.STRUCT, null, "Struct")); + } + + @Test + void toleratesAnUnreportedTypeName() { + // TYPE_NAME should always be present, but a null must surface as the normal rejection + // rather than a NullPointerException from the normalizer + assertThrows(UnsupportedOperationException.class, () -> generatorFor(JDBCType.VARCHAR, 10, null)); + assertThrows(UnsupportedOperationException.class, () -> generatorFor(JDBCType.VARBINARY, 10, "")); + } + + // ------------------------------------------------------------------------------ capabilities + + @Test + void enablesBulkLoadWithNoOpConstraintHandling() throws Exception { + assertTrue(support.supportsBulkLoad()); + + // BigQuery keys are always NOT ENFORCED, so there is nothing to disable — and nothing that + // needs a connection. Passing null proves the implementation never touches one. + BulkLoadHandle handle = support.disableConstraints(null, null); + assertTrue(handle.description().contains("NOT ENFORCED")); + support.enableConstraints(null, null, handle); + } + + @Test + void prefersTheConnectionsOwnCommitBehavior() { + assertTrue(support.prefersConnectionDefaultCommit()); + } + + @Test + void recommendsNoBatchRewriteParameter() { + // batch rewrite is unconditional in tbc-bq-jdbc, so there is no URL parameter to suggest + assertNull(support.batchRewriteUrlParameter()); + } + + @Test + void readsNoConstraints() { + // BigQuery has no CHECK constraints or enum/domain types + assertEquals(Map.of(), support.readConstraints(null, null, null)); + } +} diff --git a/bloviate-core/src/test/java/io/bloviate/ext/DatabaseSupportSelectionTest.java b/bloviate-core/src/test/java/io/bloviate/ext/DatabaseSupportSelectionTest.java index 4f8284d..ac1fdec 100644 --- a/bloviate-core/src/test/java/io/bloviate/ext/DatabaseSupportSelectionTest.java +++ b/bloviate-core/src/test/java/io/bloviate/ext/DatabaseSupportSelectionTest.java @@ -59,12 +59,26 @@ void selectsSqliteByProductName() { assertInstanceOf(SQLiteSupport.class, DatabaseSupport.forProduct("SQLite")); } + @Test + void selectsBigQueryByProductName() { + // the tbc-bq-jdbc driver reports this exact string + assertInstanceOf(BigQuerySupport.class, DatabaseSupport.forProduct("BigQuery (TBC Driver)")); + } + + @Test + void selectsBigQueryForOtherBigQueryDrivers() { + // Simba's driver reports "Google BigQuery"; the substring match catches it, which is + // usually what you want -- see the documented caveat about its differing type names. + assertInstanceOf(BigQuerySupport.class, DatabaseSupport.forProduct("Google BigQuery")); + } + @Test void matchingIsCaseInsensitive() { assertInstanceOf(MySQLSupport.class, DatabaseSupport.forProduct("mysql")); assertInstanceOf(PostgresSupport.class, DatabaseSupport.forProduct("POSTGRESQL")); assertInstanceOf(MariaDBSupport.class, DatabaseSupport.forProduct("mariadb")); assertInstanceOf(SQLiteSupport.class, DatabaseSupport.forProduct("SQLITE")); + assertInstanceOf(BigQuerySupport.class, DatabaseSupport.forProduct("BIGQUERY")); } @Test diff --git a/bloviate-core/src/test/resources/create_tables.bigquery.sql b/bloviate-core/src/test/resources/create_tables.bigquery.sql new file mode 100644 index 0000000..e22d8b9 --- /dev/null +++ b/bloviate-core/src/test/resources/create_tables.bigquery.sql @@ -0,0 +1,60 @@ +-- BigQuery test schema for BigQueryFillerTest. +-- +-- Substituted tokens: ${dataset} (the target dataset) and ${suffix} (a per-run id, so repeated or +-- concurrent runs never collide). Every table carries a 2-hour expiration so a failed run cannot +-- leave billable tables behind. +-- +-- Only natively bindable types appear here. DatabaseFiller fills every table in the dataset, so a +-- single DATETIME/JSON/GEOGRAPHY/ARRAY/STRUCT column would fail the whole fill; rejection of those +-- is covered by BigQuerySupportTest instead. +-- +-- Keys are declared NOT ENFORCED, which is the only form BigQuery accepts. They are never enforced +-- at write time, but the driver surfaces them through getPrimaryKeys/getImportedKeys, which is what +-- drives Bloviate's dependency graph and foreign-key value alignment. + +CREATE TABLE `${dataset}.region_${suffix}` ( + r_regionkey INT64 NOT NULL, + r_name STRING(25), + r_comment STRING(152), + PRIMARY KEY (r_regionkey) NOT ENFORCED +) OPTIONS(expiration_timestamp = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 2 HOUR)); + +CREATE TABLE `${dataset}.nation_${suffix}` ( + n_nationkey INT64 NOT NULL, + n_regionkey INT64 NOT NULL, + n_name STRING(25), + n_comment STRING(152), + PRIMARY KEY (n_nationkey) NOT ENFORCED, + FOREIGN KEY (n_regionkey) REFERENCES `${dataset}.region_${suffix}`(r_regionkey) NOT ENFORCED +) OPTIONS(expiration_timestamp = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 2 HOUR)); + +-- Every natively bindable type, in both its bare and parameterized form where one exists, so the +-- size and precision clamps are exercised against the real service. +CREATE TABLE `${dataset}.standard_types_${suffix}` ( + c_int64 INT64, + c_float64 FLOAT64, + c_numeric NUMERIC, + c_numeric_scaled NUMERIC(10, 2), + c_bignumeric BIGNUMERIC, + c_bool BOOL, + c_string STRING, + c_string_sized STRING(20), + c_bytes BYTES, + c_bytes_sized BYTES(16), + c_date DATE, + c_time TIME, + c_timestamp TIMESTAMP +) OPTIONS(expiration_timestamp = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 2 HOUR)); + +-- Any real BigQuery table is partitioned and clustered; this proves the batch/DML path works +-- against such a destination rather than only against a plain heap table. +CREATE TABLE `${dataset}.events_${suffix}` ( + e_id INT64 NOT NULL, + e_name STRING(50), + e_amount NUMERIC(12, 2), + e_ts TIMESTAMP NOT NULL, + PRIMARY KEY (e_id) NOT ENFORCED +) +PARTITION BY DATE(e_ts) +CLUSTER BY e_id +OPTIONS(expiration_timestamp = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 2 HOUR)); diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 69b906a..5cedb69 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -263,6 +263,15 @@ applies there too. > correctly-parameterized URL if you construct the `DataSource` yourself. CockroachDB ignores the > parameter, so no warning is emitted there. +> **Tip — BigQuery.** Rewriting is unconditional in tbc-bq-jdbc, so there is no parameter to set and +> no warning to emit. Instead, size `batchSize` against BigQuery's limit of 10,000 query parameters +> per query: the effective rows per job is `10_000 / columnCount`, so the default 128 leaves most of +> a job unused. Start from `batchSize = 5000`, and set the driver's `batchLoadThreshold` to match to +> move large batches onto its NDJSON load-job path. Leave `CommitStrategy` at its default — +> `BigQuerySupport` opts out of engine-managed transactions because `setAutoCommit(false)` starts a +> BigQuery session and silently disables that load path, and each `executeBatch` is already one +> atomic job. Bloviate logs a warning if a commit strategy is configured anyway. + ## Bulk load (unordered fill) The parallel path normally barriers between topological levels, so a **deep, narrow foreign-key diff --git a/docs/DATABASE_SUPPORT.md b/docs/DATABASE_SUPPORT.md index 3df22cf..ba936ef 100644 --- a/docs/DATABASE_SUPPORT.md +++ b/docs/DATABASE_SUPPORT.md @@ -12,6 +12,7 @@ explicitly, or let Bloviate detect it from the connection. | MariaDB | `MariaDBSupport` | Extends `MySQLSupport` (MariaDB speaks the MySQL wire protocol) | | H2 | `H2Support` | Standard JDBC types **plus** `UUID` and `JSON` (embedded; no Docker) | | SQLite | `SQLiteSupport` | Standard JDBC types via type affinity (embedded; no Docker) | +| BigQuery | `BigQuerySupport` | Scalar types only — see [BigQuery](#bigquery) for what is excluded | | Generic JDBC | `DefaultSupport` | Standard JDBC types only | All of them resolve the cross-database defaults for the common JDBC types (integers, decimals, @@ -44,6 +45,87 @@ per SQLite convention, and every value round-trips through affinity rules. Forei default (`PRAGMA foreign_keys = ON` enables them), but Bloviate orders fills by the foreign-key graph regardless. +## BigQuery + +Requires the [tbc-bq-jdbc](https://github.com/Two-Bear-Capital/tbc-bq-jdbc) driver +(`vc.tbc:tbc-bq-jdbc`, 4.3.0 or later), which is not yet on Maven Central — install it locally with +`./mvnw clean install` in that repo, or use its GitHub Releases jar. + +```java +String url = "jdbc:bigquery:my-project/my_dataset?authType=ADC"; +``` + +**Supported:** `STRING`, `BYTES`, `INT64`, `FLOAT64`, `NUMERIC`, `BIGNUMERIC`, `BOOL`, `DATE`, +`TIME`, `TIMESTAMP`. + +**Not supported:** `DATETIME`, `JSON`, `GEOGRAPHY`, `INTERVAL`, `RANGE`, `ARRAY`, `STRUCT`. These +fail fast with a message naming the column, before any rows are written. + +The reason is that BigQuery's *write* surface is narrower than its read surface. The driver reads +`JSON`/`GEOGRAPHY`/`INTERVAL` back as `VARCHAR` and `DATETIME` as `TIMESTAMP`, but has no parameter +binding that produces those types, and BigQuery will not implicitly coerce a `STRING` or `TIMESTAMP` +parameter into them. Filling them requires SQL-side construction (`PARSE_JSON(?)`, +`ST_GEOGFROMTEXT(?)`, `CAST(? AS DATETIME)`), which the fill engine does not yet emit. + +To fill one anyway, supply a per-column generator through `ColumnConfiguration` or +`GeneratorRegistry.registerColumnNamePattern`. The driver does implement `Connection.createArrayOf` +and `Connection.createStruct`, so a hand-written generator can write composites today. + +> `GeneratorRegistry.registerTypeName` is a poor fit here: it matches type names exactly, and +> BigQuery reports the raw `INFORMATION_SCHEMA` text (`string(20)`, `numeric(10, 2)`, +> `array`), so only unparameterized names ever match. + +> Generators registered by column-name pattern — including everything `bloviate-datafaker` +> contributes — rank **above** the support's own type mapping. A `GEOGRAPHY` column whose name +> matches such a pattern will get that generator instead of the error above, and fail at insert +> time. Exclude those columns explicitly if you use pattern-based generators. + +### Generated value sizes + +BigQuery reports `COLUMN_SIZE` as the *type* maximum, not a declared width: a bare `STRING` reports +2,097,152 and a bare `BYTES` reports 10,485,760. Sizes are therefore clamped downward — 256 +characters and 128 bytes respectively — while a declared `STRING(20)` is honored exactly. Similarly, +`BIGNUMERIC` reports precision 76 and scale 38, but the driver binds every `BigDecimal` as `NUMERIC`, +so generated values are clamped to `NUMERIC`'s (38, 9); a `NUMERIC`-range value is always valid in a +`BIGNUMERIC` column. + +### Required driver settings + +Both are the driver's defaults; overriding either breaks the fill. + +- **`includeStructFields=false`** — with struct fields spliced in, `getColumns` reports dotted + sub-field rows alongside their parent and the generated `INSERT` is structurally invalid. +- **`metadataLazyLoad=false`** — with lazy metadata, an unfiltered `getTables`/`getColumns` returns + nothing and the fill silently does no work. + +### Keys and fill order + +BigQuery accepts `PRIMARY KEY`/`FOREIGN KEY` only as `NOT ENFORCED`, but the driver still surfaces +them through `getPrimaryKeys`/`getImportedKeys`, so Bloviate orders fills by the foreign-key graph +and aligns child values with their parents exactly as it does elsewhere. Declare keys in your DDL if +you want referentially consistent data; without them the graph has no edges and foreign-key columns +get independent random values. + +Because nothing is ever enforced, `UNORDERED_BULK` is free here — there is no enforcement to suspend +and no ordering requirement, so `BigQuerySupport` enables it with no-op constraint handling. + +### Performance and cost + +Every statement is a BigQuery job, so per-row inserts are prohibitively slow and batching is +mandatory. The driver collapses a JDBC batch into a single multi-row `INSERT` automatically (no URL +parameter needed), chunked to stay under BigQuery's 10,000-parameters-per-query limit — so the +effective rows per job is `10_000 / columnCount`, and the default `batchSize` of 128 leaves most of +that on the table. Start from `batchSize = 5000`. + +Setting `batchLoadThreshold` to a matching value moves large batches onto the driver's NDJSON +load-job path, which avoids DML quotas and per-job query cost entirely. That path requires +auto-commit to stay on, which is why `BigQuerySupport` opts out of engine-managed transactions: +`setAutoCommit(false)` starts a BigQuery session and silently disables it. **Leave +`CommitStrategy` at its default** — each `executeBatch` is already one atomic job, so an explicit +strategy buys nothing and costs the load path. Bloviate logs a warning if you set one anyway. + +> Filling a BigQuery dataset writes real data to real storage and runs real jobs. Both cost money. + > **PostgreSQL connection requirement:** the vendor types above are bound as their text > representations, and PostgreSQL won't implicitly cast `varchar` to `uuid`/`jsonb`/`bit`/etc. > Open the connection with `stringtype=unspecified` so the server infers each column's type: @@ -66,3 +148,8 @@ DatabaseSupport support = DatabaseSupport.forConnection(connection); > `MariaDBSupport`. The legacy MySQL Connector/J driver reports `MySQL` even against a MariaDB > server and resolves to `MySQLSupport`; since `MariaDBSupport` adds no divergent behavior, the two > are equivalent. + +> **BigQuery note:** `BigQuerySupport` matches any product name containing `bigquery`, which covers +> both tbc-bq-jdbc (`BigQuery (TBC Driver)`) and Simba (`Google BigQuery`). It is written against +> tbc-bq-jdbc's metadata, though, and discriminates on the raw `INFORMATION_SCHEMA` type text, so +> some columns may not resolve under Simba's driver. diff --git a/pom.xml b/pom.xml index 4b6ced1..fd22d21 100644 --- a/pom.xml +++ b/pom.xml @@ -86,6 +86,14 @@ 3.5.9 2.4.240 3.53.2.0 + + 4.3.0 1.37 @@ -258,6 +266,11 @@ sqlite-jdbc ${sqlite.version} + + vc.tbc + tbc-bq-jdbc + ${tbc-bq-jdbc.version} + From 7d054dd06ede7876ed9a3877777fb0638721341e Mon Sep 17 00:00:00 2001 From: Tim Veil <3260845+timveil@users.noreply.github.com> Date: Mon, 3 Aug 2026 15:46:08 -0400 Subject: [PATCH 2/2] docs(ext): fix a dangling javadoc and two overstated claims Review follow-up, all three documentation-only. warnIfEngineManagedCommitDiscouraged was inserted directly above warnIfBulkIgnored's signature, which orphaned that method's javadoc: the bulk-load block ended up documenting nothing while warnIfBulkIgnored had no javadoc of its own. Reordered so each block sits with its method. BigQuerySupport.batchRewriteUrlParameter opened with "Not overridden deliberately" on a method that is, in fact, overridden. The intent was that it changes no behaviour over the interface default; say that instead. DATABASE_SUPPORT.md presented the 128-byte BYTES clamp as something a user would observe, but ByteGenerator caps itself at 25 bytes, so that clamp currently changes no generated value. The constant's own javadoc already said so; the guide now agrees, and distinguishes the string clamp (a real ~8x reduction under the generator's 2000-character limit) from the BYTES one (a stated bound sitting above the generator's). Co-Authored-By: Claude Opus 5 (1M context) --- .../java/io/bloviate/db/DatabaseFiller.java | 14 +++++++------- .../java/io/bloviate/ext/BigQuerySupport.java | 13 +++++++++---- docs/DATABASE_SUPPORT.md | 18 +++++++++++++----- 3 files changed, 29 insertions(+), 16 deletions(-) diff --git a/bloviate-core/src/main/java/io/bloviate/db/DatabaseFiller.java b/bloviate-core/src/main/java/io/bloviate/db/DatabaseFiller.java index 51d2e8f..0164e1e 100644 --- a/bloviate-core/src/main/java/io/bloviate/db/DatabaseFiller.java +++ b/bloviate-core/src/main/java/io/bloviate/db/DatabaseFiller.java @@ -589,6 +589,13 @@ private void warnIfPartitionsIgnored() { * Bulk loading needs per-worker session control, so it only applies to the {@code threads > 1} * {@link DataSource} path; elsewhere the engine fills in dependency order. */ + private void warnIfBulkIgnored() { + if (configuration.bulkLoadStrategy().isUnordered()) { + logger.warn("UNORDERED_BULK is ignored on the sequential fill path; use the DataSource " + + "constructor with threads(n) > 1 for unordered bulk loading"); + } + } + /** * Warns once per fill when an explicit commit strategy is configured against a support that * would rather the engine stayed out of transaction management. The caller's choice is still @@ -606,13 +613,6 @@ private void warnIfEngineManagedCommitDiscouraged() { } } - private void warnIfBulkIgnored() { - if (configuration.bulkLoadStrategy().isUnordered()) { - logger.warn("UNORDERED_BULK is ignored on the sequential fill path; use the DataSource " - + "constructor with threads(n) > 1 for unordered bulk loading"); - } - } - /** * The commit strategy used by parallel workers. A pooled worker connection must not be left on * the connection's autocommit (that would commit per batch and lose the engine-managed diff --git a/bloviate-core/src/main/java/io/bloviate/ext/BigQuerySupport.java b/bloviate-core/src/main/java/io/bloviate/ext/BigQuerySupport.java index 8a82447..ee7a2e3 100644 --- a/bloviate-core/src/main/java/io/bloviate/ext/BigQuerySupport.java +++ b/bloviate-core/src/main/java/io/bloviate/ext/BigQuerySupport.java @@ -240,10 +240,15 @@ public boolean prefersConnectionDefaultCommit() { } /** - * Not overridden deliberately: rewriting a batch into a multi-row {@code INSERT} is - * unconditional in tbc-bq-jdbc, so there is no URL parameter to recommend. Load-job tuning - * ({@code batchLoadThreshold}) is documented rather than advertised here, because this hook's - * contract is a batch-rewrite toggle. + * Returns {@code null}: rewriting a batch into a multi-row {@code INSERT} is unconditional in + * tbc-bq-jdbc, so there is no URL parameter to recommend and the fill engine should stay quiet + * rather than warn about a missing one. + * + *

This overrides nothing behaviorally — the interface default is already {@code null} + * — but it is stated explicitly so the reasoning is recorded where someone would look for + * it. In particular, load-job tuning ({@code batchLoadThreshold}) does not belong here: + * this hook's contract is a batch-rewrite toggle, and that property takes an integer, so + * advertising it would make the engine emit wrong advice. * * @return always {@code null} */ diff --git a/docs/DATABASE_SUPPORT.md b/docs/DATABASE_SUPPORT.md index ba936ef..9f6e74b 100644 --- a/docs/DATABASE_SUPPORT.md +++ b/docs/DATABASE_SUPPORT.md @@ -83,11 +83,19 @@ and `Connection.createStruct`, so a hand-written generator can write composites ### Generated value sizes BigQuery reports `COLUMN_SIZE` as the *type* maximum, not a declared width: a bare `STRING` reports -2,097,152 and a bare `BYTES` reports 10,485,760. Sizes are therefore clamped downward — 256 -characters and 128 bytes respectively — while a declared `STRING(20)` is honored exactly. Similarly, -`BIGNUMERIC` reports precision 76 and scale 38, but the driver binds every `BigDecimal` as `NUMERIC`, -so generated values are clamped to `NUMERIC`'s (38, 9); a `NUMERIC`-range value is always valid in a -`BIGNUMERIC` column. +2,097,152 and a bare `BYTES` reports 10,485,760. Sizes are therefore clamped downward — to 256 +characters and 128 bytes — while a declared `STRING(20)` is honored exactly. + +Those caps are ceilings, not target lengths, and the generators impose their own limits underneath: +`SimpleStringGenerator` never exceeds 2000 characters and `ByteGenerator` never exceeds 25 bytes. So +the string clamp is a further ~8x reduction that you will observe, whereas the `BYTES` clamp sits +above the generator's own limit and does not currently change any generated value. It is stated as a +bound so the intent survives a change to that generator. + +`BIGNUMERIC` is clamped for a different reason: it reports precision 76 and scale 38, but the driver +binds every `BigDecimal` as `NUMERIC`, so generated values are clamped to `NUMERIC`'s (38, 9). The +binding is what forces this, not the destination column — a `NUMERIC`-range value is always valid in +a `BIGNUMERIC` column. ### Required driver settings