Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ This is a Maven-based Java 25 project. Run `./mvnw` from the repository root to
- `TableFiller`: Handles filling individual tables with generated data

2. **Database Support (`io.bloviate.ext`)**:
- Database-specific implementations: `PostgresSupport`, `MySQLSupport`, `CockroachDBSupport`, `DefaultSupport`
- Database-specific implementations: `PostgresSupport`, `MySQLSupport`, `MariaDBSupport`, `CockroachDBSupport`, `H2Support`, `SQLiteSupport`, `BigQuerySupport`, `DefaultSupport`
- Each provides database-specific SQL generation and data type mapping

3. **Data Generators (`io.bloviate.gen`)**:
Expand Down
36 changes: 34 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,8 +132,40 @@ Docker must be running.
```

Test schemas live under `bloviate-core/src/test/resources/` (TPCC, AuctionMark, Wikipedia, and
others). `BaseDatabaseTestCase` provides the shared `DataSource` plumbing and the
fidelity assertions used by the TPC-C tests.
others). `BaseDatabaseTestCase` provides the shared `DataSource` plumbing, the classpath script
runner, and the fidelity assertions used by the TPC-C tests.

### BigQuery

`BigQueryFillerTest` is the one test Docker cannot cover. BigQuery has no usable emulator — the
tbc-bq-jdbc driver deliberately removed its emulator tier because the emulator diverged far enough
from the service to hide real defects — so the test needs a live Google Cloud project, and it is
**skipped by default**.

It is gated twice, and both gates must pass:

1. `BLOVIATE_BQ_PROJECT` is set (with Application Default Credentials available), and
2. the driver is on the classpath, which only happens under `-Pbigquery`.

The second gate exists so that setting the env var without the profile skips cleanly instead of
failing with "No suitable driver". `vc.tbc:tbc-bq-jdbc` is not on Maven Central yet, which is why it
is declared in an opt-in profile rather than as an ordinary test dependency — a default build must
stay resolvable for everyone. Bump `tbc-bq-jdbc.version` in the root POM by hand; Dependabot and
`versions:display-dependency-updates` cannot resolve that coordinate.

```bash
# once, in a clone of https://github.com/Two-Bear-Capital/tbc-bq-jdbc
./mvnw clean install

gcloud auth application-default login
export BLOVIATE_BQ_PROJECT=my-gcp-project

./mvnw verify -Pbigquery -pl bloviate-core -Dtest=BigQueryFillerTest
```

Each run creates its own dataset and drops it afterwards (with a one-day default table expiration as
a backstop), because `DatabaseFiller` fills **every** table it finds in the connection's schema.
Running it writes real data and runs real jobs, both of which cost money.

## Databases for Testing

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ by seed, and runs inside your JUnit/Testcontainers pipeline.
**Full documentation, guides, and examples live at [bloviate.io](https://bloviate.io).**

- [Quick Start](https://bloviate.io/guides/quickstart/) — install and fill a database or flat file
- [Database Support](https://bloviate.io/guides/database-support/) — PostgreSQL, MySQL, MariaDB, CockroachDB, H2, SQLite
- [Database Support](https://bloviate.io/guides/database-support/) — PostgreSQL, MySQL, MariaDB, CockroachDB, H2, SQLite, BigQuery
- [Configuration](https://bloviate.io/guides/configuration/) — per-table/column control, distributions, seeds, parallelism
- [Generators](https://bloviate.io/guides/generators/) — registry, realistic data, composite keys, TPC-C
- [Testing Integrations](https://bloviate.io/guides/integrations/) — JUnit Jupiter and Testcontainers
Expand All @@ -31,7 +31,7 @@ by seed, and runs inside your JUnit/Testcontainers pipeline.
- **Deterministic by seed** — same seed + schema ⇒ byte-identical data, even under parallel fills
- **Per-column control** and **pluggable generators**; realistic semantic values via Datafaker
- **Parallel & partitioned fills** for large datasets, with referential integrity preserved
- **PostgreSQL, MySQL, MariaDB, CockroachDB, H2, SQLite**, plus CSV/TSV/pipe **flat-file** output
- **PostgreSQL, MySQL, MariaDB, CockroachDB, H2, SQLite, BigQuery**, plus CSV/TSV/pipe **flat-file** output
- First-class **JUnit Jupiter** (JUnit 5/6) and **Testcontainers** integrations

See the [full feature tour and guides on bloviate.io](https://bloviate.io).
Expand Down
26 changes: 26 additions & 0 deletions bloviate-core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -168,4 +168,30 @@

</dependencies>

<profiles>
<!--
BigQuery Support (live Google Cloud project, no Docker, no emulator)

Opt-in only. tbc-bq-jdbc is not on Maven Central yet, so declaring it here rather than in
<dependencies> keeps the default build resolvable for everyone. BigQuerySupport itself
needs no driver on the compile classpath, and BigQueryFillerTest talks only to
java.sql/DriverManager, so both compile without this profile; the test simply skips.

./mvnw clean install # in the tbc-bq-jdbc repo, once
export BLOVIATE_BQ_PROJECT=my-gcp-project
gcloud auth application-default login
./mvnw verify -Pbigquery -pl bloviate-core
-->
<profile>
<id>bigquery</id>
<dependencies>
<dependency>
<groupId>vc.tbc</groupId>
<artifactId>tbc-bq-jdbc</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</profile>
</profiles>

</project>
37 changes: 33 additions & 4 deletions bloviate-core/src/main/java/io/bloviate/db/DatabaseFiller.java
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,8 @@ public void fill() throws SQLException {

visualizeGraph(reversedGraph, database.catalog());

warnIfEngineManagedCommitDiscouraged();

// recommend the driver batch-rewrite URL parameter once per fill if it is missing
if (connection != null) {
warnIfBatchRewriteMissing(connection);
Expand Down Expand Up @@ -594,6 +596,23 @@ private void warnIfBulkIgnored() {
}
}

/**
Comment thread
timveil marked this conversation as resolved.
* Warns once per fill when an explicit commit strategy is configured against a support that
* would rather the engine stayed out of transaction management. The caller's choice is still
* honored — this only surfaces the cost, which is otherwise invisible (on BigQuery, an engine-
* managed transaction opens a session and silently disables the driver's load-job path).
*/
private void warnIfEngineManagedCommitDiscouraged() {
if (configuration.databaseSupport().prefersConnectionDefaultCommit()
&& configuration.commitStrategy().managesTransaction()) {
logger.warn("{} recommends leaving transaction management to the connection, but commit "
+ "strategy [{}] was configured; the engine will manage transactions as asked, "
+ "which may be slower and can disable driver bulk-load paths",
configuration.databaseSupport().getClass().getSimpleName(),
configuration.commitStrategy().mode());
}
}

/**
* The commit strategy used by parallel workers. A pooled worker connection must not be left on
* the connection's autocommit (that would commit per batch and lose the engine-managed
Expand All @@ -603,12 +622,22 @@ private void warnIfBulkIgnored() {
* large partition open in one server-side transaction (unbounded WAL/undo growth and lock
* accumulation), which is the scale failure the parallel/bulk path most needs to avoid. Any
* explicitly configured strategy (including {@link CommitStrategy#perTable()}) is honored as-is.
*
* <p>A {@link io.bloviate.ext.DatabaseSupport#prefersConnectionDefaultCommit() support that
* prefers the connection's own commit behavior} suppresses the upgrade, so
* {@code CONNECTION_DEFAULT} stays as configured. That is for engines where an engine-managed
* transaction is pure cost rather than protection — see the hook's documentation.
*
* <p>Package-private so the mapping can be unit-tested without a database.
*/
private CommitStrategy effectiveParallelCommitStrategy() {
CommitStrategy effectiveParallelCommitStrategy() {
CommitStrategy configured = configuration.commitStrategy();
return configured.mode() == CommitStrategy.Mode.CONNECTION_DEFAULT
? CommitStrategy.everyNBatches(DEFAULT_PARALLEL_COMMIT_BATCHES)
: configured;
if (configured.mode() != CommitStrategy.Mode.CONNECTION_DEFAULT) {
return configured;
}
return configuration.databaseSupport().prefersConnectionDefaultCommit()
? configured
: CommitStrategy.everyNBatches(DEFAULT_PARALLEL_COMMIT_BATCHES);
}

/**
Expand Down
Loading
Loading