Tests: Add integration tests with CrateDB by amotl · Pull Request #192 · panodata/omniload

amotl · 2026-07-10T01:24:23Z

About

CrateDB is an emerging open-source data warehouse that can store data in the terabyte ranges and is largely compatible with PostgreSQL.

Details

An earlier attempt to validate CrateDB caused too much noise, because the database didn't support the DATE type. Now that this has changed recently, warehouse operations become actually viable.

Tests: Adjust software tests for CrateDB bruin-data/ingestr#284
PostgreSQL compatibility: AAUIW support for storing data using the Postgres DATE data type crate/crate#19414

@DHRUV6029: Thank you very much for submitting the relevant patch to CrateDB. 👍

Support PostgreSQL DATE as a storable column type crate/crate#19588

Status

The adjusted test suite currently needs to skip a few test cases that raise the venerable DestinationSchemaTampered exception of dlt. We think dlt-cratedb might need further adjustments beyond the patch recently submitted by @florinutz.

pytest --no-cov -k cratedb
87 passed, 71 skipped, 54 warnings in 124.39s (0:02:04)

References

coderabbitai · 2026-07-10T01:24:30Z

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)

coderabbit-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4e181141-902f-4d57-ae27-bda6fa313eeb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch cratedb

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

codecov · 2026-07-11T22:05:40Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.36%. Comparing base (dc5aa50) to head (116d76d).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #192      +/-   ##
==========================================
+ Coverage   54.33%   54.36%   +0.03%     
==========================================
  Files         203      203              
  Lines        9574     9574              
==========================================
+ Hits         5202     5205       +3     
+ Misses       4372     4369       -3

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

amotl · 2026-07-11T22:09:25Z

  "pytest-cov<8",
  "pytest-xdist[psutil]<4",
-  "testcontainers[mysql,postgres]>=4.9.1,<4.15",
+  "testcontainers[cratedb,mysql,postgres] @ git+https://github.com/testcontainers/testcontainers-python.git@main",


Thank you very much for adding a native CrateDB adapter to Testcontainers for Python. With testcontainers 4.15, we will be able to use it in downstream packages swiftly.

Suggested change

"testcontainers[cratedb,mysql,postgres] @ git+https://github.com/testcontainers/testcontainers-python.git@main",

"testcontainers[cratedb,mysql,postgres]>=4.15,<4.16",

/cc @florinutz, @alexanderankin

amotl · 2026-07-11T22:10:45Z

  "dataclasses-json<0.7",
  "dlt<1.29",
-  "dlt-cratedb<0.2",
+  "dlt-cratedb @ git+https://github.com/zerotired/dlt-cratedb.git@fix-dynamic-install",


@florinutz: It looks like even when using the newest updates in the dlt-cratedb repository, including yours, the venerable DestinationSchemaTampered error is still present across the board.

amotl · 2026-07-11T22:15:15Z

+    if dest_uri.startswith("cratedb://"):
+        pytest.skip(
+            "Fails on CrateDB with `DestinationSchemaTampered`, see "
+            "https://github.com/crate/dlt-cratedb/issues/14"
+        )
+


@florinutz: It looks like even when using the newest updates in the dlt-cratedb repository, including yours, the venerable DestinationSchemaTampered error is still present across the board.

Like demonstrated here, we had to skip all test cases that are still failing on CrateDB. It would be so sweet to resolve them in one way or another, so CrateDB can become a premium-grade data warehouse within omniload and other applications using dlt. 🙏

read-the-docs-community · 2026-07-11T23:03:40Z

Documentation build overview

📚 omniload | 🛠️ Build #33547231 | 📁 Comparing 116d76d against latest (1beee34)

🔍 Preview build

4 files changed

+ getting-started/file-format-routing.html
+ supported-sources/cbor.html
+ supported-sources/msgpack.html
± supported-sources/index.html

amotl

Let me flag two more errors.

amotl · 2026-07-11T23:08:44Z

+    if dest_uri.startswith("cratedb://"):
+        pytest.skip(
+            'Fails on CrateDB with `"_id" conflicts with system column`, '
+            "see https://github.com/crate/dlt-cratedb/issues/19"
+        )
+


Currently we need to skip the MongoDB -> CrateDB integration tests, because of the venerable “_id” conflicts with system column pattern error that apparently still happens.

Problem importing from MongoDB crate/dlt-cratedb#19

/cc @florinutz

amotl · 2026-07-11T23:10:31Z

    def append_test(dest_uri, dynamodb, schema: str):
+
+        if dest_uri.startswith("cratedb://"):
+            pytest.skip(
+                "Fails on CrateDB with twice the amount of expected results: AssertionError: assert 6 == 3"
+            )
+


That is another type of error we haven't seen before with dlt and CrateDB. It needs to be investigated.

amotl · 2026-07-11T23:15:55Z

-    assert res[0] == ("message1",)
-    assert res[1] == ("message2",)
-    assert res[2] == ("message3",)
+    if dest_uri.startswith("cratedb://"):
+        messages_db = [res[0][0], res[1][0], res[2][0]]
+        assert "message1" in messages_db
+        assert "message2" in messages_db
+        assert "message3" in messages_db
+    else:
+        assert res[0] == ("message1",)
+        assert res[1] == ("message2",)
+        assert res[2] == ("message3",)


Note to self: Maybe adjust the database query instead to emit ordered results?

kimberly-WY

Good improvement overall. One edge case: empty string vs null aren't treated the same downstream — might need a normalization step.

hampsterx · 2026-07-20T22:10:13Z

The DestinationSchemaTampered skips are release-lag; there's a separate merge-semantics gap underneath.

Reproduced against CrateDB 6.3.0 nightly with a minimal dlt pipeline (timestamp + primary key):

write disposition	dlt-cratedb 0.1.1 (released)	dlt-cratedb `main`
append	OK	OK
merge	`DestinationSchemaTampered`	OK
merge (delete-insert)	`DestinationSchemaTampered`	OK

Root cause is exactly #51: the released 0.1.1 (2026-03-10) predates crate/dlt-cratedb#49 / crate/dlt-cratedb#51, so to_db_datetime_type still mutates the live schema column in place, tripping the tamper guard on the merge staging pass. main renders against a copy. Both fixes are in CHANGES.md under "Unreleased", so a dlt-cratedb 0.1.2 release removes the crash with no further schema work.

But a release won't turn all five skips green. A 2-run merge on main (run 1 {1,2}, run 2 {2,3}) leaves {2,3}: merge and delete-insert are both redirected to full replace (CrateDbStagingReplaceJob; true upsert tracked by crate/dlt-cratedb#6). So db_to_db_merge_with_primary_key and db_to_db_delete_insert_*, which assert an earlier row surviving, will move from crash to a real-upsert mismatch.

Suggest: after the release, drop the skips on any case that only needs the load to complete, and re-point the rest from crate/dlt-cratedb#14 (fixed) to crate/dlt-cratedb#6 (the real open item) as xfail.

amotl · 2026-07-20T22:57:20Z

@hampsterx: Thanks for the detailed report about why and where dlt-cratedb might need further improvements.
@florinutz: Can you possibly fix one or two other issues of dlt-cratedb if the suggestions by @hampsterx make sense to you?

coderabbitai · 2026-07-20T22:57:46Z

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

repo="crate/dlt-cratedb"
for endpoint in \
  "repos/$repo/issues/6" \
  "repos/$repo/issues/14" \
  "repos/$repo/pulls/49" \
  "repos/$repo/pulls/51" \
  "repos/$repo/releases?per_page=10"
do
  echo "===== $endpoint ====="
  gh api "$endpoint" --jq '
    if type == "array" then
      .[] | {tag_name, name, published_at, prerelease, draft}
    else
      {number, title, state, merged_at, closed_at, html_url, body}
    end
  ' || true
done

Length of output: 7160

@amotl, @hampsterx’s suggestions make sense and are supported by the current upstream state:

crate/dlt-cratedb#51 was merged and closed #14, but it has not yet been released—the latest release remains v0.1.1.
After a release containing #51, remove skips for cases that only verify that merge, delete-insert, or refresh="drop_resources" complete without crashing.
Keep the tests that assert actual merge/upsert retention semantics as xfail, referencing the still-open crate/dlt-cratedb#6. The current full-replace fallback is intentionally not equivalent to a real merge.

@florinutz, the highest-value follow-up would be #6: implement CrateDB-compatible merge/delete SQL (or another correct upsert strategy), then convert those remaining semantic xfails into passing integration tests.

🐇 ✅

coderabbitai · 2026-07-20T22:57:52Z

Tip

For best results, initiate chat on the files or code changes.

@amotl: Yes—the suggestions make sense.

@florinutz, the most impactful follow-ups in dlt-cratedb would be:

Release the schema-tamper fix currently on main (the non-mutating datetime-type handling from #49/Kafka: Expand default_msg_processor into a miniature decoding unit #51). This should unblock merge staging runs that presently fail with DestinationSchemaTampered.
Implement true merge/upsert semantics rather than redirecting merge and delete-insert to full replacement (tracked in #6). That is the remaining blocker for tests that verify existing primary-key rows survive a subsequent merge.

Once the first item is released, this PR can unskip load-completion cases. The stateful merge assertions should remain xfail and reference #6 until upsert behavior is available.

🐇👍

coderabbitai · 2026-07-20T22:58:19Z

Tip

For best results, initiate chat on the files or code changes.

@amotl: Yes, those follow-ups make sense.

@florinutz: The two highest-impact improvements in dlt-cratedb would be:

Release the already-merged schema-tamper fix from #51, so merge staging no longer raises DestinationSchemaTampered.
Implement true merge/upsert semantics for merge and delete-insert (tracked by #6), replacing the current full-replace fallback.

After the release, this PR can unskip tests that only require successful loads. Tests asserting preservation of existing primary-key rows should remain xfail until #6 is resolved.

🐇👍

amotl force-pushed the cratedb branch 4 times, most recently from 49dd807 to 190f1af Compare July 11, 2026 21:56

amotl commented Jul 11, 2026

View reviewed changes

amotl added 2 commits July 12, 2026 01:01

Tests: Update to testcontainers 4.15

0ea82af

Tests: Add integration tests for CrateDB destination

116d76d

amotl force-pushed the cratedb branch from 190f1af to 116d76d Compare July 11, 2026 23:02

amotl commented Jul 11, 2026

View reviewed changes

This was referenced Jul 12, 2026

Tests: Probe pgrust instead of PostgreSQL #201

Closed

PostgreSQL compatibility: AAUIW support for storing data using the Postgres DATE data type crate/crate#19414

Closed

Chore: Fix packaging for dynamic installs from Git repository crate/dlt-cratedb#56

Merged

amotl changed the title ~~CrateDB integration tests~~ Tests: Add integration tests with CrateDB Jul 15, 2026

amotl added quality All things quality assurance and software tests. help wanted Extra attention is needed labels Jul 15, 2026

amotl mentioned this pull request Jul 15, 2026

CrateDB #186

Open

kimberly-WY reviewed Jul 16, 2026

View reviewed changes

	"testcontainers[cratedb,mysql,postgres] @ git+https://github.com/testcontainers/testcontainers-python.git@main",
	"testcontainers[cratedb,mysql,postgres]>=4.15,<4.16",

Uh oh!

Conversation

amotl commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

About

Details

Status

References

Uh oh!

coderabbitai Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

codecov Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

amotl Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amotl Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amotl Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

read-the-docs-community Bot commented Jul 11, 2026

Documentation build overview

Uh oh!

amotl left a comment

Choose a reason for hiding this comment

Uh oh!

amotl Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

amotl Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amotl Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

kimberly-WY left a comment

Choose a reason for hiding this comment

Uh oh!

hampsterx commented Jul 20, 2026

Uh oh!

amotl commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 20, 2026

Uh oh!

coderabbitai Bot commented Jul 20, 2026

Uh oh!

coderabbitai Bot commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

amotl commented Jul 10, 2026 •

edited

Loading

coderabbitai Bot commented Jul 10, 2026 •

edited

Loading

codecov Bot commented Jul 11, 2026 •

edited

Loading

amotl Jul 11, 2026 •

edited

Loading

amotl Jul 11, 2026 •

edited

Loading

amotl Jul 11, 2026 •

edited

Loading

amotl Jul 11, 2026 •

edited

Loading

amotl commented Jul 20, 2026 •

edited

Loading