Add relassert

[smvv]

Fixes https://github.com/duckdblabs/duckdb-internal/issues/9787

I want to repeatedly run extensions tests to detect (more) ASAN failures in CI. Some tests can fail "randomly" (if the right code paths are executed) and that affects duckdb CI on main and in PRs (and some customers that build duckdb from main).

The extension author is not always aware of failure when bumping the extension in duckdb, and re-running tests should reduce the chance of merging code that fails once every ~ten runs.

I'm trying the new CLI flag --stabilize-tests in this extension first to see what we discover. It could be that we move the changes elsewhere (like extension-ci-tools) in the future.

EDIT:
We can't use --stabilize-tests because the repo relies on python scripts to (re)populate schemas+tables, which are then modified by the duckdb tests

[Tishj]

I think (most of?) these issues are fixed by #1088 , will merge this into main to verify

[Tishj]

There is one leak of 224 bytes, and it's very likely this one: duckdb/duckdb-aws#146

[Tishj]

@smvv I can't get the suppressions to work, most likely because the symbols are missing. I can't figure out how to get the symbols in however

[Tishj]

Suppressions used:
  count      bytes template
      1        224 CRYPTO_zalloc

Awesome, I'll wait for the rest of the CI to finish but then I'll merge it 🎉

[smvv]

@Tmonster @Tishj i've changed some workflows so most tests under a release assert build (enabling the Address and Leak Sanitizers). Can you review the current PR? Thanks!

If you want, I can help with reducing CI time (in another PR). Some rough ideas:

• replace pip install (takes 50s) with uv.
• data generator script takes 167s and seems to run sequentially (?). I'm assuming we can parallelize most of the data generation (if they don't have any implicit dependencies/ordering).

An alternative is create/insert the data in each test case so can parallelize and re-run the tests. Both require test isolation (so each test uses a new database, table, etc depending on what's needed to isolate tests) and should make test jobs ~4x faster.

Let me know if this would help. If wanted, I think it's useful to chat about how to achieve this without much addition work for you (like how to review, convert tests incrementally, etc).

[Tishj]

Yea those are great ideas
As I mentioned when we last spoke, the old way of testing is 3 step:

• generate with spark (python)
• read/write with duckdb (sql)
• verify with spark (python)

I'm migrating all of that to the last step, slowly removing the first 2
That way we can have dedicated end to end tests, with targeted (re)generation of tables that are needed.

That would also solve the idempotency problem, and allow us to incorporate retries of tests + parallelization