Skip to content

Add VARIANT appender shredding test#161

Open
wmTJc9IK0Q wants to merge 1 commit into
duckdb:mainfrom
wmTJc9IK0Q:variant-append
Open

Add VARIANT appender shredding test#161
wmTJc9IK0Q wants to merge 1 commit into
duckdb:mainfrom
wmTJc9IK0Q:variant-append

Conversation

@wmTJc9IK0Q

Copy link
Copy Markdown

What

Adds TestQueryAppenderVariantShredding (appender_test.go), a test that bulk-inserts
semi-structured key-value data with keys not known at compile time into a
VARIANT column through the appender API, and proves the DuckDB storage engine
shreds the values.

Why

The direct data-chunk appender and prepared-statement binds both reject VARIANT
(TestAppenderRejectsVariant, appender.go, statement.go): Go vector setters can't
emit VARIANT and there's no C-API helper to construct a variant. This test documents
and locks in the one path that does work and verifies it end to end, including the storage-level shredding behavior introduced in DuckDB 1.5.

How it works

  • Insertion: NewQueryAppender with a temp table holding an INTEGER id and a
    VARCHAR JSON document; the INSERT casts col2::JSON::VARIANT, producing a proper
    OBJECT(x, y) variant. Keys come from runtime data (map[string]anyjson.Marshal),
    not a compile-time struct, so arbitrary/unknown keys are supported.
  • Proof of shredding (A/B): on an on-disk database attached with
    STORAGE_VERSION 'v1.5.0', identical rows are appended into two tables that differ only
    by variant_minimum_shredding_size (-1 disabled vs 0 forced) at their respective
    CHECKPOINT — the setting is GLOBAL and consulted at checkpoint time.
    pragma_storage_info then shows typed value segments present only in the shredded
    table (zero in the unshredded baseline), which is direct storage-engine evidence.
  • Round-trip: confirms variant_typeof(col) = OBJECT(x, y) and col.x/col.y extract
    the original values.

Notes

  • Relies on the json extension bundled with the duckdb-go bindings (already used by
    examples/json and TestJSON); a SELECT '{}'::JSON warmup loads it before the
    appender INSERT.
  • Integer field values are used deliberately: they yield a distinct typed value segment
    that is provably absent without shredding, giving an unambiguous signal.

Adds TestQueryAppenderVariantShredding, which appends semi-structured
key-value data with runtime-determined keys into a VARIANT column via
NewQueryAppender (temp VARCHAR column cast col2::JSON::VARIANT), then
proves the storage engine shreds it.

The direct data-chunk appender and prepared binds reject VARIANT (no Go
vector setter, no C-API to construct a variant), so the query appender is
the working path. The test is an A/B on an on-disk STORAGE_VERSION v1.5.0
database: identical rows are appended into two tables differing only by
variant_minimum_shredding_size (-1 vs 0) at CHECKPOINT time. Shredding is
observed via pragma_storage_info (typed value segments present only when
shredded) and confirmed field-addressable (variant_typeof OBJECT(x, y),
col.x round-trip).

@taniabogatsch taniabogatsch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good to me! Could you just have a look at the failing linter?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants