feat: support auto-create table metadata for bulk writer - #118
Open
v0y4g3r wants to merge 2 commits into
Open
Conversation
Add a WithAutoCreateSchema writer option so GreptimeDB Enterprise can auto-create a missing table from the Arrow schema metadata attached to bulk (Flight) writes. Each column is described by name with its semantic type (tag/field/timestamp, overridable via TagColumn/FieldColumn/ TimestampColumn), an optional comment, and an optional type override; unlisted columns fall back to the written table's own schema, the timestamp column is written non-nullable, and JSON columns are flagged greptime:type=Json. Unmatched or ambiguous spec columns fail fast, with exact-name-first then sanitized-name matching for both sanitized and unsanitized tables. Expose the option through NewBulkWriter, BulkClient.BulkWriteWithOptions, and Client.BulkWriteWithOptions, and add JSON column support to the bulk Arrow converter to match the server contract and the Java SDK. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds client-side support for GreptimeDB Enterprise’s Flight bulk auto-create-table feature by allowing callers to attach per-column metadata to the Arrow schema produced during bulk writes, plus adds JSON column support in the Arrow converter used by bulk.
Changes:
- Add bulk writer options (notably
bulk.WithAutoCreateSchema) to enrich Arrow field metadata for server-side auto-create during Flight bulk inserts. - Add
Client.BulkWriteWithOptions/BulkClient.BulkWriteWithOptionsto pass writer options through the existing endpoint-selection + health-reporting path. - Add JSON column support in the Arrow converter (map JSON to Arrow Binary and encode JSON values as UTF-8 bytes), with new unit tests and documentation/examples.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| table/types/arrow.go | Map JSON to Arrow Binary and encode JSON values into binary arrays. |
| table/types/arrow_test.go | Add a unit test validating JSON→Arrow Binary encoding. |
| bulk/bulk.go | Introduce WriterOption, plumb options through writer creation and single-shot bulk writes, and apply auto-create schema enrichment. |
| bulk/autocreate.go | Implement auto-create schema metadata attachment to Arrow field metadata (semantic type, comment, Json type flag, timestamp non-nullable). |
| bulk/autocreate_test.go | Add unit tests for metadata attachment, fallback behavior, and invalid schema handling. |
| client.go | Add Client.BulkWriteWithOptions and route BulkWrite through it. |
| README.md | Document auto-create support and how to enable it. |
| examples/bulkwrite/README.md | Add an auto-create usage section and explain semantic types + JSON handling. |
| examples/bulkwrite/main.go | Update example to demonstrate BulkWriteWithOptions + WithAutoCreateSchema. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
264
to
273
| case *array.BinaryBuilder: | ||
| if value == nil { | ||
| b.AppendNull() | ||
| } else if s, ok := value.ValueData.(*gpbv1.Value_StringValue); ok { | ||
| // JSON values are transported as strings (see BuildJSON); store | ||
| // their UTF-8 bytes in the binary column. | ||
| b.Append([]byte(s.StringValue)) | ||
| } else { | ||
| b.Append(value.GetBinaryValue()) | ||
| } |
Comment on lines
+214
to
+220
|
|
||
| if len(exactByName) > 0 { | ||
| return nil, fmt.Errorf( | ||
| "auto-create schema columns %v not found in the written table", | ||
| slices.Sorted(maps.Keys(exactByName)), | ||
| ) | ||
| } |
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's this PR for?
GreptimeDB Enterprise added auto-create table support for Flight bulk inserts (GreptimeTeam/greptimedb-enterprise#845): when the target table does not exist, the server builds a
CreateTableExprfrom the Arrow schema's field metadata. This PR adds the client-side support for specifying that metadata when creating a bulk writer.Changes
bulk: newWithAutoCreateSchemawriter option (NewBulkWriter(ctx, opts...)). EachAutoCreateColumnis matched to a written column by name (exact match first, then sanitized name, so both sanitized and unsanitized tables work) and attached to the Arrow schema as field metadata:greptime:semantic_type=tag/field/timestamp(overridable viaTagColumn/FieldColumn/TimestampColumn; defaults to the written table's own schema)greptime:comment(optional)greptime:type(optional; JSON columns are flaggedJsonautomatically)client.go: newClient.BulkWriteWithOptions(same endpoint selection/health reporting asBulkWrite)table/types: JSON column support in the bulk Arrow converter (mapped to Arrow Binary with UTF-8 bytes, mirroring the Java SDK and the server contract)examples/bulkwriteand README updatedUsage
Verification
go build ./...,go vet ./...go test ./...(89 passed)golangci-lint run ./...(0 issues)