Skip to content

[Go SDK] Convert logical type arguments to and from schema protos - #40153

Open
Eliaaazzz wants to merge 4 commits into
apache:masterfrom
Eliaaazzz:go-schema-logical-arguments
Open

Eliaaazzz wants to merge 4 commits into
apache:masterfrom
Eliaaazzz:go-schema-logical-arguments

Conversation

@Eliaaazzz

@Eliaaazzz Eliaaazzz commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Part of #39684. Depends on #40152 (the first commit); review the last commit.

schema.LogicalType had argT and argV fields that nothing populated, reflectTypeToFieldType left argument_type and argument unset (the TODO(BEAM-9615) in schema.go), and fieldTypeToReflectType ignored them. This PR wires the argument through:

  • schema.ToLogicalTypeWithArgument creates a logical type with an argument. The argument must be a value of a built in Go type with an atomic schema type other than []byte, which covers the INT32 arguments of the standard parameterized logical types.
  • The registry keys logical types by URN and argument. A schema field with a URN and argument resolves to the logical type registered for that pair, and falls back to a logical type registered for the URN without an argument, so pass through types such as beam:logical_type:var_char:v1 can be registered once for any length. Go ignores non atomic arguments. The BYTE schema type is signed, so a uint8 argument is converted through int8, and a floating point NaN is rejected at registration because it never compares equal.
  • FromType emits argument_type and argument for parameterized logical types, and top level parameterized logical types carry the argument in a beam:schema:go:logical_argument:v1 schema option so ToType can recover the Go type.
  • FromType returns an error instead of panicking with a nil pointer when a struct kind logical type with a non row storage type is used as a top level type.
  • fieldvalue.go converts atomic values between reflect.Value and FieldType/FieldValue, and rejects BYTE and INT16 values outside their range.
  • When top level logical type lookup misses, the storage row cannot resolve to the Go type of another registered logical type.

Tests: schema conversion cases for a parameterized field and a parameterized top level type, TestToType_LogicalTypeArguments for matching, missing, unknown, ignored and non atomic arguments, TestLogicalTypeArgumentPrecedence for two arguments and a fallback under one URN (fields and top level), registration validation, and round trip tests for the atomic value conversion. Tests also cover out of range BYTE and INT16 values, and unregistered top level logical types after RegisterType, for value, pointer and defined pointer Go types.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@codecov

codecov Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.23810% with 31 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.59%. Comparing base (76f6e6d) to head (a33316f).
⚠️ Report is 121 commits behind head on master.

Files with missing lines Patch % Lines
...s/go/pkg/beam/core/runtime/graphx/schema/schema.go 71.42% 17 Missing and 7 partials ⚠️
.../pkg/beam/core/runtime/graphx/schema/fieldvalue.go 93.00% 5 Missing and 2 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master   #40153       +/-   ##
=============================================
- Coverage     56.16%   28.59%   -27.57%     
=============================================
  Files          1124      380      -744     
  Lines        178336    70970   -107366     
  Branches       1489        0     -1489     
=============================================
- Hits         100170    20297    -79873     
+ Misses        75645    48835    -26810     
+ Partials       2521     1838      -683     
Flag Coverage Δ
go 28.59% <85.23%> (-0.54%) ⬇️
java ?
python ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @lostluck for label go.

This pull request likely touches a core component ("core" label). Please review with scrutiny.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@github-actions

Copy link
Copy Markdown
Contributor

Reminder, please take a look at this pr: @lostluck

The reflection based row coder used the varint encoding for int16 and
uint16 fields. Java and Python encode the INT16 schema type as 2 big
endian bytes, so rows with such fields could not be exchanged with the
other SDKs.

Fixes apache#40151. Part of apache#39684.
Logical types registered with an argument now emit argument_type and
argument in their FieldType, and schemas from other SDKs are matched on
the URN and the argument value. A logical type registered without an
argument still matches any argument, so pass through types keep
working. Top level logical types carry the argument in a schema option.

Part of apache#39684.
@Eliaaazzz
Eliaaazzz force-pushed the go-schema-logical-arguments branch from b979fd6 to d4c5f0b Compare October 8, 2026 01:07
@github-actions

Copy link
Copy Markdown
Contributor

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @jrmccluskey for label go.

This pull request likely touches a core component ("core" label). Please review with scrutiny.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants