Repository navigation
Conversation
The reflection based row coder used the varint encoding for int16 and uint16 fields. Java and Python encode the INT16 schema type as 2 big endian bytes, so rows with such fields could not be exchanged with the other SDKs. Fixes apache#40151. Part of apache#39684.
Logical types registered with an argument now emit argument_type and argument in their FieldType, and schemas from other SDKs are matched on the URN and the argument value. A logical type registered without an argument still matches any argument, so pass through types keep working. Top level logical types carry the argument in a schema option. Part of apache#39684.
… micros_instant types schema.RegisterLogicalTypeConversion registers a logical type identified by a URN together with the conversions between its Go type and its storage type, and derives the row coder from the storage type. schema.RegisterLogicalTypeCoder does the same for logical types whose wire encoding differs from the row encoding of the storage type. The standard beam:logical_type:date:v1 and beam:logical_type:micros_instant:v1 logical types are registered by default as schema.Date and schema.MicrosInstant. The standard coder regression test now builds the expected row from the schema in the coder payload, decodes payload bytes above 0x7f, counts encoding mismatches as failures, and runs the micros_instant case. Part of apache#39684.
schema.MillisInstant encodes as the Java InstantCoder does: 8 big endian bytes of the milliseconds shifted by the minimum int64. schema.Decimal encodes as the Java BigDecimalCoder does: the scale as a varint followed by the length prefixed two's complement bytes of the unscaled value. Both cases of standard_coders.yaml now run in the Go regression test. Part of apache#39684.
…gth string and bytes logical types schema.RegisterPassThroughLogicalType maps a logical type URN to the Go type of its representation without mapping that Go type back, so a schema field with the URN decodes with the plain row encoding and the length argument is ignored. The standard fixed_char, var_char, fixed_bytes and var_bytes logical types are registered this way, and their standard_coders.yaml case now runs in the Go regression test. Part of apache#39684.
…crosecond and nanosecond precision schema.TimestampMillis, TimestampMicros and TimestampNanos are beam:logical_type:timestamp:v1 with precision 3, 6 and 9, stored as a row of seconds and subseconds. Each precision is its own Go type because the row coder is built from the Go type. The three timestamp cases of standard_coders.yaml now run in the Go regression test. Part of apache#39684.
1 of 3 tasks
1 of 17 tasks
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #39684. Depends on #40156 (the first five commits); review the last commit.
beam:logical_type:timestamp:v1is a timestamp with a precision argument: the number of subsecond digits, stored as a row ofseconds(INT64) and non negativesubseconds(INT16 below precision 5, INT32 otherwise). The Go row coder is built from the Go type alone, so each precision needs its own Go type: this PR addsschema.TimestampMillis,schema.TimestampMicrosandschema.TimestampNanos(precision 3, 6 and 9, the values Java exposes as constants), registered withRegisterParameterizedLogicalTypeConversion. A schema with another precision fails withunknown logical type ... with argument, and further precisions can be registered with one line each.Encoding a value with more subsecond digits than the type carries returns an error, as the Java
Timestamplogical type does; decoding validates the subseconds range. The INT16subsecondsfield of the millisecond type depends on the big endian INT16 encoding of the first PR in the series.The three
timestamp:v1cases of standard_coders.yaml now run in the Go regression test, which means every logical type row case in the file runs.Tests: schema conversion for the three precisions (argument and representation), byte level row encoding against the yaml examples (including timestamps before the epoch), precision and range errors, and an unsupported precision.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.