Skip to content

perf(bigquery): [T2-3] support typed FieldValue backing to avoid binary-to-String-to-binary round-trips in ArrowDeserializer - #28

Open
lqiu96 wants to merge 1 commit into
mainfrom
perf/bq-t2-03-typed-fieldvalue-arrow
Open

lqiu96 wants to merge 1 commit into
mainfrom
perf/bq-t2-03-typed-fieldvalue-arrow

Conversation

@lqiu96

@lqiu96 lqiu96 commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Summary & Bottleneck

In java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ArrowDeserializer.java (arrowVectorToFieldValue), every binary primitive extracted from an Apache Arrow vector is eagerly converted into a String and wrapped in FieldValue.of(Attribute.PRIMITIVE, stringVal):

  1. Redundant binary-to-String-to-binary round-trips on $O(\text{rows} \times \text{cols})$ cells:
    • IntVector and BigIntVector values are formatted via Integer.toString or Long.toString, only for FieldValue.getLongValue() to immediately re-parse the string via Long.parseLong.
    • Float8Vector values are formatted via Double.toString, only for FieldValue.getDoubleValue() to re-parse the string via Double.parseDouble.
    • TimeStampVector epoch microsecond values are formatted via formatTimestampMicros(long) into "<seconds>.<microseconds>", only for FieldValue.getTimestampValue() to construct and scale a BigDecimal to recover the original long microseconds.
    • DecimalVector (BigDecimal) values are formatted via toPlainString(), only for FieldValue.getNumericValue() to construct new BigDecimal(String).
    • VarBinaryVector (byte[]) values are Base64-encoded via BaseEncoding.base64().encode(byte[]), only for FieldValue.getBytesValue() to Base64-decode them back into byte[].

Changes

  • java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/FieldValue.java:
    • Add package-private typed factory methods (ofLong, ofDouble, ofTimestampMicros, ofNumeric, ofBytes) that retain the typed value in rawValue.
    • Return the typed value directly in getLongValue(), getDoubleValue(), getTimestampValue(), getTimestampInstant(), getNumericValue(), and getBytesValue() without allocating or parsing a String.
    • Lazily materialize and cache the canonical String representation in getValue() and getStringValue() (and use getValue() in equals(), hashCode(), and toString()) so string access and equality checks remain 100% compatible with string-backed FieldValue instances.
  • java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ArrowDeserializer.java:
    • Construct FieldValue instances using ofTimestampMicros, ofLong, ofDouble, ofBytes, and ofNumeric in arrowVectorToFieldValue.

Expected Impact

  • Eliminates $O(\text{rows} \times \text{cols})$ intermediate String formatting, Base64 encoding/decoding, and Long/Double/BigDecimal re-parsing when reading typed columns from Arrow streams via getLongValue(), getDoubleValue(), getTimestampValue(), getNumericValue(), and getBytesValue().
  • Preserves full backwards compatibility when callers invoke getStringValue(), getValue(), equals(), or hashCode().

Verification

  • Added unit tests in java-bigquery/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/FieldValueTest.java and java-bigquery/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/ArrowDeserializerTest.java.
  • Ran mvn test -pl java-bigquery/google-cloud-bigquery -Dtest=FieldValueTest,FieldValueListTest,ArrowDeserializerTest,ArrowQueryPageFetcherTest,ConnectionImplTest (59 tests run, 0 failures).

…ry-to-String-to-binary round-trips in ArrowDeserializer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant