Skip to content

fix(bigquery): [T1-4] propagate useInt64Timestamp to listTableData/getQueryResults and optimize FieldValue.getTimestampValue - #9

Open
lqiu96 wants to merge 1 commit into
mainfrom
perf/bq-t1-04-timestamp-parsing-and-option
Open

lqiu96 wants to merge 1 commit into
mainfrom
perf/bq-t1-04-timestamp-parsing-and-option

Conversation

@lqiu96

@lqiu96 lqiu96 commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Summary & Bottleneck

  • QueryRequestInfo.toPb() configures formatOptions.useInt64Timestamp on fast-path jobs.query RPCs when DataFormatOptions.useInt64Timestamp(true) is enabled. However, HttpBigQueryRpc never sets formatOptions.useInt64Timestamp on tabledata.list (listTableDataSkipExceptionTranslation and listTableDataWithRowLimitSkipExceptionTranslation) or jobs.getQueryResults (getQueryResultsSkipExceptionTranslation and getQueryResultsWithRowLimitSkipExceptionTranslation).
  • Because BigQueryImpl.listTableData passes serviceOptions.getDataFormatOptions().useInt64Timestamp() into FieldValueList.fromPb, any call to listTableData or Page 2+ query pagination with useInt64Timestamp(true) receives floating-point seconds in scientific notation from the backend while FieldValue.getTimestampValue() attempts to parse an integer microsecond string, throwing NumberFormatException.
  • In addition, FieldValue.getTimestampValue() allocates a BigInteger on every cell when useInt64Timestamps is true, and allocates three BigDecimal objects per cell (new BigDecimal, scaleByPowerOfTen(6), and setScale(0, RoundingMode.HALF_UP)) when useInt64Timestamps is false. This scales as $O(\text{rows} \times \text{timestamp_cols})$.

Changes

  • Set .setFormatOptionsUseInt64Timestamp(...) from this.options.getDataFormatOptions().useInt64Timestamp() in listTableDataSkipExceptionTranslation, listTableDataWithRowLimitSkipExceptionTranslation, getQueryResultsSkipExceptionTranslation, and getQueryResultsWithRowLimitSkipExceptionTranslation in java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpBigQueryRpc.java.
  • Use Long.parseLong(stringValue) (with BigInteger fallback) when useInt64Timestamps is true in java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/FieldValue.java.
  • Add a zero-allocation primitive decimal-seconds parser (parseDecimalSecondsToMicros) in java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/FieldValue.java for standard [-]seconds[.fraction] strings with HALF_UP rounding at the microsecond digit, falling back to BigDecimal only for scientific notation or >12-digit integer seconds.
  • Add unit tests in java-bigquery/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/FieldValueTest.java and java-bigquery/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/spi/v2/HttpBigQueryRpcTest.java.

Expected Impact

  • Fixes NumberFormatException on listTableData and multi-page query pagination when DataFormatOptions.useInt64Timestamp(true) is enabled.
  • Eliminates 1 BigInteger + int[] allocation per timestamp cell when useInt64Timestamps is true, and eliminates 3 BigDecimal allocations per standard decimal timestamp cell when useInt64Timestamps is false ($O(\text{rows} \times \text{timestamp_cols})$).

Verification

  • Ran FieldValueTest, HttpBigQueryRpcTest, and BigQueryImplTest via Maven (239 tests run, 0 failures).

@lqiu96
lqiu96 force-pushed the perf/bq-t1-04-timestamp-parsing-and-option branch 5 times, most recently from 3e08f71 to 4fd8299 Compare October 13, 2026 22:01
@lqiu96
lqiu96 force-pushed the perf/bq-t1-04-timestamp-parsing-and-option branch from 4fd8299 to e7fccb2 Compare October 13, 2026 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant