Skip to content

perf(bigquery): [T2-2] stream Jackson tokens directly into FieldValueList without intermediate TableRow - #30

Open
lqiu96 wants to merge 1 commit into
perf/bigquery-unwrap-cell-jsonfrom
perf/bq-t2-02-direct-stream-fieldvaluelist
Open

lqiu96 wants to merge 1 commit into
perf/bigquery-unwrap-cell-jsonfrom
perf/bq-t2-02-direct-stream-fieldvaluelist

Conversation

@lqiu96

@lqiu96 lqiu96 commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Summary & Bottleneck

Even after PR #4 (perf/bigquery-unwrap-cell-json) unwraps TableCell JSON envelopes in HttpBigQueryRpc, parseTableRows (java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpBigQueryRpc.java:L2205-L2260) still allocates a new TableRow() (TableRow + GenericData's RegularEnumSet + ArrayMap = 3 objects / 80 bytes per row), a transient ArrayList<Object> per row in parseRowCells, a transient Collections.singletonMap("f", ...) + ArrayList<Object> per RECORD cell, and a transient ArrayList<Object> per REPEATED cell. Immediately afterward, BigQueryImpl.transformTableData (java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryImpl.java:L2220-L2233) and ConnectionImpl.getIterableFieldValueList (java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/ConnectionImpl.java:L826-L839) iterate over result.getRows(), allocate a second ArrayList<FieldValue> + two Iterators + ImmutableList.copyOf per row in FieldValueList.fromPb, and discard all intermediate TableRow, ArrayList<Object>, and SingletonMap wrappers.

Changes

  • Stream Jackson JsonParser tokens in HttpBigQueryRpc.parseTableRows / parseRowCells / parseCellFieldValue (java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpBigQueryRpc.java) directly into pre-sized ImmutableList<FieldValue> rows and nested FieldValueList values without allocating intermediate TableRow, ArrayList<Object>, or SingletonMap instances.
  • Wrap parsed row cell lists in a lightweight per-page StreamedTableRowList carrier (16 bytes per page) and expose @InternalApi HttpBigQueryRpc.extractFieldValueLists(Iterable<TableRow>, Schema, boolean) so BigQueryImpl.transformTableData and ConnectionImpl.getIterableFieldValueList wrap each pre-built ImmutableList<FieldValue> directly into a FieldValueList in zero-copy $O(1)$ per flat row (resolving RECORD sub-field schemas and RANGE element types only when present in the schema), while StreamedTableRowList.get(int) lazily materializes TableRow if accessed by external callers.
  • Reuse static NULL_PRIMITIVE and NULL_PRIMITIVE_INT64_TIMESTAMPS flyweights in FieldValue.of (java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/FieldValue.java) and short-circuit FieldValueList.withSchema(FieldList) (java-bigquery/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/FieldValueList.java) when the schema is already attached.
  • Add unit tests in HttpBigQueryRpcTest (java-bigquery/google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/spi/v2/HttpBigQueryRpcTest.java) verifying HttpBigQueryRpc.extractFieldValueLists across primitive, null, repeated, record, and repeated-record cells.

Expected Impact

  • Complexity / Scaling: $O(\text{rows} \times \text{cols})$.
  • Eliminates 3 objects (TableRow + RegularEnumSet + ArrayMap = ~80 bytes) per row, 2 intermediate ArrayList + Object[] buffers per row, 2 Iterators per row, intermediate SingletonMap("f", ...) and ArrayList<Object> wrappers on RECORD and REPEATED cells, and all FieldValue allocations on NULL cells.

Verification

  • Verified with mvn test -pl java-bigquery/google-cloud-bigquery -Dtest=HttpBigQueryRpcTest,FieldValueTest,FieldValueListTest,BigQueryImplTest,ConnectionImplTest -Dclirr.skip=true -Denforcer.skip=true -Dcheckstyle.skip=true (267 tests passing).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant