Skip to content

Start BigQuery support - #18

Draft
otegami wants to merge 2 commits into
red-data-tools:mainfrom
otegami:support-bigquery
Draft

Start BigQuery support#18
otegami wants to merge 2 commits into
red-data-tools:mainfrom
otegami:support-bigquery

Conversation

@otegami

@otegami otegami commented Apr 9, 2026

Copy link
Copy Markdown
Member

This is the first step toward BigQuery backend support.
More features will be added incrementally.

CI has passed on my fork here.

Done

Not done

FYI

  • Tests assume GCP project and dataset already exist (ADBC BigQuery driver requires dataset_id for all SQL execution)
    • I will write the documentation about how to set up GCP's access keys.

otegami added 2 commits April 9, 2026 12:33
This is the first step toward BigQuery backend support.
More features will be added incrementally.

Done:
- Basic connectivity and query execution (TestModel passes)
- Backtick quoting for identifiers
- insert_all without ON CONFLICT (BigQuery does not support it)

Not done:
- Type support (boolean, string, date, etc.)
- Ingest (ADBC BigQuery driver does not implement ingest API)

FYI:
- Tests assume GCP project and dataset already exist
  (ADBC BigQuery driver requires dataset_id for all SQL execution)
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
- name: "Prepare test environment: BigQuery"
if: matrix.ruby-version == '3.4' && steps.gcp-auth.outcome == 'success'
uses: columnar-tech/setup-dbc@v1

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use dbd for installing an ADBC driver for BigQuery.
ref: https://github.com/columnar-tech/dbc
ref: https://github.com/columnar-tech/setup-dbc

Comment on lines +90 to +94
uses: google-github-actions/auth@v3
continue-on-error: true
with:
project_id: ${{ secrets.GCP_PROJECT_ID }}
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}

@otegami otegami Apr 9, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Set up GCP authentication as follows.
And then add some authorizations for the above role.

@otegami

otegami commented Apr 9, 2026

Copy link
Copy Markdown
Member Author

The CI is currently failing, so I'll address that in a separate PR.

@otegami
otegami marked this pull request as draft April 9, 2026 15:28
@otegami
otegami marked this pull request as draft April 9, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant