Fix S3 persistence tests for OIDC session token and multi-account credentials - #16892
Open
silvarohan18 wants to merge 3 commits into
Open
Fix S3 persistence tests for OIDC session token and multi-account credentials#16892silvarohan18 wants to merge 3 commits into
silvarohan18 wants to merge 3 commits into
Conversation
The S3 tests call set_s3_credentials(key, secret) but omit the session_token parameter that already exists in the API. When running with OIDC-based temporary credentials (e.g., GitHub Actions), the missing session token causes AWS to reject requests. Read AWS_SESSION_TOKEN from the environment alongside the existing key and secret, and pass it to set_s3_credentials(). When the env var is not set (static IAM credentials), it passes None/NULL which preserves existing behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests accessing test.0xdata.com (DEV account) use AWS_CREDS_PREFIX_S3_DEV, tests accessing h2o-public-test-data (PROD account) use AWS_CREDS_PREFIX_S3_PROD. Both fall back to AWS_CREDS_PREFIX and then to unprefixed env vars for backward compatibility with Jenkins. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Support S3_CREDS_TEST_PATH and S3_CREDS_TEST_PREFIX env vars to allow different CI environments to use different S3 buckets and credential sources. Defaults unchanged (test.0xdata.com with DEV credentials). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
set_s3_credentials()in S3 persistence tests, required for OIDC-based credential flows (temporary credentials include a session token)AWS_CREDS_PREFIX_S3_PROD/AWS_CREDS_PREFIX_S3_DEV), with backward-compatible fallback toAWS_CREDS_PREFIXand then empty stringContext
Backport of #16891 to
rel-3.46.0.The h2o-3-enterprise nightly CI migrated from long-lived IAM user credentials to OIDC role assumption. OIDC produces temporary credentials (access key + secret key + session token), but the S3 tests only passed access key and secret key to
set_s3_credentials(). The S3 test data also spans two AWS accounts (PROD forh2o-public-test-data, DEV fortest.0xdata.com), requiring per-test credential selection via environment variable prefixes.Changes
pyunit_set_s3_credentials.py— UsesAWS_CREDS_PREFIX_S3_DEVprefix (test.0xdata.com is in DEV account); passes session token toset_s3_credentials()pyunit_import_s3_parquet.py— UsesAWS_CREDS_PREFIX_S3_PRODprefix (h2o-public-test-data is in PROD account); passes session token toset_s3_credentials()runit_pubdev_6188.R— UsesAWS_CREDS_PREFIX_S3_DEVprefix; passes session token toh2o.set_s3_credentials()All prefix lookups fall back gracefully:
AWS_CREDS_PREFIX_S3_DEV→AWS_CREDS_PREFIX→"", preserving compatibility with Jenkins IAM user credential flows.Test plan
pyunit_import_s3_parquet.pypasses in nightly CI with PROD OIDC credentialspyunit_set_s3_credentials.pyblocked by bucket policy (DevOps ticket pending fors3:GetObjectontest.0xdata.com)runit_pubdev_6188.Rblocked by OIDC token expiration in long R test suites (DevOps ticket pending forMaxSessionDurationincrease)