fix(config): resolve GCP credentials from a file path, aggregate union parse errors - #4375
Open
nanjeshramesh wants to merge 1 commit into
Open
Conversation
…n parse errors GcpServiceAccountCredentials/GcpOAuthCredentials.parse_native_representation() called json.loads() directly on the given string, so pointing `credentials` at a file path (the common GOOGLE_APPLICATION_CREDENTIALS convention) failed to parse. Since GCS/GDrive type it as Union[GcpServiceAccountCredentials, GcpOAuthCredentials], the resolver silently fell through to try OAuth next, which also failed, and only OAuth's confusing error surfaced to the user - even though the value was a valid service account file all along. - gcp_credentials.py: read the file's contents first when the given string is an existing file path, for both ServiceAccount and OAuth credentials. - resolve.py: when every spec in a Union fails to parse, report why each one failed instead of only the last one tried. This is generic to all Union-typed credentials in dlt, not just GCP. Added tests for both. Full tests/common/configuration/ suite passes (249), ruff check/format clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
GcpServiceAccountCredentials/GcpOAuthCredentials.parse_native_representation()calledjson.loads()directly on the given string, so pointingcredentialsat a file path (thecommon
GOOGLE_APPLICATION_CREDENTIALSconvention) failed to parse. Since GCS/GDrive typecredentials as
Union[GcpServiceAccountCredentials, GcpOAuthCredentials], the resolversilently fell through to try OAuth next, which also failed - and only OAuth's confusing
error surfaced to the user, even though the value was a valid service account file all along.
Two changes:
gcp_credentials.py: read the file's contents first when the given string is an existingfile path, for both ServiceAccount and OAuth credentials.
resolve.py: when every spec in aUnionfails to parse, report why each one failedinstead of only the last one tried. This is generic to all Union-typed credentials in dlt,
not just GCP, so it improves error messages beyond this one bug.
Related Issues
Additional Context
Added tests for both changes (file-path resolution for both credential types, and the
aggregated union error message). Full
tests/common/configuration/suite passes locally(249 tests),
ruff check/ruff format --checkclean.