Repository navigation
[Python] Add examples for AWS credentials from a secret manager - #40592
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #40592 +/- ##
=============================================
+ Coverage 56.16% 79.93% +23.76%
=============================================
Files 1124 488 -636
Lines 178336 82557 -95779
Branches 1489 0 -1489
=============================================
- Hits 100170 65991 -34179
+ Misses 75645 16566 -59079
+ Partials 2521 0 -2521
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
fe8564f to
d97e59a
Compare
There was a problem hiding this comment.
📋 Review Summary
This PR adds three clear and well-documented examples demonstrating how to configure --aws_credentials_secret_manager to resolve AWS credentials from Google Cloud Secret Manager when reading from S3 or AWS Secrets Manager. The examples are concise, follow Apache Beam conventions, and include helpful IAM permission documentation.
🔍 General Feedback
- In
aws_credentials_secret_read.py, printing the retrieved secret directly triggers CodeQLpy/clear-text-logging-sensitive-dataalerts (causing the CodeQL check to fail) and models an anti-pattern if copied into production code. Logging or printing non-sensitive metadata (such as secret length) instead of the raw secret value would resolve the security check while still verifying secret retrieval. - Using
json.dumps({'name': known_args.secret_id})inaws_credentials_secret_read.pyis safer than manual f-string interpolation when constructing the JSON secret specification.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
|
Assigning reviewers: R: @jrmccluskey for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Add examples under apache_beam/examples/secret_manager that set --aws_credentials_secret_manager to read AWS credentials from Google Cloud Secret Manager: * aws_credentials_s3_read.py: reads an S3 file outside a pipeline. * aws_credentials_s3_pipeline.py: reads an S3 file in a pipeline (DirectRunner or Dataflow). * aws_credentials_secret_read.py: reads a secret from AWS Secrets Manager with Secret.parse_secret_option and Secret.from_json.
d97e59a to
15d9d5b
Compare
| where requirements.txt contains: | ||
| boto3 | ||
| google-cloud-secret-manager |
There was a problem hiding this comment.
Small nit, I would suggest providing a requirements.txt file to run the examples here. Not necessary as long as everything is documented, just a consideration
There was a problem hiding this comment.
Thanks! Added the requirements file.
Add examples under apache_beam/examples/secret_manager that set
--aws_credentials_secret_managerto read AWS credentials from Google Cloud Secret Manager:aws_credentials_s3_read.py: reads an S3 file outside a pipeline.aws_credentials_s3_pipeline.py: reads an S3 file in a pipeline (DirectRunneror Dataflow).aws_credentials_secret_read.py: reads a secret from AWS Secrets Manager withSecret.parse_secret_optionandSecret.from_json.Part of #40528.