Skip to content

[Python] Add examples for AWS credentials from a secret manager - #40592

Merged
shunping merged 5 commits into
apache:masterfrom
shunping:secret-manager-10
Oct 16, 2026
Merged

shunping merged 5 commits into
apache:masterfrom
shunping:secret-manager-10

Conversation

@shunping

@shunping shunping commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Add examples under apache_beam/examples/secret_manager that set --aws_credentials_secret_manager to read AWS credentials from Google Cloud Secret Manager:

  • aws_credentials_s3_read.py: reads an S3 file outside a pipeline.
  • aws_credentials_s3_pipeline.py: reads an S3 file in a pipeline (DirectRunner or Dataflow).
  • aws_credentials_secret_read.py: reads a secret from AWS Secrets Manager with Secret.parse_secret_option and Secret.from_json.

Part of #40528.

@codecov

codecov Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.93%. Comparing base (76f6e6d) to head (16a8777).
⚠️ Report is 211 commits behind head on master.

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #40592       +/-   ##
=============================================
+ Coverage     56.16%   79.93%   +23.76%     
=============================================
  Files          1124      488      -636     
  Lines        178336    82557    -95779     
  Branches       1489        0     -1489     
=============================================
- Hits         100170    65991    -34179     
+ Misses        75645    16566    -59079     
+ Partials       2521        0     -2521     
Flag Coverage Δ
go ?
java ?
python 79.93% <ø> (+0.37%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@shunping
shunping marked this pull request as ready for review October 15, 2026 18:03

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR adds three clear and well-documented examples demonstrating how to configure --aws_credentials_secret_manager to resolve AWS credentials from Google Cloud Secret Manager when reading from S3 or AWS Secrets Manager. The examples are concise, follow Apache Beam conventions, and include helpful IAM permission documentation.

🔍 General Feedback

  • In aws_credentials_secret_read.py, printing the retrieved secret directly triggers CodeQL py/clear-text-logging-sensitive-data alerts (causing the CodeQL check to fail) and models an anti-pattern if copied into production code. Logging or printing non-sensitive metadata (such as secret length) instead of the raw secret value would resolve the security check while still verifying secret retrieval.
  • Using json.dumps({'name': known_args.secret_id}) in aws_credentials_secret_read.py is safer than manual f-string interpolation when constructing the JSON secret specification.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

Comment thread sdks/python/apache_beam/examples/secret_manager/aws_credentials_secret_read.py Outdated
Comment thread sdks/python/apache_beam/examples/secret_manager/aws_credentials_secret_read.py Outdated
@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @jrmccluskey for label python.
Backup reviewer: damccorm (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Add examples under apache_beam/examples/secret_manager that set
--aws_credentials_secret_manager to read AWS credentials from Google Cloud
Secret Manager:
* aws_credentials_s3_read.py: reads an S3 file outside a pipeline.
* aws_credentials_s3_pipeline.py: reads an S3 file in a pipeline
  (DirectRunner or Dataflow).
* aws_credentials_secret_read.py: reads a secret from AWS Secrets Manager
  with Secret.parse_secret_option and Secret.from_json.
@shunping
shunping marked this pull request as ready for review October 15, 2026 21:23

@jrmccluskey jrmccluskey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, one nit

Comment on lines +36 to +38
where requirements.txt contains:
boto3
google-cloud-secret-manager

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small nit, I would suggest providing a requirements.txt file to run the examples here. Not necessary as long as everything is documented, just a consideration

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Added the requirements file.

@shunping shunping changed the title Add examples for AWS credentials from a secret manager [Python] Add examples for AWS credentials from a secret manager Oct 16, 2026
@shunping
shunping merged commit e41f0ce into apache:master Oct 16, 2026
86 checks passed
@shunping
shunping deleted the secret-manager-10 branch October 16, 2026 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants