Skip to content

[CI] Avoid duplicate Python snapshot builds - #40856

Merged
Amar3tto merged 1 commit into
apache:masterfrom
bvolpato:bvolpato/reuse-python-snapshot-builds
Oct 30, 2026
Merged

Amar3tto merged 1 commit into
apache:masterfrom
bvolpato:bvolpato/reuse-python-snapshot-builds

Conversation

@bvolpato

@bvolpato bvolpato commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

The distroless Python image task already builds and pushes its matching base image. The snapshot matrix also runs that base task separately, repeating the SDK image build (example run). In that sample, the standalone Python jobs occupied about 5.6 hours of total runner time.

Remove the standalone Python base image jobs. The remaining distroless jobs still publish both image families with the same Python versions, tags, and architectures.

Validated the Gradle task graphs and Docker task configuration, along with workflow syntax. Full SDK image publication was not run locally.

@github-actions github-actions Bot added build size/xs XS: <10 lines changed labels Oct 28, 2026
@bvolpato
bvolpato force-pushed the bvolpato/reuse-python-snapshot-builds branch from 5e57483 to 1760453 Compare October 28, 2026 23:57
@bvolpato

Copy link
Copy Markdown
Contributor Author

Concrete duplication evidence from the October 28 scheduled snapshot run, on master at aea45221b65a0c2e6fef2bf5c8f0f6f3b0753dd1:

Job Requested task Base-image task executed
Standalone Python 3.11 :sdks:python:container:py311:docker :sdks:python:container:py311:docker at 00:20:35 UTC
Distroless Python 3.11 :sdks:python:container:distroless:py311:docker :sdks:python:container:py311:docker at 00:20:30 UTC, followed by the distroless task at 01:32:19 UTC

Both logs show the base-image task executing, without UP-TO-DATE or FROM-CACHE. Both also show a completed manifest push to gcr.io/apache-beam-testing/beam-sdk/beam_python3.11_sdk:aea45221b65a0c2e6fef2bf5c8f0f6f3b0753dd1. The standalone job finishes that push at 01:32:18 UTC, and the distroless job at 01:32:15 UTC. They produce different manifest digests, confirming separate builds and publication to the same image tag.

The dependency is explicit in distroless/common.gradle:

dockerPrepare.dependsOn ":sdks:python:container:py${pythonVersionSuffix}:docker"

The remaining distroless matrix jobs therefore still run the matching base-image tasks. The workflow arguments retain the registry, tag list, arm64,amd64 platforms, and -Ppush-containers. This PR removes only the four standalone Python base-image matrix entries.

In that run, the standalone jobs took 77m50s for Python 3.11, 98m44s for 3.12, 78m43s for 3.13, and 78m06s for 3.14. That is 5h33m23s of aggregate runner-job time. This is the observed cost of those jobs in one run. A completed publication run with this change is still needed to measure the actual reduction.

@bvolpato
bvolpato marked this pull request as ready for review October 30, 2026 02:46

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR removes the standalone python:container:py31x:docker matrix entries from the Publish Beam SDK Snapshots GitHub Actions workflow. Because each python:container:distroless:py31x:docker task already depends on :sdks:python:container:py31x:docker (via dockerPrepare.dependsOn) and inherits the same root project properties (-Ppush-containers, -Pdocker-tag-list, -Pcontainer-architecture-list, -Pdocker-pull-licenses), both the base and distroless Python container images are still built and pushed with identical configurations while saving significant CI runner time and avoiding concurrent pushes to the same image tags.

🔍 General Feedback

  • The Gradle task dependency graph in sdks/python/container/distroless/common.gradle and BeamDockerPlugin.groovy confirms that :sdks:python:container:py31x:docker builds and pushes the multi-arch base images with all specified tags prior to building and pushing the distroless images.
  • No issues found; the change is clean, accurate, and well-scoped.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @Amar3tto for label build.
Backup reviewer: Abacn (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@Amar3tto
Amar3tto merged commit 84cd1ea into apache:master Oct 30, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants