Repository navigation
The CloudML Benchmarks Dataflow job is flaky #38780
Description
The CloudML Benchmarks Dataflow is failing over 50% of the time.
Please visit https://github.com/apache/beam/actions/workflows/beam_CloudML_Benchmarks_Dataflow.yml?query=is%3Afailure+branch%3Amaster to see all failed workflow runs.
See also Grafana statistics: http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1&viewPanel=9&var-Workflow=CloudML%20Benchmarks%20Dataflow
Activity
github-actions commentedon Jun 13, 2026
Reopening since the workflow is still flaky
github-actions commentedon Aug 25, 2026
Reopening since the workflow is still flaky
aIbrahiim commentedon Aug 25, 2026
.take-issue
aIbrahiim commentedon Aug 26, 2026
the workflow is failing because tfx_bsl==1.21.0 pulls apache-beam[gcp] from PyPI, and after PR #39820, so pip can no longer keep our local Beam dev SDK version so the job is submitted with 2.77.0.dev, but workers end up with 2.75.0, and then Beam throws the environment mismatch error
I tried to put --no-deps in requirements.txt but it failed because this pip version does not accept that option inside requirements file.
then I used -c constraints.txt and forced Beam version but it also failed with ResolutionImpossible (protobuf conflict between TFX/TensorFlow stack and older google-api-core constraints)
cc: @damccorm @Abacn @mutianf
aIbrahiim commentedon Aug 26, 2026
update, I no longer use --no-deps or constraints and instead, after TFT requirements install, and force-reinstall the staged SDK on the driver (--force-reinstall --no-deps), and also pass --extra_package with the same SDK so workers keep the correct version after requirements install. I tested it and the workflow passed https://github.com/aIbrahiim/beam/actions/runs/32958188765