Skip to content

Fix OpenTelemetry dependency versions in generated Dataflow runner Maven POM - #40530

Merged
stankiewicz merged 1 commit into
apache:masterfrom
damccorm:fix-dataflow-runner-otel-pom
Oct 12, 2026
Merged

stankiewicz merged 1 commit into
apache:masterfrom
damccorm:fix-dataflow-runner-otel-pom

Conversation

@damccorm

@damccorm damccorm commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Projects consuming org.apache.beam:beam-runners-google-cloud-dataflow-java:2.78.0-SNAPSHOT with Maven fail to resolve the runner's transitive dependencies because the published POM is rejected as invalid:

[WARNING] The POM for org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.78.0-... is invalid, transitive dependencies (if any) will not be available
[ERROR] 'dependencies.dependency.version' for io.opentelemetry:opentelemetry-exporter-otlp:jar is missing.
[ERROR] 'dependencies.dependency.version' for io.opentelemetry:opentelemetry-sdk-extension-autoconfigure:jar is missing.

Because Maven skips all transitive dependencies when a dependency's POM fails model validation, required runner dependencies (such as com.google.api.services.dataflow.Dataflow) are missing at runtime, causing PipelineOptionsFactory initialization and pipeline submission to fail. This also broke the PostRelease Nightly Snapshot workflow (#30505).

Root Cause

  1. Custom POM generation in BeamModulePlugin.groovy:
    • When generating <dependencies>, BeamModulePlugin only emits <version> if it.version != null on the direct dependency entry in library.java.
    • When generating <dependencyManagement>, BeamModulePlugin only imports BOMs (Category.REGULAR_PLATFORM / Category.ENFORCED_PLATFORM) that are directly declared in that subproject's implementation (or shadow) configuration.
  2. Missing BOM import and versionless OpenTelemetry declarations:
    • In BeamModulePlugin.groovy, opentelemetry_exporter_otlp and opentelemetry_extension_autoconfigure (along with opentelemetry_api and opentelemetry_sdk) were declared without explicit versions (// opentelemetry-bom sets version).
    • Other modules using versionless opentelemetry_* dependencies (sdks/java/core, sdks/java/io/google-cloud-platform, sdks/java/io/kafka, sdks/java/extensions/opentelemetry-gcp-auth-extension) explicitly declare platform(library.java.opentelemetry_bom), whereas runners/google-cloud-dataflow-java/build.gradle only declared enforcedPlatform(library.java.google_cloud_platform_libraries_bom) (which does not manage opentelemetry-exporter-otlp or opentelemetry-sdk-extension-autoconfigure).
    • When Include OpenTelemetry dependencies in published Dataflow runner POM #40352 promoted opentelemetry_exporter_otlp and opentelemetry_extension_autoconfigure from runtimeOnly to implementation in runners/google-cloud-dataflow-java/build.gradle, Gradle still resolved their versions transitively via :sdks:java:core during compilation, but beam-runners-google-cloud-dataflow-java.pom was generated without <version> on those two dependencies and without opentelemetry-bom-alpha in <dependencyManagement>.

Fix

  1. Import opentelemetry_bom in runners/google-cloud-dataflow-java/build.gradle: Adds implementation platform(library.java.opentelemetry_bom) and permitUnusedDeclared platform(library.java.opentelemetry_bom) (matching sdks/java/core, sdks/java/io/google-cloud-platform, sdks/java/io/kafka, and sdks/java/extensions/opentelemetry-gcp-auth-extension) so io.opentelemetry:opentelemetry-bom-alpha is imported in <dependencyManagement> of beam-runners-google-cloud-dataflow-java.pom.
  2. Set explicit $opentelemetry_version on OpenTelemetry entries in BeamModulePlugin.groovy: Sets :$opentelemetry_version (kept in sync with libraries-bom by bomupgrader.py) on opentelemetry_api, opentelemetry_sdk, opentelemetry_exporter_otlp, and opentelemetry_extension_autoconfigure (following Fix pom involving maven-archetype and opentelemetry #38518 for opentelemetry_context), and removes the duplicate opentelemetry_sdk_testing map key, ensuring no module can emit a versionless OpenTelemetry dependency in its published POM.

@damccorm

Copy link
Copy Markdown
Contributor Author

R: @stankiewicz

@damccorm
damccorm marked this pull request as ready for review October 12, 2026 18:56
@github-actions

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

@stankiewicz
stankiewicz merged commit 6a05766 into apache:master Oct 12, 2026
25 of 29 checks passed
@damccorm
damccorm deleted the fix-dataflow-runner-otel-pom branch October 12, 2026 19:41
@codecov

codecov Bot commented Oct 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.27%. Comparing base (76f6e6d) to head (5b56e4a).
⚠️ Report is 137 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #40530      +/-   ##
============================================
+ Coverage     56.16%   65.27%   +9.10%     
- Complexity     2288    13921   +11633     
============================================
  Files          1124     1720     +596     
  Lines        178336   107838   -70498     
  Branches       1489    11245    +9756     
============================================
- Hits         100170    70388   -29782     
+ Misses        75645    33265   -42380     
- Partials       2521     4185    +1664     
Flag Coverage Δ
go ?
java 65.27% <ø> (-10.52%) ⬇️
python ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants