[OpenTelemetry] Enable OpenTelemetry stitching with Logs for Dataflow worker by stankiewicz · Pull Request #39625 · apache/beam

stankiewicz · 2026-08-05T08:43:58Z

This change will add trace, spanId and sampled fields supported by Logging if Spans are available in current context and if feature is enabled (disabled by default).

Outcome is trace link visible in logging and logging visible in trace:

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

codecov · 2026-08-05T09:38:13Z

Codecov Report

❌ Patch coverage is 66.66667% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 55.95%. Comparing base (f2c622b) to head (bf6b257).
⚠️ Report is 45 commits behind head on master.

Files with missing lines	Patch %	Lines
...w/worker/logging/DataflowWorkerLoggingHandler.java	75.00%	0 Missing and 4 partials ⚠️
...rker/logging/DataflowWorkerLoggingInitializer.java	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #39625      +/-   ##
============================================
- Coverage     58.15%   55.95%   -2.20%     
+ Complexity    13085     2275   -10810     
============================================
  Files          2521     1113    -1408     
  Lines        264850   175079   -89771     
  Branches      10788     1465    -9323     
============================================
- Hits         154023    97968   -56055     
+ Misses       105051    74647   -30404     
+ Partials       5776     2464    -3312

Flag	Coverage Δ
java	`75.62% <66.66%> (+11.36%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

scwhittle

Do you also want to add this to LogRecordHandler in BeamFnLoggingClient.java?
That is used with the dataflow portable runner harness.

scwhittle · 2026-08-05T12:28:23Z

            .setTimestamp(Instant.ofEpochMilli(record.getMillis()))
            .setSeverity(severityFor(record.getLevel()));
-
+    SpanContext spanContext = Span.current().getSpanContext();


maybe better to just get the current span context if logOpenTelemetryTraceSpanIdAndSampled is enabled in case there is some overhead like a thread-local

good point, will do

scwhittle · 2026-08-05T12:28:35Z

      writeIfNotEmpty(generator, "logger", record.getLoggerName());
      writeIfNotEmpty(generator, "exception", formatException(record.getThrown()));
+      SpanContext spanContext = Span.current().getSpanContext();
+      if (logOpenTelemetryTraceSpanIdAndSampled.get() && spanContext.isValid()) {


scwhittle · 2026-08-05T12:30:00Z

      writeIfNotEmpty(generator, "exception", formatException(record.getThrown()));
+      SpanContext spanContext = Span.current().getSpanContext();
+      if (logOpenTelemetryTraceSpanIdAndSampled.get() && spanContext.isValid()) {
+        generator.writeStringField("trace", spanContext.getTraceId());


does isValid imply that the trace id and span id are not empty? otherwise use writeIfNotEmpty

correct, it means that there is something non empty and not default (different to "00000000000000000000000000000000")

scwhittle · 2026-08-05T12:30:30Z

+      if (logOpenTelemetryTraceSpanIdAndSampled.get() && spanContext.isValid()) {
+        generator.writeStringField("trace", spanContext.getTraceId());
+        generator.writeStringField("spanId", spanContext.getSpanId());
+        generator.writeBooleanField("traceSampled", spanContext.isSampled());


would it be better to reduce overhead by just writing this if it is true and omitting it if false?

maybe this should be trace_sampled as well instead of camel-case to match custom_data?

hmm the documented agent format is also inconsistent https://docs.cloud.google.com/logging/docs/structured-logging#structured_logging_special_fields

uses trace_sampled and spanId

but json is format used between dataflow worker and agent that reads data and sends asynchronously to clod logging. Happy to change field names.

on skipping writing when sampled=false - sounds like good idea, will implement that as well.

scwhittle · 2026-08-05T12:31:12Z

      @Nullable DataflowExecutionState executionState,
      ImmutableMap<String, String> defaultResourceLabels) {
    Struct.Builder payloadBuilder = Struct.newBuilder();
+    //


github-actions · 2026-08-05T13:07:14Z

Assigning reviewers:

R: @kennknowles for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

scwhittle · 2026-08-07T09:10:09Z

+        if (spanContext.isValid() && spanContext.isSampled()) {
+          generator.writeStringField("trace", spanContext.getTraceId());
+          generator.writeStringField("spanId", spanContext.getSpanId());
+          generator.writeBooleanField("trace_sampled", spanContext.isSampled());


I meant to just omit
"trace_sampled": false
since that is probably common and we could just infer it was false.

This is a little different since we're also not logging the other fields.
Is there still value in adding the trace and spanId to the log if it is not-sampled? If not, this seems better to avoid unnecessary writes/parsing. but perhaps there could be some use for these fields, if they could correlate to other logs even if it isn't sampled? That is indicated by https://docs.cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#FIELDS.trace_sampled

I think it would be better then to log trace/spanId if the context is valid. And just conditionally log trace_sampled if it is valid and true. Alternatively we could add another option controlling if we only log sampled trace/spanids.

thanks, will update that today.

scwhittle

lgtm with comment

scwhittle · 2026-08-10T13:18:39Z

+    Tracer tracer = openTelemetry.getTracer("foo");
+    Span span = tracer.spanBuilder("test").startSpan();
+    try (Scope scope = span.makeCurrent()) {
+      SpanContext spanContext = Span.current().getSpanContext();


if spanContext is not sampled, the below assertEquals will fail since we won't have trace_sampled entry

can you have the test handle both or otherwise make sure it is always sampled?

100% sample is default but I've added it explicitly.

… direct logging and file based

stankiewicz requested a review from scwhittle August 5, 2026 08:44

github-actions Bot added java runners dataflow labels Aug 5, 2026

scwhittle requested changes Aug 5, 2026

View reviewed changes

github-actions Bot added the Next Action: Reviewers label Aug 5, 2026

stankiewicz force-pushed the tace_logs_stitch branch 2 times, most recently from 20804a3 to 873e645 Compare August 5, 2026 13:51

stankiewicz requested a review from scwhittle August 5, 2026 13:51

scwhittle requested changes Aug 7, 2026

View reviewed changes

stankiewicz force-pushed the tace_logs_stitch branch 3 times, most recently from 1028d85 to c52573d Compare August 10, 2026 12:21

github-actions Bot added build and removed build labels Aug 10, 2026

stankiewicz requested a review from scwhittle August 10, 2026 12:41

stankiewicz force-pushed the tace_logs_stitch branch from c52573d to 30117bc Compare August 10, 2026 13:08

github-actions Bot added build and removed build labels Aug 10, 2026

scwhittle approved these changes Aug 10, 2026

View reviewed changes

stankiewicz force-pushed the tace_logs_stitch branch from 30117bc to bf6b257 Compare August 10, 2026 13:21

github-actions Bot added build and removed build labels Aug 10, 2026

Enable OpenTelemetry stiching with Logs for Dataflow worker, both for…

a73ff29

… direct logging and file based

stankiewicz force-pushed the tace_logs_stitch branch from bf6b257 to a73ff29 Compare August 10, 2026 14:42

github-actions Bot added build and removed build labels Aug 10, 2026

stankiewicz merged commit f0da6f3 into apache:master Aug 10, 2026
28 of 29 checks passed

stankiewicz deleted the tace_logs_stitch branch October 13, 2026 12:25

Conversation

stankiewicz commented Aug 5, 2026

GitHub Actions Tests Status (on master branch)

Uh oh!

codecov Bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

scwhittle left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stankiewicz Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scwhittle left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Aug 5, 2026 •

edited

Loading

stankiewicz Aug 5, 2026 •

edited

Loading