[Spark] Run ValidatesRunner tests serially, runner metrics are JVM wide by tkaymak · Pull Request #40428 · apache/beam

tkaymak · 2026-10-06T09:39:51Z

Since #40324, ValidatesRunner methods run on 8 threads per test JVM. The Spark runners keep metrics in a JVM wide MetricsAccumulator and allow one SparkContext per JVM, so every Spark VR and PVR PostCommit is red since 2026-10-01.

This sets beam.validatesRunner.parallelThreads=1 for the Spark runner and Spark job server test tasks, as the Dataflow runner sets its own value. The trigger files run the eight affected suites on this PR.

Local check, :runners:spark:4:validatesStructuredStreamingRunnerBatch on PAssertTest, FlattenTest and CreateTest: 37 failures on master, 46 tests and 0 failures with this change.

R: @Abacn
cc @reuvenlax

Fixes apache#40427.

tkaymak · 2026-10-06T09:39:57Z

R: @Abacn

github-actions · 2026-10-06T09:40:39Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

tkaymak · 2026-10-06T12:11:14Z

All eight Spark ValidatesRunner and PVR PostCommits are green on this PR, they were red on master since 2026-10-01:
VR Spark https://github.com/apache/beam/actions/runs/37444521250
VR Spark4 https://github.com/apache/beam/actions/runs/37444521417
VR SparkStructuredStreaming https://github.com/apache/beam/actions/runs/37444521359
PVR Spark Batch https://github.com/apache/beam/actions/runs/37444521275
PVR Spark3 Streaming https://github.com/apache/beam/actions/runs/37444521185
PVR Spark4 Batch https://github.com/apache/beam/actions/runs/37444521383
PVR Spark4 Streaming https://github.com/apache/beam/actions/runs/37444521204
PVR Spark4 StructuredStreaming https://github.com/apache/beam/actions/runs/37444521314

Abacn · 2026-10-06T15:27:46Z

Thanks for identifying the issue and the fix. I didn't expect #40324 would affect Spark ValidateRunner tests, it should only affect Dataflow. I suggest we set "beam.validatesRunner.parallelThreads" as "1" by default (or treat as 1 if the property is absent), and set to "8" in Dataflow ValidateRunner tests that currently running on parallel (only batch tests?) only

Abacn · 2026-10-06T15:27:58Z

cc: @reuvenlax

reuvenlax · 2026-10-06T17:32:23Z

Do we know whether Flink has a similar issue, or is the Flink Runner better behaved?

…

On Tue, Oct 6, 2026 at 8:47 AM Yi Hu ***@***.***> wrote: *Abacn* left a comment (apache/beam#40428) <#40428 (comment)> Thanks for identifying the issue and the fix. I didn't expect #40324 <#40324> would affect Spark ValidateRunner tests, it should only affect Dataflow. I suggest we set "beam.validatesRunner.parallelThreads" as "1" by default (or treat as 1 if the property is absent), and set to "8" in Dataflow ValidateRunner tests that currently running on parallel (only batch tests?) only — Reply to this email directly, view it on GitHub <#40428?email_source=notifications&email_token=AFAYJVI3LZNP7JI2T4ROWQ35SUHXPA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMBRHE3DCMRTHEYKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-6019612390>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFAYJVK5SC2AIPBQGDFX3JD5SUHXPAVCNFSNUABEKJSXA33TNF2G64TZHM2TAOJQGQZDINJ3JFZXG5LFHM2TOMRWGI3DIMJXGWQXMAQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AFAYJVIRV6JY2PRFCJO335L5SUHXPA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMBRHE3DCMRTHEYKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/AFAYJVJAXXYTESK3XD656E35SUHXPA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMBRHE3DCMRTHEYKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

reuvenlax · 2026-10-06T17:37:12Z

lgtm

Abacn · 2026-10-06T17:40:45Z

Yeah Flink looks fine:

https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Flink.yml?query=event%3Aschedule

Spark has a well known limitation that is only one active SparkContext is allowed / honored in a JVM at a time, and caused the cross-test contamination.

reuvenlax · 2026-10-06T17:42:42Z

Ok - then seems reasonable to just disable Spark. Do we still regularly run any other runner ValidatesRunner suites?

…

On Tue, Oct 6, 2026 at 10:41 AM Yi Hu ***@***.***> wrote: *Abacn* left a comment (apache/beam#40428) <#40428 (comment)> Yeah Flink is fine: https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Flink.yml?query=event%3Aschedule Spark has a well known limitation that is only one active SparkContext is allowed / honored in a JVM at a time, and caused the cross-test contamination. — Reply to this email directly, view it on GitHub <#40428?email_source=notifications&email_token=AFAYJVNI27EZYPWSZ3XX2KL5SUVDHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMBSGE4TMOJQGIZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-6021969023>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFAYJVLQTVABDOZZNJZYMET5SUVDHAVCNFSNUABEKJSXA33TNF2G64TZHM2TAOJQGQZDINJ3JFZXG5LFHM2TOMRWGI3DIMJXGWQXMAQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AFAYJVMU2PL26QX74JMYFC35SUVDHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMBSGE4TMOJQGIZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/AFAYJVKJ6F6P4Q4KBNKQBED5SUVDHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMBSGE4TMOJQGIZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

Abacn

then seems reasonable to just disable Spark.

SGTM and will merge after test passing

codecov · 2026-10-06T18:35:24Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.74%. Comparing base (69bdf5f) to head (3c86ba8).
⚠️ Report is 24 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #40428      +/-   ##
============================================
- Coverage     59.04%   58.74%   -0.30%     
+ Complexity    15624    13903    -1721     
============================================
  Files          2797     2583     -214     
  Lines        280750   271419    -9331     
  Branches      12488    11226    -1262     
============================================
- Hits         165764   159443    -6321     
+ Misses       108540   105984    -2556     
+ Partials       6446     5992     -454

Flag	Coverage Δ
java	`65.24% <ø> (-0.20%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

reuvenlax · 2026-10-06T19:42:20Z

But Flink is fine with the changes. Unless another runner is also failing, I would leave the default as it is. We definitely don't want to rollback DirectRunner, as Java PreSubmit has historically been slow.

…

On Tue, Oct 6, 2026 at 12:31 PM Tobias Kaymak ***@***.***> wrote: *tkaymak* left a comment (apache/beam#40428) <#40428 (comment)> Done, beam.validatesRunner.parallelThreads now defaults to 1 in BeamParallelJunit4Runner independent of beam.test.parallelThreads, Dataflow keeps its explicit value and the Spark gradle changes are reverted. One place this reaches beyond Spark is :runners:direct-java:needsRunnerTests in PreCommit Java, where the ValidatesRunner methods ran 8 wide since #40324 <#40324>, so I set 8 there and in the Direct validatesRunner task to keep the PreSubmit speedup, happy to drop it if you prefer Dataflow only. The 8 Spark PostCommits rerun on this push, the Direct VR trigger file is bumped as well. — Reply to this email directly, view it on GitHub <#40428?email_source=notifications&email_token=AFAYJVOOD3AV2VTEO5ICXT35SVCC5A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMBSGM4TGMZVGYY2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-6023933561>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFAYJVN4XQYJ6XLTWSUXCMT5SVCC5AVCNFSNUABEKJSXA33TNF2G64TZHM2TAOJQGQZDINJ3JFZXG5LFHM2TOMRWGI3DIMJXGWQXMAQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AFAYJVNLFTQ64X4YRIZXKYL5SVCC5A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMBSGM4TGMZVGYY2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/AFAYJVMILUJDHTHZNFQHZMT5SVCC5A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMBSGM4TGMZVGYY2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

Abacn · 2026-10-06T20:36:57Z

The current PR was merged with only overriding Spark validate runner test to be sequential and we are good for now.

PreCommit Java doesn't run ValidateRunner test. There is a separate ValidateRunner Direct run at PostCommit time

[Spark] Run ValidatesRunner tests serially, runner metrics are JVM wide

3c86ba8

Fixes apache#40427.

github-actions Bot added runners spark labels Oct 6, 2026

tkaymak mentioned this pull request Oct 6, 2026

[Spark 4] Support stateful ParDo in Structured Streaming via transformWithState #40281

Merged

Abacn reviewed Oct 6, 2026

View reviewed changes

Abacn approved these changes Oct 6, 2026

View reviewed changes

Abacn merged commit 32d54e8 into apache:master Oct 6, 2026
29 of 30 checks passed

tkaymak changed the title ~~[Spark] Run ValidatesRunner tests serially, runner metrics are JVM wide~~ Default beam.validatesRunner.parallelThreads to 1, runners opt in Oct 6, 2026

tkaymak changed the title ~~Default beam.validatesRunner.parallelThreads to 1, runners opt in~~ [Spark] Run ValidatesRunner tests serially, runner metrics are JVM wide Oct 6, 2026

tkaymak mentioned this pull request Oct 6, 2026

[Bug]: Spark ValidatesRunner PostCommits red since parallel test threads (#40324) #40427

Closed

2 tasks done

Conversation

tkaymak commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tkaymak commented Oct 6, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

tkaymak commented Oct 6, 2026

Uh oh!

Abacn commented Oct 6, 2026

Uh oh!

Abacn commented Oct 6, 2026

Uh oh!

reuvenlax commented Oct 6, 2026 via email

Uh oh!

reuvenlax commented Oct 6, 2026

Uh oh!

Abacn commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

reuvenlax commented Oct 6, 2026 via email

Uh oh!

Abacn left a comment

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Oct 6, 2026

Codecov Report

Uh oh!

Uh oh!

reuvenlax commented Oct 6, 2026 via email

Uh oh!

Abacn commented Oct 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tkaymak commented Oct 6, 2026 •

edited

Loading

Abacn commented Oct 6, 2026 •

edited

Loading