[Bug]: Spark ValidatesRunner PostCommits red since parallel test threads (#40324) #40427

What happened?

Since #40324 (merged 2026-10-01), every Spark ValidatesRunner PostCommit on master is red. Last green runs were 2026-10-01 16:47 UTC, first red 2026-10-01 22:46 UTC.

Affected: beam_PostCommit_Java_ValidatesRunner_Spark, beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming, beam_PostCommit_Java_ValidatesRunner_Spark4, beam_PostCommit_Java_PVR_Spark_Batch, beam_PostCommit_Java_PVR_Spark3_Streaming, beam_PostCommit_Java_PVR_Spark4_Batch, beam_PostCommit_Java_PVR_Spark4_Streaming, beam_PostCommit_Java_PVR_Spark4_StructuredStreaming. Flink and Direct stay green.

Symptoms: 100 to 200 tests per run fail with Expected 1 successful assertions, but found 4 (or 0), the failing set differs between runs. The PVR suites fail with Only one SparkContext should be running in this JVM.

Cause: #40324 sets beam.test.parallelThreads=8 for every Test task. BeamParallelJunit4Runner.selectExecutor falls back from beam.validatesRunner.parallelThreads to that value, so every runner's ValidatesRunner suite became parallel without an opt in. Dataflow opts in explicitly in its build.gradle. The Spark runners keep metrics in a JVM wide MetricsAccumulator singleton, cleared on every run(), and allow one SparkContext per JVM, so concurrent pipelines mix or reset each other's PAssert counters.

Fix: set beam.validatesRunner.parallelThreads=1 for the Spark runner and Spark job server test tasks, as the Dataflow runner already sets its own value. Per pipeline metrics in the Spark runners would allow parallel tests later. A default of 1 for beam.validatesRunner.parallelThreads in BeamParallelJunit4Runner would protect other runners too.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Java SDK
• Component: Spark Runner