Flaky test report: committed-code failures on 2026-06-10

[andrross]

Summary

10 distinct tests failed against committed code (Timer or Post Merge Action builds on main) in the 24 hours ending 2026-06-10T10:00 UTC. None reproduced locally with the original seed, consistent with non-deterministic (timing/environment-dependent) flakiness.

Summary Table (sorted by total builds affected)

#	Test	Builds Affected	First Seen	Trend	Reproduced?
1	RemoteStoreStatsIT	217	2024-03-27	Worsening — recent months show increased failure rate	No — passed with seed
2	MixedClusterClientYamlTestSuiteIT	148	2024-03-25	Worsening — recent months show increased failure rate	N/A — requires multi-version cluster
3	SharedClusterSnapshotRestoreIT	108	2024-08-31	Worsening — recent months show increased failure rate	No — passed with seed
4	SecureReactorNetty4HttpServerTransportTests	67	2025-02-19	Improving — recent months show decreased failure rate	No — passed with seed
5	IngestFromKafkaIT	54	2025-10-15	Worsening — recent months show increased failure rate	No — passed with seed
6	AggregationProfilerIT	26	2025-08-07	Worsening — recent months show increased failure rate	No — passed with seed
7	SegmentReplicationUsingRemoteStoreIT	12	2024-07-15	Worsening — recent months show increased failure rate	No — passed with seed
8	IndexRecoveryIT	12	2024-06-05	Worsening — recent months show increased failure rate	No — passed with seed
9	AdmissionControlSingleNodeTests	8	2025-04-14	Stable — consistent low-level flakiness	No — passed with seed
10	IngestPipelineFromKafkaIT	3	2026-03-30	Stable — consistent low-level flakiness	No — passed with seed

Detailed Findings

1. RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs

Field	Value
Build	#79997
Seed	98627D977CB6673C:5595A84B3AA9B0FA
Error	java.lang.AssertionError
First failure	2024-03-27
Total builds affected	217
Recent monthly failures (2026)	2026-01=7, 2026-02=2, 2026-03=9, 2026-04=8, 2026-05=14, 2026-06=12
Reproduced locally	No — passed with seed
Trend	Worsening — recent months show increased failure rate

2. MixedClusterClientYamlTestSuiteIT.test {p0=cluster.health/10_basic/cluster health with closed index}

Field	Value
Build	#79975
Seed	6AD408F9F4AB03EB:E28037235A576E13
Error	java.lang.AssertionError: Failure at [cluster.health/10_basic:182]: expected [2xx] status code but api [cluster.health] returned [408 Request Timeout] [{"cluster_name":"v3.6.1","status":"red","timed_o
First failure	2024-03-25
Total builds affected	148
Recent monthly failures (2026)	2026-01=1, 2026-04=9, 2026-05=11, 2026-06=13
Reproduced locally	N/A — requires multi-version cluster
Trend	Worsening — recent months show increased failure rate

3. SharedClusterSnapshotRestoreIT.testSnapshotFileFailureDuringSnapshot

Field	Value
Build	#80058
Seed	CA37A560C385DC0F:F545090A3AB8BA12
Error	`java.lang.AssertionError:
Expected: <0L>

 but: was <1L>` |

| First failure | 2024-08-31 |
| Total builds affected | 108 |
| Recent monthly failures (2026) | 2026-01=3, 2026-03=6, 2026-04=10, 2026-05=9, 2026-06=5 |
| Reproduced locally | No — passed with seed |
| Trend | Worsening — recent months show increased failure rate |

4. SecureReactorNetty4HttpServerTransportTests.testBadRequest

Field	Value
Build	#79964
Seed	E43DC6930946AF27:F50C8D304839DBDC
Error	reactor.core.Exceptions$ReactiveException: reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
First failure	2025-02-19
Total builds affected	67
Recent monthly failures (2026)	2026-06=2
Reproduced locally	No — passed with seed
Trend	Improving — recent months show decreased failure rate

5. IngestFromKafkaIT.testAllActiveOffsetBasedLag

Field	Value
Build	#80024
Seed	C12A3978C955D4CA:62CBB61E56B3475
Error	java.lang.AssertionError
First failure	2025-10-15
Total builds affected	54
Recent monthly failures (2026)	2026-03=8, 2026-04=13, 2026-05=25, 2026-06=6
Reproduced locally	No — passed with seed
Trend	Worsening — recent months show increased failure rate

6. AggregationProfilerIT.testTopHitsAggregationFetchProfiling {p0={"search.concurrent_segment_search.enabled":"true"}}

Field	Value
Build	#80029
Seed	2ACBF530BFD46BA8:51059F0E421632E3
Error	org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
First failure	2025-08-07
Total builds affected	26
Recent monthly failures (2026)	2026-03=2, 2026-04=1, 2026-05=5, 2026-06=4
Reproduced locally	No — passed with seed
Trend	Worsening — recent months show increased failure rate

7. SegmentReplicationUsingRemoteStoreIT.testPrimaryStopped_ReplicaPromoted

Field	Value
Build	#79924
Seed	3A9E9A3DB968E533:2B969B5AA61A120B
Error	java.lang.AssertionError: Count is 2 hits but 3 was expected. Total shards: 1 Successful shards: 1 & 0 shard failures:
First failure	2024-07-15
Total builds affected	12
Recent monthly failures (2026)	2026-04=3, 2026-05=4, 2026-06=1
Reproduced locally	No — passed with seed
Trend	Worsening — recent months show increased failure rate

8. IndexRecoveryIT.testOngoingRecoveryAndClusterManagerFailOver

Field	Value
Build	#80020
Seed	B82047EE4FA4447F:AE78B75A37F1BB9E
Error	java.lang.AssertionError: timed out waiting for green state
First failure	2024-06-05
Total builds affected	12
Recent monthly failures (2026)	2026-05=4, 2026-06=3
Reproduced locally	No — passed with seed
Trend	Worsening — recent months show increased failure rate

9. AdmissionControlSingleNodeTests.testAdmissionControlRejectionDisabledMode

Field	Value
Build	#80024
Seed	C12A3978C955D4CA:10A2707B0AD35D4
Error	java.lang.AssertionError: expected:<3> but was:<6>
First failure	2025-04-14
Total builds affected	8
Recent monthly failures (2026)	2026-03=1, 2026-06=1
Reproduced locally	No — passed with seed
Trend	Stable — consistent low-level flakiness

10. IngestPipelineFromKafkaIT.testCombinedScriptTransformations

Field	Value
Build	#79989
Seed	78A1D1604AFDB051:7F2405C1BA8BBD63
Error	org.awaitility.core.ConditionTimeoutException: Condition org.opensearch.plugin.kafka.KafkaUtils$$Lambda/0x000000002a6bb978 was not fulfilled within 1 minutes.
First failure	2026-03-30
Total builds affected	3
Recent monthly failures (2026)	2026-03=1, 2026-06=2
Reproduced locally	No — passed with seed
Trend	Stable — consistent low-level flakiness

Methodology

1. Queried gradle-check-* indices on the OpenSearch metrics cluster for test_status=FAILED in the past 24h, filtered to committed-code builds (invoke_type=Timer, git_reference=main or invoke_type=Post Merge Action).
2. Queried full historical failure data across all build types (including PR builds) with monthly date histogram and cardinality aggregation on build_number.
3. Extracted test seeds from Jenkins test reports (the SeedInfo.seed([...]) line in stack traces).
4. Attempted local reproduction: ./gradlew :<module>:<task> --tests "<class>.<method>" -Dtests.seed=<SEED>.
5. None of the 9 locally-runnable tests reproduced. The MixedClusterClientYamlTestSuiteIT requires a multi-version backward-compatibility cluster and could not be run locally with a simple gradle command.

Observations

• None of the seeds are deterministic reproducers. This is expected for timing-dependent flakes — RandomizedRunner seeds control Random streams but not thread scheduling, network timing, GC pauses, or JIT warmup.
• CI runner migration (April 2026): The move from m5.8xlarge to m7a.8xlarge correlates with upticks in MixedClusterClientYamlTestSuiteIT, SegmentReplicationUsingRemoteStoreIT, and IndexRecoveryIT.
• RemoteStoreStatsIT is the most chronic offender (217 builds since March 2024) with consistent ~5-15 builds/month. No sign of improvement.
• IngestFromKafkaIT is worsening rapidly (2 builds in Oct 2025 → 25 in May 2026).
• SecureReactorNetty4HttpServerTransportTests had a spike in Sep 2025 (33 builds) but has been mostly quiet since. The June 2026 occurrence may be a recurrence.