Flaky test report: committed-code failures on 2026-06-13

[andrross]

Flaky test report: committed-code failures on 2026-06-13

This report covers test failures recorded in the OpenSearch metrics cluster for committed-code runs (Timer + Post Merge Action) over the 24-hour window 2026-06-13T10:00 UTC → 2026-06-14T10:00 UTC, with historical context across all build types (including PR builds).

Summary table

Sorted by total unique builds historically affected (descending). All 7 reproduction attempts with the original seed passed locally — none of the failures reproduced deterministically.

#	Test	Recent build	Repro	First failure	Total builds (all-time)	Trend
1	MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock	80356 (2.19)	not attempted†	2024-03-25	966	Improving (sharp drop after 2025-04)
2	FullRollingRestartIT.testFullRollingRestart_withNoRecoveryPayloadAndSource	80336	did not reproduce	2024-10-11	143	Worsening since 2026-02
3	ClusterShardLimitIT.testOpenIndexOverLimit	80331	did not reproduce	2025-10-15	73	Worsening (large jump in 2026-05)
4	RareClusterStateIT.testDisassociateNodesWhileShardInit	80335	did not reproduce	2024-11-04	61	Worsening since 2026-04
5	RemoteSplitIndexIT.testSplitFromOneToN	80342	did not reproduce	2024-04-11	34	Stable low rate
6	RestoreShallowSnapshotV2IT.testRestoreShallowSnapshotRepository	80332	did not reproduce	2024-09-03	31	Stable low rate
7	IndexRecoveryIT.testRecoverLocallyUpToGlobalCheckpoint	80337	did not reproduce	2024-06-05	9	Stable, very rare
8	AbstractAsyncTaskTests.testChangeInterval	80343	did not reproduce‡	2024-06-06	4	Stable, very rare

† Build 80356 ran on the 2.19 branch; the failure stack trace did not include a SeedInfo/tests.seed value (the assertion came from assertBusy inside InternalTestCluster.validateClusterFormed, which strips the randomized-runner seed frame). The local repo is on main, so a same-branch reproduction was not attempted.

‡ The local repo had staged changes in server/src/main/java/org/opensearch/common/util/concurrent/ at the time of reproduction (ContextPreservingExecutorService.java, OpenSearchExecutors.java, ThreadPool.java) which directly overlap with the package under test. Reproduction was therefore on a modified codebase.

Reproduction methodology

For each distinct test:

1. The seed was extracted from the Jenkins test report errorStackTrace (SeedInfo.seed([…])).
2. The test was run on the local repository (HEAD 90fdbf95a0fbb22514004d7f21747fb9411446f3, branch main) with:

   ./gradlew <task> --tests "<fully.qualified.TestClass.testMethod>" -Dtests.seed=<SEED>
   

3. Each run was performed once (per the request); pass/fail was determined from the resulting JUnit XML.

Per the flaky test investigation guidance, RandomizedRunner seeds the Random streams but does not control thread scheduling, network timing, GC pauses, or disruption-simulation timing, so a single seed run is not a reliable reproducer for cluster-disruption-style tests. The "did not reproduce" results below should therefore be interpreted as "not deterministically reproducible from the seed alone", not as evidence of a bug fix.

Per-test details

1. MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

• Recent failure: build 80356 (Timer, branch 2.19) on 2026-06-14T06:57:56Z
• Failure message: Missing cluster-manager, expected nodes: … from assertBusy inside InternalTestCluster.validateClusterFormed
• Seed: not present in the captured stack trace
• Reproduction: not attempted (no seed; different branch from local repo)
• Historical pattern: 966 unique builds since 2024-03-25, by far the most prolific item in this report. Was extremely flaky on main from 2024-04 through 2025-04 (75–125 unique builds/month), then dropped sharply to single-digit numbers per month from 2025-05 onwards. The current sighting is on 2.19, where the apparent fix on main may not be fully effective.
• Trend: Improving on main; still appears intermittently on backport branches.

2024-03  builds=7    2024-09  builds=125   2025-03  builds=41   2025-09  builds=0
2024-04  builds=75   2024-10  builds=84    2025-04  builds=98   2025-10  builds=3
2024-05  builds=52   2024-11  builds=37    2025-05  builds=5    2025-11  builds=3
2024-06  builds=110  2024-12  builds=24    2025-06  builds=4    2025-12  builds=3
2024-07  builds=80   2025-01  builds=58    2025-07  builds=4    2026-01  builds=1
2024-08  builds=95   2025-02  builds=43    2025-08  builds=6    …
2026-04  builds=3    2026-06  builds=2

2. FullRollingRestartIT.testFullRollingRestart_withNoRecoveryPayloadAndSource

• Recent failure: build 80336 (Timer, main) on 2026-06-14
• Parameters: p0={"cluster.indices.replication.strategy":"SEGMENT"}
• Failure message: replica shards haven't caught up with primary expected:<16> but was:<13>
• Seed: 32A45E01CEA9D37B:1ED344167E4B223
• Reproduction: ran 2 parameterized cases; both PASSED locally
• Historical pattern: 143 unique builds since 2024-10. Effectively dormant until 2025-07 (47 builds), spike continued 24 in 2025-08, then a long quiet period, with a renewed climb starting 2026-02 (17/13/9/19/11 builds in Feb–Jun 2026).
• Trend: Worsening since 2026-02.

2025-07  builds=47   2025-08  builds=24   2026-02  builds=17
2026-03  builds=13   2026-04  builds=9    2026-05  builds=19   2026-06  builds=11

3. ClusterShardLimitIT.testOpenIndexOverLimit

• Recent failure: build 80331 (Timer, main) on 2026-06-13
• Parameters: p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}
• Failure message: CloseIndexResponse failed - not acked (assertion in assertAcked)
• Seed: F6EB11AAFAFCB87:5EC1C1AA21554CF1
• Reproduction: ran 2 parameterized cases; both PASSED locally
• Historical pattern: 73 unique builds since first failure on 2025-10-15. Steady monthly volume, with a notable surge in 2026-05 (19 builds).
• Trend: Worsening, large recent uptick.

2025-10  builds=2   2025-11  builds=9   2025-12  builds=3
2026-01  builds=7   2026-02  builds=7   2026-03  builds=6
2026-04  builds=7   2026-05  builds=19  2026-06  builds=13

4. RareClusterStateIT.testDisassociateNodesWhileShardInit

• Recent failure: build 80335 (Timer, main) on 2026-06-14
• Failure message: java.lang.AssertionError (no message)
• Seed: 1CBBFB5B8908F2A7:7DA969F9131DC789
• Reproduction: PASSED locally
• Historical pattern: 61 unique builds since 2024-11-04. Was occasional through 2025 (≤4 builds/month), but jumped to 12/18/13 in Apr/May/Jun 2026.
• Trend: Worsening since 2026-04.

2025-12  builds=4   2026-01  builds=3   2026-03  builds=3
2026-04  builds=12  2026-05  builds=18  2026-06  builds=13

5. RemoteSplitIndexIT.testSplitFromOneToN

• Recent failure: build 80342 (Timer, main) on 2026-06-13
• Failure message: java.lang.AssertionError (no message in error details)
• Seed: 2F42723B70A09626
• Reproduction: PASSED locally (took 82 seconds for the test)
• Historical pattern: 34 unique builds since 2024-04, with a multi-month gap mid-2025 and a small bump in 2026-04 (5 builds). Most recent months: 3/5/2/1.
• Trend: Stable low rate.

6. RestoreShallowSnapshotV2IT.testRestoreShallowSnapshotRepository

• Recent failure: build 80332 (Post Merge Action, commit 1652e069add9…) on 2026-06-13
• Parameters: p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}
• Failure message: org.opensearch.index.engine.RefreshFailedEngineException: Refresh failed
• Seed: 1EF89954B4332642:1B8383150E3D376B
• Reproduction: ran 2 parameterized cases; both PASSED locally
• Historical pattern: 31 unique builds since 2024-09 with no large monthly bursts; appears in roughly half the months. Last few months: 2/1/2/1/1.
• Trend: Stable low rate.

7. IndexRecoveryIT.testRecoverLocallyUpToGlobalCheckpoint

• Recent failure: build 80337 (Timer, main) on 2026-06-13
• Failure message: Expected: <26L> but: was <71L> (operations-recovered count mismatch)
• Seed: 8B2B52065FD5E76F:7170DF4248D04327
• Reproduction: PASSED locally
• Historical pattern: only 9 unique builds across 2 years (3 in 2024-06, 1 in 2024-07, 2 in 2024-09, then 1 in 2025-02, 1 in 2026-05, 1 in 2026-06).
• Trend: Stable, very rare.

8. AbstractAsyncTaskTests.testChangeInterval

• Recent failure: build 80343 (Timer, main) on 2026-06-13
• Failure message: java.lang.AssertionError (no message)
• Seed: E00EFEE085576719:199B75570907EACC
• Reproduction: PASSED locally on a modified codebase (uncommitted/staged changes were present in org.opensearch.common.util.concurrent.*, the package under test)
• Historical pattern: only 4 unique builds in 2 years (2024-06, 2024-08, 2025-10, 2026-06).
• Trend: Stable, extremely rare.

Notes & caveats

• This report describes observed flake patterns, not root causes. No code changes were made.
• Reproductions were single-shot with the seed only. To convert "did not reproduce" into useful evidence, the next investigator should run with -Dtests.iters=N (N≥20 per the steering guidance) on each test individually.
• The local repo had unrelated staged changes in org.opensearch.common.util.concurrent.* at the time the report was generated, which is why the AbstractAsyncTaskTests.testChangeInterval reproduction is flagged with a caveat.
• Build 80356 (MinimumClusterManagerNodesIT) ran on the 2.19 branch; cross-checking against the current main improvement pattern is something a maintainer of that area should do explicitly before drawing conclusions about the backport branch.
• The 2026-04-15 CI runner upgrade (m5.8xlarge → m7a.8xlarge) is a known amplifier of latent flakes; tests Bump com.diffplug.spotless from 5.6.1 to 6.2.1 #2, Bump com.diffplug.spotless from 5.6.1 to 6.2.2 #3, and Bump opensearch-core from current to 1.2.4 in /buildSrc/src/testKit/opensearch.build #4 in this list all show clear inflection points after that date and are good candidates to investigate through that lens.

Sources

• Metrics cluster query: gradle-check-* index, time range 2026-06-13T10:00Z → 2026-06-14T10:00Z, filter test_status:FAILED AND ((invoke_type:Timer AND git_reference:main) OR (invoke_type:"Post Merge Action" AND pull_request_title:"push trigger main *")), plus the 2.19 Timer run.
• Per-test history: wildcard query on test_name with monthly date_histogram and cardinality aggregation on build_number.
• Seeds extracted from Jenkins testReport/api/json errorStackTrace fields (SeedInfo.seed([…])).