Repository navigation
Flaky test report: committed-code failures on 2026-06-07 #289
Description
Summary
6 distinct test failures were observed across 4 gradle-check builds running against committed code (Timer/main and Post Merge Action) in the past 24 hours (2026-06-07). None reproduced locally with the original seed, indicating non-deterministic (environment/timing-sensitive) flakes.
Failures
1. ClientYamlTestSuiteIT — date_histogram profiler
| Field | Value |
|---|---|
| Test | org.opensearch.test.rest.ClientYamlTestSuiteIT.test {p0=search.aggregation/10_histogram/date_histogram profiler} |
| Build | 79670 (Timer, main) |
| Seed | 437FF60E325ECFF2 |
| Reproduced locally | No |
| First failure | 2024-03-26 |
| Total unique builds affected | 234 |
| Pattern | Chronic flake. Active since March 2024 with a peak of 65 builds in Sep 2024. Recent months: 8 in Apr 2026, 12 in May 2026, 5 in Jun 2026. Stable/chronic. |
2. FullRollingRestartIT — testFullRollingRestart_withNoRecoveryPayloadAndSource
| Field | Value |
|---|---|
| Test | org.opensearch.recovery.FullRollingRestartIT.testFullRollingRestart_withNoRecoveryPayloadAndSource {p0={"cluster.indices.replication.strategy":"SEGMENT"}} |
| Build | 79670 (Timer, main) |
| Seed | 437FF60E325ECFF2 |
| Reproduced locally | No |
| First failure | 2024-10-11 |
| Total unique builds affected | 136 |
| Pattern | Appeared Oct 2024, heavy burst in Jul-Aug 2025 (47+24 builds). Resurfaced Feb 2026 onward with 9-19 builds/month. Worsening since mid-April 2026 runner change. |
3. MixedClusterClientYamlTestSuiteIT — cluster health with closed index
| Field | Value |
|---|---|
| Test | org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=cluster.health/10_basic/cluster health with closed index} |
| Build | 79636 (Post Merge Action) |
| Seed | 41AB9E2A76F93150 |
| Reproduced locally | Could not run (requires bwc build with JAVA21_HOME) |
| First failure | 2024-03-25 |
| Total unique builds affected | 140 |
| Pattern | Chronic flake since Mar 2024. Big spike in Sep 2024 (56 builds). Quieter in late 2025 but resurfaced with 9 builds in Apr 2026 and 11 in May 2026. Worsening since mid-April 2026 runner change. |
4. IngestFromKinesisIT — testKinesisIngestion_RewindByOffset
| Field | Value |
|---|---|
| Test | org.opensearch.plugin.kinesis.IngestFromKinesisIT.testKinesisIngestion_RewindByOffset |
| Build | 79629 (Post Merge Action) |
| Seed | E8356F7D14169C51 |
| Reproduced locally | No |
| First failure | 2025-03-24 |
| Total unique builds affected | 128 |
| Pattern | Burst of 51 failures in Mar 2025, subsided, then another burst of 51 in Mar 2026. Sporadic in between. Bursty/episodic. |
5. RestoreShallowSnapshotV2IT — testHashedPrefixTranslogMetadataCombination
| Field | Value |
|---|---|
| Test | org.opensearch.remotestore.RestoreShallowSnapshotV2IT.testHashedPrefixTranslogMetadataCombination {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"false"}} |
| Build | 79652 (Post Merge Action) |
| Seed | C85B3FCE78DA4C96 |
| Reproduced locally | No |
| First failure | 2024-11-28 |
| Total unique builds affected | 77 |
| Pattern | Steady low-level flake (3-8 builds/month) since Nov 2024. No clear trend change. Stable/chronic. |
6. RareClusterStateIT — testDisassociateNodesWhileShardInit
| Field | Value |
|---|---|
| Test | org.opensearch.cluster.coordination.RareClusterStateIT.testDisassociateNodesWhileShardInit |
| Build | 79652 (Post Merge Action) |
| Seed | C85B3FCE78DA4C96 |
| Reproduced locally | No |
| First failure | 2024-11-04 |
| Total unique builds affected | 54 |
| Pattern | Low-frequency flake until Apr 2026 (12 builds) and May 2026 (18 builds). Worsening — correlates with mid-April runner change to m7a.8xlarge. |
Summary Table (sorted by builds affected)
| Test | Builds Affected | First Seen | Trend | Reproduced |
|---|---|---|---|---|
| ClientYamlTestSuiteIT (date_histogram profiler) | 234 | 2024-03-26 | Stable/chronic | No |
| MixedClusterClientYamlTestSuiteIT (cluster health) | 140 | 2024-03-25 | Worsening | N/A (env) |
| FullRollingRestartIT (SEGMENT replication) | 136 | 2024-10-11 | Worsening | No |
| IngestFromKinesisIT (RewindByOffset) | 128 | 2025-03-24 | Bursty/episodic | No |
| RestoreShallowSnapshotV2IT (hashed prefix) | 77 | 2024-11-28 | Stable/chronic | No |
| RareClusterStateIT (disassociate nodes) | 54 | 2024-11-04 | Worsening | No |
Notes
- None of the 5 runnable tests reproduced with the original seed locally, confirming these are non-deterministic timing-sensitive flakes.
- Three tests (FullRollingRestartIT, MixedClusterClientYamlTestSuiteIT, RareClusterStateIT) show a clear worsening pattern starting April 2026, correlating with the CI runner migration from m5.8xlarge to m7a.8xlarge.
- IngestFromKinesisIT shows a distinctive bursty pattern (51 failures concentrated in single months) suggesting it may be triggered by specific infrastructure or dependency changes.
- All tests have been failing for 6+ months, indicating these are well-known chronic flakes rather than new regressions.
Activity