Skip to content

Harden the legacy-row partial-index plan test: fixed anchor, size guard, plan in the failure message - #5061

Merged
erikdarlingdata merged 1 commit into
devfrom
test/query-store-legacy-index-plan-time-of-day
Oct 4, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
test/query-store-legacy-index-plan-time-of-day

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

Summary

TheShippedCheck_PlansOnThePartialIndex_AndAnswersExactly failed once on a dev push's Build run on 2026-10-04 at about 00:08Z, at Assert.NotEmpty(indexReads): a scan of query_store_stats was found, but none read the partial index. It passed on re-run, and the assert printed no plan. It failed the same way on another PR's Build run about 15 minutes later, a PR that changes only web JavaScript. Both failures fell in the first half hour after 00:00 UTC, after a day of green runs.

The test seeded to the wall clock, so the one uncompressed chunk was "today since 00:00 UTC". Just after midnight that is one pass, about 60 rows on one heap page.

I could not reproduce the failure. With the old shape (only the time of day varied), anchors 00:01, 00:03, 00:08 (14 runs), 00:31, 00:38, 01:08, 01:38, 02:08, 03:00, 06:00, 12:00 and 23:55 all passed. At 00:08 the planner picked an Index Only Scan on the partial index for the one-pass chunk:

->  Index Only Scan using _hyper_*_chunk_idx_query_store_stats_server_time_null_start on _hyper_*_chunk s_4  (cost=0.12..1.73 rows=1)

With index and bitmap scans disabled, the competing Seq Scan on that chunk costs 0.00..2.05. So the near-empty chunk is a close call (1.73 vs 2.05), which is a plausible way for a small difference in statistics, visibility-map state or the platform to flip the plan on CI. That is an inference, not something I reproduced. The partial index is empty in this test (no row has a NULL start), so its cost stays about constant as the chunk grows, while the Seq Scan's cost grows with the chunk's pages. A full-day chunk therefore widens the gap instead of sitting on it. The earliest time of day at which the old shape passes is therefore not a meaningful number: it passed at every time I tried.

What changes

Tests only, in QueryStoreLegacyRowIndexLiveTests.cs:

  • The seed ends at a fixed anchor (23:50 UTC on a fixed past day) instead of DateTime.UtcNow. The newest, uncompressed chunk always holds a full day of passes (about 2,880 rows), whatever time the test runs. Nothing else in the test compared against the real "now".
  • A guard asserts that the newest chunk holds at least 1,000 rows, with a message, so a later seed change cannot bring the tiny-chunk shape back.
  • Assert.NotEmpty(scans) and Assert.NotEmpty(indexReads) became Assert.True(..., message + "\n" + plan), so the next failure shows the plan.
  • No enable_seqscan = off, no forced plans, no VACUUM. The test still asserts that the shipped check plans on the partial index and never heap-filters the NULL test. I did not try VACUUM, since there was no failure to stabilise.

Test plan

  • Old shape, anchors listed above: pass, no RED. (This is the limit: I have no RED.)
  • With the fix: the class passes 3 runs in a row, and with temporary anchor inputs of 00:05, 06:00, 12:00 and 23:55 (reverted before commit).
  • TuningHourlyCreateLiveTests: 5 of 5 pass. LivePostgresCollectionHygieneTests cannot load WPF assemblies on this machine.
  • Full suite without DARLING_TEST_PG: 22046 total, 476 failed, and every failing test name is already on the known local-failure list for dev (the comm -23 difference is empty).
  • --no-incremental build: 0 warnings.

Limits

  • The root cause is not proven; see the Summary. If the test fails again, the message now includes the plan, which will settle it.
  • The size guard runs only when TimescaleDB is present, as the chunk split does.

CHANGELOG

SECTION: None
ENTRY: None: tests only

Seed the three days to 23:50 UTC on a fixed past day instead of the wall clock,
so the uncompressed newest chunk always holds a full day of rows. Guard its
size, and print the plan when the plan asserts fail.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 4, 2026 00:45
@erikdarlingdata
erikdarlingdata merged commit 9098950 into dev Oct 4, 2026
19 of 20 checks passed
@erikdarlingdata
erikdarlingdata deleted the test/query-store-legacy-index-plan-time-of-day branch October 4, 2026 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant