Skip to content

Store: refresh a plan or text dimension's last_seen at most every 6 hours, not every hour - #4502

Merged
erikdarlingdata merged 2 commits into
devfrom
perf/dim-last-seen-guard-6h
Sep 27, 2026
Merged

erikdarlingdata merged 2 commits into
devfrom
perf/dim-last-seen-guard-6h

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Refs #4503.

Why

On one production store, collect.query_plan_dim's last_seen touch — the watermark a hot dimension row's re-sighting refreshes on every collection cycle it is referenced in — took 894,656 non-HOT updates in 71 hours under the original 1-hour guard, about 4.87 million full-page images, roughly 42 KB of WAL per row. last_seen is indexed (the prune's range scan needs it), so this update can never go HOT regardless of fillfactor, and dropping or reshaping the index isn't an option because the prune's ORDER BY last_seen LIMIT depends on it. The guard already caps a continuously-seen row to 24 touches a day; widening it to 6 hours caps the same row to 4 — a further 6x cut in how often this cost can recur per row.

What changes

  • PayloadDimensions.LastSeenRefreshGuardHours is a new named constant (6), and PayloadDimensions.LastSeenRefreshGuardInterval renders it into SQL once. UpsertSql's four INTERVAL '1 hour' literals — the WHERE NOT EXISTS pre-filter and the ON CONFLICT ... WHERE guard, for both the compressed-content branch and the shared text/plan-xml branch — now interpolate that one constant, so both dimension tables (query_plan_dim, query_text_dim) and both content shapes take the same width from the same place.
  • Doc comments on the guard and on DarlingRetention.ComputeDimensionCutoff/ComputeMapCutoff's margin discussion now name the constant and the measured WAL evidence in place of the old "one hour" wording.
  • The dimension prune's retention margin (ChunkIntervalDays + 1 = 2 days) already covers this guard and the separate Query Store liveness-touch guard (currently 12 hours) with room to spare; it did not need widening. A new test class, PayloadDimensionGuardMarginTests, pins that both guards fit inside the margin directly — against the constants and against ComputeDimensionCutoff's actual cutoff arithmetic at several plan-content-retention widths — rather than leaving the relationship to comments that could drift.
  • Existing live tests that exercised the old 1-hour boundary (PayloadDimensionLiveTests) were updated to the new 6-hour boundary: a re-touch at +3h now asserts no refresh, and +7h asserts a refresh, replacing the old +30min/+2h cases.

Test plan

  • PayloadDimensionTests — the SQL string pins for UpsertSql, updated to INTERVAL '6 hours'.
  • PayloadDimensionLiveTests — the churn-guard live test and the WHERE NOT EXISTS pre-filter live test, both re-timed to the new boundary; both GREEN on this branch.
  • PayloadDimensionGuardMarginTests (new) — pure pins that the dim upsert guard and the Query Store liveness touch guard both fit inside the dimension prune's trailing margin, and that a row stamped at the wider guard's age survives the retention cutoff at several plan-content-retention widths.
  • PlanContentRetentionTests, DarlingRetentionTests, DocCommentHygieneTests — unaffected pins re-run to confirm no collateral break.
  • RED on dev at runtime: the new live test run against dev's 1-hour guard fails the +3 h case; last_seen is refreshed there.
  • Mutation: LastSeenRefreshGuardHours set back to 1 on this branch → the same case RED; restored → GREEN.
  • Darling.Tests builds with 0 warnings.

CHANGELOG

SECTION: Changed
ENTRY:

collect.query_plan_dim and collect.query_text_dim refresh their
last_seen watermark under a guard that skips the UPDATE when a
digest was already touched recently. last_seen is indexed (the
prune's ORDER BY last_seen LIMIT needs it), so every touch that
survives the guard is a non-HOT update to a large, scattered heap.
Measured on one production store at the original 1-hour width,
query_plan_dim took 894,656 such touches in 71 hours, about 4.87
million full-page images, roughly 42 KB of WAL per row.

The guard already caps a continuously-seen row to 24 UPDATEs a day;
widening it to 6 hours caps the same row to 4, a further 6x cut on
top of the write savings the guard already provides. The width is
now one named constant, PayloadDimensions.LastSeenRefreshGuardHours,
rendered into all four guard sites in UpsertSql (the pre-filter and
the conflict arm, for both the compressed and text branches) so it
cannot drift between sites or between the two dimension tables.

The dimension prune's retention margin already reserves a full day
past the fact-retention horizon for a trailing last_seen stamp, to
cover exactly this kind of guard plus the Query Store liveness
touch's own guard. Both guards stay well inside that one-day margin
after this change, so no widening of the margin itself is needed -
a new test pins that relation directly against both guards and
against the retention cutoff arithmetic, rather than leaving it to
the doc comments.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 27, 2026 22:17
@erikdarlingdata
erikdarlingdata merged commit d247029 into dev Sep 27, 2026
17 of 18 checks passed
@erikdarlingdata
erikdarlingdata deleted the perf/dim-last-seen-guard-6h branch September 27, 2026 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant