Repository navigation
Store: refresh a plan or text dimension's last_seen at most every 6 hours, not every hour - #4502
Merged
Merged
Conversation
collect.query_plan_dim and collect.query_text_dim refresh their last_seen watermark under a guard that skips the UPDATE when a digest was already touched recently. last_seen is indexed (the prune's ORDER BY last_seen LIMIT needs it), so every touch that survives the guard is a non-HOT update to a large, scattered heap. Measured on one production store at the original 1-hour width, query_plan_dim took 894,656 such touches in 71 hours, about 4.87 million full-page images, roughly 42 KB of WAL per row. The guard already caps a continuously-seen row to 24 UPDATEs a day; widening it to 6 hours caps the same row to 4, a further 6x cut on top of the write savings the guard already provides. The width is now one named constant, PayloadDimensions.LastSeenRefreshGuardHours, rendered into all four guard sites in UpsertSql (the pre-filter and the conflict arm, for both the compressed and text branches) so it cannot drift between sites or between the two dimension tables. The dimension prune's retention margin already reserves a full day past the fact-retention horizon for a trailing last_seen stamp, to cover exactly this kind of guard plus the Query Store liveness touch's own guard. Both guards stay well inside that one-day margin after this change, so no widening of the margin itself is needed - a new test pins that relation directly against both guards and against the retention cutoff arithmetic, rather than leaving it to the doc comments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #4503.
Why
On one production store,
collect.query_plan_dim'slast_seentouch — the watermark a hot dimension row's re-sighting refreshes on every collection cycle it is referenced in — took 894,656 non-HOT updates in 71 hours under the original 1-hour guard, about 4.87 million full-page images, roughly 42 KB of WAL per row.last_seenis indexed (the prune's range scan needs it), so this update can never go HOT regardless of fillfactor, and dropping or reshaping the index isn't an option because the prune'sORDER BY last_seen LIMITdepends on it. The guard already caps a continuously-seen row to 24 touches a day; widening it to 6 hours caps the same row to 4 — a further 6x cut in how often this cost can recur per row.What changes
PayloadDimensions.LastSeenRefreshGuardHoursis a new named constant (6), andPayloadDimensions.LastSeenRefreshGuardIntervalrenders it into SQL once.UpsertSql's fourINTERVAL '1 hour'literals — theWHERE NOT EXISTSpre-filter and theON CONFLICT ... WHEREguard, for both the compressed-content branch and the shared text/plan-xml branch — now interpolate that one constant, so both dimension tables (query_plan_dim,query_text_dim) and both content shapes take the same width from the same place.DarlingRetention.ComputeDimensionCutoff/ComputeMapCutoff's margin discussion now name the constant and the measured WAL evidence in place of the old "one hour" wording.ChunkIntervalDays + 1= 2 days) already covers this guard and the separate Query Store liveness-touch guard (currently 12 hours) with room to spare; it did not need widening. A new test class,PayloadDimensionGuardMarginTests, pins that both guards fit inside the margin directly — against the constants and againstComputeDimensionCutoff's actual cutoff arithmetic at several plan-content-retention widths — rather than leaving the relationship to comments that could drift.PayloadDimensionLiveTests) were updated to the new 6-hour boundary: a re-touch at +3h now asserts no refresh, and +7h asserts a refresh, replacing the old +30min/+2h cases.Test plan
PayloadDimensionTests— the SQL string pins forUpsertSql, updated toINTERVAL '6 hours'.PayloadDimensionLiveTests— the churn-guard live test and theWHERE NOT EXISTSpre-filter live test, both re-timed to the new boundary; both GREEN on this branch.PayloadDimensionGuardMarginTests(new) — pure pins that the dim upsert guard and the Query Store liveness touch guard both fit inside the dimension prune's trailing margin, and that a row stamped at the wider guard's age survives the retention cutoff at several plan-content-retention widths.PlanContentRetentionTests,DarlingRetentionTests,DocCommentHygieneTests— unaffected pins re-run to confirm no collateral break.devat runtime: the new live test run againstdev's 1-hour guard fails the +3 h case;last_seenis refreshed there.LastSeenRefreshGuardHoursset back to 1 on this branch → the same case RED; restored → GREEN.Darling.Testsbuilds with 0 warnings.CHANGELOG
SECTION: Changed
ENTRY:
last_seenat most every 6 hours instead of every hour, so there are up to 6× fewer of these non-HOT updates. Retention is unchanged: the dimension prune's 2-day margin still covers the guard.REF:
[Store: refresh a plan or text dimension's last_seen at most every 6 hours, not every hour #4502]: Store: refresh a plan or text dimension's last_seen at most every 6 hours, not every hour #4502