Repository navigation
The store-job cadence alert now fires before the refresh-slot watch, so the wrong remedy arrives first #3179
Description
After #3178's grid re-derivation, the store-job cadence alert and the refresh-slot watch fire in the opposite order from the one #3044 was added to establish, and an operator now sees the wrong remedy first.
The ordering, verified at dev tip
| line | value | source |
|---|---|---|
| #2136 store-job cadence knob | 900 s | RefreshSlotPercentOfHourlyCadence = 25 → 25% of the 3,600 s cadence |
| #3044 refresh-slot watch | 1,050 s | RefreshSlotWarningSeconds => RefreshPhaseSlotSeconds * 5 / 6 |
| the wall | 1,260 s | RefreshPhaseSlotSeconds => HeaviestRefreshWindowMinutes * 60 (21 min) |
So 900 < 1050 < 1260, and the recorded ceiling (HeaviestHourlyRefreshObservedCeilingSeconds = 896) sits just under the cadence knob.
Why it matters
#2136's remedy text tells the operator to extend schedule_interval. For a continuous-aggregate refresh policy that value is also the end_offset — and on the hourly tier it additionally shortens the rollup horizon — so following it changes what the aggregate materializes. #3066 corrected exactly that advice for this job family.
#3044 exists to give a correct-remedy signal for this job. With the knob firing 150 s earlier, the operator now gets the wrong-remedy alert first, every time, and the correct one second. That inverts the reason the watch was added.
This is a regression introduced by #3178, not a pre-existing defect it exposed
The watch line moved 750 → 1,050 as a consequence of the new 1,260 s wall. #2136's knob stayed at 900 because V57 seeded store_job_cadence_warn_percent integer NOT NULL DEFAULT 25 and DarlingSelfAlertTests pins the C# constant equal to that rung's text. The ordering was correct before the re-derivation and is incorrect after it.
Why it cannot be fixed as a constant edit
Restoring the ordering means moving the knob below the watch, and the knob's seed lives in a shipped migration rung — changing it needs a new rung, not a constant change. Nor can the watch simply move down to clear it: it must stay above the 896 s recorded ceiling, and 896 < window × 50 < 900 has no integer solution, so no percentage lands the watch in the gap.
Options, not prescribed
- A new rung moving the cadence knob's default above the watch line (and deciding what happens to stores whose operators have set it by hand).
- Suppressing or re-wording Query Store aggregate tax scales serially with fleet size — measure, alert, and evaluate before large onboardings #2136's remedy for
policy_refresh_continuous_aggregatejobs specifically, so the first alert an operator sees carries advice that is correct for the job family — Fixes #3060 #3066's precedent. - Accepting the ordering and documenting it, if the cadence alert's earlier arrival is judged more useful than its remedy is harmful.
Not blocking any merge. Worth deciding before the release, since it changes what an operator is told first during exactly the midnight window #3112 documents.
Activity
erikdarlingdata commentedon Sep 8, 2026
Claude posting for Erik Darling
Verified every figure here independently at
devtip rather than taking them —RefreshSlotPercentOfHourlyCadence = 25(TimescaleSupport.cs:1788),HeaviestHourlyRefreshObservedCeilingSeconds = 896(:2131),RefreshPhaseSlotSeconds => HeaviestRefreshWindowMinutes * 60(:2144),RefreshSlotWarningSeconds => RefreshPhaseSlotSeconds * 5 / 6(:2196). The ordering holds at 900 < 1050 < 1260, and there is no integer window for which896 < window * 50 < 900, so the gap is genuinely unreachable by any percentage.One detail that makes the first option more involved than described, found while checking the V57 claim.
The rung's text is pinned twice, and only one of the two follows the constant.
DarlingSelfAlertTests.cs:2638builds it from the constant —$"store_job_cadence_warn_percent integer NOT NULL DEFAULT {TimescaleSupport.RefreshSlotPercentOfHourlyCadence}"— so it tracks a change automatically.StoreSelfMetricsTests.cs:117asserts the same rung with a hardcoded literal —"ADD COLUMN IF NOT EXISTS store_job_cadence_warn_percent integer NOT NULL DEFAULT 25".So a new rung moving the knob reddens the hardcoded pin while the derived one silently follows. That is not an obstacle — it is one more file — but it is worth knowing before anyone estimates the change as "a rung plus a constant", and the hardcoded copy is the one that will drift the next time this value moves for any other reason.
It is also the same shape as several findings from this batch: two assertions over one fact, where only one is derived.
HeaviestHourlyRefreshObservedCeilingSeconds' own doc argues for exactly this — a value a reader cannot re-derive is worse than one they can — so if that rung is touched, deriving the second pin from the constant is the cheap half of the work.No change requested to the issue otherwise. The regression framing is right and load-bearing: the watch moved 750 to 1,050 as a consequence of the new wall while V57 held the knob at 900, so this is something #3178 introduced rather than a pre-existing defect it revealed.
erikdarlingdata commentedon Sep 8, 2026
Claude posting for Erik Darling
Correcting this issue's central claim before anyone builds on it: the wrong remedy does not arrive first, and there is no second alert. The ordering figures are right; the consequence drawn from them is not. I verified the numbers on this issue and did not verify the harm, which is the same omission that produced the claim.
1 — #2136's remedy is already correct for this job family, and has been since #3066.
DarlingSelfAlertEvaluator.cs:1709branches onTimescaleSupport.ScheduleIntervalDoublesAsEndOffset(job.JobName)and gives refresh policies this text:The generic "extend the job's schedule_interval deliberately" is the
elsebranch, for jobs where widening is safe.ScheduleIntervalDoublesAsEndOffsetmatchesRefreshPolicyProcNameor that name followed by a space, and the job in question ispolicy_refresh_continuous_aggregate query_store_stats_interval_hourly— so it matches, and the correct branch fires.DarlingSelfAlertTests.cs:2695pins it:Assert.DoesNotContain("extend the job's schedule_interval", refresh.DetailText).So #3066 was not a precedent for a fix still to be made. It already made this one, and the first thing an operator sees is job-specific advice that points at re-phasing the grid — the actual remedy.
2 — The refresh-slot watch is a log line, not an alert, and deliberately so.
LogHeaviestRefreshSlotHeadroomwrites toILoggerfromDarlingWorker.cs:4375. Its own doc states the choice: "At or past the slot it is an Error … the highest level a log line has, and still not an alert, because the action it calls for is re-deriving #3035's grid rather than anything an operator does tonight."So there is no pair of alerts to order. There is one alert at 900 s carrying correct advice, and a log line at 1,050 s.
What is actually true, and where it belongs
The ordering inversion is real — 900 < 1,050 < 1,260, and no integer window satisfies
896 < window * 50 < 900. But its consequence is smaller and different: the slot-specific signal is a log line nobody reads, which is the already-open #3044 surfacing question, not a new remedy problem. A proposal for that is filed in the queue atinbox/3044-20260908T032511Z-refresh-slot-surfacing-proposal.md, and it establishes that #2136 is the only thing that reaches an operator here — with a 150-second blind spot between the two lines.One thing this issue did surface that is genuinely stale.
TimescaleSupport.cs:4222still reads "records 864 s, which is 24.0% of the same 3,600 s cadence".devhas 896 s, which is 24.9%. That paragraph is #3178's collateral and is worth fixing on its own.Suggested disposition: narrow this to the stale-figure fix and fold the ordering observation into the #3044 surfacing decision, or close it into that. I would rather do either than leave a correct-numbers/wrong-consequence issue open for someone to implement.
erikdarlingdata commentedon Sep 8, 2026
Claude posting for Erik Darling
Closed per Erik: folded into the #3044 refresh-slot surfacing proposal, which is where the surviving substance belongs.
Why closed rather than implemented. The threshold figures here are correct — 900 < 1,050 < 1,260 with the recorded ceiling at 896, verified independently at
devtip, and no integer window satisfies896 < window * 50 < 900. The consequence drawn from them is not:DarlingSelfAlertEvaluator.cs:1709branches onScheduleIntervalDoublesAsEndOffset(job.JobName)and tells refresh policies "Do NOT widen this job's schedule_interval … Narrow the refresh window or re-phase the grid instead." The predicate matchespolicy_refresh_continuous_aggregateplus a trailing space, so it fires forquery_store_stats_interval_hourly, andDarlingSelfAlertTests.cs:2695pins it. Fixes #3060 #3066 did not set a precedent for that fix — it made it.LogHeaviestRefreshSlotHeadroomwrites toILogger, deliberately: "still not an alert, because the action it calls for is re-deriving Compression policies still drift finish-to-start; pin them to the phase grid to finish #3012's deconfliction #3035's grid rather than anything an operator does tonight."So there was never a pair of alerts to be out of order, and no wrong remedy arriving first.
What survived, and where it went. The real finding is that the slot-specific reading has no reader at all — which is exactly #3044's expired premise, already under a filed proposal. #3178 changed the shape of that gap rather than creating a new problem: the blind band moved from 750→900 to 900→1,050, and the recorded ceiling now sits 4 s below the knob instead of 146 s above the watch. That is amended into the proposal at
inbox/3044-20260908T032511Z-refresh-slot-surfacing-proposal.md, along with a warning not to bake in 750, 1,050 or 896 — the watch derives from the slot and moves with any future geometry, while the knob is a seeded migration default that does not.Two things carried forward rather than lost:
TimescaleSupport.cs:4222is stale collateral from Re-derive the refresh grid so contention cannot depend on list position, and re-derive the compression grid with it #3178 — "records 864 s, which is 24.0%" againstdev's 896 s and 24.9%. Handed to the 27 of 43 servers fell 7 minutes behind for one hour: query_store's target reads slowed 9.8x fleet-wide while plan_correction, same shape and same run drop, stayed flat #3112 lane, which is already in that file.RefreshSlotPercentOfHourlyCadenceand once as a hardcoded25inStoreSelfMetricsTests.cs:117. Whoever eventually moves that knob touches both, and the hardcoded copy is the one that drifts.The miss worth recording. Two of us verified this issue's numbers and neither verified its harm. The figures were checked at
devtip and the claim they supported was never tested against the code that would have refuted it in one grep. Right measurements, wrong meaning — the same shape as several findings this week, and the reason a correct table is not a correct issue.