Skip to content

The CPU alert gate decides "same sample" by equality, not by order: a fall-back's repeated hour and the V134 local→UTC frame switch are new samples rather than an hour of silence, and both SKUs hand it sample_time_utc where the row has one (#3744) - #3774

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/3744-cpu-gate-utc-identity
Sep 20, 2026

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

The lie

The CPU alert's persistence gate identifies each ring-buffer sample by its instant and counts it as fresh only when that instant is strictly greater than the last one it counted — AlertEngine.ObservePersistenceAsync, sampleUtc > priorRecord.LastObservedSampleUtc (#3282's rule against a 30 s sweep re-reading a 60 s sample). The instant it was fed is cpu_utilization_stats.sample_time, the monitored server's local wall clock, under a field named CpuSampleTimeUtc. On a zone that observes DST the local clock repeats an hour every autumn, so from the fall-back instant every new sample carries a stamp at or below the last one recorded, > says "not fresh", and for roughly an hour the gate neither advances a streak nor clears an open incident — on every non-UTC server, once a year, at the one instant nobody is watching.

#3730 (V134) put the same instant in UTC beside the local stamp and left this one reader on the local column deliberately, with the reason written into DarlingWorker.LatestCpuSql's doc and pinned by TimeHonestyRungTests: switching the identity to the UTC twin flips the frame exactly once, at the upgrade, and under > every server EAST of UTC would read its first UTC instant as older than its last local one and freeze for one offset's worth of hours — the same defect on upgrade day instead of on fall-back day. Both are one thing: an identity compared by order across two clock frames.

The fix shape — equality, not a frame tag (and why)

The brief's design carried a frame tag on AlertPersistenceRecord and treated a frame change as fresh-by-definition, keeping > within a frame. config.alert_persistence_state cannot grow a column this wave (no rung), so that meant the frame in the row KEY — a second metric name (High CPU plus a UTC-suffixed sibling), a two-key seed, an orphaned legacy row per SQL Server target in both SKUs' tables forever (the seam has no delete), and CpuPersistenceMetric's stated "one spelling" property made false. Evaluated against the code, > turned out to be buying nothing that justified any of it.

Every caller hands the gate the identity of the newest stored row — an ORDER BY … DESC LIMIT 1 or a MAX(collection_time) over a table its collector only appends to. Between two sweeps that identity can stay equal (same row — not fresh) or change to a different row's; the only way it can change to an earlier value is a clock running backwards behind the stamp, and both real cases of that — the autumn fall-back and the V134 frame switch — are new samples that MUST count. So the gate now asks the question it always meant: is this the same sample? — sampleUtc != LastObservedSampleUtc.

With order gone, the identity can be the honest clock. Darling's LatestCpuSql projects sample_time_utc beside sample_time (trailing, so the pre-existing ordinals read what they always read) and ReadLatestCpuAsync hands the gate twin ?? local; Lite's overview read does the same, ServerSummaryItem carries both under names that say which clock each is, and MainWindow.AlertEngine applies the identical ??. Both ORDER BYs stay on the local stamp on purpose: every row has it, and a cross-batch order on the twin, or on a COALESCE, would compare a pre-rung local stamp against a post-rung UTC one and, east of UTC, sort a stale pre-rung row as newest for one offset's worth of hours after the upgrade. CpuSampleTimeUtc and LastObservedSampleUtc keep their names and are documented: true wherever the store can say the instant (every row since V134 / v63), documented-local where it cannot; there is no frame field beside them because nothing would read it, and a field nothing reads is a field that drifts.

Upgrade day, plainly

A store already at V134 has been writing the twin on every row since its first post-rung poll, so the first sweep of this build hands a UTC identity to a record holding a local one: one sample counted once more (or, east of UTC, a new sample counted exactly once), and from there != within the UTC frame. No freeze in either direction, no pause, no operator step. A server whose newest CPU row predates the rung stays on the local identity until its collector writes a post-rung row, then switches the same way. The stale local instant left in last_observed_sample_at is overwritten by the first fresh sample, as it always was.

What it does NOT do

  • No rung, no schema change, no key change: the persistence table is untouched.
  • It does not move either ORDER BY to the twin (see above), so the one-batch fall-back quirk TimeHonestyRungTests shows — inside the single poll that straddles the transition the EDT-side sample sorts as newest by local stamp, a reading stale by one sample, once a year — remains; with the gate on equality it costs nothing downstream.
  • It does not rename CpuSampleTimeUtc or LastObservedSampleUtc (the issue's item 3 accepted "true where V134 has written the column and documented-local before that"); renaming the record's field would touch both stores and the PostgreSQL CPU gate in DarlingWorker for no behaviour.
  • It does not change the PostgreSQL-target CPU gate in DarlingWorker (pg_cpu_utilization.sample_time is UTC from Performance Insights; it never had a frame problem).
  • It does not make CPU alerting see the fall-back hour end to end, and the reason is a second, separate mechanism found while verifying this one: CpuUtilizationCollector watermarks on sample_time (MAX(sample_time) in both hosts' watermark reads) and its ring-buffer ReadAsync drops every sample <= Watermark client-side — so after a fall-back the collector itself discards the repeated hour before any row reaches the store, and the gate is blind for that hour because there is nothing to see. That is a collector defect with its own upgrade-day story (a watermark that changes frame re-ingests or skips one offset's worth of samples, as cpu_utilization stores each sample's UTC instant beside its server-local stamp and server_properties stores the engine's time-zone id beside its offset (Darling V134 / Lite v63), so UTC-window readers stop deriving an offset that is an hour wrong across DST (#3653 item 13, Q7 + Q8) #3730 noted when it left the watermark alone), in a file shared by both SKUs with its own pins, and it is reported for its own issue rather than folded in here. This PR is the necessary half: once the collector ingests that hour, a gate on > would have gone blind on exactly those rows.

Tests

Executed here (mactest harness inside the worktree, Darling.Tests assembly name, xunit v3 in-process runner):

  • AlertEngineTests — 179 total, all green, including the three new fixtures: Cpu_AFallBack_RunsTheStampBackwards_AndTheStreakStillAdvances (01:58 → 01:59 → 01:00 fires at the bar; an open incident clears in the repeated hour; a repeated instant still holds and writes nothing), Cpu_TheUpgradeFrameSwitch_EastOfUtc_IsFreshExactlyOnce_AndNeverFreezes (local record meets a UTC instant three hours earlier across a restart: counted once, re-read held and unwritten, then != within the frame), Cpu_TheUpgradeFrameSwitch_WestOfUtc_CountsTheSameSampleOnceMoreAtMost. Red-first verified: with the predicate flipped back to > the fall-back and east fixtures fail and the west bound pin passes, as designed; the No alert requires its condition to PERSIST: real fire/resolve pairs 55-87 seconds apart on both engines — and it is not the re-fire class closed seven times #3282 Cpu_RepeatedSampleInstant_DoesNotAdvanceTheStreak pin is unchanged and green.
  • TimeHonestyRungTests — the "keeps the local stamp and says why" pin flipped to TheLatestRowReads_KeepTheLocalTiebreak_TheGatesReadProjectsTheTwin_AndTheWorkerSaysWhy (three display reads never name the twin; the gate's read projects it exactly once, in the SELECT list, orders on the local stamp, no COALESCE; the worker's doc carries the new phrases and not the retired one; the reader folds twin ?? local, pinned as source).
  • TimeHonestyRungLivePostgresTests — executed against a TimescaleDB 2.28.1 / PG 18 container (mine, torn down): LatestCpuSql returns four columns, ordinal 2 the local stamp and ordinal 3 the twin equal to the planted UTC instant; after removing the post-rung batch the pre-rung row reads a NULL twin (the fallback arm, fixture 5).
  • LatestCpuReadShapeSqlTests (13) and LatestCpuReadShapeLivePostgresTests (3, against the rig) — ordering, no store-clock predicate, SELECT prefix all hold with the fourth column.
  • Census pack: RepoFileAdoptionTests, StragglerCommandTimeoutTests, LiveCleanupConversionRatchetTests, MeasurementContractCensusTests, McpPageContractTests, FleetIdentifierScrubTests, BuiltinAlertPersistenceRungTests, ReadmeDerivedCountPinTests — all green. DocCommentHygieneTests 77, one red that is the documented harness artifact (TheDerivedProjectListCoversEveryProjectInTheTree sees the uncommitted harness csproj).
  • Every touched project builds -c Release -p:EnableWindowsTargeting=true at 0 warnings, including Darling.Tests and Lite.Tests.

CI-first (the Lite test assembly cannot be discovered off Windows — WindowsBase fatal during discovery even when filtered): LiteAlertForwardingTests twins Cpu_AFallBack_RunsTheStampBackwards_AndTheStreakStillAdvances_OnLiteToo and Cpu_TheUpgradeFrameSwitch_IsFreshExactlyOnce_OnLiteToo (same shared engine, through Lite's settings, snapshot and store); TimeHonestyRungTests.TheOverviewRead_ProjectsTheTwinBesideTheLocalStamp_AndTheSnapshotFoldsThem (source pins on the overview SQL, the trailing ordinal, the ServerSummaryItem pair, the MainWindow.AlertEngine fold, and the engine's !=; every asserted string was verified against the files by hand); TimeHonestyRungReadTests.TheOverviewRead_CarriesBothStampsOfTheNewestCpuRow (real LocalDataService over DuckDB: a pre-rung newest row surfaces local + null twin, a post-rung one surfaces both).

Descends from #3730 (Q7 of #3653). One line of Darling/README.md's V134 row corrected (it said the gate's read keeps the local column).

Closes #3744

…o a fall-back's repeated hour and the V134 local→UTC frame switch are new samples rather than an hour of silence; both SKUs' latest-CPU reads hand it sample_time_utc where the row has one (#3744)

The persistence gate's freshness test was `sampleUtc > LastObservedSampleUtc`, fed the target's LOCAL sample_time. Every autumn the local clock repeats an hour, so every sample in that hour read as stale and the gate neither advanced a streak nor cleared an incident. The same `>` was why #3730 (V134) left the gate on the local stamp: switching to the UTC twin flips the frame once, and east of UTC the first UTC instant reads older than the last local one — a frozen gate for one offset's worth of hours on upgrade day.

Both are one defect: an identity compared by order across two clock frames. The gate now compares for EQUALITY (a different instant is a different sample whichever clock stamped it), which fixes the fall-back in either frame and makes the frame switch cost at most one extra count of one sample per server, once. No rung: config.alert_persistence_state needs no frame column when the frame is not load-bearing.

Darling's LatestCpuSql and Lite's overview read project sample_time_utc beside sample_time; both hosts hand the gate `twin ?? local`. The ORDER BYs stay on the local stamp on purpose (every row has it; an order on the twin would compare frames across batches). Docs on the snapshot, the record and the worker say which clock the value is in and why there is no frame field.

Fixtures: fall-back (01:59 → 01:00 advances the streak and clears an incident), east-of-UTC upgrade (fresh exactly once, re-read held, `!=` within the frame), west-of-UTC upgrade (counted once more at most), the #3282 repeated-instant pin unchanged; TimeHonestyRungTests' "keeps the local stamp" pin flips to "projects the twin, orders on the local"; the live PG class asserts ordinal 3 and the pre-rung NULL fallback; Lite twins for the engine fixtures, a source pin on the overview read and snapshot fold, and a DuckDB pin through GetServerSummaryAsync.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant