[erikdarlingdata]

The lie

The CPU collector's watermark was the monitored server's LOCAL sample_time, and its ring-buffer dedup dropped every sample at or below it. From the autumn fall-back instant the local clock repeats an hour, so every sample for the next hour carried a stamp at or below the newest one stored before the transition and was discarded as already collected — no cpu_utilization_stats row for roughly an hour on every non-UTC server, once a year. #3744 / #3774 fixed the alert GATE's half (identity by equality); this is the collector's half, and it was invisible because #3730's live straddle test planted its batch through WritePayload, below the dedup.

Mechanism

CpuUtilizationCollector.ReadAsync cannot filter its computed stamp in SQL, so it dedups client-side against context.Watermark, which both hosts read as MAX(<WatermarkColumn>) — MAX(sample_time), a local wall-clock instant. Verified at source on dev 050b50e1: CpuUtilizationCollector.cs WatermarkColumn => "sample_time" and sampleTime <= context.Watermark.Value; DarlingCollectorRunner.GetLastCollectedTimeAsync / BuildServerWatermarkSql; Lite RemoteCollectorService.GetLastCollectedTimeAsync. Column 3 of every arm is sample_time_utc (V134 / Lite v63), the same instant in UTC, NULL only on rows written before the rung.

The fix — the watermark's frame is carried, not guessed

A new definition member, ICollectorDefinition.UtcWatermarkColumn (base default null; only cpu_utilization declares it, as sample_time_utc), and a new context member, CollectorContext.WatermarkFromUtcColumn (init-only bool, false unless the host sets it).

• Both hosts branch on the declaration. A definition WITHOUT a UTC twin goes through the unchanged GetLastCollectedTimeAsync and its byte-identical SQL. A definition WITH one goes through a new GetLastCollectedTimeWithFrameAsync (Darling, with the same collectedSince bound and explicit command deadline as its sibling; Lite, DuckDB), which runs ONE statement — SELECT MAX(sample_time_utc), MAX(sample_time) FROM cpu_utilization_stats WHERE server_id = $1 — and returns the twin's value with FromUtcColumn = true when any row carries it, else the local maximum with false, else (null, false): the first-run fallback exactly as before.
• The collector's dedup compares the row IN THE WATERMARK'S FRAME: SampleTimeUtc against a UTC watermark, SampleTime against a local one. UTC never repeats an hour, so under a UTC watermark the repeated local hour lands sample by sample.

Why this shape and not the alternatives

The issue named three shapes. MAX(sample_time_utc) is NULL on a store with no post-rung rows, and a NULL watermark is the first-run fallback — re-pointing the watermark outright would re-collect the whole TOP (60) under stamps the store already holds, a duplicate hour per server, once. A COALESCE inside one MAX, or converting the local watermark by the batch's own sample_time - sample_time_utc offset, each compare across frames on the upgrade run; the best bound either can offer is one offset's worth of samples. The stated frame's bound is zero: the first post-upgrade run finds no twin, gets the local watermark and compares local-to-local — the pre-#3778 rule exactly, nothing duplicated, nothing dropped beyond what that rule always dropped — stores rows carrying the twin, and every run after it is UTC-to-UTC forever. The twin wins whenever it exists without comparing the two maxima, because comparing them would itself be the cross-frame comparison this exists to avoid (and in the fall-back shape the local maximum belongs to an OLDER sample — the live test plants exactly that).

The mechanism is generic (a second watermark column with its frame) rather than a CPU-only special case, and its boundary is stated: honoured on the server-scoped read only; the per-database and per-item refreshes read the declared column alone, so a definition must not declare a UTC twin together with a PerDatabaseWatermarkColumn — pinned.

What nobody promised: a fall-back that straddles the FIRST post-upgrade run itself is dropped exactly as it would have been the day before, because that run is still on the local rule. One run, one server class, one hour, and the run after it is on UTC. The class remarks carry the upgrade-day story beside the #3730 essay rather than contradicting it.

What it does NOT do

• Not the gate (The CPU alert gate's observation identity is the target's LOCAL sample_time, so the gate is blind for ~1 h after every autumn fall-back on a non-UTC server; it should key on sample_time_utc with a one-time frame re-anchor (Q7 residue) #3744 / The CPU alert gate decides "same sample" by equality, not by order: a fall-back's repeated hour and the V134 local→UTC frame switch are new samples rather than an hour of silence, and both SKUs hand it sample_time_utc where the row has one (#3744) #3774). Not the de-skew readers (V134). Not sample_time's frame: it stays the server's local clock, the display frame Lite's chart plots, and both clock pins in CollectorTimestampFrameTests hold unchanged.
• Not Azure SQL DB. The Azure arm binds the watermark server-side as @last_sample_time against drs.end_time, which is UTC and is the value of BOTH stored columns there; the query text and the bound parameter are pinned identical in either frame, and the Azure arm never dedups client-side in either.
• No schema rung. No change to any other collector's watermark read: cpu_utilization is the only declaring definition (census-pinned) and the non-declaring branch calls the untouched sibling with its exact-string-pinned SQL.
• Not the per-database / enumerated watermark refreshes (boundary above).

Tests

Executed on this Mac in Lite.Tests- and Darling.Tests-named net10.0 harnesses (Collectors is net10.0; the DuckDB reads run against a real DuckDB; the PostgreSQL round trip ran against a throwaway TimescaleDB 2.28.1 / PG 18 store migrated to V135). Each behavioural pin was shown red under a deliberate mutation (the old local-only dedup; the pair read preferring the local column; the DuckDB SQL with its columns swapped) and green on the fix.

• Lite.Tests/CpuUtilizationCollectorDefinitionTests — the three acceptance sequences through the REAL ReadAsync: (1) America/New_York 2026-11-01, 62-row ring buffer newest-first, all 60 repeated-hour samples land under the UTC watermark and the two already-stored ones are dropped; the CONTROL under the local watermark lands zero (the mechanism is the frame, not the value). (2) The upgrade day: eleven pre-rung rows, run 1 on the local rule lands 12:11–12:12 with zero re-landed, run 2 finds the twins and is on UTC, fifteen distinct samples each exactly once. (3) Azure: identical query text and parameter in either frame, no client dedup in either. Plus the local-frame and UTC-frame dedup pins side by side, the kept-not-guessed never-path (NULL twin under a UTC watermark), and the census: only cpu_utilization declares UtcWatermarkColumn, none declares it beside a PerDatabaseWatermarkColumn.
• Lite.Tests/TimeHonestyRungTests — the runner-branch source pin and both DuckDB SQL strings; WatermarkFrameReadTests on a real DuckDB through the real protected reads: empty store, pre-rung only (local, LOCAL), one post-rung row flipping the frame even when a pre-rung local stamp is LATER (the in-hour upgrade shape), scoped per server, and the plain read unchanged beside it.
• Darling.Tests/TimeHonestyRungTests — BuildServerWatermarkSql pinned as the exact strings it was (bounded and unbounded) and BuildServerWatermarkPairSql as the exact new strings; the runner-branch source pin; both readers' signatures (the pair read accepts collectedSince, so ServerWatermarkReadFloorTests covers it by construction). The live round trip now reads the runner's pair read at three moments: pre-rung rows only (local, LOCAL), after the real COPY has stored the twin (the newest INSTANT in UTC while the plain read still answers the EDT-side row's later local stamp — the fall-back shape), and after the post-rung rows are gone again.
• Darling.Tests/CollectorTimestampFrameTests — prose updated; a frame-of-dedup source pin beside the frame-per-column clock pins.
• Darling.Tests/DarlingCollectorRunnerTests — the The server-scoped watermark read is discarded entirely for collectors that declare a per-database watermark #2797 IL gate pin now tracks both server-scoped readers, and asserts the pair read is actually called somewhere.

Closes #3778