Repository navigation
Measurement-layer campaign: the delta honesty contract (11 findings, one keystone) #3540
Description
Activity
Filed-in-passing from the #3561 fix (PR #3569): the deprecated Dashboard's schema-computed cntr_value_per_second column uses integer division, truncating fractional rates on its chart/MCP display surface. Pre-existing, display-only, deprecated tier - tracked here as a measurement-lane checklist item rather than its own issue.
- added a commit that references this issue
on Sep 18, 2026 Keystone (A1) landed in #3595 — Darling V127 / Lite v60:
sample_interval_secondsonwait_stats,file_io_stats,latch_stats,spinlock_stats; the four collectors write it; twenty read sites across service MCP, viewer, Lite, the anomaly detectors and Lite's baseline provider now treat a stored 0 as NULL and a pre-upgrade NULL as "keep the LAG-derived interval". Census:Lite.Tests/DeltaFamilyIntervalColumnTestspins that every delta family in the interval-carrying set persists the column and names the four that still do not. Doc-pointer follow-up in #3600.Follow-ups this landing exposes, parked here (each is measurement-lane scope; the checklist items below inherit them):
- A6 CAGG follow-up —
TimescaleSupport.cs:427CreateWaitStatsBaselineSqlstill sums the restart zero and counts it as a sample. A CAGG query cannot be altered in place and a rebuild forfeits ~35 days of baseline history that the 30-day raw retention cannot refill (the PERFMON_BATCH_REQ_SEC facts and the anomaly floor read a per-interval delta as per-second #3527 precedent). The documented shape: a new aggregate under a new name withWHERE sample_interval_seconds IS DISTINCT FROM 0,WITH NO DATA+--backfill-rollups, retire the old viaRetiredBaselineRelations(Retire the orphaned cpu_utilization_baseline / file_io_baseline continuous aggregates (#1995 cleanup) #2007 shape). This is the A6 item's first concrete step. - Four still-naked delta families —
ProcedureStatsCollector,MemoryGrantsCollector,PgWaitStatsCollector,PgStatementStatsCollectorpersist deltas without an interval. Each is a future rung (one un-landed rung at a time); pinned by the census'sStillNakedlist so the set cannot grow silently. Contract rule 4 is complete when that list is empty. - Compose compiler —
Darling/PerformanceMonitor.Darling.Service/Compose/MeasureCatalog.cs/ComposeCompiler.cs:660: the Cumulative archetype'sAVG/MIN/MAX(delta)counts unknowable rows; now that six families carry the column, the compiler can emitsample_interval_seconds IS DISTINCT FROM 0for Cumulative measures on interval-carrying sources (touches perfmon/query_stats too — its own small lane). - Latch/spinlock snapshot grids (both SKUs) show raw deltas with no interval column, so a (0,0) restart row displays as "delta 0". Cosmetic; surface the interval or render the unknowable row as such (A7's viewer item).
- A6 CAGG follow-up —
A4 landed in #3614 — every delta family a host monitors is seeded from the store at service start (latest row per key inside the 15-minute window via
DISTINCT ONfor collectors that do not write every key every pass; the bounded latest-collection probe for the rest), the per-group pass window that the #2235 series-age rescue reads is seeded for ALL families including the original four, seeding runs under a per-family guard so one slow table cannot cost the others their continuity, andClearServeris wired into Lite's server-removal cleanup and Darling's reconcile-remove branch. CensusLite.Tests/DeltaFamilySeedingCensusTestsreads both hosts' seeders against the calculator's family list and each collector's own delta-group names — an eleventh family cannot ship unseeded (contract rule 7). Measured on a migrated container: a 645k-rowpg_statement_statswindow seeds in 372 ms, every plan bounded to the window's chunk.One residual that is a rung, not a follow-up fix:
query_statscannot be key-seeded from either store.PerformanceMonitor.Collectors/QueryStatsCollector.cs:475keys deltas on{SqlHandle}:{StatementStartOffset}:{StatementEndOffset}:{PlanHandle}, butPayloadColumns(:329-385) persistssql_handle/plan_handleand neither offset — no stored row can reproduce the key, and a seed under any other key seeds nothing silently. Every delta family is seeded from the store at service start, and the series-age rescue finally has passes to read, so a restart no longer fabricates one interval of quiet for six families (#3540 A4) #3614 seeds its pass window only (which is what the Per-query CPU attribution unusable on a plan-churning instance: reads see 18 execs where Datadog sees 43% of the box; literal fragmentation defeats top-N #2235 rescue reads, so young plans are credited on the first post-restart pass); plans older than the restart gap still re-baseline once. Fix = one rung on both SKUs addingstatement_start_offset integer, statement_end_offset integertoquery_stats, the collector writing them at the tail (+DuckDbSchemaEquivalenceTests.IntentionalAppendedColumns), then a per-keyQueryStatsSeedSql. The census'sPassWindowOnlyset andThePassWindowOnlyExemption_RestsOnTheOffsetsNotBeingStoredgo red the moment the columns land, so the seed cannot be forgotten. This joins the four still-naked families as a rung candidate — five rungs queued behind one un-landed-at-a-time.- A5 adjacency (not built):
DarlingWorker.ReconcileServers's "definition changed" branch (~:3270-3280) drops the runtime and reconnects under the sameserver_idwithout clearing baselines — same identity, possibly a different physical server behind a changed connection string. Belongs to the epoch-detection item.
Status 17:45Z — A1 keystone merged (#3595, V127) and A4 seeding merged (#3614). In flight: V128
delta-family-interval-completionlane (interval on the four still-naked families + query_stats statement offsets; PR not yet open). Remaining after that: A5 identity-epoch detection (structural), A6 CAGG follow-up (new wait_stats_baseline aggregate + retirement), A7 viewer perfmon/gauge rendering, A8 rollup rate denominators, A10 baseline contamination guard, A11 assorted, the 10-rule contract census (rules 4 and 7 are now pinned; the rest are not). Issue stays open by design.- added a commit that references this issue
on Sep 18, 2026 V128
delta-family-interval-completion/ Lite v61 landed in #3630 (merged 2a1e367, 19:19:58Z). The last four families (procedure_stats,memory_grant_stats,pg_wait_stats,pg_statement_stats) carrysample_interval_seconds;query_statscarriesstatement_start_offset/statement_end_offset(byte offsets,-1= end of batch, stored verbatim) so both hosts' restart seeds rebuild its delta key. Contract rule 4 is complete —DeltaFamilyIntervalColumnTests.StillNakedis empty and asserted empty; #3614'sPassWindowOnlyexemption is likewise empty and asserted. Verified: fresh ladder V1→V128, a simulated pre-V128 store on compressed hypertables with 20 CAGGs (one rung applied, ALTERs succeed, CAGGs refresh), four release fixtures 3.3.0→3.7.0 climb to 128.Residuals from that lane (all parked under the remaining checklist items):
- A11a — the raw duration-trend SQL in
ViewerDataService.QueryTrends,DarlingTrendReader.QueryDurationTrendSql, LiteGetQueryDurationTrendAsync/GetExecutionCountTrendAsyncstill LAG-derives the interval thoughquery_statshas carried it from the start; the three-state idiom applies; pinned as "reported, not rewritten". OversizedPlanBacklog.QueryStatsFallbackSql(:244) stays hash-keyed — an exact offsets join is possible for V128+ rows; not taken because readers ask at the hash grain and pre-V128 rows are NULL.- Compose
MeasureCatalogCumulative archetype (V127's follow-up) — all ten families now carry the column, so theIS DISTINCT FROM 0guard can be emitted uniformly.
- A11a — the raw duration-trend SQL in
- added 3 commits that reference this issue
on Sep 18, 2026 Closing — the keystone and its completion are landed; contract rules 4 and 7 are pinned; the structural remainder moves to #3653 #3653.
Shipped from this issue (all merged to
devon 2026-09-18):- A1 keystone —
sample_interval_secondson wait_stats / file_io_stats / latch_stats / spinlock_stats, twenty read sites NULL-not-0 — V127 / Lite v60, The four naked delta families store the interval their deltas accrued over, so a restart's fabricated zero reads as unknowable instead of 0.00 ms/sec (#3540, V127 / Lite v60) #3595 (+ The delta calculator's doc names the census that actually guards the still-naked delta-family list (#3540 follow-up) #3600) - A1 completion — the last four families (procedure_stats, memory_grants, pg_wait_stats, pg_statement_stats) +
query_statsstatement offsets so the key seed can find its keys;StillNakedandPassWindowOnlyboth empty and asserted — V128 / Lite v61, Every delta family now stores the interval its deltas accrued over, and query_stats stores the statement offsets its delta key is made of, so no restart zero reads as a measurement anywhere and the query seed can finally find its keys (#3540, V128 / Lite v61) #3630 - A4 every delta family seeded from the store at service start,
_passesseeded so the Per-query CPU attribution unusable on a plan-churning instance: reads see 18 execs where Datadog sees 43% of the box; literal fragmentation defeats top-N #2235 series-age rescue fires,ClearServerwired into both hosts' remove paths, seeding census — Every delta family is seeded from the store at service start, and the series-age rescue finally has passes to read, so a restart no longer fabricates one interval of quiet for six families (#3540 A4) #3614 - A8's first-point-NULL for differenced series landed inside Every delta family now stores the interval its deltas accrued over, and query_stats stores the statement offsets its delta key is made of, so no restart zero reads as a measurement anywhere and the query seed can finally find its keys (#3540, V128 / Lite v61) #3630 and Zero is a measurement: health parsers say whether their source was ever seen, a regression with no baseline stays null, and the first point of a differenced trend is no longer a fabricated 0 (#3541 A12) #3642; the perfmon per-second divisor (PERFMON_BATCH_REQ_SEC facts and the anomaly floor read a per-interval delta as per-second #3527/Deprecated Dashboard analysis has the #3527 perfmon divisor defect (per-interval delta read as per-second) #3561), the Lite cadence clamp (Lite accepts collector cadences above the delta gap policy, fabricating permanent quiet #3532) and the QueryStore ordinal slip were wave 1.
Deferred to #3653: A5 identity-epoch detection (the only genuinely structural item), A6 the
wait_stats_baselinenew-aggregate + retirement, A7 viewer gauge/interpolation rendering + thecntr_typerung, A8 rollup denominators, A10 the baseline contamination guard + orphanedIsRealDeltaRow, A11 assorted (raw duration-trend LAG, datid key, local-time CPU stamps, DST offset, Dashboard integer division), the Compose Cumulative archetype, and the eight unpinned contract rules.- A1 keystone —
- added 12 commits that reference this issue
on Sep 18, 2026
Adversarial review of every delta/rate/trend computation across both SKUs, against every discontinuity class (restart, gap, failover, stats reset, chunk boundary, first sample). Verdict: the delta ENGINE (CollectorDeltaCalculator: measured 3600s gap policy, series-age rescue, the (0,0) unknowable marker) is honest and battle-hardened — but the honesty is destroyed at the schema for 4 of 6 delta families, and two unit lies feed the analysis brain. Individually filed: the perfmon per-second divisor, the Lite cadence clamp, the QueryStore ordinal slip.
The keystone — A1: interval columns for the naked delta families. wait_stats, file_io_stats, latch_stats, spinlock_stats persist deltas with NO sample_interval_seconds column, so the calculator's (delta=0, interval=0) "unknowable" marker cannot survive the write, and readers LAG-divide fabricated zeros into confident 0.0 at exactly the moments (restarts) they're unknowable. The calculator's own doc claims "every consumer maps 0 to NULL via NULLIF(sample_interval_seconds, 0)" — false for these four. perfmon/query_stats already have the column (the pattern to copy). Once landed, four findings below become one-line WHERE/NULLIF changes. Schema change -> upgrade-folder rules, both SKUs.
Remaining findings (checklist; verified at path:line; re-verify at pickup):
_passesnever seeded (the series-age rescue is inert on the exact cycle it exists for). ClearServer has zero callers in Darling.Service (re-onboard <1h fabricates a delta).Credits: the delta calculator itself; NO assumed-cadence divisor anywhere in either SKU's trend path; the QS cumulative-snapshot solution (the reference for cumulative sources); DarlingPgTrendReader as the reference discontinuity-honest reader ("nothing on the SQL Server side matches it — that asymmetry IS the roadmap"); clock-skew immunity by construction; the Azure id-space fix with no live siblings.
From the 2026-09 brains-review campaign.