Perfmon analysis reads divide the per-interval delta by its measured interval

[erikdarlingdata]

Fixes #3527

What

The analysis engine consumed perfmon delta_cntr_value — the per-COLLECTION-INTERVAL delta — as if it were per-second, overstating rates 60x at a 60s cadence and 300x at 5min. Every consuming read now divides by the interval, consistently, in both SKUs, so the window statistic, the baseline population, and the requests/sec thresholds (BatchRequestFloor 500, BatchRequestFallback 5000 — their comments were already right, the inputs weren't) meet in one unit:

1. Fact collectors (PgFactCollector.Resources.cs + DuckDbFactCollector.Resources.cs, byte-similar): PERFMON_BATCH_REQ_SEC / PERFMON_COMPILATIONS_SEC / PERFMON_RECOMPILATIONS_SEC now select the row's measured sample_interval_seconds (get_perfmon_trend: delta_value returns a fabricated 0 on ~38% of points while cntr_value advances #2234) and emit delta / interval. Metadata keeps cntr_value and delta_cntr_value raw and adds sample_interval_seconds, so the value is auditable. Rows with interval <= 0 — the unknowable-delta marker (first sighting, counter reset, gap past the delta policy) — are filtered in the WHERE, so rn = 1 lands on the newest USABLE row and an unknowable rate is never emitted as 0 or as the raw delta.
2. Anomaly windows (PgAnomalyDetector.BatchRequestWindowSql + Lite's inline twin, kept Lite-verbatim): AVG/MAX(delta_cntr_value * 1.0 / NULLIF(sample_interval_seconds, 0)) with sample_interval_seconds > 0; interval-0 rows are not samples.
3. Baselines (batch_requests arm only, both providers): Lite divides by the stored measured interval on the raw rows. Darling's arm reads the collect.perfmon_baseline continuous aggregate, which materializes only (collection_time, delta_cntr_value) — and a CAGG cannot grow a column without a drop-and-rebuild that would forfeit 31 of the 35 days of baseline history the 4-day raw tier can't refill — so it derives interval_sec from LAG(collection_time) over the collapsed series, byte-for-byte the established WaitMsPerSec idiom, with the io-arm's ::DOUBLE PRECISION cast (numeric STDDEV overflows System.Decimal). The restart-exclusion signature stays on the RAW delta in both SKUs (its > 1000 bar predates the division), and the QUALIFY-rewrite window-before-filter invariants are preserved and still pinned.

Baseline-unit consistency (the constraint this change lives or dies on)

No stored baseline state needs migration — verified. Both providers compute baselines ON READ (BaselineProvider/PgBaselineProvider.GetBaselineQuery runs per call) and cache only in a process-local dictionary with a 1-hour TTL; no baseline statistic is persisted anywhere. Darling's one materialized artifact, the perfmon_baseline CAGG, stores per-collection raw deltas — unit-neutral facts, not statistics — so all 35 days of materialized history serve the new per-second unit immediately, no backfill. The upgrade's process restart clears both in-memory caches, so no pass can ever compare a per-second window against a cached old-unit baseline. Historical rows written before #2234 carry the assumed-cadence 60 rather than a measured interval; they divide to an approximately-right rate (vs exactly-wrong raw) and age out of the 30-day window.

Divisor provenance differs only where the store forces it: Lite (all three reads) and Darling's fact/window reads use the per-row measured interval; Darling's baseline derives it from collection-time gaps. Same unit everywhere.

Tests

Both SKUs, per the issue's fixtures:

• interval 60 / delta 6000 produces a fact value of 100 (Lite DuckDB behavioral; Darling live-PG end-to-end EndToEnd_PerfmonFacts_DivideDeltaByMeasuredInterval_AgainstDevPostgres)
• interval-0 rows are skipped at the fact read (newest usable row wins; a counter with only interval-0 rows emits NO fact), at the window read (not a sample), and at the baseline read (not population)
• the anomaly floor comparison operates on per-second values: a raw delta of 3000 (past the 500 floor) whose rate is 300/sec stays quiet (Lite DetectBatchRequestAnomalies_LowVolumeSpike_NoAnomaly), and Darling's live EndToEnd_BatchRequestArm_PerSecondBaselineAndWindow_AgainstDevPostgres proves the LAG-derived baseline mean (90/sec), the per-second window peak (10000, not 600000), the skipped interval-0 window row, and the absolute fallback bar judging honest rates
• ungated SQL pins on the divisor + filters in all three Darling queries, extending the existing QUALIFY-rewrite structural pins

Local results: Lite.Tests 3932/3932 green. Darling.Tests 9702/9703 against a throwaway PG 18 + TimescaleDB rig (CI worker sizing, migrations, gated *_AgainstDevPostgres all run) — the one failure is NpgsqlRootCertificateValidationTests.ChainShape_VerifyFullWithPrintedRoot..., a TLS-harness platform test that fails identically with no rig and with none of these files involved (pre-existing local-environment condition, untouched by this change). Builds: 0 warnings.

Proposed CHANGELOG entry (not applied here)

- **Perfmon rates are honest per-second values in analysis** ([#3527]) - The PERFMON_*_SEC facts, the batch-request anomaly window, and both SKUs' batch-request baselines read the per-collection-interval delta as if per-second (60-300x overstatement); every read now divides by the measured sample_interval_seconds (Darling's baseline derives it from collection-time gaps, since the continuous aggregate stores no interval), and interval-0 rows — where no delta was knowable — are skipped instead of read as rates.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QMYFB4qv5tqGwMmdPsa4Zx

[claude]

Non-blocking observation on the BatchRequests baseline arm's interval derivation.

The collector (CollectorDeltaCalculator.Core, counter-reset branch) forces both delta_cntr_value = 0 and sample_interval_seconds = 0 together whenever a delta is unknowable (reset, gap, first sighting) — that joint invariant is what Lite's new AND sample_interval_seconds > 0 filter relies on to correctly drop these rows everywhere.

But perfmon_baseline (the CAGG/fallback view) never stored sample_interval_seconds, so this arm can't apply that filter — it derives interval_sec from LAG(collection_time) instead, i.e. purely from the wall-clock gap between collections. A counter reset that happens to occur on a completely normal collection cadence (SQL Server restarts, counter resets to 0, but the next perfmon sweep still lands ~60s later as scheduled) produces interval_sec ≈ 60 here — not NULL/0 — so the row survives into clean as a legitimate-looking 0 req/sec sample, unless the pre-existing prior_delta > 1000 restart heuristic happens to catch it (it only does so when the server was previously above ~1000/collection-interval).

So for a low-throughput server (prior delta ≤ 1000) that experiences a counter reset, Lite now correctly excludes that row from the baseline population, while Darling's BatchRequests arm can still admit it as a fabricated zero-rate sample — a real (if narrow) parity gap against the PR's stated invariant that "interval-0 rows... are skipped... at the baseline read" in both SKUs.

Flagging for awareness rather than blocking: the robust-tier scaffold (median/MAD) this feeds is already designed to tolerate occasional outlier contamination, and this exact derived-interval approximation is inherited from the pre-existing WaitMsPerSec idiom, so it's not a new class of bug this PR introduces — just a spot where the new "interval-0 must never be read as a rate" guarantee isn't quite airtight for this one arm, given the CAGG's storage constraint.

[claude]

LGTM — reviewed the delta/interval-division fix across both SKUs' fact collectors, anomaly windows, and baselines (PgFactCollector.Resources.cs, PgAnomalyDetector.cs, PgBaselineProvider.cs, and their Lite/DuckDB twins). Verified the arithmetic end-to-end against the PR's own fixtures (delta 6000 / interval 60 = 100, restart-exclusion staying on the raw delta, interval-0 rows correctly dropped before rn=1 / from the window / from the baseline population on the raw-table reads), checked filter-before-project safety for the new divisions, and confirmed the perfmon_baseline CAGG still groups by raw collection_time so the LAG-derived interval_sec reflects the true collection cadence rather than the hourly bucket. Left one non-blocking inline comment on a narrow parity gap in Darling's BatchRequests baseline arm (counter-reset rows at normal cadence can still slip in as fabricated zero-rate samples, since perfmon_baseline never stored the true per-row interval) — bounded in impact by the existing median/MAD robust statistics and inherited from the pre-existing WaitMsPerSec idiom, not a new bug class.