Latest-value lookbacks follow the collector's cadence: a day, or twice its interval if longer (#3896)

[erikdarlingdata]

Closes #3896.

Follow-up to #3931, which merged the lookback as a flat day. This PR adds the cadence-aware lookback requested in #3931's review. It also carries #3928 as a second commit, under the rule that small in-lane follow-ups go in the PR: the PostgreSQL target's config reads get the same planning fix without changing a single answer.

Why

#3931 bounds each analysis latest-value read to one day before the window's end. It covers the database-size, percent-autogrowth, disk-space, memory-clerk, plan-cache and memory_stats facts, and the autogrowth drill-down. But operators can edit collector cadences: Darling's config_collector_schedules.frequency_minutes has no upper bound, and Lite and the Dashboard keep editable schedules too. A steady-state collection advances by exactly its interval (#1553). So a database_size_stats, memory_clerks or plan_cache_stats collector scheduled daily or slower loses its facts on every pass between 24 h after its last sample and its next run. That's a one-pass flap, and it resolves and then re-fires the findings and notifications built on those facts.

#3928: the PostgreSQL target's five reads of the newest pg_server_config snapshot had no lower bound at all. The table keeps a year of hourly snapshots in one-day chunks, so a year in, TimescaleDB planned every chunk on every read: 4.0-8.3 s of planning per read at 366 chunks on the rig. Four of the reads run on every analysis pass, and the clock read runs once for every baseline the pass computes.

What changes

• The lookback is a day, or twice the collector's interval if that is longer (AnalysisContext.LatestValueLookbackFor). Twice the interval keeps the newest sample in range between runs, with a whole interval to spare for a late or failed one. Every shipped cadence is under twelve hours, so default behavior is unchanged: the flat day, as Analysis latest-value reads look back a day instead of scanning a server's whole history, and stop counting dropped databases (#3896) #3931 shipped it.
• The cadence is the one the collector actually runs at. It's resolved once per pass and stamped on AnalysisContext.LatestValueStarts, and every read binds LatestValueStartFor(<its collector>). The drill-down reads the same stamp, so a list and the count it details stay bounded alike.
  • Darling reads config_collector_schedules (per-server over fleet-wide) through the rule the scheduler runs by. That rule moved from StoreConfigProvider to CollectorScheduleDefaults.ResolveFrequencyMinutes, and ResolveSchedule delegates to it, so the scheduler and the analysis can't resolve two different cadences. If the overrides can't be read, the pass falls back to the defaults and logs a warning.
  • Lite resolves its ScheduleManager through the storage-hash to connection-GUID match the alert adapter already used. That match is now one method, ScheduleManager.GetFrequencyForStorageServer. AnalysisService takes the resolver, and the scheduled pass, the MCP host and the Recommendations tab all pass it. With no resolver, every collector takes its shipped default.
  • Dashboard reads config.collection_schedule. Its frequency 0 means "every tick", so it takes the day.
• On-load collectors. A collector moved to on-load only (frequency 0) anchors on its newest capture at or before the window's end, however old, so it doesn't lose the fact a day after every connect.
• The bound stays a plain timestamp parameter, computed in C#, so TimescaleDB still excludes chunks at plan time at any lookback.
• No SQL text changes for these reads: they keep Analysis latest-value reads look back a day instead of scanning a server's whole history, and stop counting dropped databases (#3896) #3931's collection_time >= $2 AND collection_time <= $3, and the Lite twins stay byte-identical.

#3928: the PostgreSQL target's config reads (Darling only; Lite and the Dashboard don't monitor PostgreSQL)

• The config, posture, blocking-settings and memory families' reads and the baseline clock's (PgTargetBaselineProvider.PgTargetClockSql) bound both the snapshot's MAX(collection_time) and the row scan from below. Bounding only the MAX isn't enough, because the outer collection_time = (…) can exclude chunks only at run time.
• Every read runs the day first, and unbounded only when that found nothing (PgTargetFactCollector.ConfigSnapshotLowerBounds: the day, then -infinity). The newest snapshot in the day is the newest snapshot the unbounded anchor finds, so the two runs can't disagree:
  • a target whose config collector has been dark for days still states its newest snapshot, with its age on the fact;
  • a historical window still reads the snapshot at or before its own end;
  • posture still reads the newest snapshot the target has.
• That removes the ruling PostgreSQL-target config reads plan every retained pg_server_config chunk: seconds per read once a store holds a year of snapshots #3928 asked for. Posture's "the snapshot, not the window" rule and the clock's UTC fallback behave exactly as before.
• These reads take a flat day plus the fallback, not their collector's cadence. They anchor on one snapshot per server, so there's no stale series for a bound to drop, and a cadence slower than a day costs the fallback run, never a fact.

Test plan

• Darling LatestValueLookbackTests (53, up from 27):
  • the lookback arithmetic: 1, 60 and 720 minutes take the day; 721, 1,440 and 10,080 take twice the interval; 0 takes none, and every shipped cadence takes the day;
  • the scheduler and the analysis resolving one cadence, through both ResolveFrequencyMinutes and StoreConfigProvider.ResolveSchedule, for per-server, fleet, negative and delta-capped overrides;
  • each read bound by the collector whose table it reads;
  • live on the rig:
    • database_size_stats overridden to 1,440 minutes: a 25 h-old sample still yields FILE_AUTOGROWTH_PERCENT, DISK_SPACE and the drill-down file, and a 49 h-old one does not, while a server left at the hourly default loses the same 25 h sample exactly as before;
    • an on-load override anchors on the newest capture: 3 days old, the ghost from the capture before excluded, a capture after the window's end kept out;
    • a fleet-wide override widens every server, and a server's own row wins over it;
    • plan shape with the override's 48 h bound: the shipped read plans 3 one-day chunks where the pre-Analysis 'latest value' reads scan a server's entire retained history: half of every analyze_server call, and the database-size fact counts dropped databases (all three SKUs) #3896 read plans 11 (default cadence: 2 against 13).
• Lite LatestValueLookbackTests (6): the daily override (25 h kept, 49 h dropped, default dropped) and the on-load anchor on DuckDB, through the production resolver over a real ServerManager and ScheduleManager.
• Dashboard LatestValueLookbackSqlTests (9): each read bound by its own collector's stamp, the stamp taken before the first read and in the drill-down, and the arithmetic off config.collection_schedule (1,440 → 48 h; 1, 0 and no row → 24 h).
• Red-watch: with the lookback flattened back to a day, 8 Darling, 2 Lite and 1 Dashboard tests fail, which are the cadence tests.
• DARLING01 has no schedule override for these five collectors (its only config_collector_schedules row enables long_query_completions). Its reads therefore keep Analysis latest-value reads look back a day instead of scanning a server's whole history, and stop counting dropped databases (#3896) #3931's measured bounds and numbers: DatabaseSizeSql 1,167 → 56 ms, and 1.31 s → 66 ms per fact pass.
• PostgreSQL-target config reads plan every retained pg_server_config chunk: seconds per read once a store holds a year of snapshots #3928, PgTargetConfigSnapshotBoundTests (14 new: 12 ungated, 2 live):
  • each of the five statements carries the bound on both the anchor and the row scan; each call site runs the day first and the fallback second; a census holds the five as every analysis read of the table;
  • live: with ten days of hourly snapshots seeded (eleven one-day chunks), the day plans 4 chunk scans and the fallback plans 30 on the rig's suite store (asserted: at most 4, at least 11), and the newest snapshot's value wins;
  • live: a target dark for 72 h still emits its config and posture facts (snapshot_age_s 259,200) and its clock zone; a window that ended five days ago reads its own snapshot for config and the clock, and posture reads today's.
  • Existing pins updated: PgTargetClockTests (the shared anchor text, token for token), PgTargetKnobsTests, PgTargetMemoryTests, PgTargetPostureTests, PgTargetBlockingTests, and the V138 scope census in PgServerConfigScopeRungTests (whose anchor-subquery pattern now allows the lower bound; still 12 reads).
  • The pre-existing PgTargetClockLiveTests already crossed the fallback: its New York snapshot is 31 days older than the window's end.
• PostgreSQL-target config reads plan every retained pg_server_config chunk: seconds per read once a store holds a year of snapshots #3928 red-watch: with the fallback dropped from ConfigSnapshotLowerBounds, 4 tests fail, including the pre-existing clock e2e.
• PostgreSQL-target config reads plan every retained pg_server_config chunk: seconds per read once a store holds a year of snapshots #3928 measured, same rows both ways:
  • Rig (PostgreSQL 18.4, TimescaleDB 2.28.1), 366 one-day chunks with the older ones compressed, custom plans with bound parameters: planning 4.0-8.3 s unbounded against 0.6-1.1 ms with the day (9-21 ms cold). Execution 15-55 ms against 0.1-0.15 ms (0.5-3 ms cold).
  • DARLING01 (PostgreSQL 18.6, TimescaleDB 2.30.1, 17 chunks), read-only: planning 7.1-7.6 ms against 0.76-0.88 ms, and execution 1.0-1.7 ms against 0.25-0.43 ms. The plain-EXPLAIN chunk listing the new live test counts is the same shape on 2.30.1: 2 chunks per scan with the day, 17 with the fallback.
• Full suites on this branch (dev at c5be29e plus the two commits), all builds 0 warnings:
  • Darling.Tests with DARLING_TEST_PG (rig: PostgreSQL 18.4 + TimescaleDB 2.28.1): 12,732, 0 failed, 24 skipped (the runtime and upgrade tests gated on variables this rig leaves unset). An earlier full run, before the census fix, failed 3: the V138 scope census, fixed in the PostgreSQL-target config reads plan every retained pg_server_config chunk: seconds per read once a store holds a year of snapshots #3928 commit; and two load flakes that pass alone, PgWaitSamplerLiveTests (PgWaitSamplerLiveTests can count another test's lock wait: 31 samples from a 30-snapshot cycle under full-suite load #3939) and StoreToastAndCheckpointerLivePostgresTests (filed as Test flake: StoreToastAndCheckpointerLivePostgresTests reads 96 % live after deleting half the rows under full-suite load #3977).
  • Lite.Tests: 5,134, 0 failed (on the cadence commit; the PostgreSQL-target config reads plan every retained pg_server_config chunk: seconds per read once a store holds a year of snapshots #3928 commit touches no Lite or shared code).
  • Dashboard.Tests: 818, 0 failed (same).

Left open

• TRACE_FLAGS keeps reporting a trace flag for weeks after it is turned off #3929 (TRACE_FLAGS keeps a flag after it's turned off) needs a design decision, and the cheapest fix depends on A Darling server connected for more than 30 days loses its config facts: on-load snapshots age out and nothing re-collects them #3930's. If on-load collectors re-run on a slow cadence, the trace-flags read can take this PR's cadence-aware lookback, with no sentinel row and no migration. If they don't, it needs a sentinel row per capture (a change to what every trace_flags reader sees) or a collection_log anchor (a new index, so a store migration).
• A Darling server connected for more than 30 days loses its config facts: on-load snapshots age out and nothing re-collects them #3930 (on-load config snapshots age out on a connection held for more than 30 days) needs a ruling between re-collecting on a slow cadence, exempting the newest capture from the purge, and raising retention. Each changes the scheduler or retention in all three SKUs, not a read.
• get_pg_server_config, get_pg_logging_audit and the viewer's PostgreSQL config panel plan every retained pg_server_config chunk #3974 (new): get_pg_server_config, get_pg_logging_audit and the viewer's PostgreSQL config panel have the same unbounded anchor. The fallback carries over to two of those statements. OverrideSql needs a shape decision first, because it's usually empty, so its bounded run can't tell "no snapshot in the day" from "no overrides".

🤖 Generated with Claude Code

https://claude.ai/code/session_018Z4T51J4E6L8fjriPHpsif