Skip to content

Count archived history in the analysis warm-up, and read v_ views everywhere (#1809) - #1811

Merged
erikdarlingdata merged 2 commits into
devfrom
feature/1809-analysis-spans-archives
Jul 28, 2026
Merged

erikdarlingdata merged 2 commits into
devfrom
feature/1809-analysis-spans-archives

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Closes #1809.

The reported bug — confirmed exactly

AnalysisService.GetTotalDataSpanHoursAsync measured MIN..MAX(collection_time) on RAW wait_stats — the hot store only. The 512 MB threshold calls ArchiveAllAndResetAsync, which copies everything to Parquet and empties the hot store, so the measured span restarts at every reset. On a multi-server install that resets more often than daily, the Recommendations warm-up can never reach 24 hours — while the archived history sits in Parquet, readable through the v_ union views every other tab already uses. The reporter's option 1 (count archived + hot) is what shipped; no new knob.

Why it's a 23-site sweep, not a one-word fix

The broken gate was accidentally SHIELDING a second instance of the same defect. The fact collector read raw tables in 22 more places: the analysis-window reads (query stats/snapshots, memory, perfmon, file IO, blocking, deadlocks, storage, sessions, jobs, memory clerks/pressure, plan cache, CPU scheduler) and the on-load config snapshots (server/database config, trace flags, server properties) — which after a reset are EMPTY until the next app start. Fix only the gate and post-reset runs would analyze a thin hot window: the exact fraction-of-period distortion the 24-hour floor exists to prevent. So the gate and the reads move together — all 23 sites now read their v_ archive views.

Safety of the swap, verified not assumed: the views are column-identical to their tables by construction (only config_alert_log's view adds a source column, and analysis never reads it); the QUALIFY-dedup views make re-collected post-reset rows appear exactly once; there is no SELECT * anywhere in Lite/Analysis.

Guards

  • Behavioral fixture (real DuckDB + Parquet, the ArchiveViewDedupTests harness): 28h archived + 2h hot across a simulated reset → span counts ~30h; a genuinely young install (2h, no archive) still gates.
  • Catalog-driven sweep guard: for every table in ArchiveService.ArchivableTables, a raw FROM {table} anywhere in Lite/Analysis fails with the file named — a new collector's table is guarded the day it exists.
  • Watched red: reverting the span query to the raw table fails BOTH the fixture test and the sweep guard.

Test plan

  • New tests 3/3 green; red under mutation
  • Full Lite.Tests: 1601/1601
  • Lite builds, 0 warnings
  • CI

🤖 Generated with Claude Code

erikdarlingdata and others added 2 commits July 28, 2026 13:22
…rywhere (#1809)

The 24h sufficiency check measured the span on raw wait_stats; the 512MB
ArchiveAllAndResetAsync empties the hot store, so multi-server installs
that reset sub-daily never finished warm-up. The check now reads
v_wait_stats (hot + parquet).

The broken gate was also shielding the same defect downstream: the fact
collector read raw tables in 22 more places, including the on-load
config snapshots that are EMPTY after a reset until the next app start.
Gate-only would have analyzed a thin post-reset window - the exact
distortion the floor exists to prevent. All 23 sites now read their v_
archive views, and a catalog-driven sweep guard (ArchivableTables) fails
on any raw FROM in Lite/Analysis, naming the file.

Verified: behavioral fixture proves the span counts hot + archived
parquet across a simulated reset and still gates a young install;
watched red by reverting the query (both guards fired); full Lite.Tests
1601/1601; zero warnings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@erikdarlingdata
erikdarlingdata merged commit 8ed852e into dev Jul 28, 2026
4 checks passed
@erikdarlingdata
erikdarlingdata deleted the feature/1809-analysis-spans-archives branch July 28, 2026 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant