Reconcile each raw hypertable's chunk interval once a day from its ingest (#4211)

[erikdarlingdata]

Part of #4211.

Why

RawChunkIntervalPlanner (#4332, already on dev) derives each raw hypertable's target
chunk_time_interval from its own ingest rate and a RAM budget, but it is pure: no database calls, no
schedule, nothing wired to a live store. Every raw hypertable still keeps the fixed interval it was
created with, so a table whose write rate grows or shrinks over time stays on a chunk size sized for its
first day. This PR is #4211's second half (lane I2, finished by I2b after I2 stopped with the work
uncommitted): it reads live ingest and catalog state, calls the planner, applies its decisions with
set_chunk_time_interval, and records what happened.

What changes

Per the ruling on #4211 (issuecomment-5836205190), the five pieces:

1. The reconciler - Darling/PerformanceMonitor.Darling.Storage/RawChunkIntervalReconciler.cs. Reads
   every raw hypertable's ingest rate (compressed chunks' before_compression_total_bytes, heap and index
   together, over their wall-clock span - never the 2.29-only hypertable_compression_stats columns),
   current interval, and when it last moved (from the rung history table, so the once-a-day hold survives a
   restart). Calls RawChunkIntervalPlanner.Plan, applies every changing decision inside its own
   transaction, and records the rung history plus a per-run row carrying pg_stat_wal.wal_bytes (ruling
   decision 8 / review finding L4, so a later stage can correlate WAL growth against interval moves). Runs
   at startup and once a day, riding the same daily-purge tick every other maintenance chore uses; never
   throws past one run - a bad reading degrades to "no changes" next time.
2. The off switch - Darling/PerformanceMonitor.Darling.Service/DarlingConfig.cs,
   RawChunkIntervalReconcileEnabled (default true, file-only, takes effect on restart).
3. The wiring - Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs. Gates the whole pass on
   _timescaleAvailable && config.RawChunkIntervalReconcileEnabled before the daily-purge tick calls the
   reconciler, and resolves budget B: ResolveRawChunkIntervalBudgetBytesAsync reads the same raw RAM
   figure DarlingManagedPostgres.DeriveMemorySettings sizes Postgres from on a managed store (through
   DarlingStoreHostProfile.GatherHostFacts, which itself calls
   DarlingManagedPostgres.TryReadWindowsPhysicalMemoryBytes - never a second GlobalMemoryStatusEx
   call), and reads shared_buffers/effective_cache_size live off pg_settings via
   RawChunkIntervalPlanner.BringYourOwnBudgetBytes on a bring-your-own store.
4. Migration V144 - Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs adds
   collect.raw_chunk_interval_rung_history (one row per rung change: table, when, from/to hours, reason,
   and the three inputs the decision was made from) and collect.raw_chunk_interval_reconcile_runs (one row
   per run, whether or not it changed anything, with wal_bytes). StorageVersion.cs moves 143 to 144, and
   the probe sentinel lands in Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs
   (MapProbedSchemaVersion's new top arm, ordinal 119).
5. Tests - Darling/Darling.Tests/RawChunkIntervalReconcilerLiveTests.cs (end-to-end narrow-then-hold
   against a real TimescaleDB, the bring-your-own budget path, the off-switch source check) and
   Darling/Darling.Tests/RawChunkIntervalRungHistoryRungTests.cs (the V144 rung pin).

I2b's own fix on top of I2's checkpoint (e45e705): RawChunkIntervalRungHistoryRungTests.cs (V144)
correctly took over the "I am the top rung" claim, but QueryStoreIntervalLatestRungTests.cs (V143) still
asserted it - RungVersion == SchemaVersion and "no reader.GetBoolean(119) in the viewer" - both of which
broke the moment V144 appended ordinal 119. Stepped V143's file down the same way V141's file
(CollectionCaveatsRungTests.cs) already stepped down when V142 landed on it: RungVersion < SchemaVersion, an atThisRung/behind boolean probe instead of all-true, and a nextArm check against
V144's arm. This was the entire cause of the 2 failures the checkpoint's first run reported.

Pre-merge fixes

From the PR tender's pre-arm check, all in da47a539 (report: PR comment 5838942976):

• The planner's reasons are now formatted with the invariant culture (string.Create(CultureInfo.InvariantCulture, ...)). This PR stores the two "moved" reasons in collect.raw_chunk_interval_rung_history.reason, and {x:N0} followed the host's culture: a German Windows host would have written 1.234.567 B. A planner test runs a narrowing and a widening plan under de-DE and checks for the invariant text. The test was revert-proven.
• ReconcileAsync's summary said it never throws for one table's failure. It does: that table's transaction rolls back, tables already applied stay applied and recorded, and the rest wait for the next daily pass. The worker's catch logs it. The comment now says that.
• V144's summary said the reconciler's caller purges the two new tables. Nothing does. The comment now says neither is purged: the run table gains one row a day and the history one row per move, and the history has to keep at least MinimumDaysBetweenMoves days anyway.

CHANGELOG entry

SECTION: Changed
ENTRY:
- **Raw hypertables now re-tune their own chunk interval once a day from actual ingest** ([#4344]) - Every raw hypertable used the same fixed one-day chunk interval whatever its write rate, so a busy table's open chunk, with its indexes, could outgrow the memory meant to hold it. Darling now reads each raw hypertable's compressed ingest rate daily, alongside a RAM-derived budget, and narrows the interval one rung at a time through `set_chunk_time_interval` when the store-wide total is over budget, and widens it one rung when the total would stay under half the budget; a new rung history table records every change, alongside a per-run WAL-volume record. On by default; set `rawChunkIntervalReconcileEnabled: false` in the config file to turn it off.
REF:
[#4344]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/4344

Test plan

• dotnet build Darling/Darling.Tests/Darling.Tests.csproj - 0 Warning(s), 0 Error(s).
• Targeted classes (each its own -class filter, against a fresh darlingtest on port 55991):
  *RawChunkInterval* (24), *DarlingRetention* + *TimescaleSupport* + *PgMigration* +
  *SchemaVersion* (145), *ViewerSchemaVersionGateTests* (32), *StorageCommandTimeoutTests* (19),
  *StartupCommandTimeoutTests* (14), *LivePostgresCollectionHygieneTests* (11),
  *LiveCleanupConversionRatchetTests* (15), *McpPayloadContractCensusTests* (68),
  *DocCommentHygiene* (77). All 0 failed on the first pass, before origin/dev was merged in.
• Revert proof: temporarily forced IntervalLastChangedUtc to always be null instead of reading
  it from the history table, rebuilt, ran
  EndToEnd_NarrowsOneRungOverBudget_ThenHoldsForOneDayFromHistory_AndMigrationReapplyIsANoOp_AgainstDevPostgres
  alone - phase 2 (the same-day hold) failed (Expected: 0, Actual: 1, meaning it moved a second time
  instead of holding). Restored the read, rebuilt, reran - passed.
• Budget B source check: confirmed by reading the call chain, not just the comments -
  DarlingWorker.ResolveRawChunkIntervalBudgetBytesAsync on a managed store calls
  DarlingStoreHostProfile.GatherHostFacts(), which calls
  DarlingManagedPostgres.TryReadWindowsPhysicalMemoryBytes directly (DarlingStoreHostProfile.cs:264).
  The startup sizing path (DarlingManagedPostgres.GetTotalPhysicalMemoryBytes /
  TryGetAuthoritativePhysicalMemoryBytes, which feeds DeriveMemorySettings) calls the same
  TryReadWindowsPhysicalMemoryBytes (DarlingManagedPostgres.cs:3331). Same read, not a second
  GlobalMemoryStatusEx call. On a bring-your-own store, BringYourOwnBudgetBytes_ReadFromPgSettings_AgainstDevPostgres
  already exercises the live pg_settings path. No code change was needed for this check.
• Merged origin/dev (eac444e, PostgreSQL-target baselines PR) - clean merge, no conflicts. dev is
  still at schema version 143, so V144 needed no renumbering.
• Full Darling.Tests suite, run against a freshly recreated darlingtest, twice:
  • First full run (before the V143-rung-handoff fix): Total: 14174, Errors: 0, Failed: 2, Skipped:
    48, Not Run: 1, Time: 826.060s. Both failures were
    QueryStoreIntervalLatestRungTests.TheRungIsRegisteredAtTheTopOfADenseLadder and
    .TheProbeMapsAFullyMigratedStoreToThisTopRung - fixed as described above.
  • Second full run (after the fix, same fresh database): Total: 14174, Errors: 0, Failed: 0,
    Skipped: 48, Not Run: 1, Time: 968.524s.
  • The "Not Run: 1" is stable across both runs and is not from this PR: the test project has exactly two
    [Fact(Explicit = true)] tests in the whole suite
    (McpSchemaCompatServiceLeakRaceTests - an intentionally-manual race repro for Lite budget test sometimes sees get_alert_history's dataService parameter in a full run #4075 - and
    RollupCoverageRoutingTests - marked explicit pending LA-3b), neither of which this PR touches.
  • The 48 skips are all live tests gated on environment variables this rig doesn't set (csvlog targets,
    a second UTC-logging target, DARLING_TEST_PGRUNTIME, etc.) - the standard shape for this suite.
• Lite.Tests - not run. This PR touches only Darling's own PostgreSQL/TimescaleDB store; Lite has no
  PostgreSQL storage layer, so there is no Lite parity surface for a chunk-interval rule.
• Installer.Tests - never run per lane orders.

Not done

Per the brief, these are left for lane I3:

• Design: size raw chunk interval, compress_after and the hourly CAGG refresh window from ingest rate; a large store keeps 1–2 days of raw uncompressed and it outgrows RAM #4211 ruling decision 9's pins. The ladder's top rung is already wired to
  TimescaleSupport.ChunkIntervalDays rather than a restated literal (RawChunkIntervalPlanner.cs:69-71
  already does this), but the brief calls out that the regression pin for it, plus some stale text, are
  still owed.
• Review findings L1-L3. L4 (the per-run WAL-bytes record) is in this PR; L1-L3 were not in I2's checkpoint
  and are not addressed here.

For the coordinator

• I2's rig (port 55991, C:\GitHub\worktrees\rig-i2) was reused per the brief and has now been stopped and
  the directory removed.
• The only defect found in I2's checkpoint was the V143-rung-handoff gap above; everything else (the
  reconciler, the off switch, the wiring, the migration, budget sourcing) matched the ruling and needed no
  changes.

I3 (decision 9 pins and stale text)

Branch: pushed on top of I2b's commit (072cf2b6 -> 117cb49f). Build: 0 Warning(s), 0 Error(s). Targeted
classes (*FleetReadsAreBoundedByTheFleet*, *TimescaleSupport*, *RawChunkInterval*, *DarlingRetention*,
StartupCommandTimeoutTests, DocCommentHygiene*): 237 total, 0 failed, 21 skipped (live Postgres tests, no
rig per the brief). DocCommentHygieneTests run alone afterward: 77/77 passed (it catches a broken <see cref>, among other things).

The pin. FleetReadsAreBoundedByTheFleetTests.cs's EventWindowFloor_IsOneChunkBeforeTheWindow_AsNaiveUtc
(~line 119) asserted TimeSpan.FromDays(TimescaleSupport.ChunkIntervalDays) == EventWindowFloor.SkewAllowance.
Now asserts TimeSpan.FromDays(1) directly, with a comment saying why: since #4211 ruling decision 1,
SkewAllowance is its own clock-skew tolerance, not the raw chunk width, so a table
RawChunkIntervalPlanner narrows must be free to move its chunk_time_interval without dragging this pin
along.

Stale text in TimescaleSupport.cs. Reworded 7 spots - all comments or one log template, no behavior
change:

1. CompressScheduleInterval's doc (~line 90): "12 hours [is] the default on EVERY store shape" was wrong
   for a narrowed table - TimescaleDB's own uncapped default for a 12h or 6h chunk would compute 6h or 3h, not
   12h. Scoped the claim to day-or-wider chunks and named the narrowed-table exception.
2. CompressionPhaseMaxPerMinute's doc (~line 4559): "every hypertable's newest 1-day chunk becomes eligible
   at the same UTC midnight" is still literally true - every ladder rung (1-day, 12h, 6h) divides a day
   evenly, so midnight stays a shared boundary - but it was incomplete: a narrowed table also picks up
   boundaries of its own. Reworded to say both.
3. The "why a spread rather than one shared minute" paragraph (~line 4619): the same "CompressAfterDays and
   ChunkIntervalDays are both 1" framing, now stale since only CompressAfterDays stays fixed at 1 day.
   Reworded, and noted a narrowed table's extra boundaries argue for more spread, not less.
4. The "what sits on either side" paragraph (~line 6896): "twenty-three hours of the day those policies find
   nothing eligible" no longer holds for a narrowed table, which has real work at 2 or 4 hours instead of 1.
   Scoped to "a table still at the 1-day chunk ceiling", with a "fewer for a narrowed table" clause.
5. AggregateCompressionBandFirstHour's doc (~line 6938): "the 00: hour's compression band is the one that
   does a day's compressing" implied that is the only heavy hour; a narrowed table also does real work at
   noon (12h chunks) or every 6 hours. Reworded to say the 00: hour is the one every raw table shares, and
   named the extra hours a narrowed table adds that this constant does not try to clear. Raw and aggregate
   compression lock different chunks per the paragraph directly above this one, so this reads as a
   resource-contention question rather than a lock-safety gap - flagging it for the coordinator in case it is
   worth a closer look, not as a bug I found evidence of.
6. The RefreshOverrun log template (~line 10662): "every hypertable's newest {Days}d chunk becomes eligible
   at the same UTC midnight, so one tick a day carries a full day's rewrite while the other twenty-three find
   nothing" printed CompressAfterDays (which stays 1 day, correctly, per the brief's own rule), but the
   surrounding prose claimed a universal once-daily pattern. Reworded to describe the delay-after-close
   mechanism generally, scoped "one tick a day" to the 1-day ceiling, and named what a narrowed table does
   instead.
7. CompressionActivity.AssignedPhaseMinute's doc (~line 11205): referenced "the daily chunk close of a
   1-day raw chunk" to explain why the log in Lite overview: show Online/Offline status #6 does not fit an aggregate job. Reworded to match Lite overview: show Online/Offline status #6's new
   framing.

Checked, left alone:

• L1 (RollupBackfill.SliceWidth's Compression waits for the next fixed tick, so a just-closed chunk can sit uncompressed most of a day (81 GB observed); deadlock correlation to watch #1778 note) and the EventWindowFloor.SkewAllowance own-constant, the
  CompressAfterDays_IsAtLeastTheHourlyRefreshStartOffset pin, and the DarlingRetentionTests ceiling note
  the brief named - all already correct on this branch (Derive each raw hypertable's chunk interval from ingest (#4211) #4332).
• Every log template that prints CompressAfterDays (~lines 8187, 8908, 10586, and the #1778 parenthetical
  at ~8622) - left as is, because A (the compress delay) stays fixed at 1 day on every table regardless of
  narrowing, so those read correctly as written.
• L3's "7.0 GB against a predicted 6.8 GB" measurement. I searched RawChunkIntervalPlanner.cs and
  RawChunkIntervalReconciler.cs in full, plus the wiring in DarlingWorker.cs/DarlingConfig.cs, and
  grepped the whole repo for both figures. They appear only in issue Design: size raw chunk interval, compress_after and the hourly CAGG refresh window from ingest rate; a large store keeps 1–2 days of raw uncompressed and it outgrows RAM #4211's own body (the design writeup),
  not in any code comment. Neither file claims the rate model was validated more widely than that one reading,
  so nothing needed rewording there.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FVjn4PBJN71NQXdFo6ZxNQ

[erikdarlingdata]

Pushed three pre-merge fixes to this branch as one commit: da47a539 (new head da47a5392972261d4b5d8a47fd4bdfb0ea5b4587, was 117cb49f).

Item 1: invariant-culture reason text

Darling/PerformanceMonitor.Darling.Storage/RawChunkIntervalPlanner.cs — wrapped all seven Decision reasons in string.Create(CultureInfo.InvariantCulture, $"...") and added using System.Globalization;. Text unchanged otherwise. Final line numbers: 215, 221, 252, 260, 291, 298, 308.

Added using System.Globalization; to Darling/Darling.Tests/RawChunkIntervalPlannerTests.cs and one new fact, ReasonText_FormatsNumbersInvariantly_UnderACommaDecimalCulture (line 379), plus a private nested CultureScope : IDisposable copied from PgLogEventMetricsTests's pattern. The fact switches to de-DE, runs one narrowing Plan (budget 1,000,000 B, one table at 1,000,000 B/h, 24h) and one widening Plan (budget 2,400,000 B, one table at 100,000 B/h, 6h, last changed 3 days ago), and asserts each changing decision's Reason equals the exact invariant string:

• narrowing: "moved to 12 h: store-wide open-chunk bytes exceeded the 1,000,000 B budget (rate-ordered, 1,000,000 B/h)"
• widening: "moved up to 12 h: the store holds 1,200,000 B, under half the 2,400,000 B budget"

Revert-proofed by hand: dropped the string.Create wrap from the narrowing ("moved to") reason alone, rebuilt, ran the new fact by itself, and it failed:

Darling.Tests.RawChunkIntervalPlannerTests.ReasonText_FormatsNumbersInvariantly_UnderACommaDecimalCulture [FAIL]
  Assert.Equal() Failure: Strings differ
                                         ↓ (pos 57)
  Expected: ···"hunk bytes exceeded the 1,000,000 B budget (rate-o"···
  Actual:   ···"hunk bytes exceeded the 1.000.000 B budget (rate-o"···
                                         ↑ (pos 57)

That is de-DE's dot grouping leaking into the persisted reason, exactly the bug this item fixes. Restored the wrap, rebuilt, reran: passes again.

Item 2: ReconcileAsync's summary now says what it actually does on a per-table failure

Darling/PerformanceMonitor.Darling.Storage/RawChunkIntervalReconciler.cs, lines 110-118 (ReconcileAsync's <summary>). Confirmed by reading the method: each changing decision gets its own BeginTransactionAsync/CommitAsync pair (lines 170-202), with no catch anywhere in the foreach loop or around it. Reworded the summary from "Never throws for a planning or apply failure on one table" to state a failure throws out of the method, that table's transaction rolls back, tables already committed earlier in the same pass stay applied and recorded, and the remaining tables and the run row (inserted only after the loop, lines 211-215) wait for the next daily pass. Also names the actual catch: DarlingWorker.cs line 7804-7806 (catch (Exception ex) { _logger.LogError(ex, "raw chunk interval reconcile failed"); }). Comment only, no code change.

Item 3: V144's doc comment no longer claims a purge that doesn't exist

Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs, lines 2046-2058 (the V144Sql doc comment; V144Sql itself untouched). Grepped Darling.Storage and Darling.Service for both table names first: the only hits outside this doc comment and the CREATE TABLE/index statements are one SELECT ... MAX(changed_at) read (RawChunkIntervalReconciler.cs line 68) and two INSERTs (lines 98, 106) — nothing deletes from either table. Reworded to say neither table is purged, sized the growth (the run table gains one row per daily pass, about 365 a year; the history one row per move), and noted the history has to keep at least MinimumDaysBetweenMoves days anyway because the next pass reads MAX(changed_at) from it (RawChunkIntervalPlanner.cs line 83, value 1). Left the existing "outside CollectorCatalog.All" reasoning (self-telemetry about the store's own tuning, not collected monitoring data) as-is.

Build and test

Darling/Darling.Tests/Darling.Tests.csproj -c Debug: Build succeeded, 0 Warning(s), 0 Error(s).

• *RawChunkIntervalPlannerTests*: Total: 19, Errors: 0, Failed: 0, Skipped: 0
• *DocCommentHygiene*: Total: 77, Errors: 0, Failed: 0, Skipped: 0

No live tests run (none needed for these three items; CI runs the full suite). Installer.Tests not run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3