Store host profile model + --check-settings verb + --validate-config registry fix (#4214)

[erikdarlingdata]

Part of #4214.

Why

The managed store already sizes PostgreSQL from the host. None of that was visible from the product. Answering "is this store sized right" needed a remote shell, pg_settings.sourceline, and hand-counting conf blocks. This lane builds part 1: the host/store profile model, per-setting source attribution and verdicts, and the --check-settings CLI verb. It leaves the get_store_host MCP read, the web Store/Host panel, and cloud identity via IMDS to a later lane, per the brief.

What changes

New file Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs:

• Cross-platform host facts: OS platform, containerization, CPU count (Environment.ProcessorCount), RAM, and the data volume (size, free space, filesystem, all via DriveInfo).
• RAM on Windows reuses the exact GlobalMemoryStatusEx read DarlingManagedPostgres sizes itself from. It is extracted to TryReadWindowsPhysicalMemoryBytes, so there is no second P/Invoke of the same API. RAM on Linux reads /proc/meminfo MemTotal, plus cgroup v2 memory.max with a fallback to v1 memory.limit_in_bytes. Both reads go through pure functions over already-read file text, so a Windows test covers every branch: meminfo missing, meminfo unparseable, v2's max, and v1's near-long.MaxValue sentinel.
• Store facts: PostgreSQL and TimescaleDB versions, live store size, lifetime buffer hit ratio, lifetime temp bytes, and the Design: size raw chunk interval, compress_after and the hourly CAGG refresh window from ingest rate; a large store keeps 1–2 days of raw uncompressed and it outgrows RAM #4211 metric. That metric is the total bytes and chunk count of every hypertable chunk timescaledb_information.chunks reports as not yet compressed, summed with pg_total_relation_size.
• Eight settings checked: shared_buffers, work_mem, effective_cache_size, maintenance_work_mem, timescaledb.max_background_workers, max_worker_processes, max_connections, max_wal_size. Each reuses the product's own DeriveMemorySettings, DeriveWorkerSettings, or DeriveWalSettings for "the value this host would derive right now," so no formula is duplicated. Mid-lane I found that max_wal_size is not the fixed 4GB literal the v4 block suggests: the v12 block supersedes it with a value derived from free disk space. The check follows DeriveWalSettings, not a constant.
• Source attribution never escalates privileges. For a managed store it reads postgresql.conf's managed blocks and postgresql.auto.conf directly from disk, reusing DarlingManagedPostgres.ReadConfAssignments, which already follows PostgreSQL's own include and precedence order. It does not read pg_settings.sourcefile, because that column needs superuser or pg_read_all_settings, a right the least-privilege mcp role does not have (see DarlingStoreMetricsReader.JobExecutionLoggingSql's remarks, which this lane's code comments point back to). A generic line-vs-managed-block scan walks every marker in a new DarlingManagedPostgres.AllManagedConfMarkers list, so a future version block needs no update here. A bring-your-own store gets not-managed for every setting, plus pg_settings.source.
• Four verdicts, exactly as ruled: matches, stale-after-hardware-change, operator-override, not-managed.

--check-settings CLI verb (DarlingCliCommands.CheckSettingsAsync, wired in Program.cs). It prints the profile and the per-setting table; --json selects the automation form. It returns three distinct exit codes, kept separate per the ruling so a caller can tell a sizing problem from a connectivity problem: 1 for a config error, 2 for an unreachable store, 3 for one or more stale-after-hardware-change verdicts.

--validate-config fix, from the issue's comment. It now probes the store's registry (config_monitored_servers, read via the newly-internal StoreConfigProvider.ReadMonitoredServersAsync) when the store is reachable. A darling.json server absent from the registry now gets a warning instead of failing the run. When the store cannot be reached, it falls back to darling.json's own list and says so on stdout. This is a behavior change: before this PR, --validate-config and --test-connection always probed the file's list and exited 1 on a stale entry. That is the exact false failure the issue's comment reported on a 42-server production store.

Small in-lane fixes, both mechanical and in files this PR already touches:

• Extracted DarlingManagedPostgres.TargetMaxConnections = 200, so the v4 block and the new check share one source instead of the literal 200 living in two places.
• Made FindHardwareSizingBlockEnd and ReadConfAssignments reachable (internal), so the new file can reuse them instead of re-implementing them.
• Added DarlingStoreHostProfile.ComputeEffectiveMemoryLimitBytes to TsqlConventionGuardTests.KnownTruncatedRanges (the member-range census correctly flagged it as a new expression-bodied one-liner, the same shape as the other entries already in that list).

This lane (H1b) added, on top of H1's core:

• Darling/Darling.Tests/DarlingStoreHostProfileLiveTests.cs: one live test that builds the profile against a real managed data directory and a real running server, proving all four verdicts resolve correctly end to end — not just the pure classification. Matches comes from a real v8-marker block plus pg_reload_conf() for three sighup/user-context settings; stale-after-hardware-change needs no live change at all, because PostgreSQL's untouched max_connections default (100) already disagrees with today's derivation (200) once a managed block claims it; operator-override comes from a real ALTER SYSTEM; not-managed comes from a bring-your-own config against the same connection. Restores the conf files (and reloads) in finally via LiveStoreCleanup.
• Darling/Darling.Tests/DarlingCliCommandsHostCheckTests.cs: an exit-code test per --check-settings outcome (config parse error, unreachable store, a bring-your-own store where nothing is ever stale in both text and --json, a managed store with one stale setting) and per --validate-config registry path (a seed-only server warns and exits 0, a registry server that fails still fails, an unreachable store falls back to the file's list and says so). Proved live that the stale-settings exit-code test fails if its branch is broken (flipped the Any(... StaleAfterHardwareChange) check to Matches, watched it fail, reverted) — no diff remains from that step.
• README: a new "Check Host/Store Sizing Settings" section next to "Validate the Config", and a paragraph on --validate-config's registry-vs-file behavior.
• Fixed LivePostgresCollectionHygieneTests flagging the new CLI test class for reaching the shared store without [Collection("live-postgres")].

EXPLAIN (ANALYZE, BUFFERS) on the uncompressed-chunks read (UncompressedChunkSizeSql), seeded on the rig's probe database: 3 hypertables, 10 daily chunks each (30 chunks total), 13 compressed and 17 left uncompressed.

Aggregate (actual time=19.675..19.682 rows=1 loops=1)
  Buffers: shared hit=508
  -> Subquery Scan on finalq (actual rows=17 loops=1)   [the catalog read itself: 28 buffers]
Planning: Buffers: shared hit=1016, Planning Time: 14.437 ms
Execution Time: 19.853 ms

All 508 buffer hits were cache hits (no disk reads). The catalog view itself (timescaledb_information.chunks) costs 28 of those; the rest comes from pg_total_relation_size() resolving each of the 17 uncompressed chunks' size — relation metadata (extent/fork sizes), never a scan of chunk row data. 20ms total for 30 chunks matches the code's own justification for paying this live rather than reading the hourly-sweep figure: a one-shot CLI/once-per-start cost, not a hot path. No bound was added; the read does not scale into chunk contents, only chunk count.

CHANGELOG entry

SECTION: Added
ENTRY:

• --check-settings reports whether a managed store's sizing still matches this host ([Store host profile model + --check-settings verb + --validate-config registry fix (#4214) #4271]) - a new CLI verb prints the host and store facts (RAM, CPUs, PostgreSQL/TimescaleDB versions, store size, buffer hit ratio, uncompressed TimescaleDB chunk bytes against RAM) and a verdict for each of the eight sizing-relevant settings: matches, stale after a hardware change, operator override, or not managed. --json prints the same facts for scripting. Separate exit codes for a config error, an unreachable store, and a stale setting let an install or upgrade script gate on the outcome that matters to it.
  SECTION: Fixed
  ENTRY:
• --validate-config (--test-connection) no longer fails a healthy store over a stale entry in darling.json ([Store host profile model + --check-settings verb + --validate-config registry fix (#4214) #4271]) - once the store is reachable, this now probes its own registry of monitored servers instead of the config file's first-bootstrap seed list. A server that is only in the file (never registered, or since removed from the registry) is now a warning, not a failure. A store that cannot be reached still falls back to the file's list, and says so on the console. This closes the false failure reported on a 42-server production store whose darling.json still listed two long-removed servers.
  REF:
  [Store host profile model + --check-settings verb + --validate-config registry fix (#4214) #4271]: Store host profile model + --check-settings verb + --validate-config registry fix (#4214) #4271

Test plan

• dotnet build Darling/PerformanceMonitor.Darling.Service/PerformanceMonitor.Darling.Service.csproj: 0 Warning(s), 0 Error(s).
• dotnet build Darling/Darling.Tests/Darling.Tests.csproj: 0 Warning(s), 0 Error(s).
• DarlingStoreHostProfileTests, 43 tests, all passing (pure classification, every RAM/cgroup parse branch, verdict classification, pg_settings unit normalization, CLI verb recognition).
• DarlingStoreHostProfileLiveTests (new, gated on DARLING_TEST_PG + DARLING_TEST_PGRUNTIME): 1 test, all four verdicts proved end to end against the rig. Passing.
• DarlingCliCommandsHostCheckTests (new): 7 tests, all --check-settings and --validate-config exit-code paths. Passing. Proved live that the stale-settings test fails against a broken branch, then reverted.
• EXPLAIN (ANALYZE, BUFFERS) run on seeded hypertables/chunks on probe — numbers above; no bound needed.
• README updated (--check-settings section, --validate-config registry behavior); plain-English checker run once, the one genuinely long new sentence split.
• git merge origin/dev (clean, no conflicts), darlingtest dropped/recreated, full Darling suite run once: 13944 total, 0 errors, 7 failed, 29 skipped. Two failures were this branch's: a hygiene-collection miss and a census list needing the new pure function added (both fixed, files changed 0 net diff besides the additions; re-ran the affected classes clean, 76/76 passing). The other 5 (SerialLoopStoreSizeSourceTests, EventWindowedReadsAreBoundedLivePostgresTests, CaptureDownChunkOrderTests, ServerListAndSummaryPlanShapeTests, LiveCleanupBatchTests) were called plan-shape/chunk-count assertions in files this PR never touches. That call was wrong for the first of the five; see below.
• SerialLoopStoreSizeSourceTests is this branch's own defect, not pre-existing. Re-ran it alone on a freshly created darlingtest. PgDatabaseSizeRunsOnlyWhereItsCostHasABudget failed because DarlingStoreHostProfile.StoreSizeSql (pg_database_size) is a real fourth call site the pin's census did not know about yet, not a false positive. It is a genuine fourth regime: --check-settings's own one-shot, user-invoked read under CliStoreReadSeconds (10s), never the serial loop. It was documented and added to the pin's allow-list with that reasoning, mirroring the existing three entries' style, not weakened. 20/20 passing after the fix.
• Startup logging is now built (ruling 9). DarlingStoreHostProfile.GatherStartupProfileAsync gathers only the cheap parts: host facts, the live pg_settings read, the managed conf files. It never calls GatherStoreFactsAsync, so the store-size/chunk-total reads that scale with the store stay out of the startup path entirely. DarlingWorker.LogStoreHostProfileAsync calls it once, ahead of the collection loop's while (pinned by source order). It runs under its own new deadline, ServiceCommandDeadlines.StartupHostProfileSeconds (CliStoreReadSeconds + 5 = 15s, documented as its own regime rather than reusing BootstrapSeconds), logs the profile at info, warns once per stale-after-hardware-change setting, and never throws (pinned: no throw anywhere in the method's source). New file StartupHostProfileLogTests.cs has 4 tests: the log-line shape against a fixture whose Store is filled with sentinel values that must never print, the store-facts exclusion, the never-throws failure path, and the call-site ordering. Plus 1 new live test in DarlingStoreHostProfileLiveTests.cs proving the exclusion holds against a real running server. All passing.
• Ran every class in every file this lane (H1c) touched, plus every pin named in this lane's brief, together: StartupCommandTimeoutTests, StorageCommandTimeoutTests, LivePostgresCollectionHygieneTests, TsqlConventionGuardTests, DocCommentHygiene, DarlingStoreHostProfileTests, DarlingCliCommandsHostCheckTests. 185 total, 0 failed. SerialLoopStoreSizeSourceTests (20), StartupHostProfileLogTests (4) and DarlingStoreHostProfileLiveTests (2) ran separately above, all passing.
• git merge origin/dev again for this lane's own work (clean, no conflicts; dev had since touched EventWindowedReadsAreBoundedLivePostgresTests.cs and PlanChunkScans.cs), darlingtest dropped/recreated. Then ran H1b's 4 other remaining-failure classes alone, skipping EventWindowedReadsAreBoundedLivePostgresTests (another lane owns it): SerialLoopStoreSizeSourceTests, CaptureDownChunkOrderTests, ServerListAndSummaryPlanShapeTests, LiveCleanupBatchTests. 29 total, 0 failed. None of H1b's other 4 reproduced after this merge, so there was nothing left to check against dev's CI for them.
• Full suite not re-run by this lane. H1b ran it after merging dev (13944 total, see above), the targeted classes above cover every file this lane touched plus every named pin, and CI runs the full suite again on push.

What the coordinator should double-check

1. Startup logging is built and pinned; see the test plan above. Nothing is outstanding from the original brief.
2. SerialLoopStoreSizeSourceTests's new fourth allow-list entry (DarlingStoreHostProfile.cs, in the test file). Please confirm the reasoning holds: --check-settings is genuinely this call site's only caller, and startup deliberately never reaches it.
3. The rig (C:\GitHub\worktrees\rig-th, port 55983) is stopped. It now also carries a darling database (managed-mode connections always target that fixed name) and a probe database with 3 seeded/partly-compressed hypertables, left in place for reuse. Both are throwaway rig state, not customer data.
4. One judgment call worth checking: for a bring-your-own store, "data volume" reports the volume under the service's own AppContext.BaseDirectory, not a store data directory. A BYO store's data directory is not something this process can know, and the store may be remote. Documented in DarlingStoreHostProfile.ResolveVolumeAnchor's comment as the smallest reasonable call, flagged for revisiting once part 2 builds the web panel.
5. ServiceCommandDeadlines.StartupHostProfileSeconds is a new constant (15s = CliStoreReadSeconds + 5). Please confirm the derivation in its doc comment reads right: it backstops GatherSettingProfilesAsync's own CliStoreReadSeconds-bound query rather than setting an independent number, the same relationship CliBudgetBackstopSeconds already has to CliStoreReadSeconds.

No monitored or production server was touched. Installer.Tests was not run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

[erikdarlingdata]

Fix report

Two CI failures fixed in commit 9f6f98e, pushed to feature/4214-store-host-check.

Failure 1: LiveCleanupConversionRatchetTests (6 offenders)

Diagnosis confirmed. DarlingCliCommandsHostCheckTests.cs used root.Delete(recursive: true) (a DirectoryInfo instance method) in 7 finally blocks. The ratchet excludes only blocks containing File., Directory., or .Kill(. The instance call contains none of these tokens, so all 6 blocks that lacked any other exclusion were flagged.

Fix: changed all 7 instances to Directory.Delete(root.FullName, recursive: true). The static call contains Directory., which satisfies the ratchet's file-teardown exclusion. The 7th instance (in the CheckSettingsAsync_ManagedStoreWithAStaleBlock finally) already had a File.WriteAllText call, so it was not an offender — changing it to the static form is consistent but cosmetic for that block.

Result: LiveCleanupConversionRatchetTests — Total: 15, Failed: 0.

Failure 2: DarlingMissingCredentialMessageTests — Assert.Equal(6) → 7

Diagnosis confirmed. The test NoVerbStillCarriesItsOwnFirstRunAdvice counts calls to DarlingStoreBootstrapEvidence.MissingStoreCredentialMessage( in the CLI source. The new verb added by #4214 added a seventh call site, raising the actual count to 7.

Fix: updated the expected value from 6 to 7 at DarlingCliCommandsTests.cs:1667.

Result: DarlingMissingCredentialMessageTests — Total: 16, Failed: 0.

Build

dotnet build Darling/Darling.Tests/Darling.Tests.csproj -c Debug — 0 Error(s), 0 Warning(s).