Skip to content

Test: the managed stale-block --check-settings test migrates its own darling store, so it passes on a bare rig - #4691

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/darling-tests-bare-rig-and-readcount
Sep 29, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
fix/darling-tests-bare-rig-and-readcount

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

No issue filed; both defects surfaced in the full-suite run for PR #4681 and were ruled test defects. This PR fixes item 1 only. Item 2 is parked below with the diagnosis so far (see "Item 2: not done").

Why

DarlingCliCommandsHostCheckTests.CheckSettingsAsync_ManagedStoreWithAStaleBlock_ReturnsStaleSettingsExitCode_Gated failed on a bare rig with 42P01: relation "collect.managed_conf_verdicts" does not exist.

The managed connect path always targets a database named darling. The test created an empty darling database when none existed, then --check-settings read collect.managed_conf_verdicts from it. CI never showed this because darling-pg sets DARLING_TEST_PG with Database=darling (build.yml, "Run Darling PG tests"), and the live-postgres collection fixture migrates that very database before the first test runs. On any rig whose suite database has another name (mine: darlingtest), nothing migrated darling, so the test passed only after some other test had.

What changes

One test file, Darling/Darling.Tests/DarlingCliCommandsHostCheckTests.cs: the test now runs PgMigrations.MigrateAsync on darling itself, as the first statement inside the existing try, so the finally still drops a database the test created even if the migration throws. MigrateAsync is idempotent (the fixture makes the same call), so it is a no-op where the store is already established. The connection uses Pooling=false so it cannot pin the database against the DROP. No product code changed. Every assertion and the existing DARLING_TEST_PG skip are unchanged. No new skip was added.

Test plan

Rig: PostgreSQL 18.6 from the repo's pg-runtime.zip, port 5573, timezone/log_timezone UTC, shared_preload_libraries = 'timescaledb,pg_stat_statements' (what darling-pg sets at build.yml line 1003; with only timescaledb preloaded, item 2's test would skip), workers 74 / 85 as CI. Suite database darlingtest; no darling database existed before any run below.

  • RED, before the fix, on the bare rig: -method "*CheckSettingsAsync_ManagedStoreWithAStaleBlock*" failed with Npgsql.PostgresException : 42P01: relation "collect.managed_conf_verdicts" does not exist.
  • GREEN, after the fix, same bare rig (darling absent before and after): -class "*DarlingCliCommandsHostCheckTests*" Total 7, Failed 0, Skipped 0. The test ran on its own in the first pass, so it depends on no other test's setup or on order.
  • Assertion still bites: planted a product break (CheckSettingsAsync returning Ok for the stale arm). The test failed with Expected: 3, Actual: 0. Reverted; rebuilt.
  • Build: 0 Warning(s), 0 Error(s).
  • Pin classes touching live-class hygiene (LivePostgresCollectionHygieneTests, LiveCleanupConversionRatchetTests): 103 tests, 0 failed.
  • Full Darling.Tests suite: not run locally; CI runs it.
  • Item 2: not started as code (see below).

CI: which job runs each test

Both tests run in the darling-pg job (build.yml, windows-latest, 3 shards by class-name hash) and neither is skipped there:

  • Item 1's test: skipped only when DARLING_TEST_PG is empty. darling-pg sets it (Port=5541;Database=darling).
  • Item 2's test (WaitRateTileReadCountLiveTests): skipped only when pg_stat_statements is missing from shared_preload_libraries or TimescaleDB is missing. darling-pg preloads timescaledb,pg_stat_statements.

Item 2: not in this PR (diagnosis so far)

WaitRateTileReadCountLiveTests.Waits_DetectAnomaliesAsync_ReadsWaitRateTileWindowSqlOnce_NotTwice. Baseline on this UTC rig: passes alone (Total 2 with item 1's test, 1 failed = item 1's). I did not reproduce the full-suite failure. PR #4681's report says only "passes when its class is re-run alone on a freshly created database (load-sensitive under the full run)"; it records no calls value. Getting that value (0 or 2) comes first, because it separates the two stories:

  • Ruled out by reading the code:
    • Other tests cannot add calls: the reset and the count are both scoped to the test's private scratch database oid.
    • The one blind cluster-wide pg_stat_statements_reset() is StoreStatementStatsLiveTests.cs:166, and it runs on that test's own managed cluster on a free port, not the shared rig.
    • The other resets (OverviewFleetHealthSingleFlightLiveTests, QueryStoreTrendRoutingCachedLiveTests, StoreSizeCacheLiveTests) are scoped to their own database oid.
    • PgAnomalyDetector.DetectWaitAnomalies reads WaitRateTileWindowSql once (PgAnomalyDetector.cs:671) on its own connection; no retry logic and no static cache there (PgBaselineProvider's cache is per instance).
  • Lead hypothesis, unproven: eviction. On this rig (and in darling-pg, which sets neither) pg_stat_statements.max is 5000 and track_utility is on. Every parallel own-store class replays the migration ladder in its own scratch database, which creates thousands of distinct entries and repeated dealloc passes. A calls = 1 row evicted between the detector run and the count sums to 0. A value of 2 would contradict this story and would need a reconcile before going further.
  • To name the interferers: run -class *WaitRateTileReadCountLiveTests* together with migration-heavy own-store classes (candidates above) while watching pg_stat_statements_info.dealloc on a probe connection.
  • Proposed fix (not applied): stop counting in cluster-shared pg_stat_statements. Count on the test's own NpgsqlDataSource, built with NpgsqlDataSourceBuilder and a test-local ILoggerFactory (or an ActivityListener on Npgsql's ActivitySource filtered by db.name = the scratch database, since a listener is process-wide), counting executed command texts that contain both peak_ms_per_sec and v_wait_stats. Keep Assert.Equal(1, calls) literally. RED: plant a second ExecuteReader of WaitRateTileWindowSql in DetectWaitAnomalies and show the test fails. The preload-skip guard becomes dead and would be removed and named as a guard removal; the class doc comment would be rewritten to match. Still to verify: that Npgsql's command log carries the SQL text without parameter logging.

CHANGELOG

None: test-only change.

Notes

  • Item 2 is left for a separate change.
  • A rig that matches darling-pg preloads pg_stat_statements too.

…ing store it creates

CheckSettingsAsync_ManagedStoreWithAStaleBlock_ReturnsStaleSettingsExitCode_Gated created an
empty "darling" database when none existed, then read collect.managed_conf_verdicts from it and
died with 42P01 on any rig whose DARLING_TEST_PG database is not itself named darling. CI hid it:
darling-pg's DARLING_TEST_PG names Database=darling, which the live-postgres collection fixture
migrates before the first test. The test now runs PgMigrations.MigrateAsync on darling itself (idempotent,
Pooling=false so the finally's DROP is not pinned), so it no longer depends on another test or on order.
Every assertion is unchanged. Surfaced on PR #4681.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKkGR7ZBZavtGbVYcK58GC
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 29, 2026 00:28
@erikdarlingdata
erikdarlingdata merged commit 872e7c9 into dev Sep 29, 2026
19 of 20 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/darling-tests-bare-rig-and-readcount branch September 29, 2026 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant