Wait for the baseline to land instead of assuming one sweep seeded it

[erikdarlingdata]

Fixes #2374. Root-caused rather than retried — the mechanism is documented in the product's own comments, which is what made it findable.

What actually happens

The baseline seed rides GetDatabaseStateDeviationsAsync's best-effort maintenance block. That block takes the write connection with a 5-second lock acquisition and, on TimeoutException, skips seed / #2189 heal / #2203 forget / prune and runs the deviation read anyway. That is deliberate and well argued in place — skipping is the only lossless option when archival holds the lock, since letting it escape would either crash the sweep or read as "every database recovered".

The lock is private static readonly — process-wide — and xunit runs test classes in parallel. The method's own comment already records this exact interaction biting once:

That is how this surfaced — an unrelated server-tags test timed out acquiring the write lock while this method held the read lock, on a static lock shared by the whole process.

So a bare sweep can return having written no expectation at all. A test that then acts as though it has a baseline is asserting on a precondition it never established, and it fails far from the cause: the deviation read still succeeds and simply has nothing to deviate FROM. That is the whole of the failure — Assert.Single() on an empty collection, three lines after a sweep that quietly did nothing.

The fix

SweepUntilAsync already solved this one step later, for the #2189 heal, with exactly this reasoning and in 23 tests' company. BaselineAsync is that helper for the seed, applied at the six sites that establish a first baseline.

It settles on the recorded state, not on row count, and that detail is load-bearing: GetDatabaseStateExpectationsAsync LEFT JOINs the newest snapshot, so every current database comes back whether or not anything was ever seeded for it. An unseeded database is a present row with an empty expected state — "did I get rows?" is always true and never the question. Re-reading it is a genuine retry rather than a re-check, because it re-runs the seed side effect itself ("reuse the seed/prune side-effect").

The test that gained the most

CriticalAndTransientFirstObservations_StayPending was passing partly by luck. A skipped maintenance block seeds nothing, which is indistinguishable from all three databases being correctly pending — so its two "" assertions passed for entirely the wrong reason and only the Healthy assertion could report the problem. It now settles on Healthy first, which proves the seed ran before asking what it declined to learn.

Why this cannot mask a regression

SweepUntilAsync's property, unchanged: a seed that is genuinely broken never records the state, every cycle runs, and the caller's own assertion fails on the same empty result it sees today. No tolerance, no tuned threshold — the semantics are "eventually", and the count only needs to exceed one.

What I deliberately did not do

Not making the lock per-instance. It looks like the obvious fix and it is wrong: production creates many DuckDbInitializer instances against the same App.DatabasePath — MainWindow, DatabaseStateOverridesWindow, and fourteen in DuckDbAlertHistoryStore — so the static lock is exactly what makes those mutually exclusive. Narrowing it would trade a flaky test for a real data race.

That does leave the observation from the issue standing: SharedDuckDbFixture chose IClassFixture to keep "xUnit's cross-class parallelism intact", and the process-wide lock hands much of that back regardless. Worth revisiting on its own terms — it is the likely reason Lite.Tests takes 500 s — but it is a separate change from stopping this test from flaking.

[claude]

BaselineAsync doesn't actually give the same "cannot mask a regression" guarantee that its own doc comment claims by reference to SweepUntilAsync.

SweepUntilAsync returns its last result unconditionally, and every caller asserts directly on that return value (Assert.Empty(await SweepUntilAsync(...))) — so if the condition never settles, the failure surfaces immediately, at the call site, with a message about the actual final state.

BaselineAsync returns void/Task. When the loop exhausts cycles without finding the expected row, it just falls through and returns normally — no assertion, no exception, nothing indicating the baseline never landed. The test then keeps going as if the precondition held, and if a genuine regression is under test, the failure (if it occurs at all) shows up several lines later on an unrelated assertion, likely with a confusing message rather than one that points at the baseline step. That's a milder version of exactly the "asserting a precondition it never established" bug #2374 was fixing — it's just far less likely to trigger now (needs 5 consecutive best-effort skips instead of 1).

Worth having this return a bool (or the rows) so callers can assert on it directly the same way SweepUntilAsync's callers do, e.g.:

private static async Task<bool> BaselineAsync(
    LocalDataService service, string database, string expectedState, int cycles = 5)
{
    for (var attempt = 0; attempt < cycles; attempt++)
    {
        var rows = await service.GetDatabaseStateExpectationsAsync(ServerId);
        if (rows.Any(r => r.DatabaseName == database && r.ExpectedState == expectedState))
        {
            return true;
        }
    }
    return false;
}

with callers doing Assert.True(await BaselineAsync(service, "App", "ONLINE")).

[claude]

Reviewed. This is a test-only change (Lite.Tests/DatabaseStateExpectedStoreTests.cs, no T-SQL, no Lite/Darling parity concern — Darling's alert-deviation path doesn't share Lite's DuckDB static-lock/timeout race, so nothing to mirror there).

Root-cause analysis and the BaselineAsync fix are sound: converting the six sites that establish a first baseline to poll GetDatabaseStateExpectationsAsync instead of trusting a single bare sweep correctly addresses #2374, and settling on recorded state rather than row count is the right check given the LEFT JOIN in GetDatabaseStateExpectationsAsync.

Left one inline finding: BaselineAsync's doc comment claims the same "cannot mask a regression" property as SweepUntilAsync, but unlike SweepUntilAsync it doesn't return anything for the caller to assert on — if all cycles attempts are exhausted it just returns silently, so a genuine regression surfaces later on an unrelated, less diagnostic assertion instead of failing cleanly at the baseline step. Suggested returning a bool so callers can assert directly, matching SweepUntilAsync's pattern.