Skip to content

An alert-pass read outside AlertPassCommandTimeoutTests' six-file allowlist is untimed, and the pin cannot see it #2932

Description

@erikdarlingdata

An alert-pass store read sits outside AlertPassCommandTimeoutTests' six-file allowlist, so it is untimed and the pin is structurally unable to say so

Found by #2874's group B and verified independently on current dev. Two halves, filed together on purpose: fixing the site alone leaves the next one just as invisible.

The untimed site

Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs:3886, ReadLatestCpuAsync:

:3891    await using var connection = await _postgres!.OpenConnectionAsync(cancellationToken);
:3892    using var command = new NpgsqlCommand(@"
             SELECT sqlserver_cpu_utilization, other_process_cpu_utilization
             FROM cpu_utilization_stats
             WHERE server_id = $1
             ORDER BY sample_time DESC
             LIMIT 1", connection);

No CommandTimeout, so it inherits Npgsql's undocumented 30 s default — the defect class #2874 exists to sweep.

It is unambiguously an alert-pass read: its single caller is :2921, which feeds the result straight into the AlertServerSnapshot passed to engine.EvaluateServerAsync on the next line.

Why #2882's pin reports clean over it

Darling/Darling.Tests/AlertPassCommandTimeoutTests.cs:55 scopes the whole scan to an allowlist of six filenames:

DarlingAlertReadAdapter.cs, DarlingPostgresAlertReadAdapter.cs, PgAlertStateStore.cs,
PgMuteRuleStore.cs, PgAlertHistoryStore.cs, DarlingSelfAlertEvaluator.cs

DarlingWorker.cs is not among them, so the site is never read.

The pin already guards the direction that doesn't matter here. It asserts each of the six paths resolves, with a comment saying a renamed or moved alert-pass type must fail loudly "rather than silently shrinking the scan to the files that still resolve — an empty or partial sweep is how a guard starts reporting clean on code it no longer reads." That reasoning is right and the guard works: it catches a file leaving the list. There is nothing at all against a site that was never in it, which is the asymmetry that let this through.

Why an allowlist of filenames is the wrong scope, specifically

Group B established the general rule the hard way, twice in one PR, from opposite directions: .Service's budget regimes do not respect file boundaries. It over-counted by attributing sites to a regime by file, and it under-counted by excluding DarlingObservability.WriteAnalysisStateAsync because that site sits lexically inside a budgeted method while receiving the plain stoppingToken rather than the budgeted one. The reliable test is:

which cancellation token does this call site actually receive, and what re-runs it?

A filename allowlist encodes precisely the assumption that failed both times. DarlingWorker.cs is the clearest case in the project — it holds sites belonging to the startup regime, the sweep body, the analysis pass and the alert pass all at once.

Shape of the fix

Both halves, in one change:

  1. Give the site a deadline. It plausibly takes the alert pass's existing 10 s, but that number was derived from a 30 s interval and a 1,744.9 ms read, and this is a different query — a single-row ORDER BY … LIMIT 1 against cpu_utilization_stats. Confirm the derivation rather than inheriting the constant by proximity.
  2. Re-scope the pin so the next one cannot hide. Either derive the scan scope from the pass's actual call graph instead of a filename list, or keep the list and add a control that fails when an alert-pass store read exists outside it. The second is cheaper and is the one that would have caught this; the first is what stops the class.

Whatever lands, the pin needs a red-first proof that it fails on this site being untimed — a widened scope that still can't see DarlingWorker.cs would pass while guarding nothing, which is the same trap in a new coat.

Not verified

The site's real-world duration was not measured; there is no reported failure attached to it. cpu_utilization_stats is a hypertable and the query is a single-row read on a covered ordering, so the expectation is that it is fast and that this is latent exposure rather than an active problem — stated as an expectation, not a measurement.

Activity

  1. erikdarlingdata commented on Sep 4, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling

    Both halves are on dev as of 14919d58 (#2934).

    The site. DarlingWorker.ReadLatestCpuAsync takes DarlingAlertReadAdapter.AlertPassCommandTimeoutSeconds in the object-initializer form. One production line.

    The derivation, confirmed rather than inherited by proximity, which is what this issue asked for. The ceiling transfers exactly — 30 s s_alertSweepInterval, a property of the pass, and this site's only caller is EvaluateAlertsAsync under the plain stoppingToken with no enclosing CancelAfter. The floor is now measured for this query rather than borrowed from the 1,744.9 ms forced-plan read: read-only psql on the use2 store, running the SQL extracted from source and kept parameterized through PREPARE/EXECUTE, gives 11.5–17.2 ms over six round trips on the heaviest server, 13.9 ms EXPLAIN (ANALYZE) execution, and avg 12.5 ms / worst 50.6 ms across all 61 server_ids cold. So 10 s clears the measured fleet-wide worst by ~200×.

    That measurement also disproved something this issue and I both expected. The issue anticipated "a single-row read on a covered ordering… fast", stated as an expectation. It is fast, but not for that reason: the plan is Limit over a top-N Sort over a ChunkAppend of 29 compressed chunks, scanning 47,752 rows and 2,228 shared buffers to return one row, with Chunks excluded during startup: 0 because the query carries no sample_time predicate. Brute force, not an ordering. Left untouched here and worth its own issue — a sample_time > now() - interval '…' predicate would make it nearly free, and this runs per server per 30 s tick.

    The scope. Option one, not the filename list. Mixed-regime files are named by their entry point and the swept members are the transitive closure of calls out of EvaluateAlertsAsync, computed per run — so the pin's claim is literally "runs inside EvaluateAlertsAsync". The six filenames survive only as a whole-file sweep of the six dedicated files, where whole-file is strictly broader than any closure over them. The measured reason the cheaper option was not taken: the same scanner over all of DarlingWorker.cs finds 8 command sites, 6 untimed and every one owned by another budget, so a file-scoped guard would fail on correct code six times over.

    Red-first on this site, as required. Reverting the fix fails EveryAlertPassCommandReachedFromAnEntryPoint_SetsAnExplicitDeadline naming DarlingWorker.cs ReadLatestCpuAsync. And because a widened scope that still could not see the file would pass while guarding nothing, the walk itself is proven: cutting the call — leaving the site present and correctly timed — fails the closure's positive control, a mutation under which every occurrence count in the file is invariant.

    Review during the PR found two further holes in that walk, both fixed and both proven both directions: expression-bodied members were invisible (three were already live on this call graph, method-group referenced), and nested declarations were swept twice.

    Not closing #2874 — other regime groups remain there. One follow-up is filed as #2942: #2938's two-span call-site pattern needs applying to this pin, which could not be done here because ConstructionSpanFrom was not on dev yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions