pg_index_bloat has failed every run since #2618 shipped: the 200-index budget admits 286GB on target-a, and the timeout it dies on cannot be classified

[erikdarlingdata]

#2617 closed on rig evidence (160 indexes, 56ms) with its own real-cluster acceptance criteria left for "tomorrow's run": SUCCESS, 1,517 rows, ~200 measured, largest first. Nobody checked tomorrow's run. The store has now answered it eleven times.

Live state (pgmon store, read 2026-09-05)

Every pg_index_bloat run that has ever executed on the #2617 target (target-a) is status=ERROR, rows=0, message Exception while reading from stream, daily from the first-ever run at 2026-08-25T18:14:21Z (the run #2617 quotes) through 2026-09-04T19:00:27Z — eleven for eleven, including all nine runs after #2618's fix was hash-verified onto this box (#2617's close comment records the deploy at 2026-08-26T00:24Z; every subsequent daily run still fails identically). last_success: null.

The other 49 pgmon targets are all NO_PERMISSIONS (pgstatindex absent — pgstattuple never created), last_success: null. So the collector has produced zero rows fleet-wide, ever, and #2617's own closing worry stands: the #2561 exact-bloat claim remains unverified against any production cluster.

Cost on the failing target: the run is not free. Around the 2026-08-30 failure the entire sweep froze for 316 seconds (18:29:35 → 18:34:51), every 1-minute collector gapping, resuming the second the ERROR landed — and the daily run time drifts ~5 minutes later each day, which is a ~300-second slot being consumed daily. That is the collector blocking the sweep body for its full timeout before dying.

Mechanism, two layers — both verified in source on dev

1. The work budget bounds the wrong dimension for this target. PgIndexBloatCollector.cs:176-181 chose the 200-index count deliberately — "200 rather than a byte budget … A count is legible" — with MeasureCeilingBytes = 20GB (line 63) bounding any single index and largest-first ordering choosing which 200. On target-a that legibility trade fails: reading just the top 80 indexes from the store, 3 exceed the ceiling (correctly excluded, 71GB) and 77 sub-ceiling indexes sum 286GB, topping out at 19.68GB each — an independent read by another lane got the same shape (57 indexes / 265GB under a slightly different cut). The 200 largest sub-ceiling indexes are therefore hundreds of GB of pgstatindex page reads in one LEFT JOIN LATERAL statement. The 300-second CommandTimeoutSecondsOverride (line 190) cannot cover that; the count budget only bounds work when count correlates with bytes, and this target is the counterexample. #2617's own diagnosis — "a per-index ceiling and a total-work ceiling are different things and I only built the first" — happened a second time, one dimension up.

2. The timeout the fix added can never be classified. #2618's remedy 3 wanted "a classified timeout rather than a dropped connection". But the PostgreSQL classification arm in DarlingWorker.cs is:

catch (PostgresException ex) when (
    PostgresFaultOutcome(ex, collectorName, runtime.ConnectedDatabase) is { Status: not "ERROR" } outcome)

A client-side command timeout surfaces as NpgsqlException wrapping TimeoutException — not PostgresException, no SQLSTATE — so it can never enter PostgresFaultOutcome and falls to the general catch, which writes "ERROR", 0, 0, 0 with null phase blocks and the raw seven-word transport message. Two side effects worth naming: the hardcoded zeros make a 300-second death look like an instant one (it misled two independent investigations toward "connection dies at open"), and the message carries no collector/database/duration context at all — compare the authored text every classified skip on this fleet gets.

Fix shape, in this order (order matters)

1. Widen the PG fault classification to cover non-PostgresException faults — an authored CommandTimeout outcome ("client-side deadline of Ns expired; the statement was cancelled mid-read") and a ConnectionFatal one — without disturbing the existing SQLSTATE path that correctly degrades the other 49 targets to PERMISSIONS. Doing 2 first would leave the next timeout as unreadable as this one.
2. Bound bytes, not just count: cap the cycle at a cumulative sub-ceiling byte budget (keep largest-first; keep skipped_reason = not measured this cycle (work budget) for the remainder). The :176 comment's legibility argument can survive as "N indexes or B bytes, whichever first".
3. Record real elapsed time on the ERROR arm instead of literal zeros — the current shape destroys exactly the evidence that distinguishes a timeout from a connection failure.

The observation that decides

Next scheduled attempt on this target is ~2026-09-05T19:05Z (daily cadence drifting from 18:14 → 19:00 across the eleven runs — note it is NOT in the ~10:33Z block the other dailies use), and it will be the first attempt on the build deployed 2026-09-05 04:13Z. Expected under this diagnosis: identical failure, since nothing in that build touches the budget or the classification. A SUCCESS there falsifies layer 1 and leaves only layer 2 to fix.

Related: #2617 (closed on the rig), #2618 (the fix this measures), #2561 (the claim still unverified in production), #2994 (the same rollup-honesty family on this fleet), #2623.

[erikdarlingdata]

Closed by #3003, merged to dev as 7a25aab367.

What landed — five changes, not the three this issue scoped

1. A cumulative byte budget, gated on the LEFT JOIN LATERAL pgstatindex(...) join condition itself rather than on any skipped_reason path. It charges only sub-ceiling indexes, matching the LATERAL's own ceiling term — load-bearing, because the three over-ceiling indexes total 71 GB against a 20 GB budget and charging them would spend the whole budget on work never done. No starvation band: the gate is sum-including-current-row <= budget with budget == ceiling, so the first sub-ceiling index always fits.
2. Classification. The gap was not a missing SQLSTATE — it was PostgresFaultOutcome's parameter type, so no amount of SQLSTATE widening could ever have reached it. Status stays ERROR deliberately: a deadline that collected nothing belongs in error counts and the health band. ConnectionFatal stays with the general catch, which owns the reprobe.
3. Real elapsed time on both ERROR arms.
4. The wrong database was being named — both fault arms read ServerRuntime.ConnectedDatabase, stamped once at probe, so every fault reported the wrong database. Both now route through CollectorFaultDatabase.For(ex, fallback). This also fixed A missing-extension message says CREATE EXTENSION without naming which database, and the fleet has one installed in the wrong one #2638's extension message, which carried the same defect on all 49 degraded targets.
5. An Npgsql guard — the filter accepted any TimeoutException while the authored message claimed an Npgsql mechanism.

Items 4 and 5 came from review, not from this issue.

An amplification this issue understated

RunsPerDatabase is true, the per-database loop continues past a failure (failed++ / continue), and the rethrow fires only at failed == attempted. So the deadline is paid per database, not once per sweep. 316 s therefore implies one attempted database — it is not the ceiling. The widest clusters on this fleet reach 16 attempted databases, which is roughly 80 minutes of worst-case sweep freeze before this fix.

The evidence for the original diagnosis was destroyed at write time

The ERROR arm called LogCollectionAsync(..., "ERROR", 0, 0, 0, ...) and duration_ms is written as sqlMs + storageMs, so all eleven failing rows carry duration_ms = 0. The 316 s sweep freeze — inferred from other collectors' gaps — is the only surviving measurement. The 300 s timeout is established from source, not from a stored duration.

The six classified non-ERROR arms still write literal zeros and were deliberately left alone, because collector_cost and the health bands currently treat classified skips as free, and making them carry real duration changes what those surfaces report for degraded servers. That is a choice someone should make rather than inherit, and it is now its own work with the read-surface decision required before any write change.

Verification

Nine mutations, one at a time, anchor count and source hash each. A comment-only control moved both the source hash and the assembly hash while failing nothing, which is why neither is cited as evidence of behaviour. The shipped BuildQuery(ctx).Text was run verbatim against PostgreSQL 17.11 and 16.15 — Darling PostgreSQL tests exercises the store, and CI never parses collector SQL, so sum(...) FILTER (...) OVER (...) had no other route. The FILTER's negative control produced the predicted wrong answer, and the ROWS/RANGE tie was measured after reasoning got the direction backwards.

Not verified

Nothing ran against the affected target. The deciding observation is its first daily run after the next nightly reaches the box — a failure before then is the currently-deployed build, not this fix. Sweep-freeze detachment is deliberately out of scope: an independent change, and if the budget works there is nothing to detach around.

[erikdarlingdata]

Tonight's daily run fired at 19:05:33Z and failed — but this is a pre-fix control, not the deciding observation, and it should not be read as the budget gate failing.

Build provenance, checked before interpreting the run: the monitoring host's service binary is 3.6.0 with a write time of 2026-09-05T03:59:00Z. #3003 (the byte budget plus the classified deadline) merged at 2026-09-05T13:48:22Z, roughly ten hours after that binary was produced, and the most recent published nightly asset is dated 06:27:28Z — also before the merge. So the running build cannot contain the fix, whichever way the nightly was installed.

The run itself, from the store's collection log on the monitoring host:

when (UTC)	status	rows	message
09-05 19:05:33	ERROR	0	Exception while reading from stream

One row for the whole cycle rather than the per-database fan-out the collector normally writes, and the service log attributes it to the Aurora target this issue is about. That is the same shape the issue was filed on: the read dies against the target and the cycle stores nothing.

Worth separating from a second, unrelated condition in the same collector's history. The previous day's runs against other targets are PERMISSIONS with function public.pgstatindex(regclass) does not exist (SQLSTATE 42883) — that is pgstattuple not being installed on those clusters, not a deadline problem, and no budget change affects it. Only the ERROR/stream-exception rows belong to this issue.

So the deciding observation is still ahead, and it needs the first daily run after a build containing #3003 actually reaches the host. Two ways to read that run when it comes:

• the budget gate holds → the run completes inside the deadline and writes rows, with the largest indexes measured and the remainder carrying the gate's own explanatory note;
• the gate is insufficient → it fails again, and the classified deadline from the same PR should now say which deadline died rather than surfacing as an untyped stream exception, which is itself the diagnostic improvement.

The second reading is only available on the new build, which is another reason not to score tonight's failure against the fix.