Skip to content

Write the database_name column three collectors declared but never wrote - #2622

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/2599-extension-payload-arity
Aug 26, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
fix/2599-extension-payload-arity

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Found by standing up a self-hosted PostgreSQL rig — 40 tables, 160 indexes, all five instrument extensions loaded — and pointing the actual service at it. First time these collectors have had real data to write.

The bug

pg_extension_availability, pg_column_stats and pg_index_bloat all gained a database_name column in v95 (#2599). I updated the SQL, the row record and the readers, and missed WritePayload in all three:

pg_extension_availability | ERROR | Collector wrote 6 payload values but declares 7 payload columns

Every row they produce is rejected at EndPayload.

Why the existing runtime check wasn't enough

It only fires when a collector runs. All three are daily, and on the Aurora fleet two of them return zero rows for unrelated reasons — a missing grant on pg_stats, a missing extension for pgstatindex. So the store simply had nothing from them, which is indistinguishable from a quiet server.

It survived three schema versions that way. pg_extension_availability was the one collector actually broken on the fleet, and its last successful run predated the upgrade that broke it — so even the dogfood pass this morning saw a stale SUCCESS and moved on.

Pinned at build time now

A counting writer runs every collector's WritePayload against a default row and asserts the write count equals the declared column count. 50 of 66 exercised, all matching.

Collectors whose default row isn't valid — the delta-based SQL Server ones dereference fields it leaves null — count as uncovered rather than as failures, and a coverage assertion keeps that from quietly swallowing the catalog. A guard that skips most of what it claims to check is worse than none.

The columns and the writes are two halves of one statement about the table, and nothing but a test holds them together.

Worth noting

This is the second defect in two days that only a real instance could surface — after #2617, where a collector had never returned a row because I bounded each index and not the cycle. Both were invisible to CI, to a two-table container, and to the Aurora fleet. The rig is cheap and I'd keep it.

pg_extension_availability, pg_column_stats and pg_index_bloat all gained
a database_name COLUMN in v95 (#2599). I updated the SQL, the row
record and the readers, and missed WritePayload in all three - so every
row they produce is rejected at EndPayload with "wrote 6 payload values
but declares 7 payload columns".

The runtime check existed and was not enough. It only fires when the
collector RUNS; all three are daily, and on Aurora two of them return
zero rows for unrelated reasons - a missing grant and a missing
extension. So the store had nothing from them, which is indistinguishable
from a quiet server. It survived three schema versions that way.

Found by pointing the service at a self-hosted PostgreSQL with 40 tables
and 160 indexes, which is the first time these collectors had real data
to write.

Now pinned at build time across the whole catalog: a counting writer
runs every collector's WritePayload against a default row and asserts
the write count equals the declared column count. Collectors whose
default row is not valid - the delta-based SQL Server ones dereference
fields it leaves null - count as uncovered rather than as failures, and
a coverage assertion keeps that from quietly swallowing the catalog.
50 of 66 exercised today.

The columns and the writes are two halves of one statement about the
table, and nothing but a test holds them together.

Fixes the v95 regression in #2599.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

/* The guard is only worth having if it actually covered the catalog. */
Assert.True(
skipped.Count < CollectorCatalog.All.Count / 4,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This coverage threshold looks like it's sitting exactly on its own failure boundary.

CollectorCatalog.All.Count is 66 today. Integer division makes 66 / 4 == 16. The PR description states this probe currently exercises "50 of 66" collectors — i.e. skipped.Count == 16. That makes the assertion 16 < 16, which is false.

If that count is accurate, this test fails the moment it runs, on this very PR. Either:

  • the "50 of 66" figure in the PR description is stale/approximate and the real skip count is ≤15 (worth double-checking against the actual CI run rather than trusting the prose), or
  • the boundary needs to be <= (or a less exact ratio), since < a truncated-integer-division threshold is fragile — the next collector added to the catalog can flip this pass/fail with no code change to the guard itself, purely from Count / 4 re-truncating.

Worth confirming against the real CI numbers before merge; a coverage guard that flakes on its own introduction undermines the "no silent gaps" goal it's meant to enforce.

@claude

claude Bot commented Aug 26, 2026

Copy link
Copy Markdown

Reviewed the diff. Summary:

Core fix (PgColumnStatsCollector, PgExtensionAvailabilityCollector, PgIndexBloatCollector) — correct. I cross-checked all three collectors' Row field order against PayloadColumns and the ReadAsync ordinal mapping; .Value(row.DatabaseName) is inserted in the right position in all three, matching what ReadAsync already reads at ordinal 0 (from #2599) and what Darling's storage readers (e.g. DarlingPgExtensionAvailabilityReader) already expect. No SQL text changed, so no T-SQL style concerns apply here.

Lite/Darling parity — no drift. PerformanceMonitor.Collectors (including CollectorCatalog.All, which both Lite and Darling build from) is a single shared project referenced by both Lite/ and Darling/*, so this fix applies to both SKUs from one change. Postgres-collector-definition tests already live exclusively in Lite.Tests in this repo (e.g. PgExtensionAvailabilityCollectorDefinitionTests.cs) even though Darling is the consumer that actually talks to Postgres, so putting the new CollectorPayloadArityTests there too is consistent with existing convention, not a gap.

One thing worth checking before merge — left as an inline comment on Lite.Tests/CollectorPayloadArityTests.cs:131: the new coverage-threshold assertion (skipped.Count < CollectorCatalog.All.Count / 4) appears to sit exactly on its own failure boundary given the PR description's own reported "50 of 66" figure (66/4 truncates to 16, and 16 skipped is not < 16). Worth confirming against the actual CI run.

No missing-index DMV concerns, no security/injection concerns — this PR only reorders/adds .Value() calls in existing writer chains and adds a reflection-based test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant