get_store_metrics' inventory was blind to 57% of the store it reported on: it now enumerates the continuous aggregates and the named plain tables, reconciles every sweep against pg_database_size and states its coverage, and the owner's job_history count rides the sweep so the managed-mode block is a measurement too (#3582, #3574)

[erikdarlingdata]

The lie, in one number

On the largest production store, get_store_metrics' object inventory accounted for 159 GiB of a 415 GiB database — 38% — and answered "here is the store" with it. A growth investigation that trusted the tool went looking for the missing 256 GiB inside the one object the tool did measure correctly. The bytes were in two places the walk could not see. The twenty continuous-aggregate materializations, ~235 GiB (57%), because timescaledb_information.hypertables never lists a materialization: the view's own definition on 2.28.1 ends WHERE ht.compression_state <> 2 AND ca.mat_hypertable_id IS NULL. (The issue's stated mechanism — that the view lists them under _materialized_hypertable_N names and the sweep's walk keys on collector tables — is wrong on both counts; the walk has no filter at all and the view has no such rows. The effect is exactly as reported.) And the product's own plain tables the walk never named: collect.query_store_text at 15 GiB — V74 stores statement text INLINE by design — collect.query_store_plan_map, and config.config_alert_log.

A correct query producing a wrong conclusion because the envelope never stated its coverage. Same family as #3541.

What the sweep writes now

Every row is under the one metric_time a run stamps, so a run's rows join. Four new kinds, each a named constant (StoreSelfMetrics.*ObjectKind), because a drifted spelling returns zero rows to every reader rather than an error:

• continuous_aggregate — one row per aggregate in timescaledb_information.continuous_aggregates, under its user-facing view name, sized through its materialization: hypertable_detailed_size on materialization_hypertable_schema.materialization_hypertable_name for the total, chunk_compression_stats on the same regclass for pre/post bytes, and the chunk count from timescaledb_information.chunks — NOT a count over chunk_compression_stats, which returns zero rows for a hypertable whose compression is off (measured: a two-chunk materialization yielded no rows until compression was enabled). hypertable_detailed_size on a materialization is byte-exact against pg_total_relation_size over root + chunks + compressed chunk relations (measured: 155,648 = 24,576 + 49,152 + 81,920).
• table — collect.query_store_text, collect.query_store_plan_map, config.config_alert_log: pg_total_relation_size plus the planner's reltuples estimate as row_count. An estimate, deliberately, and the doc says why the two kinds differ: a dim's heap is small (its bytes are TOAST), so the dim arm's exact count(*) is cheap; query_store_text's heap IS the 15 GiB, and an exact count would be a 15 GiB read every hour for a figure whose job is "roughly how many statements have text". -1 (never analysed) maps to NULL, not to a count of minus one. Schema-qualified names, unlike every other kind, because the kind spans two schemas and shares its population with the live un-enumerated census, which names relations schema.relation.
• other — the sum of pg_total_relation_size over every user-schema relation (relkind r/m/p/S, NOT relisshared) that no named row accounts for, with the relation count in chunk_count (the pieces the sum spans, which is what the column means on a hypertable row too). What is excluded is exactly what other rows already size: the dims and named tables by (schema, relation), every hypertable root the hypertables view lists, every materialization root the aggregates view names, and every chunk relation in _timescaledb_catalog.chunk — joined by NAME, never by a regclass cast a vanished relation would make raise. So the statement sizes a few hundred relations, not ~40,000 chunks, and its cost does not follow the chunk count.
• system — the same census on the other side of the user/system line: pg_catalog, information_schema, and every _timescaledb_* schema minus the chunks. Two rows and not one, because folding them would inflate "relations the product should know by name" with sixty pg_catalog tables, and leaving them out would surface as an unreconciled gap of a percent or more on every store every hour — a false finding of exactly the shape this work exists to stop. NOT c.relisshared is load-bearing: the shared catalogs live in global/, outside pg_database_size, and summing them made the rig census EXCEED the database by 512 KiB — wrong in the direction nobody checks.

Both catch-all rows coalesce(..., 0): an empty bucket is a ZERO row, so an absent row means the statement did not run, never that there was nothing to count. A plain-PostgreSQL variant omits the TimescaleDB anti-joins (those catalogs do not exist there); on such a store the collector tables are ordinary tables and land in other, and the note says so in those words rather than flagging a low share.

The reconciliation

DarlingStoreMetricsReader.ComputeInventory — pure — takes the rows that share the store row's metric_time (the same sweep; a sweep that died half-way leaves older rows behind for the kinds it never reached, and summing those against a newer database figure would manufacture a gap) and states two coverages separately: enumerated (bytes under named objects — the issue's "inventory covers N%") and attributed (bytes any row explains). residual_bytes = pg_database_size − attributed; reconciled when both catch-all rows are present and the residual is inside the larger of 1% and 64 MiB. The floor is there because on a fresh rig the residual was 161,471 bytes on a 17 MiB database — exactly pg_database_size minus the sum of every local relation, i.e. pg_internal.init, pg_filenode.map, PG_VERSION — already 0.9% of that store and fixed per database. Rows from an older sweep are counted (stale_object_rows) and the note says the sweep is not completing.

The inventory block is on every response: database_bytes, enumerated_bytes/percent, attributed_bytes/percent, residual_bytes, tolerance_bytes, reconciled, stale_object_rows, bytes_by_kind (where "what is driving growth" is one glance — the 57% would have been the first line), unenumerated {total_bytes, relation_count}, system {...}, largest_unenumerated (live top 10 by name, over the SAME census fragments the sweep sums with, so the list and the number cannot disagree), aggregate_state, and a note in the issue's words: "named rows account for 409.0 GiB of the 415.0 GiB database (98.55%) … 3.0 GiB sits in 14 un-enumerated user-schema relation(s), the largest being … RECONCILED: every row together accounts for 99.76% …" — or "NOT RECONCILED — a finding: 10.0 GiB of pg_database_size is attributed to NO row of this inventory".

Each continuous_aggregate object carries, read LIVE from the catalog (the #2813 precedent for state the series does not carry), compression_enabled, materialized_only, source (a hierarchical aggregate's source resolved back to its parent's view name), and refresh_policy_job_id / compression_policy_job_id / retention_policy_job_id (null where none exists) — the three facts #3581 assembled by hand. timescaledb_information.jobs reports a policy on an aggregate under the aggregate's view name (its own COALESCE(ca.user_view_schema, ht.schema_name)), which is what the lookups join on; the compression predicate carries the 2.18+ columnstore hedge the rest of the codebase uses. The note adds "N of M continuous aggregate(s) have compression DISABLED and hold X GiB between them" when true. These are STATE, not series, and the table has no column that could hold a bool without bending a byte column's meaning — so they are read live and no migration rung is taken.

The managed-mode self-proof (#3574 follow-up, ruled yes)

#3585 made the job_history block say visibility: None in managed mode — the MCP host reads as the least-privilege mcp role, which the view's ownership filter shows nothing — but it could not COUNT. The hourly sweep runs on the worker's owner pool, the role that created every policy job, so it can. StoreSelfMetrics.JobHistoryInsertSql embeds the evidence SELECT verbatim (the string moved to StoreSelfMetrics.JobHistoryEvidenceSql; DarlingStoreMetricsReader.JobHistoryEvidenceSql is now an alias, so every #3585 pin still holds and there is one predicate, not two) and persists an object_kind = 'job_history' row: object_name is the role that counted; row_count is rows_observed ONLY when the view admits that role to every job's history (the two pg_has_role tests, in SQL) and NULL otherwise, because a count the filter truncated is not a count; total_runs is the unfiltered population half; schedule_interval_ms is the window width computed from the two binds; last_run_duration_ms is the newest row's age at the sweep in milliseconds. That last one is the single overload that bends a column's name. It is taken rather than a rung because the store has no timestamp column besides metric_time, V127 is claimed today, and the reader decodes it back into an absolute instant before anyone reads it — the mapping is documented once on the StoreSelfMetrics class summary, and a dedicated column is the clean follow-up. $1 is the window start (timestamptz, Kind = Utc, the #3574 inverse-discipline bind) and $2 metric_time — the one arm where $1 is not the stamp, and the doc says why.

OwnerJobHistoryEvidence.FromLatest (pure) decodes the row into Observed / Stale / Filtered / Absent. Stale is older than 3 hours — the first whole hour past the 2.47 h largest-observed sweep gap the LatestStoreSizeSql paragraph documents; a row that old means two consecutive sweeps did not land and its window no longer speaks for now, so it is shown with its age and no verdict is judged from it. The block gains owner_evidence, owner_role, owner_observed_at, owner_observed_age_hours, owner_observed_window_hours, owner_rows_observed, owner_newest_row_at, owner_jobs_run_in_window; contradiction is true from EITHER admitted reader's numbers; and the note — appended only where this connection is not itself All — says where the number came from: "THE OWNER'S OWN COUNT: the service's hourly self-metrics sweep runs on the owner pool … Reading as 'darling' at …, the sweep saw 2 row(s) … 'recording' is therefore a measurement on this store after all — the service's, taken hourly as the owner — not a GUC echo." Or the CONTRADICTION from the owner's numbers, with the same benign cause and the same way to settle it. Every #3585 field and arm is intact; the two-argument JobHistoryNote is the three-argument one with Absent.

Proven live on the rig as an unprivileged role: visibility: None, rows_observed: 0 beside owner_evidence: Observed, owner_rows_observed: 2 after the GUC was turned on and one job ran — and owner_rows_observed: 1 before, the one row being the telemetry job's failure written with the GUC off.

Two truth fixes

1. The job_history is silently empty on clusters predating ~2026-08-17, because its GUC lives in the one postgresql.conf block EnsureConfAppended cannot heal #3175 GUC-off notes said job_history "is NOT recording, so a maximum over it returns zero rows". TimescaleDB 2.28.1 writes a FAILED run's row regardless of the GUC (job_stat_history.c gates only the success path; measured: with the GUC off, the telemetry job's one failure was the view's one row). Both Off arms now say the view is recording FAILED runs only, a maximum over it is a census of failures and not of runs, an empty result means no failure was recorded, and a non-empty one is not a sign logging is secretly on — which is what the visibility arm appended right after them already (correctly) said. The reader's JobExecutionLoggingSql doc and the Off enum doc carry the same correction; the tool description's "ONLY while … is on" sentence too.
2. Darling/README.md called the mcp role "dormant until MCP is exposed on the LAN". DarlingMcpHostService builds the MCP pool from the stored mcp-role credential whenever postgres.managed = true, loopback included. Both sentences that said otherwise (the roles table and the BYO paragraph's "the only consumer") now say what the code does; the Background-workers paragraph gains the owner's-count sentence.

What this does NOT do

• No migration rung. collect.store_metrics is unchanged; the four new kinds and the documented column mapping are the whole on-disk change.
• Per-collector-table rows on plain PostgreSQL: not taken; the note explains the low enumerated share there instead.
• It does not recommend compressing the aggregates. It reports compression_enabled and the policy ids; The continuous aggregates are 235 GiB and none are compressed: the rollups outweigh their 9x-compressed source, and they were never on the compression ladder #3581 owns the decision.
• It does not touch TimescaleSupport.cs or CHANGELOG.md.
• The job_history row is deliberately excluded from objects[] and daily[]: it carries no bytes, and rendered raw its last_run_duration_ms would read as a run's duration.

Verification

Rig: Docker Desktop, timescale/timescaledb:2.28.1-pg18 (the CI recipe's versions), torn down after. The real SweepAsync against a migrated, converted, aggregate-bearing scratch store wrote 191 rows across nine kinds; the rows summed to exactly the census of every local relation, residual 161,471 bytes = pg_database_size − Σ pg_total_relation_size. The real GetStoreMetrics ran as the owner and as an unprivileged role; both shapes above are its output. The plain variant executed. Every new test's assertions were run through the shipped code in a throwaway harness before CI (this Mac cannot run net10.0-windows; CI is first execution). All four build targets 0 warnings.

Tests: StoreSelfMetricsTests — the aggregate SQL pins (materialization regclass, view name, chunks-view count, the hypertable arm's lack of any filter), the named-table pins cross-checked against the census predicate by (schema, relation), both catch-all variants over the shared fragments, the job_history arm's embedded SELECT and column mapping, the kind vocabulary distinct and written by its arms, the dialect theory over every new statement, and the live end-to-end sweep extended to assert every new kind, the reconciliation through the real reader, and the owner row decoding as Observed. DarlingMcpStoreMetricsToolsTests — the shared-SQL identity, the owner decoder's mapping and freshness bar, the owner contradiction's five flips, the owner note per shape and its exact append rule, the corrected GUC-off notes (with the On arm as positive control), ComputeInventory over one sweep (enumerated vs attributed, stale rows, the floor, missing catch-all, no store), the inventory note's five shapes, the byte formatter, the aggregate-state and top-N SQL pins, failure isolation to null (never an empty list), and the description asserted with the arms and constants behind it.

[claude]

Minor doc accuracy nit: "Names are the same compile-time constants the INSERT arms interpolate, so the census and the rows it excludes cannot drift apart" is only true for the two payload-dimension tuples. The three plain-table tuples below (('collect', 'query_store_text'), ('collect', 'query_store_plan_map'), ('config', 'config_alert_log')) are hand-typed literals, not {QueryStoreTextStore.TableName} / {QueryStorePlanMap.TableName} / {AlertLogTable} interpolations — those constants are compound ("collect.query_store_text") so they can't be split into the (schema, relation) tuple form directly. If one of those constants' table name ever changed, this predicate wouldn't follow it automatically the way the comment implies; the only thing that would catch the resulting double-count is StoreSelfMetricsTests.TableInsertSql_NamesTheThreeProductTables_AndTheCensusExcludesExactlyThose, not the "cannot drift apart" language here. Given this PR's whole thesis is about exactly this class of silent drift, worth either softening the claim or deriving the three literals from the constants (e.g. split on . at a call site) so the comment's guarantee actually holds.

[erikdarlingdata]

Fixed in 36d458e, and the nit was the PR's own thesis applied to itself. The three tuples were hand-typed because the table constants are compound ("collect.query_store_text") and cannot be split at compile time — so rather than soften the claim, the predicate now compares the concatenated (n.nspname || '.' || c.relname) against the five constants themselves ('collect.{QueryTextDimTable}', '{QueryStoreTextStore.TableName}', '{QueryStorePlanMap.TableName}', '{AlertLogTable}', …), and the "cannot drift apart" sentence is true for all five. No product relation name contains a dot, which the doc now says. TableInsertSql_NamesTheThreeProductTables_AndTheCensusExcludesExactlyThose pins that every quoted name in the predicate is one of the five constants and nothing else; the predicate shape was re-verified on a throwaway 2.28.1 container.

[claude]

Reviewed. This is a large, carefully self-verified change (real Docker/TimescaleDB rig measurements, extensive test coverage for both the sweep SQL and the reconciliation/decoding logic). No correctness bugs, SQL-injection surface (all identifiers come from format('%I.%I', ...) over catalog rows or compile-time constants, never free-form input), or performance regressions found. This is Darling/PostgreSQL-only functionality (hypertables, continuous aggregates, TimescaleDB catalogs) with no Lite equivalent to keep in parity — Lite's DuckDB store has no analogous concept, and the PR correctly doesn't touch it.

One left an inline nit on StoreSelfMetrics.cs (doc comment slightly overstates the drift-safety of NamedRelationPredicateSql for 3 of its 5 tuples — non-blocking, already covered by a test).

Separately: every other [Unreleased] entry in CHANGELOG.md gets a bullet for its issue number, and this PR explicitly opts out ("What this does NOT do: ... It does not touch ... CHANGELOG.md"). Given the scope here (a real production data-visibility gap, #3582/#3574), that's a notable departure from convention — worth confirming that's intentional (e.g. changelog entries added in a later batch step) rather than an oversight.

[claude]

LGTM — reviewed the Darling PostgreSQL store-metrics sweep, MCP reader, and MCP tool changes (StoreSelfMetrics.cs, DarlingStoreMetricsReader.cs, DarlingMcpStoreMetricsTools.cs, tests, README/instructions) for correctness, SQL injection, Lite/Darling parity, and performance. No substantive issues found; left one non-blocking inline nit and a note about the CHANGELOG.md omission.

[erikdarlingdata]

On the CHANGELOG: intentional, not an oversight. Lane PRs in this repo do not touch CHANGELOG.md; the coordinator splices one changelog PR from the entries each lane hands back (the buffer-and-splice protocol), so the [Unreleased] bullets for #3582 and #3574's follow-up land there. The entry is drafted and in this lane's report.

The two red CI guards were meta-guards over the touched file rather than behaviour (all new tests passed): the ObjectKind comparison scanner in SerialLoopStoreSizeSourceTests only knew the one kind constant that existed and now accepts every StoreSelfMetrics.*ObjectKind (its invariant is "no retyped literal"; theory rows added for a sibling const passing and a non-kind member failing), and TsqlConventionGuardTests.KnownTruncatedRanges gains the two new expression-bodied formatters, Stamp and Window, whose stranded literals are format strings — both in 36d458e, per that guard's own instruction to say which in the PR.

[claude]

LGTM — reviewed the SQL (census anti-joins for the new continuous_aggregate/table/other/system rows, the JobHistoryInsertSql two-parameter bind, reconciliation math in ComputeInventory, and the owner job_history freshness/decode logic), confirmed no SQL injection surface (all new statements are compile-time consts with positional params), confirmed no Lite/Darling parity gap (this is Postgres/TimescaleDB-only catalog introspection with no DuckDB analog), and checked the new test coverage for the boundary cases (fresh/stale cutoff, filtered vs. zero, catch-all-missing, plain-PostgreSQL variant). No correctness, security, or parity issues found.