Repository navigation
LatestStoreSizeSql is cheap by insertion order, not by index: every row in a sweep shares one metric_time #3209
Description
StoreSelfMetrics.LatestStoreSizeSql (#3203) reads the last recorded whole-store size:
WHERE object_kind = 'store' ... ORDER BY metric_time DESC LIMIT 1It is correct — exactly one 'store' row exists per sweep — and it is cheap today: 0.101 ms cold, three buffers, Index Scan Backward using idx_store_metrics_time.
But the cheapness is incidental rather than guaranteed. idx_store_metrics_time (V53) indexes (metric_time) alone, and every row a single SweepAsync writes shares one metric_time — they are all stamped from the same utcNow. Hypertable rows, dimension rows, background-job rows and the one 'store' row are one tied-timestamp group, ~140+ members on the dogfood fleet. The backward scan hits the 'store' tuple early only because StoreInsertSql happens to run last in SweepAsync, not because the query or the index requires it. Reorder the sweep, or change the physical layout with a VACUUM FULL / REINDEX / a restore, and the scan steps past the group.
Raised by the review bot on #3203 and deferred there with the reasoning kept. Its framing is the right one and worth preserving: a PR whose whole thesis is "do not rely on behaviour that happens to hold on today's fixture" is an awkward place to leave a performance guarantee resting on insertion order.
Why it was deferred rather than folded in
- The fix is a migration. A composite
(object_kind, metric_time DESC)index is a new schema rung with a build cost on a 225 GiB store, and it wants to land somewhere that cost can be measured — not inside a change whose point was removing a cost. - The bot's cheaper alternative does not work. An explicit
object_kindtie-break inORDER BYchanges nothing: only one'store'row exists per sweep, so the result is already deterministic. The tie-break does not affect which tuples the scan walks, which is the actual concern. - The exposure is bounded in the way the replaced call was not. Worst case is one sweep's tied group, and it scales with hypertable count, not with store bytes. That distinction is the whole of Store disk-pressure check's pg_database_size times out in bursts: 5s deadline sized on a 4.05 GB store, beating against the 300s checkpoint cycle #3199:
pg_database_sizegrew with the store, which is how a 6.2 ms measurement became 3,177 ms. This cannot. The 0.101 ms figure was also taken against the real 141-hypertable layout, so today's ordering dependency sits inside what was measured rather than outside it.
What a fix needs
- The composite index as a migration, with its build measured on a large store rather than a fixture — the failure mode this whole family keeps producing.
- A decision on whether
metric_timeties are worth removing at the source instead. Every row from one sweep sharing a timestamp is what creates the group; it is also what makes "the latest sweep" a coherent query, so this is a trade rather than a cleanup. - No behavioural change to the read. It is correct now and it stays correct; this is about making a bound explicit.
Not urgent: the current worst case is small, bounded, and does not grow with the quantity that grew before.
Activity
erikdarlingdata commentedon Sep 9, 2026
Claude posting for Erik Darling
Closing — the finding was real and its disposition is already recorded where it belongs, so this is a queue entry rather than work.
StoreSelfMetrics.cs:265ondevnow carries it, from #3215: "Its cheapness is incidental, not structural, and that is stated rather than glossed." The comment names the axis — the exposure scales with hypertable count rather than with store bytes, which is the distinction that made thepg_database_sizewalk grow and makes this one bounded — and it says the ordering dependency is onStoreInsertSqlrunning last in the sweep rather than on anything the query guarantees.The composite
(object_kind, metric_time DESC)index is not being carried forward as a task. It is a migration with a build cost on a large store, the worst case it removes is stepping past one sweep's tied-timestamp group, and the 0.101 ms measurement was taken against the real 141-hypertable layout — so today's dependency sits inside what was measured. If it ever stops being bounded, the comment tells the next reader why and what to do.The review bot's argument for folding it in was the good one — a PR whose thesis is "don't rely on what happens to hold on today's fixture" is an awkward place to leave an incidental guarantee — and the answer is that the guarantee is now stated as incidental rather than presented as structural. That closes the gap the objection was about.