Skip to content

LatestStoreSizeSql is cheap by insertion order, not by index: every row in a sweep shares one metric_time #3209

Closed

Description

@erikdarlingdata

StoreSelfMetrics.LatestStoreSizeSql (#3203) reads the last recorded whole-store size:

WHERE object_kind = 'store' ... ORDER BY metric_time DESC LIMIT 1

It is correct — exactly one 'store' row exists per sweep — and it is cheap today: 0.101 ms cold, three buffers, Index Scan Backward using idx_store_metrics_time.

But the cheapness is incidental rather than guaranteed. idx_store_metrics_time (V53) indexes (metric_time) alone, and every row a single SweepAsync writes shares one metric_time — they are all stamped from the same utcNow. Hypertable rows, dimension rows, background-job rows and the one 'store' row are one tied-timestamp group, ~140+ members on the dogfood fleet. The backward scan hits the 'store' tuple early only because StoreInsertSql happens to run last in SweepAsync, not because the query or the index requires it. Reorder the sweep, or change the physical layout with a VACUUM FULL / REINDEX / a restore, and the scan steps past the group.

Raised by the review bot on #3203 and deferred there with the reasoning kept. Its framing is the right one and worth preserving: a PR whose whole thesis is "do not rely on behaviour that happens to hold on today's fixture" is an awkward place to leave a performance guarantee resting on insertion order.

Why it was deferred rather than folded in

  • The fix is a migration. A composite (object_kind, metric_time DESC) index is a new schema rung with a build cost on a 225 GiB store, and it wants to land somewhere that cost can be measured — not inside a change whose point was removing a cost.
  • The bot's cheaper alternative does not work. An explicit object_kind tie-break in ORDER BY changes nothing: only one 'store' row exists per sweep, so the result is already deterministic. The tie-break does not affect which tuples the scan walks, which is the actual concern.
  • The exposure is bounded in the way the replaced call was not. Worst case is one sweep's tied group, and it scales with hypertable count, not with store bytes. That distinction is the whole of Store disk-pressure check's pg_database_size times out in bursts: 5s deadline sized on a 4.05 GB store, beating against the 300s checkpoint cycle #3199: pg_database_size grew with the store, which is how a 6.2 ms measurement became 3,177 ms. This cannot. The 0.101 ms figure was also taken against the real 141-hypertable layout, so today's ordering dependency sits inside what was measured rather than outside it.

What a fix needs

  • The composite index as a migration, with its build measured on a large store rather than a fixture — the failure mode this whole family keeps producing.
  • A decision on whether metric_time ties are worth removing at the source instead. Every row from one sweep sharing a timestamp is what creates the group; it is also what makes "the latest sweep" a coherent query, so this is a trade rather than a cleanup.
  • No behavioural change to the read. It is correct now and it stays correct; this is about making a bound explicit.

Not urgent: the current worst case is small, bounded, and does not grow with the quantity that grew before.

Activity

  1. erikdarlingdata commented on Sep 9, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling

    Closing — the finding was real and its disposition is already recorded where it belongs, so this is a queue entry rather than work.

    StoreSelfMetrics.cs:265 on dev now carries it, from #3215: "Its cheapness is incidental, not structural, and that is stated rather than glossed." The comment names the axis — the exposure scales with hypertable count rather than with store bytes, which is the distinction that made the pg_database_size walk grow and makes this one bounded — and it says the ordering dependency is on StoreInsertSql running last in the sweep rather than on anything the query guarantees.

    The composite (object_kind, metric_time DESC) index is not being carried forward as a task. It is a migration with a build cost on a large store, the worst case it removes is stepping past one sweep's tied-timestamp group, and the 0.101 ms measurement was taken against the real 141-hypertable layout — so today's dependency sits inside what was measured. If it ever stops being bounded, the comment tells the next reader why and what to do.

    The review bot's argument for folding it in was the good one — a PR whose thesis is "don't rely on what happens to hold on today's fixture" is an awkward place to leave an incidental guarantee — and the answer is that the guarantee is now stated as incidental rather than presented as structural. That closes the gap the objection was about.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions