Skip to content

Checkpoint fsync on the busy store is dirtied-file count, not IOPS and not inventory: provisioning would buy nothing, checkpoint_timeout is the near-term lever, and the durable question is which aggregates need an hourly refresh at all #3892

Closed

Description

@erikdarlingdata

The store's checkpoint fsync cost is dirtied-file count, and only one store dirties them — two levers, two measured negatives

Measured by the monitoring seat on 2026-09-22 during a record 149-second fsync hour, across all three stores:

largest SQL store second SQL store
volume gp3, 3000 IOPS / 125 MB/s baseline identical
observed IOPS 1,011 (34 % of allowance) 183
observed throughput 12.3 MB/s (10 %) 3.1 MB/s
relations / chunk tables / TimescaleDB chunks 12,429 / 2,536 / 1,268 13,024 / 2,660 / 1,322
write operations per second 689 181
checkpoint write phase 89 % 89 %
fsync 6.7× baseline

Two levers this kills before anyone spends on them

The mechanism that fits every number

fsync time scales with the count of files dirtied since the last checkpoint, not with bytes written. The busy store dirties roughly four times as many per checkpoint — 689 write operations per second against 181 on the same inventory — because it is the only store actually ingesting Query Store data: the second store's replicas hold Query Store READ_ONLY by architecture, so its query_store collector stores nothing, while the busy store writes the whole Query Store family and refreshes the aggregates built over it, every hour.

The two levers that remain, honestly separated

  1. checkpoint_timeout is still 300 s — twelve timed checkpoints an hour. At 900 s, pages dirtied repeatedly between checkpoints sync once instead of three times; it is the standard remedy for exactly this symptom, reload-only, and its cost is longer crash recovery on a monitoring store — a cheaper currency than read availability. This is the maintainer's knob, the same class as the 16 GB max_wal_size (The bundled store authors its memory knobs from the box but leaves max_wal_size at PostgreSQL's 1 GB default — derive it from disk headroom (≤16 GB) in the same versioned, heal-on-start block #3802 authors that one from disk headroom; this one belongs beside it once ruled).
  2. Fewer dirtied files per unit of work — and this is NOT the per-bucket slicing Daily continuous-aggregate refreshes run one bucket per transaction: the policy's buckets_per_batch is set and converged, so a three-day window is three commits, not one eight-million-row transaction (#3745) #3865 shipped. Slicing a refresh into one transaction per bucket spreads the same dirtied pages across more checkpoints, which for a repeatedly-dirtied page is the wrong direction; it was right for the daily tier because that tier's problem was one enormous transaction, not file count. The hourly tier's problem is different: HourlyRefreshStartOffset is one day on a one-hour bucket, so an ordinary run materializes only the invalidated buckets (one or two), but it touches every one of the aggregates built over the Query Store family, and each materialization's chunks are a day wide (Continuous-aggregate materializations are chunked at one raw chunk, like every raw table, so retention drops what it was designed to and compression reaches the newest day instead of stopping ten days short (#3620) #3621). The real questions are therefore how many of those aggregates need an hourly refresh at all, and whether the Query Store-derived ones can share a coarser cadence — a design pass on the refresh grid's membership rather than its minutes, and the honest successor to Daily band re-derivation is DEFERRED until a daily successor is added (band full 23/23) — the slicing arm landed in #3865 (buckets_per_batch = 1); the storm class was fixed by the WAL ceiling (#3802) #3745's deferred band re-derivation.

Item 2 needs no product work until item 1 is ruled, because raising checkpoint_timeout changes the measurement item 2 would be designed against. Sequence: rule the knob, re-measure the fsync tail for a day, then decide whether the grid's membership needs the pass.

Evidence: the monitoring seat's reads of 2026-09-22 (volume type and allowance, observed IOPS/throughput, relation and chunk inventories, write-operation rates, checkpoint phase split) and the checkpointer rows the store now publishes on its own card (#3826, V137).

Activity

  1. erikdarlingdata commented on Sep 22, 2026

    @erikdarlingdata
    OwnerAuthor

    Measurement gap worth stating before a future reader discovers a correlation and assumes cause: the checkpointer instrument (#3826) and the one-day materialization re-chunk (#3620/#3621) arrived in the same install. There is therefore no pre-re-chunk fsync baseline anywhere in the record, and today's 56 s → 149 s drift can be neither attributed to nor cleared of the re-chunk retroactively — the instrument's own birthday confounds the only comparison that would settle it.

    Consequence for the sequence above: if the checkpoint_timeout knob plus a membership pass do not explain the drift, the re-chunk is the next hypothesis, and it is testable only forward — a controlled interval change on one aggregate family with the instrument watching, one family at a time, with the fsync tail read before and after on the same store. Not a lane today; recorded so the experiment is designed rather than improvised when the question arrives.

    Also recorded: item 2's design is data-dependent on item 1's outcome, which is why the sequence is knob → re-measure a day → decide membership, and not the three in parallel.

  2. added
    client-siteOwned by the client-site agents (other laptop). Local sessions never pick these up.
    on Sep 23, 2026
  3. erikdarlingdata commented on Sep 23, 2026

    @erikdarlingdata
    OwnerAuthor

    Closing as a recorded finding, with no configuration change.

    Six days of the store logs' own checkpoint complete lines (about 1,800 checkpoints per store, Sep 17–23) show per-checkpoint sync time is flat or improving:

    store sync p50 sync p90 daily max
    the Query Store–ingesting SQL store 5.5–7.7 s 25 s (Sep 17) → 14–16 s (Sep 22–23) 30–88 s
    the replica SQL store ~1 s ~4 s 9–53 s

    The worst checkpoints are the nightly ones that sync about 400 files instead of the usual 190–250. That is compression dirtying many files at once, the mechanism this issue describes. None of them came with a failed or retried read.

    The checkpoint_timeout 300 → 900 change is therefore not taken. The trend it was held for never arrived, and a longer interval would make each checkpoint's file count larger, not smaller. Reopen if per-checkpoint p50/p90 climbs for several consecutive days, or if reads start failing inside checkpoint windows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    client-siteOwned by the client-site agents (other laptop). Local sessions never pick these up.enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions