Skip to content

Add a v16 managed-conf marker for a 15-minute checkpoint interval (#4246) - #4426

Merged
erikdarlingdata merged 6 commits into
devfrom
fix/4246-managed-checkpoint-interval
Sep 26, 2026
Merged

erikdarlingdata merged 6 commits into
devfrom
fix/4246-managed-checkpoint-interval

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Evidence: on the trial store, the 15-minute interval's reading at +3 hours shows WAL volume down from 14.0 to 4.6 GB/h, the full-page-image share down from 8.6% to 4.2%, and the average checkpoint sync phase down from 1.66 s to 0.24 s. That sync phase is far under the self-alert's 10-second bar. The 24-hour reading is logged after merge.

Refs #4246.

Why

The v15 marker shipped wal_compression = lz4 and held back a longer checkpoint interval, because a
longer interval risks the checkpointer's own sync-phase self-alert (DarlingSelfAlertEvaluator.CheckpointSyncBarMs,
#4037): more dirty pages per checkpoint makes the fsync phase longer, and #3892 already traced killed
reads on a production store to sync phases of 14.0s and 25.2s. One production store ran a trial of
checkpoint_timeout = 15min through ALTER SYSTEM and a reload, ahead of this PR, to measure that risk
before the setting ships to every managed store. Its +3-hour reading is at the top of this description.

What changes

Adds a v16 marker (DarlingManagedPostgres.ConfMarkerV16) that appends checkpoint_timeout = 15min
to postgresql.conf, in the same shape as the v15 block:

  • BuildCheckpointIntervalConfAppend() builds the marker line plus the setting, with no fingerprint or
    stamp line (matching v9-v11/v13/v15).
  • Added to AllManagedConfMarkers, after v15.
  • EnsureConfAppended appends it after the v15 block, on the same marker-absent guard, with an
    Information log line.
  • ManagedConfMigration covers the new marker in CoveredMarkers and classifies its line as a fixed
    managed line (checkpoint_timeout = 15min).
  • The v15 doc comment, which said the checkpoint interval was "held, not shipped," now says it ships as
    v16.
  • ManagedConfFile.RenderBody renders the v16 block after v15, in the same order as the legacy blocks. A store that has migrated to darling-managed.conf renders only that file and never runs the legacy appenders, so without this line it would never get the setting. A store migrating on first start would carry the value over and then lose it on the next start's render.

checkpoint_timeout is sighup-context in PostgreSQL, so a running store could pick it up from a
reload alone. This service still appends it before pg_ctl start, the same as every other managed
setting, so a service-owned start applies it on the very start that writes the block.

This block does not touch max_wal_size: ConfMarkerV12 (#3802) already bounds it by free disk, and a
pin (WalVolumeConfAppend_PinsV15Marker_AndSetsCompression's v16 counterpart) checks that neither
max_wal_size nor min_wal_size nor checkpoint_completion_target land in this block by accident.

Small stores and size-triggered checkpoints

max_wal_size (ConfMarkerV12, #3802) is clamp(free / 8, 1 GB, 16 GB), floored to a power of two: 1
GB, 2 GB, 4 GB, 8 GB, or 16 GB depending on free disk.

PostgreSQL starts a size-triggered checkpoint when WAL written since the last checkpoint approaches
max_wal_size. The PostgreSQL documentation for checkpoint_completion_target states the server aims
to finish each checkpoint's spread-out writes before that much WAL accumulates, which in practice means
a checkpoint fires around max_wal_size / (1 + checkpoint_completion_target) of WAL, not at
max_wal_size itself. ConfMarkerV12 pins checkpoint_completion_target = 0.9 on PostgreSQL 14 and
later (the version the managed store ships), so the trigger threshold is max_wal_size / 1.9.

Using one production store's measured post-compression WAL rate of about 4.6 GB/h:

derived max_wal_size size-trigger threshold time to threshold at 4.6 GB/h checkpoints today (5 min timer) with a 15 min timer
1 GB ~0.53 GB ~6.9 min every 5 min (timer) every ~7 min (size)
2 GB ~1.05 GB ~13.7 min every 5 min (timer) every ~14 min (size)
4 GB ~2.11 GB ~27.5 min every 5 min (timer) every 15 min (timer)
8 GB ~4.21 GB ~54.9 min every 5 min (timer) every 15 min (timer)
16 GB ~8.42 GB ~109.8 min every 5 min (timer) every 15 min (timer)

What a small store gets: at this WAL rate, the 5-minute timer fires before any size trigger on every store, so today every store checkpoints every 5 minutes. With a 15-minute timer, a store whose derived max_wal_size is 1 GB (the smallest, free-disk-limited class) reaches its size threshold first, at about 7 minutes. So it moves from 5 to about 7 minutes, not to 15, and gets a smaller share of the WAL reduction. A 2 GB store moves to about 14 minutes. A store at 4 GB or more gets the full 15 minutes. A store writing less WAL than 4.6 GB/h reaches its size threshold later, so the timer governs more often. Changing max_wal_size is out of scope here: v12 bounds it by free disk on purpose.

Compose store

The Linux compose deployment (Darling/compose/docker-compose.yml) runs the official TimescaleDB image,
which runs timescaledb-tune on every start. Per #4322, this file already overrides the handful of
settings tune gets wrong for this product (worker sizing, work_mem) with explicit flags, and leaves
everything else — including checkpoint settings — to tune's own sizing from the container's cgroup
limit. This PR does not add a checkpoint override to compose: it only ships the v16 marker in the
managed-Windows-store code path, so the compose store's checkpoint interval is unaffected. Whether timescaledb-tune sets checkpoint_timeout itself isn't verified here; if it doesn't, the compose store keeps PostgreSQL's 5-minute default.

Test plan

  • DarlingManagedPostgresTests: added CheckpointIntervalConfAppend_PinsV16Marker_AndSetsCheckpointTimeout,
    ConfMarkerV16_IsInAllManagedConfMarkers, EnsureConfAppended_AppendsV16Once_AndNotAgainOnANextStart
    (copies of the v15 pins), and updated EveryConfMarker_IsDistinct_AndNoneIsASubstringOfAnother's
    expected count from 15 to 16 markers.
  • ManagedConfMigrationTests: added ClassifyLines_UntouchedV16Block_IsOurs.
  • ManagedConfFileTests:
    • RenderBody_CarriesEveryManagedConfMarkersOwnedKeys: for every marker in AllManagedConfMarkers, it parses the keys out of that marker's own legacy block and asserts each appears in the rendered managed file, so a future marker with no render line fails it. It fails at c59e1365 (before the render line): "marker … (v16 checkpoint interval) owns key 'checkpoint_timeout' … but RenderBody's rendered body does not carry it". It passes with the fix.
    • v14, the legacy maintenance_work_mem cap, is the one documented skip. RenderBody decides it from its own inputs, but the memory-sizing cap (2047 MB) sits under the PostgreSQL 17 limit for any RAM size, so no fixture can make the render add it.
    • A pin that the render contains checkpoint_timeout = '15min'.
  • ManagedConfRehearsalTests: the fixture's managed-key count goes from 24 to 25.
  • ManagedConfFileLiveTests.SecondStart_UnchangedInputs_DoesNotRewriteFile_Gated caught the missing render line on CI, and passes with it.
  • Both classes and DocCommentHygieneTests run in-process on macOS: 0 failures introduced (the
    DarlingManagedPostgresTests run shows 5 pre-existing failures, all Windows-path assertions unrelated
    to this change, e.g. ResolveDataDirectory_TrimsATrailingSeparator expecting D:\darling\pg).
  • Darling.Tests and Lite.Tests both build 0 errors / 0 warnings-as-errors with
    -p:EnableWindowsTargeting=true.
  • RED on dev: ConfMarkerV16 and BuildCheckpointIntervalConfAppend don't exist there, so the new
    tests are a compile failure against the pre-fix code, not a runtime failure.
  • Full suite was not run given the time budget; the targeted classes above are the ones this change can
    affect (marker list, migration classifier, doc comments).

CHANGELOG entry

SECTION: Changed
ENTRY:

)

Ships checkpoint_timeout = 15min behind a new marker, copying the v15
wal_compression block's shape: builder, marker list entry, append-once
heal, and migration classification. The v15 doc comment now says the
checkpoint interval ships as v16 rather than being held back.

Refs #4246.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 26, 2026 18:32
@erikdarlingdata
erikdarlingdata merged commit 0ef31a2 into dev Sep 26, 2026
15 of 16 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4246-managed-checkpoint-interval branch September 26, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant