Raise managed store maintenance_work_mem to the measured compression floor (#1777)

[erikdarlingdata]

Closes #1777.

Field measurement on a production field instance (16 GB RAM class) showed TimescaleDB compression throughput rising about 70% when maintenance_work_mem went from the old formula's landing point (~800 MB) to 1536 MB, and gaining nothing measurable at 4096 MB. TimescaleDB's compression sort runs on maintenance_work_mem, not work_mem, so this setting directly gates how fast the background compression job moves.

Per the issue's own sequencing note: the controlled repro (same chunk, only the setting varying, to find the exact threshold between ~800 MB and 1536 MB) is deliberately not done here and stays open on #1777. This ships the formula change now because the numbers already bound it from both sides.

The formula

DarlingManagedPostgres.DeriveMemorySettings (Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs:494, the maintenanceWorkMem term at :506) goes from min(5% RAM, 1 GB) to:

maintenance_work_mem = min( max(5% RAM, 1536MB), 25% RAM, 2048MB )

• the 1536 MB floor is the measured capture point and is the fix itself. The old 5%-of-RAM term landed under the old 1 GB cap on a 16 GB host, so raising the cap alone would have changed nothing there;
• the 25%-of-RAM term is the small-host guard, so the floor cannot overcommit a box that has no business handing 1.5 GB to one maintenance operation;
• the 2 GB cap concedes nothing measurable (4096 MB bought nothing) and bounds the big-RAM case.

Landing table

SHOW output is listed separately on purpose: PostgreSQL normalizes memory units on the way out, so a conf line of 2048MB reads back as 2GB. Same setting, different string, and an operator checking the value should not be surprised by it.

Host RAM class	conf line	SHOW maintenance_work_mem;	before this change
4 GB	1024MB	1GB	204 MB
8 GB	1536MB	1536MB	409 MB
16 GB	1536MB	1536MB	819 MB
32 GB	1638MB	1638MB	1024 MB
64 GB	2048MB	2GB	1024 MB

Each row exercises a different one of the three terms, and DeriveMemorySettings_PerTier (Darling/Darling.Tests/DarlingManagedPostgresTests.cs:399) pins all of them so the interaction cannot drift silently: 4 GB is the 25% guard winning, 8/16 GB is the floor winning, 32 GB is raw 5% having overtaken the floor, 64 GB is the cap.

Flagged for review, not hidden: the floor raises small hosts proportionally more than the 16 GB host it was measured on (4 GB goes 204 -> 1024 MB, 8 GB goes 409 -> 1536 MB). That is bounded and I believe it is fine, but it is a judgement call and worth a second opinion. maintenance_work_mem is a per-operation ceiling, not a reservation: PostgreSQL grows the sort/TidStore allocation to fit the work, and a small host's chunks are small, so the ceiling is simply never reached there. PostgreSQL 17+ (the bundle pins 18.4) also made vacuum's dead-TID store grow incrementally instead of allocating the full limit up front, which is what made the old comment's "several autovacuum workers can each take up to this" the binding worry it no longer is.

Propagation: the half that reaches the field

A formula change alone only ever affects a fresh initdb, and every store that needs this is already collecting. The product already has the mechanism for exactly this: postgresql.conf is built from independently versioned marker blocks (ConfMarker v1 through v6), each checked on its own in EnsureConfAppended and appended if absent, so an existing cluster gains a missing block on its next service-owned start. postgresql.conf takes the last assignment of a setting, which is how v5 already re-states shared_buffers over v3's line without ever rewriting v3.

This adds ConfMarkerV7 (:181), BuildCompressionMemoryConfAppend (:551) and the heal branch (:981) on the same pattern: it re-states maintenance_work_mem alone. The v3 block is never touched. The append happens before pg_ctl start, so an existing store picks the new value up on that very start rather than one restart later.

Live before/after on a real store

Against the bundled runtime (PostgreSQL 18.4 + TimescaleDB 2.28.1, verified from pg_ctl.exe --version rather than the filename). A real store was provisioned through the production bootstrap, then rewound to its pre-#1777 shape: v7 block removed, and the v3 block's value set to 819MB, which is what the old formula produced on a 16 GB host.

Before - server started on the rewound conf, read straight from the server:

SELECT current_setting('maintenance_work_mem'), pg_size_bytes(current_setting('maintenance_work_mem'));
819MB|858783744

The service's next start, verbatim from the log:

info: Appended v7 compression memory to postgresql.conf (maintenance_work_mem = 2048MB from min(max(5% RAM, 1536MB), 25% RAM, 2048MB); TimescaleDB compression sorts on this setting)
info: Starting managed Postgres (listen_addresses=127.0.0.1, ssl=off, port 55977, ...)
info: Managed Postgres started
SHOW maintenance_work_mem; --> 2GB

After - the healed conf, showing the override rather than a rewrite:

904:maintenance_work_mem = 819MB
924:maintenance_work_mem = 2048MB
v7 marker count: 1

The legacy line is still there and simply outvoted, which is the whole design. This evidence box is a 128 GB host, so it lands on the 2 GB cap; the 16 GB landing is pinned by the theory rows above.

How an operator verifies it post-upgrade

1. In the service log, on the first start after upgrading, one line:

   Appended v7 compression memory to postgresql.conf (maintenance_work_mem = <N>MB from min(max(5% RAM, 1536MB), 25% RAM, 2048MB); TimescaleDB compression sorts on this setting)

   It appears once per store, ever. A store that already healed will not log it again.

2. Against the store:

SHOW maintenance_work_mem;

On a 16 GB-class host that returns 1536MB. On a 4 GB host it returns 1GB and on a 64 GB host 2GB - both correct, both the unit normalization described above, not a failed upgrade.

3. If the value is still the old one, the conf did not heal. Check that the service (not an operator) owned the last PostgreSQL start, since the append runs in the bootstrap path.

Tests

Full Darling.Tests: 3489 passed, 0 failed, 169 skipped (the skips are the DARLING_TEST_PG connection-string-gated live classes; the DARLING_TEST_PGRUNTIME-gated ones below all ran). Build: 0 Warning(s) on a -t:Rebuild sweep of PerformanceMonitor.Darling.Service and Darling.Tests.

New/changed:

• DeriveMemorySettings_PerTier - landing rows added for 4 GB and re-pinned for 2/8/16/32/64 GB.
• CompressionMemoryConfAppend_PinsV7Marker_AndOverridesAnOlderV3Line (:224) - builds a pre-maintenance_work_mem formula lands too low for compression throughput: +70% measured from raising it, plateaus by 1.5 GB #1777 v3 block and proves the effective value moves 819MB -> 1536MB by append, with the old line preserved.
• ExistingStore_AdoptsRaisedMaintenanceWorkMem_OnNextStart_Gated (:752) - the propagation E2E above, against a real server.
• The existing bootstrap E2E now asserts the v7 marker on first run, exactly one after a second start, and that the live server holds the conf's effective value.
• DarlingStoreUpgradeTests asserts one v7 marker after a major upgrade (the upgrade path shares EnsureConfAppended, so v7 rides along).

Comparisons are made in bytes via pg_size_bytes, not on the setting string. The first version of the propagation test compared strings and went red on this host with Expected: "2048MB", Actual: "2GB" - a real property of the server, caught only because the test ran against one.

Mutation evidence

Each applied, watched red, then restored:

Mutation	Result
(a) formula reverted to min(5% RAM, 1 GB)	10 red, including every landing row and both conf-block tests
(b) v7 heal branch disabled	2 red - the propagation test (existing store keeps 819MB) and the bootstrap E2E (fresh store never gains v7)
(c) 25% small-host guard dropped	3 red, and only the small-host cases: the 2 GB row, the 4 GB row and the 4 GB fallback. 8/16/32/64 GB stayed green, which is the guard proving it touches nothing else

Also in this diff

The Darling README's memory-sizing paragraph carried three stale claims that predate #1559 and had nothing to do with this issue: shared_buffers = min(25% RAM, 8GB) (it has been capped at 1 GB since #1559), the worked example "on an 8 GB box that is shared_buffers 2048MB" (it is 1024MB), and "All three appends" (there are seven blocks). Corrected while rewriting the same sentence for the new formula.