Skip to content

Managed store: the v8 hardware-sizing block never re-derives work_mem (stuck at the pre-resize value), and it re-appends itself on every hypertable-count change #4207

Closed

Description

@erikdarlingdata

Problem

The v8 block (#2845) exists so that a resized host re-derives its memory and worker sizing. The v1–v7 blocks are marker-guarded and are written only once. v8 re-states effective_cache_size, maintenance_work_mem, timescaledb.max_background_workers and max_worker_processes, but not work_mem. So after a RAM resize, work_mem stays at whatever the v3 block computed for the old hardware, indefinitely.

A second, smaller defect: the v8 block is appended again whenever its fingerprint changes, and the fingerprint includes the worker derivation, which moves with the hypertable count. Each managed store carries three v8 blocks (max_background_workers 72 → 73 → 74). postgresql.conf grows by one block every time the aggregate or hypertable count changes. The last one wins, so the effect is only clutter today, but it compounds with every schema change.

Measured

All three production managed stores, build 3.8.0-nightly.20260924.467, 2026-09-25. The hosts are 31.5 GiB (33,788,809,216 bytes), 8 vCPU, resized from 16 GiB.

  • pg_settings.work_mem = 31744 kB, sourceline = the v3 block (work_mem = 31MB, computed at 16 GiB). DeriveMemorySettings at 31.5 GiB gives clamp(RAM/512, 16 MB, 64 MB) = 63 MB.
  • effective_cache_size = 23808MB and maintenance_work_mem = 1587MB both come from the latest v8 block, so the heal works for the settings it names.
  • postgresql.conf shows the v8 marker three times on each store (lines ~925/941/948), with worker counts 72/73/74.

Cost of the stale value: pg_stat_database.temp_bytes since cluster creation is 33 TB on one store and 7 TB on another. The busiest statements spill at 31 MB (see #3953 and the file-I/O read). Not all of that would be avoided at 64 MB, but the host is sized for it.

Where

  • Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs:
    • ConfMarkerV8 doc ~:188–205: "v8 re-states the SAME formulas (it calls DeriveMemorySettings …)". The block builder emits only three of MemorySettings' four values.
    • DeriveMemorySettings ~:805–890: the work_mem formula that is never re-applied.
    • The v8 append logic keys on ConfHardwareFingerprintPrefix. It appends a new block instead of replacing the prior v8 block.

Fix shape

  • Emit work_mem in the v8 block, from the same DeriveMemorySettings call.
  • Replace instead of append. On a fingerprint change, rewrite the existing v8 block in place: find the marker, replace through the block end. Alternatively, keep v8's derived values in postgresql.auto.conf-style single-owner storage. Either way, one v8 block per store.
  • Consider separating the worker fingerprint from the hardware fingerprint, so a hypertable-count change doesn't re-trigger the memory lines.
  • Pin: a unit test that the v8 block contains all four MemorySettings values, and a conf-rewrite test that two successive fingerprint changes leave exactly one v8 block.

Activity

  1. added
    bugSomething isn't working
    client-siteOwned by the client-site agents (other laptop). Local sessions never pick these up.
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Sep 25, 2026
  2. erikdarlingdata commented on Sep 25, 2026

    @erikdarlingdata
    OwnerAuthor

    Closed by the watcher: delivered in PR #4225, merged to dev.

  3. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Sep 25, 2026
  4. erikdarlingdata commented on Sep 25, 2026

    @erikdarlingdata
    OwnerAuthor

    Reopening. #4225 fixes a store the next time its hardware changes, but not a store that was resized BEFORE the fix, and that is the population this issue is about.

    What a field store showed on the first start of a build with #4225:

    • --check-settings exits 3 with work_mem 31MB / derived 62MB: stale-after-hardware-change, and the start log says the same.
    • postgresql.conf still holds three v8 blocks, and work_mem = 31MB from the v3 block.

    Why: the v8 heal runs only when the newest fingerprint differs from the current hardware (ConfHasCurrentHardwareFingerprint). This store's hardware hasn't changed since its last v8 block. So #4225's collapse to one block, and its work_mem line, wait for the store's NEXT resize.

    Fix: the v8 heal also runs when the conf holds more than one v8 block, or when the newest v8 block differs from what this build would write for the same inputs. An old store then heals once, at its first start on the fixed build, and a second start changes nothing. ALTER SYSTEM values stay as they are.

    🤖 Generated with Claude Code

    https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

  5. added
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Sep 25, 2026
  6. erikdarlingdata commented on Sep 25, 2026

    @erikdarlingdata
    OwnerAuthor

    Closed by the watcher: delivered in PR #4342, merged to dev.

  7. erikdarlingdata commented on Sep 25, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed by #4342, merged to dev (3bb11e3).

  8. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingclient-siteOwned by the client-site agents (other laptop). Local sessions never pick these up.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions