Skip to content

Carry an operator's postgresql.conf lines below the include across a major upgrade (#4358) - #4405

Merged
erikdarlingdata merged 4 commits into
devfrom
fix/4358-carry-operator-conf-lines
Sep 26, 2026
Merged

erikdarlingdata merged 4 commits into
devfrom
fix/4358-carry-operator-conf-lines

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Refs #4358.

Why

DarlingStoreUpgrade.CarryAutoConfAsync carries only postgresql.auto.conf across a major PostgreSQL
upgrade. Lines an operator adds below the darling-managed.conf include in postgresql.conf were not
copied into the new cluster; they stayed behind in the old (retained) data directory. #4336's docs point
operators at ALTER SYSTEM as the only carried path today; #4358 closes that gap for postgresql.conf
itself.

What changes

  • ManagedConfMigration.ExtractOperatorLinesBelowInclude(string postgresqlConf) (pure, no disk I/O):
    given a postgresql.conf's text, returns every raw line strictly after the darling-managed.conf
    include, in original order, using the same PostgreSQL-accurate include detection
    Rewrite/HasManagedInclude already use (never a raw string match). A legacy conf (no working include)
    returns nothing — this change is scoped to the migrated shape.
  • DarlingStoreUpgrade.CarryOperatorConfLinesAsync: wired into UpgradeDataDirectoryAsync
    immediately after CarryAutoConfAsync (post-swap, one pattern). Reads the OLD (retained) cluster's
    postgresql.conf, extracts the below-include region, and probes each ASSIGNMENT line one at a time
    against the new binaries with postgres -C — the same per-setting isolation CarryAutoConfAsync
    gives auto.conf settings. A comment, blank line, or operator
    include/include_if_exists/include_dir directive is carried unconditionally (there is no GUC name
    to hand -C). A rejected line is logged (name only; the same secret-name-fragment withholding
    CarryAutoConfAsync already applies) and skipped — never fatal to the upgrade. Carried lines are
    appended to the new cluster's postgresql.conf AFTER the legacy blocks context.AppendManagedConf
    already wrote (step "conf-new-cluster"), under ManagedConfMigration.MovedOperatorLinesComment
    (reused — provenance need not distinguish a same-major move from a cross-major carry). Never throws: an
    unexpected exception resets postgresql.conf to its pre-carry (legacy-appended) baseline and logs a
    warning naming the retained old data directory as the manual fallback, matching CarryAutoConfAsync's
    own "never brick a completed upgrade" posture.

Test plan

  • GREEN, in-process on macOS — ManagedConfExtractOperatorLinesTests.cs (5 unit [Fact]s, no live PG):
    Total: 5, Failed: 0.
  • GREEN, in-process on macOS — DarlingStoreUpgradeTests.CarryOperatorConfLinesAsync_CarriesAfterTheLegacyBlocks_OperatorValueWins
    (ordering pin): after the carry, the new cluster's postgresql.conf holds the carried operator block
    AFTER the legacy block, parsed with ParseConfText, and the operator's log_min_duration_statement
    value wins over the legacy block's own copy of the same key.
  • GREEN, in-process on macOS — DarlingStoreUpgradeTests.CarryOperatorConfLinesAsync_RejectedLine_IsSkippedAndLogged_OthersStillCarried_NeverFatal
    (rejected-line-never-fatal pin, via the substitutable probe delegate — no real postgres.exe): one
    candidate line the probe rejects is skipped and logged (NOT carried: darling_4358_unknown_setting),
    the other two candidates are still carried, and the method returns normally.
    Both new pins run as dotnet Darling.Tests.dll -class Darling.Tests.DarlingStoreUpgradeTests -method "*CarryOperatorConfLinesAsync*" → Total: 2, Errors: 0, Failed: 0.
  • RED confirmed at d20aa68e4 (before the CarryOperatorConfLinesAsync wiring commit): copying the new
    test file into a detached worktree at that commit and building fails to COMPILE —
    CS1061: 'DarlingStoreUpgrade' does not contain a definition for 'CarryOperatorConfLinesAsync' — the
    member does not exist yet, so both new unit pins are RED by compile failure, as expected.
  • UNCHECKED, _Gated — extends ManagedConfUpgradePathTests.cs —
    UpgradeInPlace_OperatorLineBelowInclude_CarriedAndApplied_NoManagedKeyDuplicated_Gated: runs a real
    end-to-end major upgrade (DarlingManagedPostgres.EnsureRunningAsync) with an operator line
    (log_min_duration_statement = 4358) appended below the include in the OLD cluster's migrated
    postgresql.conf. Asserts the line is present in the NEW cluster's postgresql.conf after the upgrade,
    APPLIED per pg_file_settings (not pg_settings.sourcefile, which still names postgresql.conf for an
    unchanged value), and that shared_preload_libraries = 'timescaledb' appears exactly once (no managed key
    duplicated between the legacy block and the carried operator block). Builds clean (0 Warning(s), 0
    Error(s), both Darling.Tests.csproj and Lite.Tests.csproj), and is correctly SKIPPED on this macOS run
    (1 Skipped in the ManagedConfUpgradePathTests class run) because it needs DARLING_TEST_PGRUNTIME_OLD /
    DARLING_TEST_PGRUNTIME_NEWZIP and the managed runtime is Windows-only. On the pre-Carry operator lines below the darling-managed.conf include across a major PostgreSQL upgrade #4358 code the
    operator line assertion fails outright
    : DarlingStoreUpgrade has no CarryOperatorConfLinesAsync and
    UpgradeDataDirectoryAsync never calls it, so the operator's below-include line never reaches the new
    cluster at all — both the presence assertion and the pg_file_settings.applied assertion fail. CI decides
    this pin on Windows with the PG runtime fixtures set.
  • Also ran (unaffected, still GREEN): ManagedConfMigrationStateTests (Total: 6, Failed: 0),
    ManagedConfRewriteOperatorSafetyTests (Total: 5, Failed: 0), ManagedConfUpgradePathTests
    (Total: 5, Failed: 0, Skipped: 1 — the new _Gated pin).
  • Build: Darling.Tests.csproj -p:EnableWindowsTargeting=true and Lite.Tests.csproj -p:EnableWindowsTargeting=true — 0 Warning(s), 0 Error(s) on macOS, after every change in this session.

CHANGELOG entry

SECTION: Fixed
ENTRY:

…on step for #4358

ManagedConfMigration.ExtractOperatorLinesBelowInclude(postgresqlConf) returns
every raw line strictly after the darling-managed.conf include, in original
order, using the same PostgreSQL-accurate include detection Rewrite and
HasManagedInclude already use. A Legacy conf (no working include) returns
nothing -- #4358 is scoped to the migrated shape.

Unit pins (ManagedConfExtractOperatorLinesTests.cs, no live PG needed):
 - a line below the include is returned;
 - a line above the include never appears in the extracted set;
 - no include returns empty;
 - a differently-pathed but equivalent include is still recognized (parser,
   not raw string match);
 - a conf ending exactly at the include returns empty, not a stray blank.

Refs #4358.
Alongside CarryAutoConfAsync, post-swap (one pattern, per the 07:17Z ruling).
Reads the OLD (retained) cluster's postgresql.conf, extracts operator lines
below the darling-managed.conf include via
ManagedConfMigration.ExtractOperatorLinesBelowInclude, and probes each
assignment line one at a time against the new binaries with postgres -C,
the same isolation CarryAutoConfAsync gives auto.conf settings. A comment,
blank line, or operator include directive is carried unconditionally (never
probed -- there is no setting name to ask -C about). A rejected line is
logged and skipped, never fatal. Carried lines are appended AFTER the
legacy blocks context.AppendManagedConf already wrote, under
ManagedConfMigration.MovedOperatorLinesComment (reused per the ruling), so
an operator's line still wins. Never throws: an unexpected failure resets
postgresql.conf to its pre-carry baseline and logs a warning naming the
retained old data directory as the manual fallback.

Build: Darling.Tests 0 warnings / 0 errors on macOS
(EnableWindowsTargeting=true).

Pins (extending ManagedConfUpgradePathTests etc. for the wiring order and
the live _Gated upgrade-path pin) are still open -- see the PR body /
handoff note for what's left.

Refs #4358.
…e-never-fatal, and a live upgrade-path pin
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 26, 2026 12:12
@erikdarlingdata
erikdarlingdata merged commit bd8a266 into dev Sep 26, 2026
15 of 16 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4358-carry-operator-conf-lines branch September 26, 2026 12:12
erikdarlingdata added a commit that referenced this pull request Sep 26, 2026
…ow live (#4446)

The upgrade-path tests now read the preload line where the managed settings live: darling-managed.conf, not postgresql.conf.

- A new shared helper, ManagedPreloadAssert, reads both postgresql.conf and darling-managed.conf. It asserts all of these:
  - exactly one effective shared_preload_libraries line across the two files;
  - that line is in the managed file, and the include is present;
  - it names both timescaledb and pg_stat_statements.
- RuntimeAdvance_SameMajorStoreWithRealData_SwapsWithoutPgUpgradeAndKeepsEverything_Gated (both cases) uses it. It also reads pg_settings' setting and sourcefile on a non-pooled connection, and expects both libraries, sourced from darling-managed.conf.
- UpgradeInPlace_OperatorLineBelowInclude_CarriedAndApplied_NoManagedKeyDuplicated_Gated counts the preload line across both files, and keeps its pg_file_settings applied check.
- ManagedConfFileTests.RenderBody_PreloadLine_IsExactlyOne_WithBothLibraries pins the render itself: exactly one preload line, with both libraries.
- Test-only.

Refs #4336, #4405, #4358
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant