Skip to content

CI: run the store-upgrade tests on pull requests that touch the store's schema or upgrade path - #4500

Merged
erikdarlingdata merged 2 commits into
devfrom
ci/pr-store-upgrade-legs
Sep 27, 2026
Merged

erikdarlingdata merged 2 commits into
devfrom
ci/pr-store-upgrade-legs

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Why

Two gated end-to-end tests exist only to catch a store-upgrade regression before it ships: an
upgraded-in-place major upgrade over a real store with a hypertable, TOAST-sized plan XML, a
continuous aggregate and a compressed chunk, checksum-compared before and after; and a same-major
runtime swap with no pg_upgrade involved. Both are gated on three environment variables
(DARLING_TEST_PGRUNTIME_OLD/_NEWZIP/_PREVIOUS) that point at a fixture nightly CI builds and
sets every night — and only there. Pull-request CI never built that fixture and never set those
variables, so both tests silently skipped on every single pull request, including ones that changed
the exact code they exist to cover.

What changes

  • A new store-upgrade: path filter in .github/darling-paths-filter.yml, covering the store
    schema, the service project holding the upgrade orchestration and the managed-conf migration
    machinery it drives, the two fixture-build scripts, and the workflow/filter files themselves (the
    same self-validating reasoning the existing darling: filter already uses for build.yml).
  • build.yml's PG test job now builds the same upgrade fixture nightly builds, and sets the same
    three environment variables on the test step, on every shard, so a class's shard doesn't decide
    whether it runs — only when a pull request's changed files match the new filter. Every PR that
    doesn't touch this filter's paths is unaffected; no new step runs, so the build time for those legs
    is unchanged. nightly.yml is untouched.
  • A pin in DarlingPathFilterGateTests asserting the filter exists and covers the store schema
    directory and both fixture scripts; a companion test asserts an injected removal of the schema entry
    is detected. Both run green on this branch; the missing-entry case was verified RED by hand before
    restoring the file (see below).

Test plan

  • dotnet build Darling/Darling.Tests/Darling.Tests.csproj -c Release -p:EnableWindowsTargeting=true:
    0 warnings, 0 errors.
  • python3 -c "import yaml; yaml.safe_load(...)" on both workflows and the filter file: clean.
    actionlint on build.yml: no new findings (the one pre-existing shellcheck note is in
    nightly.yml, which this PR does not touch).
  • DarlingPathFilterGateTests run in-process on the built assembly: 8/8 green, including the two new
    facts.
  • Mutation, run and reverted locally: removed the Storage entry from the new filter's pattern
    list, rebuilt nothing (the test reads the YAML at run time), reran the same class — both new facts
    failed as expected (TheStoreUpgradeGate_CoversTheSchemaAndTheFixtureScriptsAndTheServiceProject and
    TheStoreUpgradeGate_MissingTheStorageEntry_IsDetected), then restored the file and reran — 8/8 green
    again.
  • Proven in pull-request CI with three throwaway draft PRs (Not for merge: CI proof (upgrade-touch-storage) #4497, Not for merge: CI proof (upgrade-touch-lite) #4498, Not for merge: CI proof (upgrade-mutation) #4499, since closed and their branches deleted), each one commit on top of this branch:
    • Not for merge: CI proof (upgrade-touch-storage) #4497, a comment-only change to PgMigrations.cs: every PG shard ran the fixture steps (the cached download ~6–10 s, the previous-major runtime build ~26 s) and passed.
      • The shards' [SKIP] lists hold none of the upgrade tests. Shard 0 skipped 6 (the live-SQL-Server and live-PostgreSQL-target E2Es, and a cert shape), shard 1 skipped 2 (the PG-log live tests), and shard 2 skipped 0.
      • So DarlingStoreUpgradeTests, DarlingManagedPostgresTests and ManagedConfUpgradePathTests ran their gated facts.
      • Totals 5,088 + 4,947 + 5,667, 0 failed.
    • Not for merge: CI proof (upgrade-mutation) #4499, a one-line break at the in-place upgrade's pg_upgrade call site (the serverOptions argument dropped; the argument-builder unit pins don't reach that call):
      • it went RED in PR CI on exactly the gated upgrade tests: DarlingStoreUpgradeTests.UpgradeInPlace_OldMajorStoreWithRealData_UpgradesAndKeepsEverything_Gated (shard 2) and ManagedConfUpgradePathTests.UpgradeInPlace_OperatorLineBelowInclude_CarriedAndApplied_NoManagedKeyDuplicated_Gated (shard 1).
      • Everything else stayed green.
    • Not for merge: CI proof (upgrade-touch-lite) #4498, a comment-only Lite change, can't show the skip before this merges. A PR's diff against dev still contains this branch's workflow and filter files, which store-upgrade: includes on purpose (so a change to the gate exercises the gate). Not for merge: CI proof (upgrade-touch-lite) #4498 therefore ran the fixture too, as designed.
      • When store-upgrade doesn't match, the new steps' if: is false and the env expressions give an empty string, which every gated test treats as unset (IsNullOrWhiteSpace → skip).
      • The first Lite-only PR after this merges shows it.
  • The cost when it matches: ~33 s per PG shard for the fixture, and the shards run in parallel.

CHANGELOG

None: CI-only change.

…'s schema or upgrade path

The gated upgrade-in-place and same-major-swap E2Es (#1706, #3906) only ran
on the nightly job: pull-request CI never built the upgrade fixture or set
the three env vars those tests read, so they silently skipped on every PR.

Adds a store-upgrade path filter derived from what those E2Es exercise (the
store schema, the service project holding DarlingManagedPostgres and
DarlingStoreUpgrade plus the managed-conf migration machinery, the two
fixture scripts, and the workflow/filter files themselves). When a PR's
changed files match it, build.yml's darling-pg job builds the same fixture
nightly.yml builds and sets DARLING_TEST_PGRUNTIME_OLD/_NEWZIP/_PREVIOUS on
the shards that carry the gated classes; other shards and non-matching PRs
are unaffected.

Pins the filter's existence and coverage in DarlingPathFilterGateTests.
nightly.yml is unchanged.
The fixture steps and the three DARLING_TEST_PGRUNTIME_* env expressions were
gated on matrix.shard != 0 because today's upgrade classes happen to hash into
shards 1 and 2. That meant a renamed class, or a new upgrade E2E that hashes
into shard 0, would silently skip the fixture with no signal. A class's shard
must never decide whether it runs.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 27, 2026 21:23
@erikdarlingdata
erikdarlingdata merged commit 133b5e2 into dev Sep 27, 2026
15 of 16 checks passed
@erikdarlingdata
erikdarlingdata deleted the ci/pr-store-upgrade-legs branch September 27, 2026 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant