Skip to content

fix(#3461): the publication pointer never moves BACKWARDS - #4273

Merged
meshweaver-cloud[bot] merged 1 commit into
mainfrom
fix/3461-pointer-never-backwards
Sep 14, 2026
Merged

meshweaver-cloud[bot] merged 1 commit into
mainfrom
fix/3461-pointer-never-backwards

Conversation

@rbuergi

@rbuergi rbuergi commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Refs #3461 — the defect Copilot found in #4269, which merged before it could be addressed there (77ec6d4, 05:33:18Z). The prefix is on the generation layout now, so this closes the one hole that layout opened.

The defect, and why it is specific to the generation layout

two overlapping publications on one prefix
flat both write one directory ⇒ the byte-level postcondition finds the other run's bytes and REFUSES. Loud.
generation they write disjoint directories ⇒ nothing to refuse, and whichever finishes last moves _current.
A (older content, slow)   resolve LIVE ──────── upload ──────── seal GA ── _current → GA
B (newer content, fast)       resolve LIVE ── upload ── seal GB ── _current → GB
                                                                      ↑ B live … then A takes it back

Readers never see a torn directory — they see a complete, sealed, self-consistent publication of older bytes, with nothing red anywhere. That is the failure this whole layout exists to prevent, reached from the other side.

🚨 The existing never-seal-backwards guard cannot see it. It runs in publish_to_target, against the publication that was live when LIVE was resolved — roughly 90 seconds before the pointer is written. A sibling that sealed in between is invisible to it.

The fix — ask it again, where it can still matter

pointer_moved_past_us, called immediately before move_pointer:

  1. re-resolve _current by the readers' own rules (resolve_publication_dir, not a second copy of them);
  2. if it now names a different generation, read that generation's source-commit.txt and order it against this bake's SOURCE_SHA through the same compare API the decision-time guard uses;
  3. ahead (newer, and containing ours) ⇒ do not move the pointer, and do not refresh the flat copy either — writing this run's older bytes there would do to pre-pointer readers exactly what the pointer refusal just declined to do to pointer-following ones.

The run still succeeds: its content is contained in what is live, so there is nothing to report as a failure. Its generation stays on the share, sealed and named by nothing — precisely the state retention is defined over. Counted as targets-superseded=N, printed on every run including zero, for the same reason targets-converged is: a number that only appears when non-zero is not a denominator.

🚨 This is a postcondition, not mutual exclusion — the same thing verify_publication says about itself, and for the same reason: there is no lease command under either storage group. What it buys is that the window shrinks from a whole publication (~90 s) to one read plus a one-line upload, and that losing that race is no worse than flat's behaviour today. An unorderable pair keeps today's behaviour and says so, exactly as the decision-time guard does — refusing on an unanswerable comparison would mean a run whose content cannot be ordered never becomes live at all.

Negative control — 4 assertions that fail on main and pass here

The harness had no gh stub, so both compare-API guards had only ever taken their "could not order" arm. STUB_GH implements the compare endpoint (ahead/behind/identical/diverged, plus the 404 that made 5 of 25 core re-seals a no-op on 2026-09-13), and the new case drives the ordered pair: the newer content publishes and points while the older run is mid-upload.

$ venv/bin/python .github/scripts/test-publish-bake-overlap.py \
      --script <origin/main 77ec6d4c74's copy, with this branch's sibling scripts>
  FAIL the pointer is left on the NEWER publication
       — _current='Systemorph-MeshWeaver-3701-1': 10/10 present, sealed=True, by bake {'newer-content': 6, …}
  FAIL …and says so, naming both commits rather than leaving a silent no-op
  FAIL …and the summary counts it, on every run, as its own denominator
  FAIL the FLAT compatibility copy was not refreshed with the older bytes either
       — flat: by bake {'older-content': 6, …}
  94 passed, 4 failed, 98 assertion(s) total.

$ venv/bin/python .github/scripts/test-publish-bake-overlap.py          # this branch
  98 passed, 0 failed, 98 assertion(s) total.

The case's own two fixture assertions — "both generations exist" and "the newer run really did seal first" — pass on main too, so the case is proven to have reached the path rather than to have failed to set it up.

The other two review findings from #4269

  • No persistent guard held the flip. test-publish-bake-overlap.py supplies BAKE_PUBLICATION_LAYOUT to the script directly and can never see a caller. NodeRepoLaneHostGuard.ThePlatformsPluginsBake_DeclaresTheGenerationLayout reads the job block out of main-cd.yml. Control: with the caller set back to flat, Failed: 1, Passed: 0; restored, Passed: 7.
  • The What's New entry overstated the guarantee. It now says, in the description and twice in the body, that pre-pointer installations still read the compatibility copy and still see the in-place overwrite, and the "what this does not change" section says this is not "the problem is gone". It also gains the forward-only paragraph.

Verification (each step able to fail)

step result
test-publish-bake-overlap.py (this branch) 98 passed, 0 failed
…against origin/main's publisher 94 passed, 4 failed — the four new ones
NodeRepoLaneHostGuard + DocumentationLinkIntegrity Passed: 7
…with the caller reverted to flat (control) Failed: 1
dotnet build test/MeshWeaver.Documentation.Test -c Release -warnaserror 0 Warning(s), 0 Error(s)
bash -n publish-bake-bundles.sh OK
check-workflow-shell.py 2 findings, 0 live, 2 allow-listed, 0 stale — the new read is the same deliberate fail-open as the decision-time one, waived with its diagnosis
check-workflow-timeouts.py / check-workflow-yaml-keys.py 87 jobs, 0 violations / 36 workflows, 0 violations

Docs: Doc/Architecture/SealedPublicationGenerations → phase 4 gains "The pointer only moves FORWARD" with the mechanism, the postcondition-not-mutual-exclusion statement and the measured control.

Exposure while this is open

Core CD publishes ~25 plugins publications/day, and same-lane concurrent publications were measured 4× in one day (2026-09-10) and 0× in the 09-12→09-13 window. So the window is rare-but-real, and only when the finish order inverts the content order.

Pairs-with: none — scripts, a guard test and docs; no public type or member leaves src/. No interface gains a member. No i18n catalog key changes.

🤖 Generated with Claude Code

#4269 put `prebuilt-bundles/<identity>/plugins` on the generation layout.
Copilot's review of it found the one thing that layout can get silently wrong
and `flat` could not, and it is real: under `flat` two overlapping publications
write ONE directory, so the byte-level postcondition finds the other run's bytes
and REFUSES; under `generation` they write disjoint directories, there is no mix
to refuse, and whichever run finishes LAST moves `_current`. When that run
carries OLDER content, every reader is handed a publication that is complete,
sealed, self-consistent and out of date — with nothing red anywhere.

The decision-time never-seal-backwards guard cannot see it: it reads the
publication that was live when publish_to_target resolved LIVE, ~90 s before the
pointer is written, so a sibling that sealed in between is invisible to it.

So `pointer_moved_past_us` asks the question again, immediately before
move_pointer: re-resolve `_current` by the readers' own rules, and if it now
names a different generation, order that generation's source-commit.txt against
this bake's SOURCE_SHA through the same compare API. `ahead` ⇒ do not move the
pointer, and do not refresh the flat copy either (writing this run's older bytes
there would do to pre-pointer readers exactly what the refusal just declined to
do to pointer-following ones). The run still SUCCEEDS — its content is contained
in what is live — and is counted as `targets-superseded=N`, printed on every run
including zero.

A postcondition, not mutual exclusion: there is no lease under either storage
group. The window shrinks from a whole publication to one read plus a one-line
upload, and losing that race is no worse than `flat` today. An unorderable pair
keeps today's behaviour and says so, exactly as the decision-time guard does.

Also from that review:
- `NodeRepoLaneHostGuard.ThePlatformsPluginsBake_DeclaresTheGenerationLayout` —
  the harness supplies BAKE_PUBLICATION_LAYOUT directly and can never see the
  caller, so nothing held the flip in place.
- the What's New entry no longer claims the race is gone for every installation:
  pre-pointer readers still see the in-place overwrite until the flat copy goes.

Refs #3461

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 14, 2026 05:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The registry/tag publication can still move back to older content, and the overlap test has an incomplete compatibility-copy assertion.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds a final generation-pointer ordering check to prevent older publications from reclaiming _current.

Changes:

  • Adds superseded-publication handling and overlap regression coverage.
  • Enforces generation layout for the relevant bake workflow.
  • Updates documentation and shell-check allowlisting.
File summaries
File Summary
test/MeshWeaver.Documentation.Test/NodeRepoLaneHostGuard.cs Guards the workflow’s generation-layout configuration.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-14-an-update-is-never-read-while-it-is-being-written.md Updates user-facing publication behavior documentation.
src/MeshWeaver.Documentation/Data/Architecture/SealedPublicationGenerations.md Documents pointer ordering and its residual races.
.github/workflow-shell.allow Allows the intentional fail-open shell check.
.github/scripts/test-publish-bake-overlap.py Adds overlap regression coverage; its compatibility-copy assertion needs strengthening.
.github/scripts/publish-bake-bundles.sh Adds the final pointer-order check; registry/tag publication still requires superseded-run handling.
Review details

Suppressed comments (1)

src/MeshWeaver.Documentation/Data/Architecture/SealedPublicationGenerations.md:463

  • The architecture heading promises that the pointer only moves forward, but the implementation deliberately fails open when the pointer/marker or compare API is unavailable and still has a check-to-write race after this re-read. The later paragraph acknowledges both cases, so this heading should describe a best-effort ordering guard rather than an unconditional guarantee.
#### 🚨 The pointer only moves FORWARD, and this is the one thing the generation layout could get silently wrong that `flat` could not

Under `flat`, two overlapping publications write ONE directory, so the byte-level postcondition
finds the other run's bytes and **refuses**. Under `generation` they write disjoint directories:
there is no mix to refuse, and whichever run finishes **last** moves `_current`. When that is the
  • Files reviewed: 6/6 changed files
  • Comments generated: 3
  • Review effort level: Lite

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +1204 to +1205
echo superseded >> "$OUTCOMES"
return 0
Comment on lines +1194 to +1196
check("the FLAT compatibility copy was not refreshed with the older bytes either",
set(s.bakes_present()) - {"<marker>"} != {"older-content"},
f"flat: {denominator(s)} — pre-pointer readers must not be handed what the pointer refused")
Comment on lines +40 to +44
**And the pointer only ever moves forward.** Two updates can finish out of order — the newer one
first — and the older one must not then take the pointer back, because that would hand every reader
a set that is complete, correct-looking and out of date. Before moving the pointer, a publication
re-checks what is live: if something newer arrived while it was working, it leaves the pointer
alone, says so by name, and its own set stays on the shelf unused.
@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

262 tests   262 ✅  2m 27s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 74a6c60.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

1 449 tests   1 449 ✅  2m 12s ⏱️
    2 suites      0 💤
    2 files        0 ❌

Results for commit 74a6c60.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

899 tests   899 ✅  3m 22s ⏱️
  2 suites    0 💤
  2 files      0 ❌

Results for commit 74a6c60.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

1 844 tests   1 652 ✅  4m 34s ⏱️
    4 suites    192 💤
    4 files        0 ❌

Results for commit 74a6c60.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    6 files      6 suites   5m 17s ⏱️
2 129 tests 2 129 ✅ 0 💤 0 ❌
2 133 runs  2 133 ✅ 0 💤 0 ❌

Results for commit 74a6c60.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

1 061 tests   1 059 ✅  6m 7s ⏱️
    3 suites      2 💤
    3 files        0 ❌

Results for commit 74a6c60.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results

   18 files     18 suites   24m 2s ⏱️
7 644 tests 7 450 ✅ 194 💤 0 ❌
7 648 runs  7 454 ✅ 194 💤 0 ❌

Results for commit 74a6c60.

@meshweaver-cloud
meshweaver-cloud Bot added this pull request to the merge queue Sep 14, 2026
Merged via the queue into main with commit 0a45bcc Sep 14, 2026
37 checks passed
@rbuergi
rbuergi deleted the fix/3461-pointer-never-backwards branch October 10, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants