fix: self-update ranks candidates by sealed-publication lineage, and tells a strand from being current (#3542, #3543) by rbuergi · Pull Request #3551 · Systemorph/MeshWeaver

rbuergi · 2026-09-07T07:49:27Z

Closes #3542
Closes #3543

Two issues, one file area, one defect family: an install's self-update picked the wrong target and then could not recover. The selector had a notion of MAGNITUDE (the version string) and none of LINEAGE (which sealed set was published last), and the check had a boolean where it needed three states.

#3542 — the key is the CD run number, not the version string

Directory.Build.props briefly read 3.1.0 on 2026-09-05, so ten sets published as 3.1.0-ci.7832…7841 and were withdrawn. Ordered by SemVer those outrank every later, sealed 3.0.0-ci.79xx set for ever — and on 2026-09-07 memex-cloud rolled itself onto 3.1.0-ci.7841, three days behind, onto a line nobody had published for two days.

Nothing in the release process can catch this. 7841 passed promote, bake, seal and register: every gate it has says yes, and none of them asks whether this core is newer than what the fleet already runs.

Removing the slip tags did not help either, which is the instructive part — SemVer §11.4 compares pre-release identifiers as text, so rc9 > ci, and the selector simply moved to 3.0.0-rc9.ci.7824, a 09-04 build. Ordering by any property of the label recreates the trap under the next label.

So VersionSelect reads the publishing run number (BuildOrdinal) — the monotonic GITHUB_RUN_NUMBER that Directory.Build.props already calls load-bearing, and the value MissedBuildFact already orders publications by — and ranks on it. Both separators (-ci. / .ci.) and the edge channel are read.

An official release carries no run number of its own (it is a promotion, a retag of a sealed set), so the two questions use the key differently, deliberately:

PickTargets needs a total order, so lineage-bearing tags rank by run number ahead of promotion tags, which rank by SemVer among themselves. Under Stable no continuous tag is eligible at all, so that path is byte-identical to before.
IsNewer is a pairwise question, so it answers on the strongest key both sides share — run number when both are continuous builds, SemVer when either side is a promotion.

That split is what keeps a Stable install running a continuous build able to reach the clean release it is waiting for, and it is why they are not one comparer: mixing the two keys pairwise inside a sort is not transitive (a slip tag beats a release beats a sealed tag beats the slip tag), and an intransitive comparer hands a sort an arbitrary answer.

🚦 Continuous follows the ci line (maintainer, 2026-09-07)

Once v3.0.0 is tagged, SemVer ranks the clean release above every 3.0.0-ci.<n>. A Continuous install would take it and then sit there while later sealed ci builds pile up below it — the same freeze, wearing the release's clothes. So SelectCandidates adds one policy rule: an install whose running build carries a run number is ON the line, and under Continuous only tags that also carry one are candidates. An install that is not on the line is not held there — it rejoins at the next line's first ci build.

🚨 This deliberately does not change IsNewer. "Is the release newer than this ci build?" is yes — Stable needs that answer to reach the release at all. "Should a Continuous install take it?" is no. One predicate cannot carry both, so the POLICY lives in SelectCandidates where the policy is in scope, and the ORDER stays a pure fact about publications. ContinuousFollowsTheCiLine_AndDoesNotJumpToTheRelease pins all four cases, including the one that matters most: right after the tag, when nothing on the line is newer yet and the release sits there looking newer.

The RECOVERY path drops the line rule on purpose — an install that cannot start a pod at all takes the best image that EXISTS.

🔗 The same wrong assumption has a SECOND call site, so the key is shared

ModulePlatformFloor.DeclineReason compares with NuGetVersionComparer — pure SemVer — and SemVer §11.4 compares pre-release identifiers as text, so "ci" < "rc". Measured here against that comparer (Compare(running, floor) < 0 ⇒ DECLINED):

running \ floor	`3.0.0-rc4`	`3.0.0-rc8`	`3.0.0-rc9`	`3.0.0`
`3.0.0-ci.1`	DECLINED	DECLINED	DECLINED	DECLINED
`3.0.0-ci.7989`	DECLINED	DECLINED	DECLINED	DECLINED
`3.0.0-ci.999999999`	DECLINED	DECLINED	DECLINED	DECLINED
`3.0.0`	ok	ok	ok	ok
`3.1.0-ci.1`	ok	ok	ok	ok

Note the last column — a clean 3.0.0 floor is refused too, so it is not only the retired rc* floors.

So the lineage notion moved out of VersionSelect into PlatformReleaseOrder, in MeshWeaver.Plugin.Packaging beside NuGetVersionComparer — the lowest assembly that both MeshWeaver.PluginCatalog (which owns the floor) and Memex.Portal.Shared (which owns the selector) already reference directly.

🚨 But I am overturning half of the design premise I was handed, with the measurement. "Newer than" and "satisfies the floor of" are not the same predicate, and the counter-example is inside this PR: for an updater 3.0.0 MUST outrank 3.0.0-ci.7977 (or Stable can never reach the release); for a floor the same two strings must give the opposite answer. Same strings, opposite required answers — so the shareable part is the key (BuildOrdinal), not the predicate. Forcing one predicate to serve both would fix the floor by breaking the update path. PlatformReleaseOrderTest pins that boundary with the measurement so the floor's own predicate is chosen knowingly rather than by reusing this one.

The genuine overlap is the same-line case — two builds of one line, compared numerically — and both sides now agree on it by construction (SameLineBuildsCompareNumerically_TheOneCaseTheFloorAlsoAsks). Since the declared floors will name a 3.0.0-ci.<n> (Plugins#1447), that is the whole of the shared surface.

PlatformReleaseOrder.Compare returns int?, not int: "unknown" is not an order. NuGetVersionComparer alone answers 1 for Compare("3.0.0", "unknown") — an unparseable core reads as zero — which is exactly how every registry tag once looked newer for ever.

#3543 — "I am current" and "my tag no longer exists" printed the same sentence

Once installed on the highest-sorting tag, nothing is ever newer. That answer also comes out of a withdrawn tag: the 3.1.0 images were untagged, so the Deployment named an image no new pod could start from, and the portal kept printing 500 tag(s) listed, none newer than the installed 3.1.0-ci.7841 — the sentence a healthy install prints. Both AKS portals had to be moved by an operator kubectl set image.

The listing needed to tell them apart was already fetched. CheckInstalledTag now asks it, and answers with three states, never two:

State	Meaning	What the check does
`Resolved`	the installed version is in the listing	"no newer release" — genuinely up to date
`Withdrawn`	the listing is populated and does NOT contain it	the eligible list becomes a recovery set: the best AVAILABLE release, backwards if need be, because an image that exists beats one that does not
`Indeterminate`	the running version does not parse, or the listing carried no platform tags	say so; change nothing

The third row is the point, and it is the recurring house rule: a check that answers a boolean about something it had to READ must not collapse a failed read into a real negative. An unreachable registry and "your tag is gone" have the same shape and opposite meanings; recovering on one bad response would roll the whole fleet backwards. (It is also what keeps the plugins repo's SelfUpdateCheckVerdictTest green — two of its cases list zero tags.)

The state is durable and visible: Admin/UpdatePolicy carries UnresolvedInstalledTag, written on every check so a healed strand clears itself; the Updates tab renders it above everything else; a recovery roll's verdict carries RECOVERY —; a strand with nothing to recover to reports SelfUpdateOutcome.InstalledTagWithdrawn at Warning, naming the operator's move.

Tests — fail before, pass after

Test	Pins
`VersionSelectTest.AMislabelledLine_NeverOutranksALaterSealedSet`	the 2026-09-07 registry: head is `3.0.0-ci.7977`, not `3.1.0-ci.7841`. Old code returns the slip tag.
`…AnInstallOnTheMislabelledTag_SeesTheLaterSealedSetAsNewer`	the strand half: `IsNewer("3.0.0-ci.7977", "3.1.0-ci.7841")`. Old code: `false`.
`…ARetiredPrereleaseLabel_DoesNotOutrankALaterRun`	the second trap the issue measured 20 min later. Old code picks `rc9.ci.7824`.
`…AnOfficialRelease_IsStillReachableFromAContinuousBuild`	the regression the lineage key must NOT introduce
`…CheckInstalledTag_SeparatesCurrent_FromWithdrawn_FromCouldNotTell`	the three states, incl. both Indeterminate shapes
`…AWithdrawnInstalledTag_TurnsTheEligibleListIntoARecoverySet` / `…AnUnreadableListing_RecoversNothing`	recovery is reachable ONLY from a proven withdrawal
`VersionSelectTest.ContinuousFollowsTheCiLine_AndDoesNotJumpToTheRelease`	the maintainer's decision, all four cases
`PlatformReleaseOrderTest` (new, 7 cases)	the lineage key, the incident, the `int?` third state, and the measured floor boundary
`SelfUpdateStrandRecoveryTest` (new, real monolith mesh, 3 cases)	the poller: a withdrawn tag rolls to the best available and stamps the node; a current install still reports "no newer release" and patches nothing; a strand with nothing eligible says `STRANDED` and names the move

"Fails before" is measured, not asserted. The pre-fix VersionSelect.cs — taken verbatim out of origin/main, not re-implemented — run against the three new behavioural cases:

== AMislabelledLine_NeverOutranksALaterSealedSet ==
  expected 3.0.0-ci.7977, pre-fix returns: 3.1.0-ci.7841
  order: 3.1.0-ci.7841 > 3.1.0-ci.7832 > 3.0.0-rc9.ci.7693 > 3.0.0-ci.7977 > 3.0.0-ci.7962 > 3.0.0-ci.7955
== AnInstallOnTheMislabelledTag_SeesTheLaterSealedSetAsNewer ==
  expected True,  pre-fix IsNewer(3.0.0-ci.7977, 3.1.0-ci.7841) = False
  expected False, pre-fix IsNewer(3.1.0-ci.7841, 3.0.0-ci.7977) = True
== ARetiredPrereleaseLabel_DoesNotOutrankALaterRun ==
  expected 3.0.0-ci.7977, pre-fix returns: 3.0.0-rc9.ci.7824
  expected False, pre-fix IsNewer(3.0.0-rc9.ci.7824, 3.0.0-ci.7977) = True

That is the incident, both halves of it, reproduced from the shipped code. The #3543 cases fail before by construction — CheckInstalledTag / SelectCandidates / UnresolvedInstalledTag do not exist on main.

Three existing fixtures pinned the defect as the contract and were corrected, each with a note explaining why: VersionSelectTest.Tags (3.1.0-ci.5 over 3.0.0-ci.51), Any_RequireCiGreenFalse_IncludesEdgeBuilds (3.1.0-edge.7 over 3.0.0-ci.51) and NonAdminUpdateStatusTest.NewerTag (9999.0.0-ci.1 over 3.0.0-ci.2340). Each paired a higher version label with an earlier run number — the version slip itself, written into test data, which a real registry cannot produce because the run number is global and monotonic.

Local: Memex.Portal.Shared, MeshWeaver.Plugin.Packaging and both test projects build -c Release -warnaserror with 0 Error(s); 136/136 portal + 24/24 ordering (PlatformReleaseOrderTest + NuGetVersionComparerTest) across VersionSelect/SelfUpdateVerdict/SelfUpdateStrandRecovery/ComboGateRoll/ComboVerdictRecording/PlatformUpdate*/NonAdminUpdateStatus/RollSelection/…; DocumentationLinkIntegrityTest + WhatsNewEntryIntegrityTest + LocalizationTest green.

One unrelated red, root-caused rather than re-run

Run 34098887738 (shard 4) failed in LateNackReenqueueCorrelationTest with

System.ArgumentException : Value does not fall within the expected range.
  at ImmutableExtensions.ToArray[T](IEnumerable`1 sequence, Int32 count)
  at ImmutableArray.CreateRange[T](IEnumerable`1 items)
  at CapturingLoggerProvider.<FirstMatching>b__0(Int64 _)

That is a real race, not noise, and it is pre-existing — the helper landed in e77b22e99, itself a fix for the previous race in the same poll — and uncatalogued, so the merge-queue steward would not have re-queued it either.

lines is a ConcurrentQueue<string> written by the hub's action-block thread while the poller reads it. ToImmutableArray() goes through ImmutableArray.CreateRange, which takes the ICollection.Count fast path: read Count, allocate exactly that many, then copy what the enumerator yields — and ConcurrentQueue's enumerator is its own, later snapshot. One line enqueued between the two steps overflows the buffer and the poll throws instead of polling again. The line that races is the very LATE_NACK_REENQUEUE warning the test waits for, so it fails precisely when it is about to succeed.

Fixed at the root: ConcurrentQueue.ToArray(), which snapshots the segments under the queue's own synchronisation and has no count/copy window. Separate commit (951df4bef) so it is reviewable on its own.

Cross-repo

Pairs-with: none — no public type or member is removed and no interface gains a member; the additions are new members on VersionSelect / SelfUpdateVerdict / UpdatePolicyContent. check-record-signatures.py locally: 9 changed C# file(s): no binary-breaking primary-constructor changes.

This is nonetheless break shape 7 — behaviour changing behind an unchanged signature — which no gate sees, so MeshWeaver.Plugins' four self-update suites were swept by hand against the new rules. All stay green: SelfUpdateCheckVerdictTest's empty listings land in Indeterminate (message still contains no newer release) and its NewerThanInstalled() is a clean release compared by SemVer; SelfUpdatePollerResilienceTest / SelfUpdateAvailabilityGateTest's 9999.0.0-ci.1 still wins because that host's installed version carries run number 0 (local) or none (CIRun).

i18n: one new key in both catalogs (ui.updateInstalledTagWithdrawn). Per AGENTS.md the mirror in MeshWeaver.Plugins/clients/react/src/i18n/ needs npm run sync:i18n -- --ref <this merge sha> afterwards; it compares against a pinned core commit, so nothing reds until then.

Measured, and what is NOT fixed here

az acr repository show-tags --name meshweaver --repository memex-portal-ai, 2026-09-07: 1262 tags — 795 staging-*, 47 3.0.0-*, zero 3.1.0-*, zero rc*. The withdrawn slip tags are already gone from all three repositories (memex-portal-ai, memex-migration, mw-plugin-test), so the specific 2026-09-07 trap is no longer reachable; this change is what makes the next slip harmless while its tags are still there.

Deliberately out of scope, and still open on #3542: the policy record losing its own policy field under its own bookkeeping writes (proposal 3), and retagging/deleting in ACR (proposal 4, already done by hand — measured above).

Follow-up, not taken here to avoid delaying this PR: a gate that a declared minMeshVersion floor is satisfiable by a platform build that actually exists in ACR. The Plugins lane deliberately cannot host it — that gate runs on forks and Dependabot, where an ACR credential would need an if: on a secret, which is the skip-trapdoor AGENTS.md forbids. A core lane that already holds ACR credentials (main-cd's bake) needs no such exemption, so it belongs there. Filed as #3554 with the measurement.

🚨 Declared floors are not going away: core #3552's link probe reads ModulePlatformFloor.DeclineReason(minMeshVersion) ?? LinkHoldReason(), so it short-circuits behind the declared floor and never runs unless that already passes.

Design and incident record: Doc/Architecture/SelfUpdateTargetSelection (new). Four existing places stated the old SemVer rule as fact and are corrected in the same change set — ReleaseStrategy, ReleaseProcess §1 (which drew the consequence "an install on an rc build sees nothing newer and stays put" as a property to rely on), RollSelection, and the Directory.Build.props ordering note. A doc another repo reads as data is one of the ten break shapes no gate sees.

🤖 Generated with Claude Code

https://claude.ai/code/session_018LzrcSMRzD2R5UcGDk5TKU

…ell a strand from being current An install's self-update picked the wrong target and then could not recover. Both halves are the same defect family: the selector had a notion of MAGNITUDE (the version string) and none of LINEAGE (which sealed set was published last), and the check had a boolean where it needed three states. #3542 — the key is the CD run number, not the version string. `Directory.Build.props` briefly read 3.1.0 on 2026-09-05, so ten sets published as 3.1.0-ci.7832…7841 and were withdrawn. Ordered by SemVer those outrank every later, sealed 3.0.0-ci.79xx set FOREVER, and on 2026-09-07 memex-cloud rolled itself onto 3.1.0-ci.7841 — three days behind, on a line nobody had published for two days. Nothing in the release process can catch that: 7841 passed promote, bake, seal and register. Every gate it has says yes, and none asks whether this core is newer than what the fleet already runs. Removing the slip tags did not help either: SemVer §11.4 compares pre-release identifiers as TEXT, so `rc9` > `ci` and the selector simply moved to 3.0.0-rc9.ci.7824, a 09-04 build. Ordering by any property of the LABEL recreates the trap under the next label. So VersionSelect now reads the publishing run number (`BuildOrdinal` — the monotonic GITHUB_RUN_NUMBER Directory.Build.props already calls load-bearing, and the value MissedBuildFact already orders publications by) and ranks on it. An official release carries no run number of its own (it is a PROMOTION, a retag of a sealed set), so the two questions use the key differently and deliberately: PickTargets needs a TOTAL order, so lineage-bearing tags rank by run number ahead of promotion tags, which rank by SemVer among themselves; IsNewer is a PAIRWISE question, so it answers on the strongest key both sides share. That split is what keeps a Stable install running a continuous build able to reach the clean release it waits for, and it is why the two are not one comparer — mixing the keys pairwise inside a sort is not transitive. #3543 — "I am current" and "my tag no longer exists" printed the same sentence. Once installed on the highest-sorting tag, nothing is ever newer. That answer also comes out of a withdrawn tag: the 3.1.0 images were untagged, so the Deployment named an image no new pod could start from, and the portal kept printing "500 tag(s) listed, none newer than the installed 3.1.0-ci.7841" — the sentence a healthy install prints. Both AKS portals had to be moved by an operator kubectl set image. The listing needed to tell them apart was already fetched. CheckInstalledTag now asks it, and answers with THREE states, never two: Resolved (up to date), Withdrawn (the eligible list becomes a RECOVERY set — the best AVAILABLE release, backwards if need be, because an image that exists beats one that does not), and Indeterminate (the running version does not parse, or the listing carried no platform tags — a failed read, which must never be collapsed into a real negative, or one bad ACR response would roll the whole fleet backwards). The state is durable and visible: Admin/UpdatePolicy carries UnresolvedInstalledTag, written on EVERY check so a healed strand clears itself; the Updates tab renders it above everything else (ui.updateInstalledTagWithdrawn, en + de); a recovery roll's verdict carries RECOVERY; a strand with nothing to recover to reports SelfUpdateOutcome.InstalledTagWithdrawn at Warning, naming the operator's move. Three test fixtures pinned the defect as the contract and were corrected, each with a note: they paired a HIGHER version label with an EARLIER run number, which is the version slip itself written into test data and cannot occur in a real registry. Closes #3542 Closes #3543 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018LzrcSMRzD2R5UcGDk5TKU

ReleaseProcess §1 and RollSelection both described the self-updater as ordering candidates by SemVer, and ReleaseProcess drew the consequence — that an install on an rc build 'sees nothing newer and stays put' — as a property to rely on. Both are now false, and a doc another repo reads as data is one of the ten break shapes no gate sees. Also corrects PickTargets' own summary: it lists ELIGIBLE tags in order, and no longer claims to be the thing that prevents a backwards roll — SelectCandidates decides that, forward-only except on the proven-withdrawn path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018LzrcSMRzD2R5UcGDk5TKU

Copilot

🟡 Changes recommended

There is at least one concrete build-affecting issue in the new test file (an unused using directive) that should be corrected before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR hardens the Memex self-update selector and verdict reporting to prevent “slip-tag” downgrades and to allow recovery when an installed image tag is withdrawn from the registry, aligning update decisions with sealed-publication lineage rather than SemVer label ordering.

Changes:

Update target selection now ranks continuous candidates by publishing run number (“sealed-publication lineage”) and adjusts IsNewer semantics to avoid non-transitive comparisons across promotions vs continuous tags.
Self-update checks now distinguish Resolved vs Withdrawn vs Indeterminate installed-tag states, enabling safe recovery only when withdrawal is proven and making the stranded state visible on Admin/UpdatePolicy.
Adds/updates unit + monolith-mesh integration tests and documents the incident and the revised selection rules.

File summaries

File	Description
test/Memex.Portal.Shared.Test/VersionSelectTest.cs	Expands coverage for lineage-based ordering, installed-tag resolution states, and recovery-vs-roll-forward behavior.
test/Memex.Portal.Shared.Test/SelfUpdateVerdictTest.cs	Pins new verdict shapes/messages for withdrawn tags, recovery qualification, and “unchecked” installed-tag state.
test/Memex.Portal.Shared.Test/SelfUpdateStrandRecoveryTest.cs	Adds end-to-end poller tests against a real monolith mesh for withdrawn-tag recovery and stranded behavior.
test/Memex.Portal.Shared.Test/NonAdminUpdateStatusTest.cs	Adjusts fixture data to reflect run-number lineage semantics.
src/MeshWeaver.Messaging.Hub/Localization/strings.en.json	Adds UI string for withdrawn-installed-tag visibility on the Updates tab.
src/MeshWeaver.Messaging.Hub/Localization/strings.de.json	Adds German localization for the same withdrawn-installed-tag UI string.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-07-your-installation-updates-to-the-latest-build-not-the-highest-number.md	Adds a What’s New entry explaining the incident and the behavioral fix.
src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateTargetSelection.md	Adds an architecture doc formalizing lineage-based selection and the three-state installed-tag check.
src/MeshWeaver.Documentation/Data/Architecture/ReleaseStrategy.md	Updates release/self-update strategy docs to reflect lineage ordering and references the new architecture page.
src/MeshWeaver.Documentation/Data/Architecture.md	Links the new Self-Update Target Selection doc from the Architecture index.
memex/Memex.Portal.Shared/Settings/UpdatePolicySettingsTab.cs	Surfaces withdrawn-installed-tag state prominently above other update status messaging.
memex/Memex.Portal.Shared/SelfUpdate/VersionSelect.cs	Implements `BuildOrdinal`, lineage-based ordering, three-state installed-tag resolution, and recovery candidate selection.
memex/Memex.Portal.Shared/SelfUpdate/UpdatePolicyNodeType.cs	Adds `UnresolvedInstalledTag` to persist and clear stranded state on every check.
memex/Memex.Portal.Shared/SelfUpdate/SelfUpdateVerdict.cs	Adds `InstalledTagWithdrawn` outcome and verdict qualifiers for recovery/unchecked states.
memex/Memex.Portal.Shared/SelfUpdate/SelfUpdateHostedService.cs	Integrates `SelectCandidates`, emits correct verdicts for the three “nothing to roll” states, and records `UnresolvedInstalledTag`.
Directory.Build.props	Updates ordering documentation comments to reflect the lineage-based rule and the incident.

Review details

Files reviewed: 18/18 changed files
Comments generated: 1
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+using MeshWeaver.Mesh;
+using MeshWeaver.Mesh.Security;
+using MeshWeaver.Mesh.Services;
+using MeshWeaver.PluginCatalog;


github-actions · 2026-09-07T08:03:11Z