Repository navigation
The prebuilt-bundle retention sweep is registered NOWHERE — AddPrebuiltBundleRetention has zero callers, so it has never run #3963
Description
AddPrebuiltBundleRetention has ZERO callers — the sweep has never run on any portal
Found while landing generation retention for #3461 (PR #3959), by checking whether the code I was
extending is reachable at runtime. It is not.
Measured
grep -rn "AddPrebuiltBundleRetention" src/ test/ → 1 hit: its own definition
grep -rn "AddPrebuiltBundleRetention" MeshWeaver.Plugins/src → 0 hits
The only other mention of PrebuiltBundleRetentionHostedService anywhere is a logger category in
src/MeshWeaver.PluginCatalog/DeploymentPinnedReferences.cs:47 — not a registration.
The control that makes this conclusive rather than a naming guess: the sibling service in the
same file family is wired up in both portal hosts.
| service | registered in Memex.Portal.Distributed/Program.cs |
in Memex.Portal.Monolith/Program.cs |
|---|---|---|
AddAssemblyCacheRetention |
✅ line 138 | ✅ line 49 |
AddPrebuiltBundleRetention |
❌ | ❌ |
So the pattern is in use and this one service was left out of it.
Why it matters
PrebuiltBundleStore is the cleanup for the prebuilt-bundle store, and it was written in response to
a real outage: memex.systemorph.com, 2026-09-08 — 482 identity directories, 13398 MiB on a 16384
MiB share, 3 MiB free. A full Azure Files share truncates writes silently: every runtime
recompile landed as Bad IL format and CD's bake read-back answered ResourceNotFound for 39 of 45
files, with nothing reporting a fault until somebody looked at the free space.
The sweep for that has existed since, with tests, a ledger, a report-only mode, an abort-on-anything-
unreadable discipline and a documented rule — and it has never executed, because no host
constructs it. Its own class comment opens with "THE PREBUILT-BUNDLE STORE GROWS BY ONE IDENTITY
DIRECTORY PER CI BUILD, AND NOTHING USED TO REMOVE ONE"; the past tense is not yet earned.
🚨 This is the same shape as a CI gate that skips: the code exists, its tests are green, the
documentation describes it as the store's cleanup, and it does nothing. Nothing anywhere reports
the absence, because "the sweep found nothing to collect" and "the sweep never ran" produce the same
observable — no ledger line, no log, no deletion.
The fix, and why it is not a one-liner to slip into another PR
The registration itself is one line per host, in MeshWeaver.Plugins
(Memex.Portal.Distributed/Program.cs, Memex.Portal.Monolith/Program.cs). But
PrebuiltBundleRetention.Delete defaults to true, so the first pass on a live portal deletes
against defaults nobody has yet checked against that portal's own store. The sibling service's own
comment states the rule for exactly this case: "the alternative is a first run that deletes on
defaults nobody has checked against their own roll cadence."
Recommended rollout, each step its own reviewed change:
- register with
PreWarm:PrebuiltBundleRetention:Delete=false— the sweep then measures and logs
what it would remove, and writes it to the store's ledger; - read the ledger on memex and memex-cloud, and confirm the protected set is what it should be
(the running identity, the release markers, the pins, the reported identities); - arm deletion.
🚨 Do not skip step 2 by reasoning about the rules. The rules are unit-tested against a temp
tree; what has never been checked is whether the live store's reference inventory reads completely
on a real portal — StampedIdentities() throws when there is no mesh hub or IMeshService, and a
PinnedPlatformReferenceSource that errors aborts the pass. Either would make the sweep abort every
time, silently, in exactly the way it does today.
Also worth checking in the same pass
Whether PreWarm:PrebuiltBundleRoot is even configured on the hosts, since the sweep is inert
without it — and whether an unregistered sweep should be a startup refusal rather than a silence,
on a deployment that consumes CI bakes.
Refs #3461, and PR #3959 which adds generation-level collection to this same sweep.
Activity
rbuergi commentedon Sep 10, 2026
Re-measured: the zero-callers premise does not hold — and the defect points the other way
Working this in #3972. Re-verifying the premise first, as asked, rather than taking it from here.
AddPrebuiltBundleRetentionHAS a caller, and has had one since the commit that created the serviceUnconditional, inside
ConfigureMemexMesh, two lines belowAddModuleGenerationsGc— and both hosts call it:Memex.Portal.Distributed/Program.cs:392,Memex.Portal.Monolith/Program.cs:79, both with aProjectReferencetomemex/Memex.Portal.Shared.MeshHostApplicationBuilderisbase(x => x.Invoke(Host.Services), address), soMeshBuilder.ConfigureServicesruns against the host collection andAddHostedServicereaches the generic host's roster.The measurement was
grep -rn … src/ test/. This repo has three roots that ship code —src/,test/,memex/— and the registration is in the third. A denominator error, and the sibling made it worse rather than catching it:AddAssemblyCacheRetentionreads as registered on the same restricted search only because it is duplicated per host in MeshWeaver.Plugins. As a positive control it confirmed the error.(Registration is per-host in Plugins for the sibling, but once, in shared core, for this one — so this sweep is actually the better-wired of the two.)
The live state is the opposite of "never run" — and worse
PrebuiltBundleRetention.Deletedefaults totrue, andPreWarm__PrebuiltBundleRetention__Deletewas templated nowhere. The portal ConfigMap enumerates keys explicitly, so a values file, an overlay or a record'sextraPortalConfigsetting it reached no container, silently — the #1660 WS3 / #2203 drop shape, on a knob whose omission deletes.Read-only on the control instance, 2026-09-10:
memex(record v75) andmemex-cloudboth carryPreWarm__PrebuiltBundleRoot=/data/prebuilt-bundlesand no Delete key in record, overlay, ConfigMap or inline env.deploy/aks/values.aks.yaml:276anddeploy/homebrew/share/values.local.defaults.yaml:60mount the same root.So the store has been swept with deletion armed, on defaults nobody read a report for — the state step 1 of this issue's own rollout was written to avoid.
#3972
Report-only via the chart (
| default "false", mirroringAssemblyCache__Retention__Delete), which also makes the knob reachable for the first time. It deletes nothing — strictly less than before.Three controls, each verified against a mutation rather than only asserted: the registration (removing line 259 fails it on its named assertion, positive control still passing); deletion-never-armed-by-omission (line removed ⇒ fail,
default "false"flipped ⇒ fail); and the report naming its denominator, so "examined 482, collectable 0" cannot read the same as "examined 0".The gap this does not close, named
DeploymentPinnedReferencesprotects exactly two things:Hosting/Deploymentrecords'pinnedImageTagandHosting/ModuleInventoryreports from registered instances. A satellite CI gate'sMW_PLATFORM_REFis neither. Such an identity is protected only by a clean_releasesmarker or the 30-day window — and satellite pins are-ci.<n>pre-releases, so the marker rule never applies. Past 30 days it is collectable: the #3876 / Memex#219 shape. That must close before deletion is armed on the registry.Also:
PrebuiltBundleRetentionStatusis written and has no reader, so "read the ledger on memex and memex-cloud" (step 2 here) is break-glass today.Full write-up committed as
Doc/Architecture/PrebuiltBundleRetention.