Bundle transfers: the 120 s budget bounded SIZE, not responsiveness — #4549 fixed the shape; the roll, the byte/rate measurement and the inverted 3 min/5 min budgets are still owed

What is failing

The plugin-registry-bundles resilience pipeline — the one PluginBundleClient resolves, with a
120 s attempt / 5 min total budget — is logging OnTimeout. An attempt on that pipeline exceeding
120 s is a different defect from #4222, which is about the 30 s plugin-registry pipeline and the
catalog listing.

The measurement

Read from Admin/_LogIncident/9ca334c1e8dad9ca on memex.systemorph.com, 2026-09-16T20:5xZ
(version 1692, occurrences 2,078, lastSeen 2026-09-16T18:53:25Z).

🚨 That incident's fingerprint masks the Source: value, so every Polly OnTimeout on every
pipeline folds onto it. Of its 10 retained samples:

samples	pipeline	when
6	`plugin-registry-bundles-standard//Standard-AttemptTimeout`	2026-09-15T20:50:45Z – 20:55:17Z
4	`plugin-registry-standard//Standard-AttemptTimeout` (#4222's)	2026-09-16 06:45Z, 08:25Z, 18:52Z, 18:53Z

The six are a burst inside four and a half minutes across four pods
(memex-portal-deployment-6546c779d4-4ngqv, -7fd9668b69-4qtkx, -7fd9668b69-4zzzs,
-7fd9668b69-jnwlv) — i.e. several replicas at once, which is the shape of a roll or a reconcile
wave, not of one unlucky request.

#4222's triage on 2026-09-15 saw this at n=1 and said outright that if bundle transfers keep
exceeding 120 s "that wants its own issue". It is now n=6 in one window. This is that issue.

Why it matters beyond the latency

It contaminates #4222's closing condition. That issue closes on "occurrences stops advancing
over a full week" on this very incident — but the counter is shared with this defect, so the
incident can keep advancing after #4222 is completely fixed, and has already produced reopens that
read as "#4222 regressed". Anyone reading a future reopen must read samples[] before believing it.

Where to look

plugin-registry-bundles is resolved by PluginBundleClient
(src/MeshWeaver.PluginCatalog/PluginBundleClient.cs:81) for both call shapes, so the log line
alone does not say which one timed out:

The bundle INDEX — GET /api/plugins/bundles/index.json. This is the endpoint the
2026-08-26 ServiceDefaults comment measured at 12–19 s TTFB for an 8.7 KB document, and its
per-request work is real and uncached: InstalledPackages and HeldPartitions are mesh
Query<MeshNode> calls per request (PluginBundleEndpoints.cs:2149, :1325), plus two
landing.GetActivation() reads and per-package module-byte filesystem probes in
ServableModules. 12–19 s does not reach 120 s on its own — but that measurement is a month old
and was taken on a smaller catalog.
The bundle BLOB transfers — megabytes each, legitimately slow, which is why the budget is
120 s in the first place.

Distinguishing them is the first task; the empty Operation Key on the event means the sample
cannot. A Logs InstanceAction window around 2026-09-15T20:50Z on the control instance should show
which route the client was on.

What this is NOT

Not the catalog listing (The plugin registry still serves the pre-#4246/#4527 catalog listing — memex.meshweaver.cloud is 934 commits behind and has never run either fix #4222). That is the 30 s plugin-registry pipeline, and its dominant
cost — the listing transferring the whole source repository — is fixed in The catalog listing transferred the whole repository to read its manifests (#4222) #4527.
Not fix: the registry's catalog listing stops re-reading the repository per request (fixes #4222) #4246's listing cache, which does not touch either bundle route.

Not investigated here

I found this while measuring #4222 and deliberately did not chase it: it is a different pipeline, a
different budget and a different code path, and folding it into #4222 is what made #4222's own
signal unreadable in the first place.

Fixed in main, not yet live. #4549 merged as 32f0523e2 — the bundle transfer now streams and is bounded on silence (a per-chunk idle deadline) rather than on total duration, which is what made a large-but-healthy transfer indistinguishable from a stalled one.

Deliberately leaving this open: the fix ships in the platform image, so the OnTimeout on plugin-registry-bundles-standard keeps firing on every portal still running a pre-fix image. The close condition is a roll, not the merge — re-read the incident's lastSeen after the portals carry a set that includes 32f0523e2.

2026-09-17 — merged, not deployed, and by the fix's own doc not yet settled

#4549 = 32f0523e2ddfa44841f71409c278594b7a047a4e, merged 2026-09-17 05:27Z, on main. It is on
neither portal: memex.systemorph.com runs afde4eab (2026-09-15 21:50Z) and
memex.meshweaver.cloud runs c84c6c05 (2026-09-12); git merge-base --is-ancestor 32f0523e2 <either>
→ NO. The 09:25Z note stands: the close condition is a roll of the CONSUMER, and it has not happened.

What landed, in src/MeshWeaver.PluginCatalog/PluginBundleClient.cs:
:660 SendAsync(…, HttpCompletionOption.ResponseHeadersRead, ct) — the body leaves the Polly
attempt; :698 → :745 CopyStallBounded — a per-chunk deadline reset; :788
TransferStallBudget = 120 s, documented as "a bound on silence, never on total size"; and
:702-706 the measurement whose absence made this unanswerable — bytes, elapsed ms, KiB/s, with
:794 ThroughputKibPerSecond returning 0 rather than fabricating a rate.

Two corrections to this issue's own body, from Doc/Architecture/BundleTransferBudget:
it was 18 adopt failures (14 on 09-15 19:58–20:54Z, 4 on 09-16 14:20–14:33Z), ten of them spaced
at exactly the reconciler's 180 s PerPackageAdoptBudget — not "6 of 10 in 4.5 min across 4 pods".
The ten-sample retention made a sequential reconcile pass look like a roll wave. And the issue's
own first task is answered: it was the blob transfers, not the index.

Still owed, in order

One transfer's real byte count and rate. The doc says outright: "It does not prove a bundle
fits 120 s, and it does not cure the eighteen failures." That evidence arrives with the roll.
🚨 Inverted budgets. RegistryUpdateReconciler.cs:932 PerPackageAdoptBudget = 3 min sits
inside ServiceDefaults.cs:119 TotalRequestTimeout = 5 min — the outer wait expires before the
inner retry can finish, so the HTTP cause is always discarded for a bare TimeoutException.
Derive one from the other; do not raise either.
The masked Source: fingerprint. This issue and The plugin registry still serves the pre-#4246/#4527 catalog listing — memex.meshweaver.cloud is 934 commits behind and has never run either fix #4222 share one occurrences counter on
Admin/_LogIncident/9ca334c1e8dad9ca, so neither can close on it. That formula lives in the log
watcher in MeshWeaver.Plugins.

🚨 The roll this issue's close condition waits on HAPPENED — and the defect did not stop

Measured read-only 2026-09-19 ~18:5xZ. This is the one reading that changes the disposition, so it is
first: do not read "#4549 merged" as "#4549 finished the job".

The consumer is on a set carrying #4549, on every replica, and has been for ~10.7 h

Ops/Status/memex, sampled 2026-09-19T18:53:11Z:


replicas	`2/2 ready`, `generations: 1`, `converged: true`, `notScraped: false`
image	`meshweaver.azurecr.io/memex-portal-ai:3.0.0-ci.8968` on both
commit	`96f88406b419215a40b3aa97cd4c414ea7b3eb19` on both
`startedAt`	`2026-09-19T08:10:37Z` (`…-6cd5d8f887-lvmpf`), `2026-09-19T08:11:39Z` (`…-6cd5d8f887-2kcwk`)

git merge-base --is-ancestor 32f0523e2 96f88406…   → yes      (#4549)

memex.meshweaver.cloud moved too, to c25f86ae (2026-09-19T07:32:15Z), also carrying 32f0523e2.

And `plugin-registry-bundles` is still timing out, right now

The incident moved node: Admin/_LogIncident/9ca334c1e8dad9ca is now status: "Superseded",
supersededBy: "833c5f2e6337f4d3" (created 2026-09-19T11:41:32Z). On the successor, read at 18:5xZ:

occurrences: 2464, lastSeen: 2026-09-19T18:51:53Z
its shape row: firstSeen: 2026-09-19T06:22:18Z, occurrences: 375
all ten retained samples are Source: 'plugin-registry-bundles-standard//Standard-AttemptTimeout' —
zero plugin-registry-standard

So on an image where the transfer is bounded on silence rather than on total duration, the 120 s
attempt budget on this pipeline is still being exceeded, ~10.5 h after the replicas came up on it.

The cadence is the reconciler's, exactly as the 09-17 comment established

Per pod the retained samples sit at a 180 s period with millisecond jitter:

pod	samples (UTC)	interval
`…-6cd5d8f887-2kcwk`	18:45:51.953 · 18:48:51.959 · 18:51:52.039	180.006 s · 180.080 s
`…-6cd5d8f887-lvmpf`	18:45:53.033 · 18:48:53.034 · 18:51:53.040	180.001 s · 180.006 s
`…-69956b6dbc-ndnxt`	18:46:02.379 · 18:49:02.380	180.001 s
`…-69956b6dbc-s246c`	18:47:06.781 · 18:50:06.781	180.000 s

That is RegistryUpdateReconciler.cs:932 PerPackageAdoptBudget = 3 min, the same spacing the 09-17
comment already attributed ("ten of them spaced at exactly the reconciler's 180 s
PerPackageAdoptBudget"). It is not a new discovery; what is new is that the spacing survived the
roll unchanged, which rules out "one unlucky large bundle" — a timer-shaped population at a fixed period
is a repeating scheduled adopt whose attempt exceeds 120 s every time it runs.

Two of the four pods (69956b6dbc-*) are not in Ops/Status/memex's roster, which lists only
6cd5d8f887-*. So the population spans at least two deployments; I did not chase which, and the two
that are memex replicas are enough to establish the image.

What this does to the three owed items

"One transfer's real byte count and rate — that evidence arrives with the roll." The roll has
arrived, so item 1 is now collectable: :702-706's bytes / elapsed / KiB/s measurement is running on
these pods and lands in the log. It has not been read yet.
🚨 The inverted budgets are now the thing standing between us and item 1. PerPackageAdoptBudget = 3 min inside ServiceDefaults.cs:119 TotalRequestTimeout = 5 min: the reconciler cancels at 180 s,
so the HTTP cause is discarded for a bare TimeoutException — which is exactly why 375 occurrences
today still cannot say which of the two call shapes timed out. Derive one from the other; do not
raise either.
The masked Source: counter is no longer shared. The plugin registry still serves the pre-#4246/#4527 catalog listing — memex.meshweaver.cloud is 934 commits behind and has never run either fix #4222 closed today on its own discriminator
(its last plugin-registry-standard sample is 2026-09-18T15:18:44Z, pre-roll; everything since is
this pipeline). So this issue now owns that counter outright, and a reopen on it is about this
defect.

One reading worth stating, because #4549's own doc raised it: on this image an attempt timeout on
plugin-registry-bundles can no longer be spent streaming — HttpCompletionOption.ResponseHeadersRead
takes the body out of the Polly attempt and CopyStallBounded bounds the copy separately. That is
consistent with the remaining 120 s being spent before the response headers arrive, i.e. candidate
(1) in this issue's "Where to look" (the bundle INDEX and its uncached per-request Query<MeshNode>
work) rather than candidate (2). It is consistent with, not established — item 2 is what is hiding the
proof.

Staying open. The close condition was a roll of the consumer; the roll happened and the symptom did not
clear, so the condition has to change to a measurement, and item 2 is the cheapest step to it.

_{Reads: one get of Ops/Status/memex, two get calls on Admin/_LogIncident nodes, four

curl /api/version, local git merge-base. Nothing written; no InstanceAction filed.}

Follow-up, same session: the population is BOTH production deployments, and the incident now reports HERE

1. Attribution completed — the second pod generation is memex-cloud

Ops/Status/memex-cloud, sampled 2026-09-19T19:03:22Z:

3/3 ready   generations: 1   converged: true   notScraped: false   restarts: 1
image  meshweaver.azurecr.io/memex-portal-ai:3.0.0-ci.8969   commit c25f86ae85b7…
imageProvenance: "3 of 3 replica(s) resolve generation + image + commit"
…-69956b6dbc-gx6z6  startedAt 2026-09-19T08:39:58Z
…-69956b6dbc-ndnxt  startedAt 2026-09-19T08:39:59Z
…-69956b6dbc-s246c  startedAt 2026-09-19T08:34:53Z

So the emitting pods split cleanly across the two production deployments, both converged, both on an
image carrying 32f0523e2 (#4549):

generation	deployment	image	commit	replicas
`…-6cd5d8f887-*`	memex	`3.0.0-ci.8968`	`96f88406`	2
`…-69956b6dbc-*`	memex-cloud	`3.0.0-ci.8969`	`c25f86ae`	3

git merge-base --is-ancestor 32f0523e2 c25f86ae… → yes, as for 96f88406. Five replicas, two
deployments, one verdict: the fix is live everywhere it can be and the attempt budget is still being
exceeded.

2. Fifteen minutes later the shape has advanced 22 more

Re-read at 19:07Z: shape occurrences 375 → 397, node lastSeen: 2026-09-19T19:06:53Z, and again
all ten retained samples plugin-registry-bundles-standard//Standard-AttemptTimeout, now spanning
19:00:52Z → 19:06:53Z across five pods — …-6cd5d8f887-2kcwk at 19:00:52.047 · 19:03:52.052 ·
19:06:52.052 is the 180 s PerPackageAdoptBudget spacing again, to ~5 ms over six minutes.

3. The incident node now points here, not at #4222

This issue's item 3 said the shared counter meant "neither can close on it". #4222 closed today on its
own discriminator, so the counter had to follow the samples or the recurrence bot would have reopened
#4222 on the next tick — which, at ~1.5 occurrences a minute, would have been within the hour.

Admin/_LogIncident/833c5f2e6337f4d3 on memex.systemorph.com, patched v304 → v306:

 "repository": "Systemorph/MeshWeaver",
-"issueNumber": 4222,
-"issueUrl": "https://github.com/Systemorph/MeshWeaver/issues/4222",
+"issueNumber": 4528,
+"issueUrl": "https://github.com/Systemorph/MeshWeaver/issues/4528",

Nothing else on the node was touched — fingerprint, shapes, foldedFrom, reporterFingerprint
(9ca334c1e8dad9ca), status: Filed and the sample ring are as they were. This is the same repointing
the 2026-09-15 comment on #4222 records being done once before (issueNumber 1134 → 4222, v1641 → v1642)
when the masked fingerprint last attributed itself to a fixed defect.

The predecessor node Admin/_LogIncident/9ca334c1e8dad9ca is deliberately left pointing at #4222: it
is status: Superseded, its lastSeen is 2026-09-18T15:18:44Z and it is the historical record of that
issue's own defect. It should not advance again; if it does, that is a plugin-registry regression and
belongs on #4222.

🚨 So a reopen on this issue from now on is about this defect and is trustworthy — which is the first
time that has been true for either issue. The masked Source: formula itself (log watcher, MeshWeaver.Plugins)
is unchanged and still folds every pipeline onto one fingerprint, so item 3 is narrowed rather than closed.

A second, independent consumer reproduces this from outside the portal — and it localises the stall to behind authentication

This was filed about the portal's own reconciler. Tonight a completely different consumer hit the same shape on the same endpoint family, which is worth adding because it rules out a portal-side cause.

Consumer 1, the portal reconciler (measured by another session this evening): #4549 is live on all five replicas of both deployments and the 120 s attempt budget is still exceeded — occurrences went 375 → 397 in fifteen minutes, all ten samples plugin-registry-bundles, at the reconciler's 180 s PerPackageAdoptBudget with about 5 ms drift. Because #4549 took the body out of the Polly attempt, the time can no longer be spent streaming, so the cause is time-to-first-byte rather than the blob transfers this page was originally about.

Consumer 2, MeshWeaver.Education's CI (measured 19:39–19:46Z): a plain curl from a GitHub runner, twice in seven minutes, against https://memex.meshweaver.cloud/api/plugins/bundles/DefaultViews/1.1.29:

curl: (28) Operation timed out after 180002 milliseconds with 0 bytes received
curl: (28) Operation timed out after 180002 milliseconds with 0 bytes received

0 bytes received is the important part. Not a slow stream, not a throttled answer — no response line at all within 180 s. That is the same time-to-first-byte failure, reached by a consumer that shares no code with the reconciler.

Where it is not

I probed the same host unauthenticated just now, read-only:

path	status	total	time-to-first-byte
`/api/version`	200	0.120 s	0.120 s
`/api/plugins/bundles/DefaultViews/1.1.29`	401	0.115 s	0.115 s
`/api/plugins`	401	0.128 s	0.126 s

So DNS, TLS, ingress, routing and the auth rejection path are all healthy and answer in about an eighth of a second. With a valid credential the same request writes nothing for 180 s. The stall is behind authentication, on the authenticated bundle path specifically.

🚨 What that does not establish: a 401 exercises the rejection path, so it cannot name which authenticated path stalls. Attributing it to the mounted share is an inference from source, not a measurement. The read that would settle it is a server-side one on the replica serving that request.

Why this raises the priority rather than just adding detail

Two independent consumers, and the second one is blocking a repository's CI right now. Education's Disposable-mesh gate is a required context and cannot pass while this hangs — and that repo's pull request is otherwise correct, having already converted a silent 916 s cap kill into a named 421 s failure that the ledger can see.

It also means the close condition on this issue cannot be "a roll of the consumer": two consumers, one of them outside the mesh entirely, both wait on the same server.

Related: #4804 covers the 503 + Retry-After exhaustion, which is a different mechanism — a throttled answer rather than no answer — and tonight's shape is the second one. Worth keeping them apart.

🤖 Generated with Claude Code

Still occurring: 271 more since the last update (2505 total).

Last seen: 2026-09-19 19:50:48Z
Pods: memex-portal-deployment-6c99c9f796-mh5cs, memex-portal-deployment-84855f7679-z4rsp, memex-portal-deployment-575dc48bfb-wq7s8, memex-portal-deployment-6bd9c46f8b-z9t6r, memex-portal-deployment-85ddbf6c7f-p4t4b, memex-portal-deployment-59479dc7d9-kmwvr, memex-portal-deployment-748bfcc86b-2m6vg, memex-portal-deployment-6fc79755d4-r96vd, memex-portal-deployment-57885dbb86-76zsg, memex-portal-deployment-6d9cdd447b-tm4ck, memex-portal-deployment-6cd98575f6-gr5ds, memex-portal-deployment-784494885b-c5sz5, memex-portal-deployment-6854bb96cb-z6bn9, memex-portal-deployment-6c9bb5d95c-jb9dq, memex-portal-deployment-69f9fc445-6fws2, memex-portal-deployment-c9584c6dd-dms29, memex-portal-deployment-cc58dc7c6-ckll8, memex-portal-deployment-67967894c4-7fvvg, memex-portal-deployment-c594497fc-dsgds, memex-portal-deployment-86bd794cb5-4rs66, memex-portal-deployment-8489f66759-zlbcg, memex-portal-deployment-745d7588ff-4mrxf, memex-portal-deployment-797df55dfb-cn4p8, memex-portal-deployment-97c56b569-tjnmj, memex-portal-deployment-bf88ffcbb-t6nhv, memex-portal-deployment-6cc6f7997b-99xkc, memex-portal-deployment-7799dd87b9-vshbj, memex-portal-deployment-6fcf6bbc47-c2nw9, memex-portal-deployment-6fcf6bbc47-8qvz6, memex-portal-deployment-79766fd4f6-w66fq, memex-portal-deployment-648c756f74-smlsb, memex-portal-deployment-745b99b846-hrk46, memex-portal-deployment-84799488cb-cb7q4, memex-portal-deployment-78cf8bd9f5-klqft, memex-portal-deployment-f96d46ddf-zn9nl, memex-portal-deployment-8669cb7675-xksqg, memex-portal-deployment-5d6f77b68c-srtqg, memex-portal-deployment-5b7b597fb5-s6xf9, memex-portal-deployment-596d7ff9bf-vtsp6, memex-portal-deployment-7c47bc9c85-xhspn, memex-portal-deployment-79756bcc4f-mb2c5, memex-portal-deployment-65f87d4594-5vsw8, memex-portal-deployment-696cc645c6-gfw6n, memex-portal-deployment-6c5bd56bf6-kqk6s, memex-portal-deployment-747d45bbf7-djm8g, memex-portal-deployment-c89ddfbb6-txnd5, memex-portal-deployment-6dfbd755f6-c5dr6, memex-portal-deployment-66cc6466bf-frwpk, memex-portal-deployment-68cd75c658-8qt9s, memex-portal-deployment-f47f8cdd-rx8xj, memex-portal-deployment-7cb5549c57-8c9gr, memex-portal-deployment-89d7df89f-lrbc4, memex-portal-deployment-89d7df89f-k9fkr, memex-portal-deployment-764d5fbf-nn6nj, memex-portal-deployment-76b967b5fb-pkw5p, memex-portal-deployment-559ddb8f8-dbpx6, memex-portal-deployment-98785f547-76qjj, memex-portal-deployment-59ccff47c5-xw4zx, memex-portal-deployment-d7dd9796b-rc4bh, memex-portal-deployment-d7dd9796b-66g44, memex-portal-deployment-588c4cbb68-jm8cz, memex-portal-deployment-55f6c4c4c5-wcccn, memex-portal-deployment-58cf8c86f7-p95sf, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-7897bfd494-x8rfw, memex-portal-deployment-7897bfd494-dl9wb, memex-portal-deployment-7d6b864b54-w4xnl, memex-portal-deployment-7d6b864b54-gr6n6, memex-portal-deployment-688b496b7-2sdt8, memex-portal-deployment-688b496b7-brp4j, memex-portal-deployment-999c9c79d-8ccj2, memex-portal-deployment-999c9c79d-g9wps, memex-portal-deployment-6dc7cf965d-8f5t6, memex-portal-deployment-64d9857f9d-bmrnj, memex-portal-deployment-64d9857f9d-v7hmx, memex-portal-deployment-df7ddfdf4-xb9bk, memex-portal-deployment-6d58c699f8-bp5bs, memex-portal-deployment-6d58c699f8-9t675, memex-portal-deployment-775748f44d-7xlkm, memex-portal-deployment-7969b8cfff-fqnk9, memex-portal-deployment-76cbbdd5f-9mtkj, memex-portal-deployment-76cbbdd5f-82mvq, memex-portal-deployment-569544576b-q6nvd, memex-portal-deployment-569544576b-89r5k, memex-portal-deployment-569544576b-9h52b, memex-portal-deployment-569544576b-wtfz5, memex-portal-deployment-569544576b-cqlnw, memex-portal-deployment-867dc45877-lgskl, memex-portal-deployment-867dc45877-h8tjc, memex-portal-deployment-867dc45877-fmcss, memex-portal-deployment-58c7bb4b9d-97pjf, memex-portal-deployment-5b6c4f89f8-rpfmq, memex-portal-deployment-58c7bb4b9d-z2xcv, memex-portal-deployment-5b6c4f89f8-hpcbf, memex-portal-deployment-5b6c4f89f8-97sbj, memex-portal-deployment-5b6c4f89f8-kz4rh, memex-portal-deployment-6f64bf4b56-gxmrr, memex-portal-deployment-6f64bf4b56-2mzrf, memex-portal-deployment-6f64bf4b56-xqw96, memex-portal-deployment-858f57dc6-8zlfh, memex-portal-deployment-547649b6f8-wmt6n, memex-portal-deployment-547649b6f8-48jrz, memex-portal-deployment-858f57dc6-wns9j, memex-portal-deployment-7b67cb5577-rctvc, memex-portal-deployment-7d8787bcc7-4htmx, memex-portal-deployment-7544fdd47b-pbrvx, memex-portal-deployment-6dc5679f77-nd67w, memex-portal-deployment-6dc5679f77-vf2qx, memex-portal-deployment-858f57dc6-bkkfv, memex-portal-deployment-858f57dc6-ff5kn, memex-portal-deployment-858f57dc6-6wggn, memex-portal-deployment-6fd666bc95-s79r4, memex-portal-deployment-6fd666bc95-f7brs, memex-portal-deployment-66ff6fc5f9-nwl9f, memex-portal-deployment-66ff6fc5f9-mnw9n, memex-portal-deployment-79b5fd5f6b-7wdn4, memex-portal-deployment-79b5fd5f6b-jg4td, memex-portal-deployment-cf988c6f5-gtqpl, memex-portal-deployment-7b5f547cb7-cp2wv, memex-portal-deployment-7b5f547cb7-ps2nz, memex-portal-deployment-6c7b9889c7-rkr8g, memex-portal-deployment-6c7b9889c7-72fmv, memex-portal-deployment-cf5dc996f-pgjtz, memex-portal-deployment-cf5dc996f-gk2bv, memex-portal-deployment-6c46b84848-hnh9v, memex-portal-deployment-6c46b84848-c9mm5, memex-portal-deployment-6bfbc8ff4b-rz9p4, memex-portal-deployment-6bfbc8ff4b-vvc82, memex-portal-deployment-86779d495b-ksxqq, memex-portal-deployment-86779d495b-qrnh6, memex-portal-deployment-595d8b9995-j5f9p, memex-portal-deployment-595d8b9995-xntj7, memex-portal-deployment-7d95465fbb-s7gds, memex-portal-deployment-7d95465fbb-qrhvg, memex-portal-deployment-5df4ccb89-z8w2k, memex-portal-deployment-5df4ccb89-sprw2, memex-portal-deployment-5df4ccb89-txn2l, memex-portal-deployment-d999f9888-sfpph, memex-portal-deployment-dbc8997cd-q2n4s, memex-portal-deployment-66cfbf759-svxh9, memex-portal-deployment-75597864c8-wwfnf, memex-portal-deployment-9bf79b774-tk88b, memex-portal-deployment-67dbc7bc9c-lswwx, memex-portal-deployment-9bf79b774-qlks7, memex-portal-deployment-9bf79b774-jwrng, memex-portal-deployment-85859db964-mgxj7, memex-portal-deployment-77cfc55bfc-5ztnt, memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-54f4c96b6b-pn9z5, memex-portal-deployment-77cfc55bfc-tgtr4, memex-portal-deployment-77cfc55bfc-gxz25, memex-portal-deployment-5b9cbf46dd-5nqbz, memex-portal-deployment-5b9cbf46dd-zn7nf, memex-portal-deployment-79ddd44d77-k8qlc, memex-portal-deployment-6c74f8458-w46gf, memex-portal-deployment-bf77c847-tctvp, memex-portal-deployment-bf77c847-dmpfd, memex-portal-deployment-bf77c847-m5l4r, memex-portal-deployment-5bf869698f-5qgvz, memex-portal-deployment-574d54c98d-cdd47, memex-portal-deployment-747c879d8c-ph572, memex-portal-deployment-8ff67f8c7-gcf24, memex-portal-deployment-7fd9668b69-sjm5v, memex-portal-deployment-bfc5dcf6f-7z88q, memex-portal-deployment-6c7d6b6b44-klrt7, memex-portal-deployment-7fd9668b69-j4fdv, memex-portal-deployment-7fd9668b69-4qtkx, memex-portal-deployment-854cd8cd6d-8bc2q, memex-portal-deployment-854cd8cd6d-r4cn5, memex-portal-deployment-7fd9668b69-lzm7l, memex-portal-deployment-7fd9668b69-v72dx, memex-portal-deployment-6546c779d4-4ngqv, memex-portal-deployment-7fd9668b69-4zzzs, memex-portal-deployment-7fd9668b69-jnwlv, memex-portal-deployment-7fd9668b69-kvwgv, memex-portal-deployment-7cb6684584-ztqz8, memex-portal-deployment-54787585bc-qfwxx, memex-portal-deployment-6c967c7c4c-2k4xb, memex-portal-deployment-6b879996c7-dstvb, memex-portal-deployment-6b879996c7-lk6fd, memex-portal-deployment-6b879996c7-cmnfg, memex-portal-deployment-6c967c7c4c-kdff5, memex-portal-deployment-bbf878dfb-gknl9, memex-portal-deployment-5564495d46-f8xhs, memex-portal-deployment-86899455c7-z9nm2, memex-portal-deployment-6cd5d8f887-lvmpf, memex-portal-deployment-6cd5d8f887-2kcwk, memex-portal-deployment-58dbb6d98d-8msww, memex-portal-deployment-58dbb6d98d-lzqrr, memex-portal-deployment-7978987466-kmrpk, memex-portal-deployment-85f5bcf79d-n6lj2, memex-portal-deployment-85f5bcf79d-cgwtr, memex-portal-deployment-69956b6dbc-ndnxt, memex-portal-deployment-69956b6dbc-gx6z6, memex-portal-deployment-69956b6dbc-s246c, memex-portal-deployment-69956b6dbc-gj8m2, memex-portal-deployment-69956b6dbc-v5r29

Recent log lines

2026-09-19 19:42:53Z memex-portal-deployment-6cd5d8f887-lvmpf fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'plugin-registry-bundles-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''
2026-09-19 19:45:53Z memex-portal-deployment-6cd5d8f887-lvmpf fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'plugin-registry-bundles-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''
2026-09-19 19:50:48Z memex-portal-deployment-69956b6dbc-s246c fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'plugin-registry-bundles-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''

_{Re-addressed by the current identity function: this incident inherited 9ca334c1e8dad9ca (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Still occurring: 412 more since the last update (2917 total).

Last seen: 2026-09-20 01:51:36Z
Pods: memex-portal-deployment-6c99c9f796-mh5cs, memex-portal-deployment-84855f7679-z4rsp, memex-portal-deployment-575dc48bfb-wq7s8, memex-portal-deployment-6bd9c46f8b-z9t6r, memex-portal-deployment-85ddbf6c7f-p4t4b, memex-portal-deployment-59479dc7d9-kmwvr, memex-portal-deployment-748bfcc86b-2m6vg, memex-portal-deployment-6fc79755d4-r96vd, memex-portal-deployment-57885dbb86-76zsg, memex-portal-deployment-6d9cdd447b-tm4ck, memex-portal-deployment-6cd98575f6-gr5ds, memex-portal-deployment-784494885b-c5sz5, memex-portal-deployment-6854bb96cb-z6bn9, memex-portal-deployment-6c9bb5d95c-jb9dq, memex-portal-deployment-69f9fc445-6fws2, memex-portal-deployment-c9584c6dd-dms29, memex-portal-deployment-cc58dc7c6-ckll8, memex-portal-deployment-67967894c4-7fvvg, memex-portal-deployment-c594497fc-dsgds, memex-portal-deployment-86bd794cb5-4rs66, memex-portal-deployment-8489f66759-zlbcg, memex-portal-deployment-745d7588ff-4mrxf, memex-portal-deployment-797df55dfb-cn4p8, memex-portal-deployment-97c56b569-tjnmj, memex-portal-deployment-bf88ffcbb-t6nhv, memex-portal-deployment-6cc6f7997b-99xkc, memex-portal-deployment-7799dd87b9-vshbj, memex-portal-deployment-6fcf6bbc47-c2nw9, memex-portal-deployment-6fcf6bbc47-8qvz6, memex-portal-deployment-79766fd4f6-w66fq, memex-portal-deployment-648c756f74-smlsb, memex-portal-deployment-745b99b846-hrk46, memex-portal-deployment-84799488cb-cb7q4, memex-portal-deployment-78cf8bd9f5-klqft, memex-portal-deployment-f96d46ddf-zn9nl, memex-portal-deployment-8669cb7675-xksqg, memex-portal-deployment-5d6f77b68c-srtqg, memex-portal-deployment-5b7b597fb5-s6xf9, memex-portal-deployment-596d7ff9bf-vtsp6, memex-portal-deployment-7c47bc9c85-xhspn, memex-portal-deployment-79756bcc4f-mb2c5, memex-portal-deployment-65f87d4594-5vsw8, memex-portal-deployment-696cc645c6-gfw6n, memex-portal-deployment-6c5bd56bf6-kqk6s, memex-portal-deployment-747d45bbf7-djm8g, memex-portal-deployment-c89ddfbb6-txnd5, memex-portal-deployment-6dfbd755f6-c5dr6, memex-portal-deployment-66cc6466bf-frwpk, memex-portal-deployment-68cd75c658-8qt9s, memex-portal-deployment-f47f8cdd-rx8xj, memex-portal-deployment-7cb5549c57-8c9gr, memex-portal-deployment-89d7df89f-lrbc4, memex-portal-deployment-89d7df89f-k9fkr, memex-portal-deployment-764d5fbf-nn6nj, memex-portal-deployment-76b967b5fb-pkw5p, memex-portal-deployment-559ddb8f8-dbpx6, memex-portal-deployment-98785f547-76qjj, memex-portal-deployment-59ccff47c5-xw4zx, memex-portal-deployment-d7dd9796b-rc4bh, memex-portal-deployment-d7dd9796b-66g44, memex-portal-deployment-588c4cbb68-jm8cz, memex-portal-deployment-55f6c4c4c5-wcccn, memex-portal-deployment-58cf8c86f7-p95sf, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-7897bfd494-x8rfw, memex-portal-deployment-7897bfd494-dl9wb, memex-portal-deployment-7d6b864b54-w4xnl, memex-portal-deployment-7d6b864b54-gr6n6, memex-portal-deployment-688b496b7-2sdt8, memex-portal-deployment-688b496b7-brp4j, memex-portal-deployment-999c9c79d-8ccj2, memex-portal-deployment-999c9c79d-g9wps, memex-portal-deployment-6dc7cf965d-8f5t6, memex-portal-deployment-64d9857f9d-bmrnj, memex-portal-deployment-64d9857f9d-v7hmx, memex-portal-deployment-df7ddfdf4-xb9bk, memex-portal-deployment-6d58c699f8-bp5bs, memex-portal-deployment-6d58c699f8-9t675, memex-portal-deployment-775748f44d-7xlkm, memex-portal-deployment-7969b8cfff-fqnk9, memex-portal-deployment-76cbbdd5f-9mtkj, memex-portal-deployment-76cbbdd5f-82mvq, memex-portal-deployment-569544576b-q6nvd, memex-portal-deployment-569544576b-89r5k, memex-portal-deployment-569544576b-9h52b, memex-portal-deployment-569544576b-wtfz5, memex-portal-deployment-569544576b-cqlnw, memex-portal-deployment-867dc45877-lgskl, memex-portal-deployment-867dc45877-h8tjc, memex-portal-deployment-867dc45877-fmcss, memex-portal-deployment-58c7bb4b9d-97pjf, memex-portal-deployment-5b6c4f89f8-rpfmq, memex-portal-deployment-58c7bb4b9d-z2xcv, memex-portal-deployment-5b6c4f89f8-hpcbf, memex-portal-deployment-5b6c4f89f8-97sbj, memex-portal-deployment-5b6c4f89f8-kz4rh, memex-portal-deployment-6f64bf4b56-gxmrr, memex-portal-deployment-6f64bf4b56-2mzrf, memex-portal-deployment-6f64bf4b56-xqw96, memex-portal-deployment-858f57dc6-8zlfh, memex-portal-deployment-547649b6f8-wmt6n, memex-portal-deployment-547649b6f8-48jrz, memex-portal-deployment-858f57dc6-wns9j, memex-portal-deployment-7b67cb5577-rctvc, memex-portal-deployment-7d8787bcc7-4htmx, memex-portal-deployment-7544fdd47b-pbrvx, memex-portal-deployment-6dc5679f77-nd67w, memex-portal-deployment-6dc5679f77-vf2qx, memex-portal-deployment-858f57dc6-bkkfv, memex-portal-deployment-858f57dc6-ff5kn, memex-portal-deployment-858f57dc6-6wggn, memex-portal-deployment-6fd666bc95-s79r4, memex-portal-deployment-6fd666bc95-f7brs, memex-portal-deployment-66ff6fc5f9-nwl9f, memex-portal-deployment-66ff6fc5f9-mnw9n, memex-portal-deployment-79b5fd5f6b-7wdn4, memex-portal-deployment-79b5fd5f6b-jg4td, memex-portal-deployment-cf988c6f5-gtqpl, memex-portal-deployment-7b5f547cb7-cp2wv, memex-portal-deployment-7b5f547cb7-ps2nz, memex-portal-deployment-6c7b9889c7-rkr8g, memex-portal-deployment-6c7b9889c7-72fmv, memex-portal-deployment-cf5dc996f-pgjtz, memex-portal-deployment-cf5dc996f-gk2bv, memex-portal-deployment-6c46b84848-hnh9v, memex-portal-deployment-6c46b84848-c9mm5, memex-portal-deployment-6bfbc8ff4b-rz9p4, memex-portal-deployment-6bfbc8ff4b-vvc82, memex-portal-deployment-86779d495b-ksxqq, memex-portal-deployment-86779d495b-qrnh6, memex-portal-deployment-595d8b9995-j5f9p, memex-portal-deployment-595d8b9995-xntj7, memex-portal-deployment-7d95465fbb-s7gds, memex-portal-deployment-7d95465fbb-qrhvg, memex-portal-deployment-5df4ccb89-z8w2k, memex-portal-deployment-5df4ccb89-sprw2, memex-portal-deployment-5df4ccb89-txn2l, memex-portal-deployment-d999f9888-sfpph, memex-portal-deployment-dbc8997cd-q2n4s, memex-portal-deployment-66cfbf759-svxh9, memex-portal-deployment-75597864c8-wwfnf, memex-portal-deployment-9bf79b774-tk88b, memex-portal-deployment-67dbc7bc9c-lswwx, memex-portal-deployment-9bf79b774-qlks7, memex-portal-deployment-9bf79b774-jwrng, memex-portal-deployment-85859db964-mgxj7, memex-portal-deployment-77cfc55bfc-5ztnt, memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-54f4c96b6b-pn9z5, memex-portal-deployment-77cfc55bfc-tgtr4, memex-portal-deployment-77cfc55bfc-gxz25, memex-portal-deployment-5b9cbf46dd-5nqbz, memex-portal-deployment-5b9cbf46dd-zn7nf, memex-portal-deployment-79ddd44d77-k8qlc, memex-portal-deployment-6c74f8458-w46gf, memex-portal-deployment-bf77c847-tctvp, memex-portal-deployment-bf77c847-dmpfd, memex-portal-deployment-bf77c847-m5l4r, memex-portal-deployment-5bf869698f-5qgvz, memex-portal-deployment-574d54c98d-cdd47, memex-portal-deployment-747c879d8c-ph572, memex-portal-deployment-8ff67f8c7-gcf24, memex-portal-deployment-7fd9668b69-sjm5v, memex-portal-deployment-bfc5dcf6f-7z88q, memex-portal-deployment-6c7d6b6b44-klrt7, memex-portal-deployment-7fd9668b69-j4fdv, memex-portal-deployment-7fd9668b69-4qtkx, memex-portal-deployment-854cd8cd6d-8bc2q, memex-portal-deployment-854cd8cd6d-r4cn5, memex-portal-deployment-7fd9668b69-lzm7l, memex-portal-deployment-7fd9668b69-v72dx, memex-portal-deployment-6546c779d4-4ngqv, memex-portal-deployment-7fd9668b69-4zzzs, memex-portal-deployment-7fd9668b69-jnwlv, memex-portal-deployment-7fd9668b69-kvwgv, memex-portal-deployment-7cb6684584-ztqz8, memex-portal-deployment-54787585bc-qfwxx, memex-portal-deployment-6c967c7c4c-2k4xb, memex-portal-deployment-6b879996c7-dstvb, memex-portal-deployment-6b879996c7-lk6fd, memex-portal-deployment-6b879996c7-cmnfg, memex-portal-deployment-6c967c7c4c-kdff5, memex-portal-deployment-bbf878dfb-gknl9, memex-portal-deployment-5564495d46-f8xhs, memex-portal-deployment-86899455c7-z9nm2, memex-portal-deployment-6cd5d8f887-lvmpf, memex-portal-deployment-6cd5d8f887-2kcwk, memex-portal-deployment-58dbb6d98d-8msww, memex-portal-deployment-58dbb6d98d-lzqrr, memex-portal-deployment-7978987466-kmrpk, memex-portal-deployment-85f5bcf79d-n6lj2, memex-portal-deployment-85f5bcf79d-cgwtr, memex-portal-deployment-69956b6dbc-ndnxt, memex-portal-deployment-69956b6dbc-gx6z6, memex-portal-deployment-69956b6dbc-s246c, memex-portal-deployment-69956b6dbc-gj8m2, memex-portal-deployment-69956b6dbc-v5r29

Recent log lines

2026-09-20 01:48:36Z memex-portal-deployment-69956b6dbc-s246c fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'plugin-registry-bundles-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''
2026-09-20 01:49:04Z memex-portal-deployment-6cd5d8f887-lvmpf fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'plugin-registry-bundles-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''
2026-09-20 01:51:36Z memex-portal-deployment-69956b6dbc-s246c fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'plugin-registry-bundles-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''

_{Re-addressed by the current identity function: this incident inherited 9ca334c1e8dad9ca (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Still occurring: 893 more since the last update (3810 total).

Last seen: 2026-09-20 21:53:15Z
Pods: memex-portal-deployment-6c99c9f796-mh5cs, memex-portal-deployment-84855f7679-z4rsp, memex-portal-deployment-575dc48bfb-wq7s8, memex-portal-deployment-6bd9c46f8b-z9t6r, memex-portal-deployment-85ddbf6c7f-p4t4b, memex-portal-deployment-59479dc7d9-kmwvr, memex-portal-deployment-748bfcc86b-2m6vg, memex-portal-deployment-6fc79755d4-r96vd, memex-portal-deployment-57885dbb86-76zsg, memex-portal-deployment-6d9cdd447b-tm4ck, memex-portal-deployment-6cd98575f6-gr5ds, memex-portal-deployment-784494885b-c5sz5, memex-portal-deployment-6854bb96cb-z6bn9, memex-portal-deployment-6c9bb5d95c-jb9dq, memex-portal-deployment-69f9fc445-6fws2, memex-portal-deployment-c9584c6dd-dms29, memex-portal-deployment-cc58dc7c6-ckll8, memex-portal-deployment-67967894c4-7fvvg, memex-portal-deployment-c594497fc-dsgds, memex-portal-deployment-86bd794cb5-4rs66, memex-portal-deployment-8489f66759-zlbcg, memex-portal-deployment-745d7588ff-4mrxf, memex-portal-deployment-797df55dfb-cn4p8, memex-portal-deployment-97c56b569-tjnmj, memex-portal-deployment-bf88ffcbb-t6nhv, memex-portal-deployment-6cc6f7997b-99xkc, memex-portal-deployment-7799dd87b9-vshbj, memex-portal-deployment-6fcf6bbc47-c2nw9, memex-portal-deployment-6fcf6bbc47-8qvz6, memex-portal-deployment-79766fd4f6-w66fq, memex-portal-deployment-648c756f74-smlsb, memex-portal-deployment-745b99b846-hrk46, memex-portal-deployment-84799488cb-cb7q4, memex-portal-deployment-78cf8bd9f5-klqft, memex-portal-deployment-f96d46ddf-zn9nl, memex-portal-deployment-8669cb7675-xksqg, memex-portal-deployment-5d6f77b68c-srtqg, memex-portal-deployment-5b7b597fb5-s6xf9, memex-portal-deployment-596d7ff9bf-vtsp6, memex-portal-deployment-7c47bc9c85-xhspn, memex-portal-deployment-79756bcc4f-mb2c5, memex-portal-deployment-65f87d4594-5vsw8, memex-portal-deployment-696cc645c6-gfw6n, memex-portal-deployment-6c5bd56bf6-kqk6s, memex-portal-deployment-747d45bbf7-djm8g, memex-portal-deployment-c89ddfbb6-txnd5, memex-portal-deployment-6dfbd755f6-c5dr6, memex-portal-deployment-66cc6466bf-frwpk, memex-portal-deployment-68cd75c658-8qt9s, memex-portal-deployment-f47f8cdd-rx8xj, memex-portal-deployment-7cb5549c57-8c9gr, memex-portal-deployment-89d7df89f-lrbc4, memex-portal-deployment-89d7df89f-k9fkr, memex-portal-deployment-764d5fbf-nn6nj, memex-portal-deployment-76b967b5fb-pkw5p, memex-portal-deployment-559ddb8f8-dbpx6, memex-portal-deployment-98785f547-76qjj, memex-portal-deployment-59ccff47c5-xw4zx, memex-portal-deployment-d7dd9796b-rc4bh, memex-portal-deployment-d7dd9796b-66g44, memex-portal-deployment-588c4cbb68-jm8cz, memex-portal-deployment-55f6c4c4c5-wcccn, memex-portal-deployment-58cf8c86f7-p95sf, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-7897bfd494-x8rfw, memex-portal-deployment-7897bfd494-dl9wb, memex-portal-deployment-7d6b864b54-w4xnl, memex-portal-deployment-7d6b864b54-gr6n6, memex-portal-deployment-688b496b7-2sdt8, memex-portal-deployment-688b496b7-brp4j, memex-portal-deployment-999c9c79d-8ccj2, memex-portal-deployment-999c9c79d-g9wps, memex-portal-deployment-6dc7cf965d-8f5t6, memex-portal-deployment-64d9857f9d-bmrnj, memex-portal-deployment-64d9857f9d-v7hmx, memex-portal-deployment-df7ddfdf4-xb9bk, memex-portal-deployment-6d58c699f8-bp5bs, memex-portal-deployment-6d58c699f8-9t675, memex-portal-deployment-775748f44d-7xlkm, memex-portal-deployment-7969b8cfff-fqnk9, memex-portal-deployment-76cbbdd5f-9mtkj, memex-portal-deployment-76cbbdd5f-82mvq, memex-portal-deployment-569544576b-q6nvd, memex-portal-deployment-569544576b-89r5k, memex-portal-deployment-569544576b-9h52b, memex-portal-deployment-569544576b-wtfz5, memex-portal-deployment-569544576b-cqlnw, memex-portal-deployment-867dc45877-lgskl, memex-portal-deployment-867dc45877-h8tjc, memex-portal-deployment-867dc45877-fmcss, memex-portal-deployment-58c7bb4b9d-97pjf, memex-portal-deployment-5b6c4f89f8-rpfmq, memex-portal-deployment-58c7bb4b9d-z2xcv, memex-portal-deployment-5b6c4f89f8-hpcbf, memex-portal-deployment-5b6c4f89f8-97sbj, memex-portal-deployment-5b6c4f89f8-kz4rh, memex-portal-deployment-6f64bf4b56-gxmrr, memex-portal-deployment-6f64bf4b56-2mzrf, memex-portal-deployment-6f64bf4b56-xqw96, memex-portal-deployment-858f57dc6-8zlfh, memex-portal-deployment-547649b6f8-wmt6n, memex-portal-deployment-547649b6f8-48jrz, memex-portal-deployment-858f57dc6-wns9j, memex-portal-deployment-7b67cb5577-rctvc, memex-portal-deployment-7d8787bcc7-4htmx, memex-portal-deployment-7544fdd47b-pbrvx, memex-portal-deployment-6dc5679f77-nd67w, memex-portal-deployment-6dc5679f77-vf2qx, memex-portal-deployment-858f57dc6-bkkfv, memex-portal-deployment-858f57dc6-ff5kn, memex-portal-deployment-858f57dc6-6wggn, memex-portal-deployment-6fd666bc95-s79r4, memex-portal-deployment-6fd666bc95-f7brs, memex-portal-deployment-66ff6fc5f9-nwl9f, memex-portal-deployment-66ff6fc5f9-mnw9n, memex-portal-deployment-79b5fd5f6b-7wdn4, memex-portal-deployment-79b5fd5f6b-jg4td, memex-portal-deployment-cf988c6f5-gtqpl, memex-portal-deployment-7b5f547cb7-cp2wv, memex-portal-deployment-7b5f547cb7-ps2nz, memex-portal-deployment-6c7b9889c7-rkr8g, memex-portal-deployment-6c7b9889c7-72fmv, memex-portal-deployment-cf5dc996f-pgjtz, memex-portal-deployment-cf5dc996f-gk2bv, memex-portal-deployment-6c46b84848-hnh9v, memex-portal-deployment-6c46b84848-c9mm5, memex-portal-deployment-6bfbc8ff4b-rz9p4, memex-portal-deployment-6bfbc8ff4b-vvc82, memex-portal-deployment-86779d495b-ksxqq, memex-portal-deployment-86779d495b-qrnh6, memex-portal-deployment-595d8b9995-j5f9p, memex-portal-deployment-595d8b9995-xntj7, memex-portal-deployment-7d95465fbb-s7gds, memex-portal-deployment-7d95465fbb-qrhvg, memex-portal-deployment-5df4ccb89-z8w2k, memex-portal-deployment-5df4ccb89-sprw2, memex-portal-deployment-5df4ccb89-txn2l, memex-portal-deployment-d999f9888-sfpph, memex-portal-deployment-dbc8997cd-q2n4s, memex-portal-deployment-66cfbf759-svxh9, memex-portal-deployment-75597864c8-wwfnf, memex-portal-deployment-9bf79b774-tk88b, memex-portal-deployment-67dbc7bc9c-lswwx, memex-portal-deployment-9bf79b774-qlks7, memex-portal-deployment-9bf79b774-jwrng, memex-portal-deployment-85859db964-mgxj7, memex-portal-deployment-77cfc55bfc-5ztnt, memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-54f4c96b6b-pn9z5, memex-portal-deployment-77cfc55bfc-tgtr4, memex-portal-deployment-77cfc55bfc-gxz25, memex-portal-deployment-5b9cbf46dd-5nqbz, memex-portal-deployment-5b9cbf46dd-zn7nf, memex-portal-deployment-79ddd44d77-k8qlc, memex-portal-deployment-6c74f8458-w46gf, memex-portal-deployment-bf77c847-tctvp, memex-portal-deployment-bf77c847-dmpfd, memex-portal-deployment-bf77c847-m5l4r, memex-portal-deployment-5bf869698f-5qgvz, memex-portal-deployment-574d54c98d-cdd47, memex-portal-deployment-747c879d8c-ph572, memex-portal-deployment-8ff67f8c7-gcf24, memex-portal-deployment-7fd9668b69-sjm5v, memex-portal-deployment-bfc5dcf6f-7z88q, memex-portal-deployment-6c7d6b6b44-klrt7, memex-portal-deployment-7fd9668b69-j4fdv, memex-portal-deployment-7fd9668b69-4qtkx, memex-portal-deployment-854cd8cd6d-8bc2q, memex-portal-deployment-854cd8cd6d-r4cn5, memex-portal-deployment-7fd9668b69-lzm7l, memex-portal-deployment-7fd9668b69-v72dx, memex-portal-deployment-6546c779d4-4ngqv, memex-portal-deployment-7fd9668b69-4zzzs, memex-portal-deployment-7fd9668b69-jnwlv, memex-portal-deployment-7fd9668b69-kvwgv, memex-portal-deployment-7cb6684584-ztqz8, memex-portal-deployment-54787585bc-qfwxx, memex-portal-deployment-6c967c7c4c-2k4xb, memex-portal-deployment-6b879996c7-dstvb, memex-portal-deployment-6b879996c7-lk6fd, memex-portal-deployment-6b879996c7-cmnfg, memex-portal-deployment-6c967c7c4c-kdff5, memex-portal-deployment-bbf878dfb-gknl9, memex-portal-deployment-5564495d46-f8xhs, memex-portal-deployment-86899455c7-z9nm2, memex-portal-deployment-6cd5d8f887-lvmpf, memex-portal-deployment-6cd5d8f887-2kcwk, memex-portal-deployment-58dbb6d98d-8msww, memex-portal-deployment-58dbb6d98d-lzqrr, memex-portal-deployment-7978987466-kmrpk, memex-portal-deployment-85f5bcf79d-n6lj2, memex-portal-deployment-85f5bcf79d-cgwtr, memex-portal-deployment-69956b6dbc-ndnxt, memex-portal-deployment-69956b6dbc-gx6z6, memex-portal-deployment-69956b6dbc-s246c, memex-portal-deployment-69956b6dbc-gj8m2, memex-portal-deployment-69956b6dbc-v5r29, memex-portal-deployment-65f8c68995-c7z8l, memex-portal-deployment-65f8c68995-z27tq, memex-portal-deployment-5f69f6b6bd-74xjv, memex-portal-deployment-5f69f6b6bd-z7tb6, memex-portal-deployment-69956b6dbc-ntksf, memex-portal-deployment-69956b6dbc-mczlv

Recent log lines

2026-09-20 21:50:15Z memex-portal-deployment-5f69f6b6bd-74xjv fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'plugin-registry-bundles-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''
2026-09-20 21:52:01Z memex-portal-deployment-69956b6dbc-mczlv fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'plugin-registry-bundles-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''
2026-09-20 21:53:15Z memex-portal-deployment-5f69f6b6bd-74xjv fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'plugin-registry-bundles-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''

_{Re-addressed by the current identity function: this incident inherited 9ca334c1e8dad9ca (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Still occurring: 45 more since the last update (3855 total).

Last seen: 2026-09-20 22:25:14Z
Pods: memex-portal-deployment-6c99c9f796-mh5cs, memex-portal-deployment-84855f7679-z4rsp, memex-portal-deployment-575dc48bfb-wq7s8, memex-portal-deployment-6bd9c46f8b-z9t6r, memex-portal-deployment-85ddbf6c7f-p4t4b, memex-portal-deployment-59479dc7d9-kmwvr, memex-portal-deployment-748bfcc86b-2m6vg, memex-portal-deployment-6fc79755d4-r96vd, memex-portal-deployment-57885dbb86-76zsg, memex-portal-deployment-6d9cdd447b-tm4ck, memex-portal-deployment-6cd98575f6-gr5ds, memex-portal-deployment-784494885b-c5sz5, memex-portal-deployment-6854bb96cb-z6bn9, memex-portal-deployment-6c9bb5d95c-jb9dq, memex-portal-deployment-69f9fc445-6fws2, memex-portal-deployment-c9584c6dd-dms29, memex-portal-deployment-cc58dc7c6-ckll8, memex-portal-deployment-67967894c4-7fvvg, memex-portal-deployment-c594497fc-dsgds, memex-portal-deployment-86bd794cb5-4rs66, memex-portal-deployment-8489f66759-zlbcg, memex-portal-deployment-745d7588ff-4mrxf, memex-portal-deployment-797df55dfb-cn4p8, memex-portal-deployment-97c56b569-tjnmj, memex-portal-deployment-bf88ffcbb-t6nhv, memex-portal-deployment-6cc6f7997b-99xkc, memex-portal-deployment-7799dd87b9-vshbj, memex-portal-deployment-6fcf6bbc47-c2nw9, memex-portal-deployment-6fcf6bbc47-8qvz6, memex-portal-deployment-79766fd4f6-w66fq, memex-portal-deployment-648c756f74-smlsb, memex-portal-deployment-745b99b846-hrk46, memex-portal-deployment-84799488cb-cb7q4, memex-portal-deployment-78cf8bd9f5-klqft, memex-portal-deployment-f96d46ddf-zn9nl, memex-portal-deployment-8669cb7675-xksqg, memex-portal-deployment-5d6f77b68c-srtqg, memex-portal-deployment-5b7b597fb5-s6xf9, memex-portal-deployment-596d7ff9bf-vtsp6, memex-portal-deployment-7c47bc9c85-xhspn, memex-portal-deployment-79756bcc4f-mb2c5, memex-portal-deployment-65f87d4594-5vsw8, memex-portal-deployment-696cc645c6-gfw6n, memex-portal-deployment-6c5bd56bf6-kqk6s, memex-portal-deployment-747d45bbf7-djm8g, memex-portal-deployment-c89ddfbb6-txnd5, memex-portal-deployment-6dfbd755f6-c5dr6, memex-portal-deployment-66cc6466bf-frwpk, memex-portal-deployment-68cd75c658-8qt9s, memex-portal-deployment-f47f8cdd-rx8xj, memex-portal-deployment-7cb5549c57-8c9gr, memex-portal-deployment-89d7df89f-lrbc4, memex-portal-deployment-89d7df89f-k9fkr, memex-portal-deployment-764d5fbf-nn6nj, memex-portal-deployment-76b967b5fb-pkw5p, memex-portal-deployment-559ddb8f8-dbpx6, memex-portal-deployment-98785f547-76qjj, memex-portal-deployment-59ccff47c5-xw4zx, memex-portal-deployment-d7dd9796b-rc4bh, memex-portal-deployment-d7dd9796b-66g44, memex-portal-deployment-588c4cbb68-jm8cz, memex-portal-deployment-55f6c4c4c5-wcccn, memex-portal-deployment-58cf8c86f7-p95sf, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-7897bfd494-x8rfw, memex-portal-deployment-7897bfd494-dl9wb, memex-portal-deployment-7d6b864b54-w4xnl, memex-portal-deployment-7d6b864b54-gr6n6, memex-portal-deployment-688b496b7-2sdt8, memex-portal-deployment-688b496b7-brp4j, memex-portal-deployment-999c9c79d-8ccj2, memex-portal-deployment-999c9c79d-g9wps, memex-portal-deployment-6dc7cf965d-8f5t6, memex-portal-deployment-64d9857f9d-bmrnj, memex-portal-deployment-64d9857f9d-v7hmx, memex-portal-deployment-df7ddfdf4-xb9bk, memex-portal-deployment-6d58c699f8-bp5bs, memex-portal-deployment-6d58c699f8-9t675, memex-portal-deployment-775748f44d-7xlkm, memex-portal-deployment-7969b8cfff-fqnk9, memex-portal-deployment-76cbbdd5f-9mtkj, memex-portal-deployment-76cbbdd5f-82mvq, memex-portal-deployment-569544576b-q6nvd, memex-portal-deployment-569544576b-89r5k, memex-portal-deployment-569544576b-9h52b, memex-portal-deployment-569544576b-wtfz5, memex-portal-deployment-569544576b-cqlnw, memex-portal-deployment-867dc45877-lgskl, memex-portal-deployment-867dc45877-h8tjc, memex-portal-deployment-867dc45877-fmcss, memex-portal-deployment-58c7bb4b9d-97pjf, memex-portal-deployment-5b6c4f89f8-rpfmq, memex-portal-deployment-58c7bb4b9d-z2xcv, memex-portal-deployment-5b6c4f89f8-hpcbf, memex-portal-deployment-5b6c4f89f8-97sbj, memex-portal-deployment-5b6c4f89f8-kz4rh, memex-portal-deployment-6f64bf4b56-gxmrr, memex-portal-deployment-6f64bf4b56-2mzrf, memex-portal-deployment-6f64bf4b56-xqw96, memex-portal-deployment-858f57dc6-8zlfh, memex-portal-deployment-547649b6f8-wmt6n, memex-portal-deployment-547649b6f8-48jrz, memex-portal-deployment-858f57dc6-wns9j, memex-portal-deployment-7b67cb5577-rctvc, memex-portal-deployment-7d8787bcc7-4htmx, memex-portal-deployment-7544fdd47b-pbrvx, memex-portal-deployment-6dc5679f77-nd67w, memex-portal-deployment-6dc5679f77-vf2qx, memex-portal-deployment-858f57dc6-bkkfv, memex-portal-deployment-858f57dc6-ff5kn, memex-portal-deployment-858f57dc6-6wggn, memex-portal-deployment-6fd666bc95-s79r4, memex-portal-deployment-6fd666bc95-f7brs, memex-portal-deployment-66ff6fc5f9-nwl9f, memex-portal-deployment-66ff6fc5f9-mnw9n, memex-portal-deployment-79b5fd5f6b-7wdn4, memex-portal-deployment-79b5fd5f6b-jg4td, memex-portal-deployment-cf988c6f5-gtqpl, memex-portal-deployment-7b5f547cb7-cp2wv, memex-portal-deployment-7b5f547cb7-ps2nz, memex-portal-deployment-6c7b9889c7-rkr8g, memex-portal-deployment-6c7b9889c7-72fmv, memex-portal-deployment-cf5dc996f-pgjtz, memex-portal-deployment-cf5dc996f-gk2bv, memex-portal-deployment-6c46b84848-hnh9v, memex-portal-deployment-6c46b84848-c9mm5, memex-portal-deployment-6bfbc8ff4b-rz9p4, memex-portal-deployment-6bfbc8ff4b-vvc82, memex-portal-deployment-86779d495b-ksxqq, memex-portal-deployment-86779d495b-qrnh6, memex-portal-deployment-595d8b9995-j5f9p, memex-portal-deployment-595d8b9995-xntj7, memex-portal-deployment-7d95465fbb-s7gds, memex-portal-deployment-7d95465fbb-qrhvg, memex-portal-deployment-5df4ccb89-z8w2k, memex-portal-deployment-5df4ccb89-sprw2, memex-portal-deployment-5df4ccb89-txn2l, memex-portal-deployment-d999f9888-sfpph, memex-portal-deployment-dbc8997cd-q2n4s, memex-portal-deployment-66cfbf759-svxh9, memex-portal-deployment-75597864c8-wwfnf, memex-portal-deployment-9bf79b774-tk88b, memex-portal-deployment-67dbc7bc9c-lswwx, memex-portal-deployment-9bf79b774-qlks7, memex-portal-deployment-9bf79b774-jwrng, memex-portal-deployment-85859db964-mgxj7, memex-portal-deployment-77cfc55bfc-5ztnt, memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-54f4c96b6b-pn9z5, memex-portal-deployment-77cfc55bfc-tgtr4, memex-portal-deployment-77cfc55bfc-gxz25, memex-portal-deployment-5b9cbf46dd-5nqbz, memex-portal-deployment-5b9cbf46dd-zn7nf, memex-portal-deployment-79ddd44d77-k8qlc, memex-portal-deployment-6c74f8458-w46gf, memex-portal-deployment-bf77c847-tctvp, memex-portal-deployment-bf77c847-dmpfd, memex-portal-deployment-bf77c847-m5l4r, memex-portal-deployment-5bf869698f-5qgvz, memex-portal-deployment-574d54c98d-cdd47, memex-portal-deployment-747c879d8c-ph572, memex-portal-deployment-8ff67f8c7-gcf24, memex-portal-deployment-7fd9668b69-sjm5v, memex-portal-deployment-bfc5dcf6f-7z88q, memex-portal-deployment-6c7d6b6b44-klrt7, memex-portal-deployment-7fd9668b69-j4fdv, memex-portal-deployment-7fd9668b69-4qtkx, memex-portal-deployment-854cd8cd6d-8bc2q, memex-portal-deployment-854cd8cd6d-r4cn5, memex-portal-deployment-7fd9668b69-lzm7l, memex-portal-deployment-7fd9668b69-v72dx, memex-portal-deployment-6546c779d4-4ngqv, memex-portal-deployment-7fd9668b69-4zzzs, memex-portal-deployment-7fd9668b69-jnwlv, memex-portal-deployment-7fd9668b69-kvwgv, memex-portal-deployment-7cb6684584-ztqz8, memex-portal-deployment-54787585bc-qfwxx, memex-portal-deployment-6c967c7c4c-2k4xb, memex-portal-deployment-6b879996c7-dstvb, memex-portal-deployment-6b879996c7-lk6fd, memex-portal-deployment-6b879996c7-cmnfg, memex-portal-deployment-6c967c7c4c-kdff5, memex-portal-deployment-bbf878dfb-gknl9, memex-portal-deployment-5564495d46-f8xhs, memex-portal-deployment-86899455c7-z9nm2, memex-portal-deployment-6cd5d8f887-lvmpf, memex-portal-deployment-6cd5d8f887-2kcwk, memex-portal-deployment-58dbb6d98d-8msww, memex-portal-deployment-58dbb6d98d-lzqrr, memex-portal-deployment-7978987466-kmrpk, memex-portal-deployment-85f5bcf79d-n6lj2, memex-portal-deployment-85f5bcf79d-cgwtr, memex-portal-deployment-69956b6dbc-ndnxt, memex-portal-deployment-69956b6dbc-gx6z6, memex-portal-deployment-69956b6dbc-s246c, memex-portal-deployment-69956b6dbc-gj8m2, memex-portal-deployment-69956b6dbc-v5r29, memex-portal-deployment-65f8c68995-c7z8l, memex-portal-deployment-65f8c68995-z27tq, memex-portal-deployment-5f69f6b6bd-74xjv, memex-portal-deployment-5f69f6b6bd-z7tb6, memex-portal-deployment-69956b6dbc-ntksf, memex-portal-deployment-69956b6dbc-mczlv, memex-portal-deployment-6b7c4f94db-4sgtk, memex-portal-deployment-5485b778c9-mqz2v, memex-portal-deployment-75b545bbd-nssvp, memex-portal-deployment-6b7c4f94db-zzvfq, memex-portal-deployment-75b545bbd-gwrc6

Recent log lines

2026-09-20 22:25:13Z memex-portal-deployment-75b545bbd-nssvp fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'Orleans.Placement/(null)/Timeout', Operation Key: '', Result: ''
2026-09-20 22:25:14Z memex-portal-deployment-75b545bbd-nssvp fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'Orleans.Placement/(null)/Timeout', Operation Key: '', Result: ''
2026-09-20 22:25:14Z memex-portal-deployment-75b545bbd-nssvp fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'Orleans.Placement/(null)/Timeout', Operation Key: '', Result: ''

_{Re-addressed by the current identity function: this incident inherited 9ca334c1e8dad9ca (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Not verified in production — leaving open. I could inspect the code and the shipped configuration, and nothing more.

Read-only session, 2026-09-21T07:45Z – 07:56Z. Instruments: the public /health and /api/version on both production portals, gh api (REST), and authenticated get/search through the systemorph MCP (memex.systemorph.com, the control instance). Nothing was written: no node, no InstanceAction.

What IS established: the fix is rolled

$ date -u '+%Y-%m-%dT%H:%M:%SZ'; curl -sS https://memex.systemorph.com/api/version
2026-09-21T07:47:25Z
{"version":"3.0.0+9017d62c9af9ae3eeae4e76e751f475a7271d7b0","commit":"9017d62c…"}

$ curl -sS https://memex.meshweaver.cloud/api/version
{"version":"3.0.0+026442fff0f2728e96bb838494559b68b5c2b797","commit":"026442ff…"}

gh api repos/Systemorph/MeshWeaver/compare/4d9cc22e8036d60191a74d7bf59a1d6e409b2dec...<image commit> → {"status":"ahead","behind_by":0} for both (ahead_by 60 and 19). #4977's merge commit is an ancestor of both running images, and memex.systemorph.com — the consumer whose pods this issue's samples came from — has been running it since it booted at 2026-09-20T22:12:07Z and 22:21:18Z (from its own /health). So ~9.5 h of production runtime on the fix at the time of measurement. That is the issue title's first owed item, "the roll", done.

What IS established: the shipped configuration

At the running commit 026442ff, memex/aspire/Memex.Portal.ServiceDefaults/ServiceDefaults.cs:146:

builder.Services.AddHttpClient("plugin-registry-bundles")
    .RemoveAllResilienceHandlers()
    .AddStandardResilienceHandler(options =>
    {
        options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(120);
        options.TotalRequestTimeout.Timeout = TimeSpan.FromMinutes(5);
        options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(240);
    });

and RegistryUpdateReconciler.PerPackageAdoptBudget = PluginBundleClient.TransferStallBudget + TimeSpan.FromMinutes(1) with TransferStallBudget = TimeSpan.FromSeconds(120).

🚨 So the title's third owed item — "the inverted 3 min/5 min budgets" — is resolved by DERIVATION, not by changing either number. The reconciler's per-package budget is still 180 s and the HTTP pipeline's total is still 300 s; what changed is that the client now holds a 120 s silence budget per stage, so it is the first clock to fire and it names the stage. Anyone reading the title expecting the 3 min / 5 min pair to have been flipped should read #4977's Doc/Architecture/BundleTransferBudget instead — the claim is "one silence budget, derived", not "the inversion removed".

What is NOT established: everything timing-shaped — i.e. the whole point of the issue

I could not exercise a real bundle transfer in production, and I could not read a single production log line. A responsiveness budget is a timing property; code inspection is not a measurement of it. Concretely, each route I tried:

1. The byte/rate measurement (the title's second owed item) is emitted at Information and never enters the mesh. From #4977's diff:

_logger?.LogInformation(
    "Bundle for {Plugin}@{Version}: {Bytes} byte(s) over HTTP from {Registry} in "
    + "{Elapsed} ms ({Throughput} KiB/s)", …);
_logger?.LogInformation(
    "Bundle index from {Registry}: {Bytes} byte(s) in {Elapsed} ms, {Count} bundle(s) "
    + "advertised on {Identity}/{Architecture}", …);

and the refusal at LogWarning:

_logger?.LogWarning(
    "{Transfer} over HTTP from {Registry} did NOT complete after {Elapsed} ms — "
    + "{Received} of {Declared} byte(s) had arrived. Cause: {Cause}", …);

Those reach Loki, not a node. Reading them needs a Logs Hosting/InstanceAction on the control instance, which is a write; this session is read-only on every live portal, so I did not file one.

2. There is no existing log window to read. On memex.systemorph.com:

search 'namespace:Ops scope:descendants nodeType:LogEntry' limit:15
→ {"count":0,"limit":15,"truncated":false,"coverage":{"scope":"partition","partitions":["ops"]},"results":[]}

Ops/Logs is the output of one Logs query, never a feed — and here it holds nothing at all, so there is no window anyone else has already opened.

3. The incident this issue was measured from is now UNREADABLE as a signal, in two separate ways.

Admin/_LogIncident/9ca334c1e8dad9ca (read 07:50Z) is "status":"Superseded", "supersededBy":"833c5f2e6337f4d3", lastSeen frozen at 2026-09-18T15:18:44.644Z, occurrences 2089 — so the counter this issue's closing condition referred to has moved house, and its own 10 retained samples now hold zero plugin-registry-bundles lines (4× plugin-registry-standard, i.e. #4222's pipeline, and 3× Orleans.Placement). The ring rotated; the six bundle samples the issue was written from are gone.

The successor Admin/_LogIncident/833c5f2e6337f4d3 (read 07:50Z): occurrences 3855, lastSeen 2026-09-20T22:25:14.465Z, lastModified 2026-09-21T03:57:08.761Z, issueNumber 4528. Its shapes[0]: firstSeen 2026-09-19T06:22:18.627Z → lastSeen 2026-09-20T22:25:14.465Z, occurrences 1766 — i.e. ≈0.74 events/min over that 40.0 h window. All 10 retained samples are Orleans.Placement/(null)/Timeout inside a three-second burst at 22:25:11–22:25:14, three minutes after the 22:12Z boot of the image carrying #4977.

Since then: nothing, for 9.5 h, where the prior rate predicts ~420 events. That looks like the storm stopping when the fix started running — and it is not evidence, because the entire incident population froze at the same moment. Over 200 Admin/_LogIncident nodes (search 'namespace:Admin scope:descendants nodeType:LogIncident' limit:200, truncated:true, so a floor), the newest lastModified of any of them is 2026-09-20T22:29:35.003Z. A 9.5-hour fleet-wide silence is equally consistent with:

the portal having been healthy since its 22:12Z restart and emitting no error-level lines at all; and
nothing having written a LogIncident since 22:29Z.

I cannot separate those read-only — I have no positive control that the log-incident watcher is still writing — so I am not reporting the freeze as a fix. (I also did not file a bug about the watcher: "it may be dead" is not established either, and the truncated 200 is an arbitrary page, not the population.)

🚨 And even an advancing counter would not attribute: the fingerprint still masks Source: (#4977 lists that as deliberately not fixed, in MeshWeaver.Plugins), so every Polly OnTimeout on every pipeline still folds onto this one incident. The contamination this issue warned about — "the counter is shared with this defect… anyone reading a future reopen must read samples[] before believing it" — is unchanged, and the retained samples no longer contain the pipeline in question.

4. No named refusal has surfaced. search 'namespace:Admin scope:descendants nodeType:LogIncident bundle transfer stalled registry' limit:25 on the control instance returns no incident naming BundleTransferException, NoResponse, StalledMidBody or OverSize. That is consistent with "no stall happened since the roll" and with "warning-level lines do not become incidents" and with "the watcher is not writing" — it discriminates nothing.

Also relevant to whoever picks this up

#4963 is still open — the registry never beginning a response to an authenticated GET /api/plugins/bundles/index.json (180 s, 0 bytes). #4977 states explicitly that it does not fix that and cannot: "a bound on bytes cannot fire on a stream that sends none, and a bound on silence can only name it." So the symptom recurring is not by itself a regression of this issue, and this issue closing would not make the stall stop.

The reading that would close this

One Logs Hosting/InstanceAction on memex.systemorph.com over the memex namespace, covering a window that contains a reconcile pass (the reconciler's period is PerPackageAdoptBudget = 180 s; a boot reconcile is the heaviest), looking for:

Bundle index from {Registry}: {Bytes} byte(s) in {Elapsed} ms — the byte/rate measurement this issue is owed, and the number that finally says whether 120 s was being spent on a large bundle or a slow registry;
Bundle for {Plugin}@{Version}: … ({Throughput} KiB/s) for at least one real bundle;
one index request per pass, not one per package — Bundle transfers fail on SILENCE at a named stage — the client owns every clock (#4528) #4977's PromiseSlot fix, which is directly readable from the line count per pass and is the claim that explains the 180 s periodicity in the original measurement;
either a BundleTransferStage-named refusal, or none at all across the window — and in the "none" case, a positive control that the window really did contain transfers (the completed lines from 1–2), so that a silent window is not read as a quiet one.

That is a write, so it needs a session that is allowed to file an InstanceAction. Until then this is code inspection wearing a verification costume, and a sev:H should not close on it.

_{Measured 2026-09-21 by Claude Opus 5, read-only, via /health, /api/version, gh api REST and the control instance's MCP (get, search). No portal was mutated; no InstanceAction was filed.}

Still occurring: 50 more since the last update (4078 total).

Last seen: 2026-09-23 18:21:01Z
Pods: memex-portal-deployment-6c99c9f796-mh5cs, memex-portal-deployment-84855f7679-z4rsp, memex-portal-deployment-575dc48bfb-wq7s8, memex-portal-deployment-6bd9c46f8b-z9t6r, memex-portal-deployment-85ddbf6c7f-p4t4b, memex-portal-deployment-59479dc7d9-kmwvr, memex-portal-deployment-748bfcc86b-2m6vg, memex-portal-deployment-6fc79755d4-r96vd, memex-portal-deployment-57885dbb86-76zsg, memex-portal-deployment-6d9cdd447b-tm4ck, memex-portal-deployment-6cd98575f6-gr5ds, memex-portal-deployment-784494885b-c5sz5, memex-portal-deployment-6854bb96cb-z6bn9, memex-portal-deployment-6c9bb5d95c-jb9dq, memex-portal-deployment-69f9fc445-6fws2, memex-portal-deployment-c9584c6dd-dms29, memex-portal-deployment-cc58dc7c6-ckll8, memex-portal-deployment-67967894c4-7fvvg, memex-portal-deployment-c594497fc-dsgds, memex-portal-deployment-86bd794cb5-4rs66, memex-portal-deployment-8489f66759-zlbcg, memex-portal-deployment-745d7588ff-4mrxf, memex-portal-deployment-797df55dfb-cn4p8, memex-portal-deployment-97c56b569-tjnmj, memex-portal-deployment-bf88ffcbb-t6nhv, memex-portal-deployment-6cc6f7997b-99xkc, memex-portal-deployment-7799dd87b9-vshbj, memex-portal-deployment-6fcf6bbc47-c2nw9, memex-portal-deployment-6fcf6bbc47-8qvz6, memex-portal-deployment-79766fd4f6-w66fq, memex-portal-deployment-648c756f74-smlsb, memex-portal-deployment-745b99b846-hrk46, memex-portal-deployment-84799488cb-cb7q4, memex-portal-deployment-78cf8bd9f5-klqft, memex-portal-deployment-f96d46ddf-zn9nl, memex-portal-deployment-8669cb7675-xksqg, memex-portal-deployment-5d6f77b68c-srtqg, memex-portal-deployment-5b7b597fb5-s6xf9, memex-portal-deployment-596d7ff9bf-vtsp6, memex-portal-deployment-7c47bc9c85-xhspn, memex-portal-deployment-79756bcc4f-mb2c5, memex-portal-deployment-65f87d4594-5vsw8, memex-portal-deployment-696cc645c6-gfw6n, memex-portal-deployment-6c5bd56bf6-kqk6s, memex-portal-deployment-747d45bbf7-djm8g, memex-portal-deployment-c89ddfbb6-txnd5, memex-portal-deployment-6dfbd755f6-c5dr6, memex-portal-deployment-66cc6466bf-frwpk, memex-portal-deployment-68cd75c658-8qt9s, memex-portal-deployment-f47f8cdd-rx8xj, memex-portal-deployment-7cb5549c57-8c9gr, memex-portal-deployment-89d7df89f-lrbc4, memex-portal-deployment-89d7df89f-k9fkr, memex-portal-deployment-764d5fbf-nn6nj, memex-portal-deployment-76b967b5fb-pkw5p, memex-portal-deployment-559ddb8f8-dbpx6, memex-portal-deployment-98785f547-76qjj, memex-portal-deployment-59ccff47c5-xw4zx, memex-portal-deployment-d7dd9796b-rc4bh, memex-portal-deployment-d7dd9796b-66g44, memex-portal-deployment-588c4cbb68-jm8cz, memex-portal-deployment-55f6c4c4c5-wcccn, memex-portal-deployment-58cf8c86f7-p95sf, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-7897bfd494-x8rfw, memex-portal-deployment-7897bfd494-dl9wb, memex-portal-deployment-7d6b864b54-w4xnl, memex-portal-deployment-7d6b864b54-gr6n6, memex-portal-deployment-688b496b7-2sdt8, memex-portal-deployment-688b496b7-brp4j, memex-portal-deployment-999c9c79d-8ccj2, memex-portal-deployment-999c9c79d-g9wps, memex-portal-deployment-6dc7cf965d-8f5t6, memex-portal-deployment-64d9857f9d-bmrnj, memex-portal-deployment-64d9857f9d-v7hmx, memex-portal-deployment-df7ddfdf4-xb9bk, memex-portal-deployment-6d58c699f8-bp5bs, memex-portal-deployment-6d58c699f8-9t675, memex-portal-deployment-775748f44d-7xlkm, memex-portal-deployment-7969b8cfff-fqnk9, memex-portal-deployment-76cbbdd5f-9mtkj, memex-portal-deployment-76cbbdd5f-82mvq, memex-portal-deployment-569544576b-q6nvd, memex-portal-deployment-569544576b-89r5k, memex-portal-deployment-569544576b-9h52b, memex-portal-deployment-569544576b-wtfz5, memex-portal-deployment-569544576b-cqlnw, memex-portal-deployment-867dc45877-lgskl, memex-portal-deployment-867dc45877-h8tjc, memex-portal-deployment-867dc45877-fmcss, memex-portal-deployment-58c7bb4b9d-97pjf, memex-portal-deployment-5b6c4f89f8-rpfmq, memex-portal-deployment-58c7bb4b9d-z2xcv, memex-portal-deployment-5b6c4f89f8-hpcbf, memex-portal-deployment-5b6c4f89f8-97sbj, memex-portal-deployment-5b6c4f89f8-kz4rh, memex-portal-deployment-6f64bf4b56-gxmrr, memex-portal-deployment-6f64bf4b56-2mzrf, memex-portal-deployment-6f64bf4b56-xqw96, memex-portal-deployment-858f57dc6-8zlfh, memex-portal-deployment-547649b6f8-wmt6n, memex-portal-deployment-547649b6f8-48jrz, memex-portal-deployment-858f57dc6-wns9j, memex-portal-deployment-7b67cb5577-rctvc, memex-portal-deployment-7d8787bcc7-4htmx, memex-portal-deployment-7544fdd47b-pbrvx, memex-portal-deployment-6dc5679f77-nd67w, memex-portal-deployment-6dc5679f77-vf2qx, memex-portal-deployment-858f57dc6-bkkfv, memex-portal-deployment-858f57dc6-ff5kn, memex-portal-deployment-858f57dc6-6wggn, memex-portal-deployment-6fd666bc95-s79r4, memex-portal-deployment-6fd666bc95-f7brs, memex-portal-deployment-66ff6fc5f9-nwl9f, memex-portal-deployment-66ff6fc5f9-mnw9n, memex-portal-deployment-79b5fd5f6b-7wdn4, memex-portal-deployment-79b5fd5f6b-jg4td, memex-portal-deployment-cf988c6f5-gtqpl, memex-portal-deployment-7b5f547cb7-cp2wv, memex-portal-deployment-7b5f547cb7-ps2nz, memex-portal-deployment-6c7b9889c7-rkr8g, memex-portal-deployment-6c7b9889c7-72fmv, memex-portal-deployment-cf5dc996f-pgjtz, memex-portal-deployment-cf5dc996f-gk2bv, memex-portal-deployment-6c46b84848-hnh9v, memex-portal-deployment-6c46b84848-c9mm5, memex-portal-deployment-6bfbc8ff4b-rz9p4, memex-portal-deployment-6bfbc8ff4b-vvc82, memex-portal-deployment-86779d495b-ksxqq, memex-portal-deployment-86779d495b-qrnh6, memex-portal-deployment-595d8b9995-j5f9p, memex-portal-deployment-595d8b9995-xntj7, memex-portal-deployment-7d95465fbb-s7gds, memex-portal-deployment-7d95465fbb-qrhvg, memex-portal-deployment-5df4ccb89-z8w2k, memex-portal-deployment-5df4ccb89-sprw2, memex-portal-deployment-5df4ccb89-txn2l, memex-portal-deployment-d999f9888-sfpph, memex-portal-deployment-dbc8997cd-q2n4s, memex-portal-deployment-66cfbf759-svxh9, memex-portal-deployment-75597864c8-wwfnf, memex-portal-deployment-9bf79b774-tk88b, memex-portal-deployment-67dbc7bc9c-lswwx, memex-portal-deployment-9bf79b774-qlks7, memex-portal-deployment-9bf79b774-jwrng, memex-portal-deployment-85859db964-mgxj7, memex-portal-deployment-77cfc55bfc-5ztnt, memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-54f4c96b6b-pn9z5, memex-portal-deployment-77cfc55bfc-tgtr4, memex-portal-deployment-77cfc55bfc-gxz25, memex-portal-deployment-5b9cbf46dd-5nqbz, memex-portal-deployment-5b9cbf46dd-zn7nf, memex-portal-deployment-79ddd44d77-k8qlc, memex-portal-deployment-6c74f8458-w46gf, memex-portal-deployment-bf77c847-tctvp, memex-portal-deployment-bf77c847-dmpfd, memex-portal-deployment-bf77c847-m5l4r, memex-portal-deployment-5bf869698f-5qgvz, memex-portal-deployment-574d54c98d-cdd47, memex-portal-deployment-747c879d8c-ph572, memex-portal-deployment-8ff67f8c7-gcf24, memex-portal-deployment-7fd9668b69-sjm5v, memex-portal-deployment-bfc5dcf6f-7z88q, memex-portal-deployment-6c7d6b6b44-klrt7, memex-portal-deployment-7fd9668b69-j4fdv, memex-portal-deployment-7fd9668b69-4qtkx, memex-portal-deployment-854cd8cd6d-8bc2q, memex-portal-deployment-854cd8cd6d-r4cn5, memex-portal-deployment-7fd9668b69-lzm7l, memex-portal-deployment-7fd9668b69-v72dx, memex-portal-deployment-6546c779d4-4ngqv, memex-portal-deployment-7fd9668b69-4zzzs, memex-portal-deployment-7fd9668b69-jnwlv, memex-portal-deployment-7fd9668b69-kvwgv, memex-portal-deployment-7cb6684584-ztqz8, memex-portal-deployment-54787585bc-qfwxx, memex-portal-deployment-6c967c7c4c-2k4xb, memex-portal-deployment-6b879996c7-dstvb, memex-portal-deployment-6b879996c7-lk6fd, memex-portal-deployment-6b879996c7-cmnfg, memex-portal-deployment-6c967c7c4c-kdff5, memex-portal-deployment-bbf878dfb-gknl9, memex-portal-deployment-5564495d46-f8xhs, memex-portal-deployment-86899455c7-z9nm2, memex-portal-deployment-6cd5d8f887-lvmpf, memex-portal-deployment-6cd5d8f887-2kcwk, memex-portal-deployment-58dbb6d98d-8msww, memex-portal-deployment-58dbb6d98d-lzqrr, memex-portal-deployment-7978987466-kmrpk, memex-portal-deployment-85f5bcf79d-n6lj2, memex-portal-deployment-85f5bcf79d-cgwtr, memex-portal-deployment-69956b6dbc-ndnxt, memex-portal-deployment-69956b6dbc-gx6z6, memex-portal-deployment-69956b6dbc-s246c, memex-portal-deployment-69956b6dbc-gj8m2, memex-portal-deployment-69956b6dbc-v5r29, memex-portal-deployment-65f8c68995-c7z8l, memex-portal-deployment-65f8c68995-z27tq, memex-portal-deployment-5f69f6b6bd-74xjv, memex-portal-deployment-5f69f6b6bd-z7tb6, memex-portal-deployment-69956b6dbc-ntksf, memex-portal-deployment-69956b6dbc-mczlv, memex-portal-deployment-6b7c4f94db-4sgtk, memex-portal-deployment-5485b778c9-mqz2v, memex-portal-deployment-75b545bbd-nssvp, memex-portal-deployment-6b7c4f94db-zzvfq, memex-portal-deployment-75b545bbd-gwrc6, memex-portal-deployment-77494bdbcc-7ndrn, memex-portal-deployment-77494bdbcc-5djgr, memex-portal-deployment-77494bdbcc-rg99v, memex-portal-deployment-77494bdbcc-6sv8l, memex-portal-deployment-77494bdbcc-bphfb, memex-portal-deployment-77494bdbcc-kkjld, memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-86dbb4d55c-jj7gc, memex-portal-deployment-857546649-rb7qd, memex-portal-deployment-857546649-xx9j4, memex-portal-deployment-857546649-vf2q6, memex-portal-deployment-c54d5847c-bf27p, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-5dd79dc746-4bhs7, memex-portal-deployment-5dd79dc746-45vzd, memex-portal-deployment-5dd79dc746-6vmxd, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-777677694-cc6zz, memex-portal-deployment-777677694-hjkc8, memex-portal-deployment-777677694-d4j8c, memex-portal-deployment-786d86cf54-895wl, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv

Recent log lines

2026-09-23 18:20:53Z memex-portal-deployment-7d7f4d84f9-wl8mv fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: '-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''
2026-09-23 18:21:00Z memex-portal-deployment-7d7f4d84f9-mvzb5 fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: '-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''
2026-09-23 18:21:01Z memex-portal-deployment-7d7f4d84f9-n7g6b fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: '-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''

_{Re-addressed by the current identity function: this incident inherited 9ca334c1e8dad9ca (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Still occurring: 4 more since the last update (4082 total).

Last seen: 2026-09-23 21:14:12Z
Pods: memex-portal-deployment-6c99c9f796-mh5cs, memex-portal-deployment-84855f7679-z4rsp, memex-portal-deployment-575dc48bfb-wq7s8, memex-portal-deployment-6bd9c46f8b-z9t6r, memex-portal-deployment-85ddbf6c7f-p4t4b, memex-portal-deployment-59479dc7d9-kmwvr, memex-portal-deployment-748bfcc86b-2m6vg, memex-portal-deployment-6fc79755d4-r96vd, memex-portal-deployment-57885dbb86-76zsg, memex-portal-deployment-6d9cdd447b-tm4ck, memex-portal-deployment-6cd98575f6-gr5ds, memex-portal-deployment-784494885b-c5sz5, memex-portal-deployment-6854bb96cb-z6bn9, memex-portal-deployment-6c9bb5d95c-jb9dq, memex-portal-deployment-69f9fc445-6fws2, memex-portal-deployment-c9584c6dd-dms29, memex-portal-deployment-cc58dc7c6-ckll8, memex-portal-deployment-67967894c4-7fvvg, memex-portal-deployment-c594497fc-dsgds, memex-portal-deployment-86bd794cb5-4rs66, memex-portal-deployment-8489f66759-zlbcg, memex-portal-deployment-745d7588ff-4mrxf, memex-portal-deployment-797df55dfb-cn4p8, memex-portal-deployment-97c56b569-tjnmj, memex-portal-deployment-bf88ffcbb-t6nhv, memex-portal-deployment-6cc6f7997b-99xkc, memex-portal-deployment-7799dd87b9-vshbj, memex-portal-deployment-6fcf6bbc47-c2nw9, memex-portal-deployment-6fcf6bbc47-8qvz6, memex-portal-deployment-79766fd4f6-w66fq, memex-portal-deployment-648c756f74-smlsb, memex-portal-deployment-745b99b846-hrk46, memex-portal-deployment-84799488cb-cb7q4, memex-portal-deployment-78cf8bd9f5-klqft, memex-portal-deployment-f96d46ddf-zn9nl, memex-portal-deployment-8669cb7675-xksqg, memex-portal-deployment-5d6f77b68c-srtqg, memex-portal-deployment-5b7b597fb5-s6xf9, memex-portal-deployment-596d7ff9bf-vtsp6, memex-portal-deployment-7c47bc9c85-xhspn, memex-portal-deployment-79756bcc4f-mb2c5, memex-portal-deployment-65f87d4594-5vsw8, memex-portal-deployment-696cc645c6-gfw6n, memex-portal-deployment-6c5bd56bf6-kqk6s, memex-portal-deployment-747d45bbf7-djm8g, memex-portal-deployment-c89ddfbb6-txnd5, memex-portal-deployment-6dfbd755f6-c5dr6, memex-portal-deployment-66cc6466bf-frwpk, memex-portal-deployment-68cd75c658-8qt9s, memex-portal-deployment-f47f8cdd-rx8xj, memex-portal-deployment-7cb5549c57-8c9gr, memex-portal-deployment-89d7df89f-lrbc4, memex-portal-deployment-89d7df89f-k9fkr, memex-portal-deployment-764d5fbf-nn6nj, memex-portal-deployment-76b967b5fb-pkw5p, memex-portal-deployment-559ddb8f8-dbpx6, memex-portal-deployment-98785f547-76qjj, memex-portal-deployment-59ccff47c5-xw4zx, memex-portal-deployment-d7dd9796b-rc4bh, memex-portal-deployment-d7dd9796b-66g44, memex-portal-deployment-588c4cbb68-jm8cz, memex-portal-deployment-55f6c4c4c5-wcccn, memex-portal-deployment-58cf8c86f7-p95sf, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-7897bfd494-x8rfw, memex-portal-deployment-7897bfd494-dl9wb, memex-portal-deployment-7d6b864b54-w4xnl, memex-portal-deployment-7d6b864b54-gr6n6, memex-portal-deployment-688b496b7-2sdt8, memex-portal-deployment-688b496b7-brp4j, memex-portal-deployment-999c9c79d-8ccj2, memex-portal-deployment-999c9c79d-g9wps, memex-portal-deployment-6dc7cf965d-8f5t6, memex-portal-deployment-64d9857f9d-bmrnj, memex-portal-deployment-64d9857f9d-v7hmx, memex-portal-deployment-df7ddfdf4-xb9bk, memex-portal-deployment-6d58c699f8-bp5bs, memex-portal-deployment-6d58c699f8-9t675, memex-portal-deployment-775748f44d-7xlkm, memex-portal-deployment-7969b8cfff-fqnk9, memex-portal-deployment-76cbbdd5f-9mtkj, memex-portal-deployment-76cbbdd5f-82mvq, memex-portal-deployment-569544576b-q6nvd, memex-portal-deployment-569544576b-89r5k, memex-portal-deployment-569544576b-9h52b, memex-portal-deployment-569544576b-wtfz5, memex-portal-deployment-569544576b-cqlnw, memex-portal-deployment-867dc45877-lgskl, memex-portal-deployment-867dc45877-h8tjc, memex-portal-deployment-867dc45877-fmcss, memex-portal-deployment-58c7bb4b9d-97pjf, memex-portal-deployment-5b6c4f89f8-rpfmq, memex-portal-deployment-58c7bb4b9d-z2xcv, memex-portal-deployment-5b6c4f89f8-hpcbf, memex-portal-deployment-5b6c4f89f8-97sbj, memex-portal-deployment-5b6c4f89f8-kz4rh, memex-portal-deployment-6f64bf4b56-gxmrr, memex-portal-deployment-6f64bf4b56-2mzrf, memex-portal-deployment-6f64bf4b56-xqw96, memex-portal-deployment-858f57dc6-8zlfh, memex-portal-deployment-547649b6f8-wmt6n, memex-portal-deployment-547649b6f8-48jrz, memex-portal-deployment-858f57dc6-wns9j, memex-portal-deployment-7b67cb5577-rctvc, memex-portal-deployment-7d8787bcc7-4htmx, memex-portal-deployment-7544fdd47b-pbrvx, memex-portal-deployment-6dc5679f77-nd67w, memex-portal-deployment-6dc5679f77-vf2qx, memex-portal-deployment-858f57dc6-bkkfv, memex-portal-deployment-858f57dc6-ff5kn, memex-portal-deployment-858f57dc6-6wggn, memex-portal-deployment-6fd666bc95-s79r4, memex-portal-deployment-6fd666bc95-f7brs, memex-portal-deployment-66ff6fc5f9-nwl9f, memex-portal-deployment-66ff6fc5f9-mnw9n, memex-portal-deployment-79b5fd5f6b-7wdn4, memex-portal-deployment-79b5fd5f6b-jg4td, memex-portal-deployment-cf988c6f5-gtqpl, memex-portal-deployment-7b5f547cb7-cp2wv, memex-portal-deployment-7b5f547cb7-ps2nz, memex-portal-deployment-6c7b9889c7-rkr8g, memex-portal-deployment-6c7b9889c7-72fmv, memex-portal-deployment-cf5dc996f-pgjtz, memex-portal-deployment-cf5dc996f-gk2bv, memex-portal-deployment-6c46b84848-hnh9v, memex-portal-deployment-6c46b84848-c9mm5, memex-portal-deployment-6bfbc8ff4b-rz9p4, memex-portal-deployment-6bfbc8ff4b-vvc82, memex-portal-deployment-86779d495b-ksxqq, memex-portal-deployment-86779d495b-qrnh6, memex-portal-deployment-595d8b9995-j5f9p, memex-portal-deployment-595d8b9995-xntj7, memex-portal-deployment-7d95465fbb-s7gds, memex-portal-deployment-7d95465fbb-qrhvg, memex-portal-deployment-5df4ccb89-z8w2k, memex-portal-deployment-5df4ccb89-sprw2, memex-portal-deployment-5df4ccb89-txn2l, memex-portal-deployment-d999f9888-sfpph, memex-portal-deployment-dbc8997cd-q2n4s, memex-portal-deployment-66cfbf759-svxh9, memex-portal-deployment-75597864c8-wwfnf, memex-portal-deployment-9bf79b774-tk88b, memex-portal-deployment-67dbc7bc9c-lswwx, memex-portal-deployment-9bf79b774-qlks7, memex-portal-deployment-9bf79b774-jwrng, memex-portal-deployment-85859db964-mgxj7, memex-portal-deployment-77cfc55bfc-5ztnt, memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-54f4c96b6b-pn9z5, memex-portal-deployment-77cfc55bfc-tgtr4, memex-portal-deployment-77cfc55bfc-gxz25, memex-portal-deployment-5b9cbf46dd-5nqbz, memex-portal-deployment-5b9cbf46dd-zn7nf, memex-portal-deployment-79ddd44d77-k8qlc, memex-portal-deployment-6c74f8458-w46gf, memex-portal-deployment-bf77c847-tctvp, memex-portal-deployment-bf77c847-dmpfd, memex-portal-deployment-bf77c847-m5l4r, memex-portal-deployment-5bf869698f-5qgvz, memex-portal-deployment-574d54c98d-cdd47, memex-portal-deployment-747c879d8c-ph572, memex-portal-deployment-8ff67f8c7-gcf24, memex-portal-deployment-7fd9668b69-sjm5v, memex-portal-deployment-bfc5dcf6f-7z88q, memex-portal-deployment-6c7d6b6b44-klrt7, memex-portal-deployment-7fd9668b69-j4fdv, memex-portal-deployment-7fd9668b69-4qtkx, memex-portal-deployment-854cd8cd6d-8bc2q, memex-portal-deployment-854cd8cd6d-r4cn5, memex-portal-deployment-7fd9668b69-lzm7l, memex-portal-deployment-7fd9668b69-v72dx, memex-portal-deployment-6546c779d4-4ngqv, memex-portal-deployment-7fd9668b69-4zzzs, memex-portal-deployment-7fd9668b69-jnwlv, memex-portal-deployment-7fd9668b69-kvwgv, memex-portal-deployment-7cb6684584-ztqz8, memex-portal-deployment-54787585bc-qfwxx, memex-portal-deployment-6c967c7c4c-2k4xb, memex-portal-deployment-6b879996c7-dstvb, memex-portal-deployment-6b879996c7-lk6fd, memex-portal-deployment-6b879996c7-cmnfg, memex-portal-deployment-6c967c7c4c-kdff5, memex-portal-deployment-bbf878dfb-gknl9, memex-portal-deployment-5564495d46-f8xhs, memex-portal-deployment-86899455c7-z9nm2, memex-portal-deployment-6cd5d8f887-lvmpf, memex-portal-deployment-6cd5d8f887-2kcwk, memex-portal-deployment-58dbb6d98d-8msww, memex-portal-deployment-58dbb6d98d-lzqrr, memex-portal-deployment-7978987466-kmrpk, memex-portal-deployment-85f5bcf79d-n6lj2, memex-portal-deployment-85f5bcf79d-cgwtr, memex-portal-deployment-69956b6dbc-ndnxt, memex-portal-deployment-69956b6dbc-gx6z6, memex-portal-deployment-69956b6dbc-s246c, memex-portal-deployment-69956b6dbc-gj8m2, memex-portal-deployment-69956b6dbc-v5r29, memex-portal-deployment-65f8c68995-c7z8l, memex-portal-deployment-65f8c68995-z27tq, memex-portal-deployment-5f69f6b6bd-74xjv, memex-portal-deployment-5f69f6b6bd-z7tb6, memex-portal-deployment-69956b6dbc-ntksf, memex-portal-deployment-69956b6dbc-mczlv, memex-portal-deployment-6b7c4f94db-4sgtk, memex-portal-deployment-5485b778c9-mqz2v, memex-portal-deployment-75b545bbd-nssvp, memex-portal-deployment-6b7c4f94db-zzvfq, memex-portal-deployment-75b545bbd-gwrc6, memex-portal-deployment-77494bdbcc-7ndrn, memex-portal-deployment-77494bdbcc-5djgr, memex-portal-deployment-77494bdbcc-rg99v, memex-portal-deployment-77494bdbcc-6sv8l, memex-portal-deployment-77494bdbcc-bphfb, memex-portal-deployment-77494bdbcc-kkjld, memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-86dbb4d55c-jj7gc, memex-portal-deployment-857546649-rb7qd, memex-portal-deployment-857546649-xx9j4, memex-portal-deployment-857546649-vf2q6, memex-portal-deployment-c54d5847c-bf27p, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-5dd79dc746-4bhs7, memex-portal-deployment-5dd79dc746-45vzd, memex-portal-deployment-5dd79dc746-6vmxd, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-777677694-cc6zz, memex-portal-deployment-777677694-hjkc8, memex-portal-deployment-777677694-d4j8c, memex-portal-deployment-786d86cf54-895wl, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-76cf776847-5hflx

Recent log lines

2026-09-23 21:13:13Z memex-portal-deployment-76cf776847-5hflx fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'plugin-registry-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''
2026-09-23 21:13:45Z memex-portal-deployment-76cf776847-5hflx fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'plugin-registry-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''
2026-09-23 21:14:12Z memex-portal-deployment-76cf776847-5hflx fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'plugin-registry-standard//Standard-TotalRequestTimeout', Operation Key: '', Result: ''

_{Re-addressed by the current identity function: this incident inherited 9ca334c1e8dad9ca (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Still occurring: 1 more since the last update (4083 total).

Last seen: 2026-09-24 05:43:17Z
Pods: memex-portal-deployment-6c99c9f796-mh5cs, memex-portal-deployment-84855f7679-z4rsp, memex-portal-deployment-575dc48bfb-wq7s8, memex-portal-deployment-6bd9c46f8b-z9t6r, memex-portal-deployment-85ddbf6c7f-p4t4b, memex-portal-deployment-59479dc7d9-kmwvr, memex-portal-deployment-748bfcc86b-2m6vg, memex-portal-deployment-6fc79755d4-r96vd, memex-portal-deployment-57885dbb86-76zsg, memex-portal-deployment-6d9cdd447b-tm4ck, memex-portal-deployment-6cd98575f6-gr5ds, memex-portal-deployment-784494885b-c5sz5, memex-portal-deployment-6854bb96cb-z6bn9, memex-portal-deployment-6c9bb5d95c-jb9dq, memex-portal-deployment-69f9fc445-6fws2, memex-portal-deployment-c9584c6dd-dms29, memex-portal-deployment-cc58dc7c6-ckll8, memex-portal-deployment-67967894c4-7fvvg, memex-portal-deployment-c594497fc-dsgds, memex-portal-deployment-86bd794cb5-4rs66, memex-portal-deployment-8489f66759-zlbcg, memex-portal-deployment-745d7588ff-4mrxf, memex-portal-deployment-797df55dfb-cn4p8, memex-portal-deployment-97c56b569-tjnmj, memex-portal-deployment-bf88ffcbb-t6nhv, memex-portal-deployment-6cc6f7997b-99xkc, memex-portal-deployment-7799dd87b9-vshbj, memex-portal-deployment-6fcf6bbc47-c2nw9, memex-portal-deployment-6fcf6bbc47-8qvz6, memex-portal-deployment-79766fd4f6-w66fq, memex-portal-deployment-648c756f74-smlsb, memex-portal-deployment-745b99b846-hrk46, memex-portal-deployment-84799488cb-cb7q4, memex-portal-deployment-78cf8bd9f5-klqft, memex-portal-deployment-f96d46ddf-zn9nl, memex-portal-deployment-8669cb7675-xksqg, memex-portal-deployment-5d6f77b68c-srtqg, memex-portal-deployment-5b7b597fb5-s6xf9, memex-portal-deployment-596d7ff9bf-vtsp6, memex-portal-deployment-7c47bc9c85-xhspn, memex-portal-deployment-79756bcc4f-mb2c5, memex-portal-deployment-65f87d4594-5vsw8, memex-portal-deployment-696cc645c6-gfw6n, memex-portal-deployment-6c5bd56bf6-kqk6s, memex-portal-deployment-747d45bbf7-djm8g, memex-portal-deployment-c89ddfbb6-txnd5, memex-portal-deployment-6dfbd755f6-c5dr6, memex-portal-deployment-66cc6466bf-frwpk, memex-portal-deployment-68cd75c658-8qt9s, memex-portal-deployment-f47f8cdd-rx8xj, memex-portal-deployment-7cb5549c57-8c9gr, memex-portal-deployment-89d7df89f-lrbc4, memex-portal-deployment-89d7df89f-k9fkr, memex-portal-deployment-764d5fbf-nn6nj, memex-portal-deployment-76b967b5fb-pkw5p, memex-portal-deployment-559ddb8f8-dbpx6, memex-portal-deployment-98785f547-76qjj, memex-portal-deployment-59ccff47c5-xw4zx, memex-portal-deployment-d7dd9796b-rc4bh, memex-portal-deployment-d7dd9796b-66g44, memex-portal-deployment-588c4cbb68-jm8cz, memex-portal-deployment-55f6c4c4c5-wcccn, memex-portal-deployment-58cf8c86f7-p95sf, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-7897bfd494-x8rfw, memex-portal-deployment-7897bfd494-dl9wb, memex-portal-deployment-7d6b864b54-w4xnl, memex-portal-deployment-7d6b864b54-gr6n6, memex-portal-deployment-688b496b7-2sdt8, memex-portal-deployment-688b496b7-brp4j, memex-portal-deployment-999c9c79d-8ccj2, memex-portal-deployment-999c9c79d-g9wps, memex-portal-deployment-6dc7cf965d-8f5t6, memex-portal-deployment-64d9857f9d-bmrnj, memex-portal-deployment-64d9857f9d-v7hmx, memex-portal-deployment-df7ddfdf4-xb9bk, memex-portal-deployment-6d58c699f8-bp5bs, memex-portal-deployment-6d58c699f8-9t675, memex-portal-deployment-775748f44d-7xlkm, memex-portal-deployment-7969b8cfff-fqnk9, memex-portal-deployment-76cbbdd5f-9mtkj, memex-portal-deployment-76cbbdd5f-82mvq, memex-portal-deployment-569544576b-q6nvd, memex-portal-deployment-569544576b-89r5k, memex-portal-deployment-569544576b-9h52b, memex-portal-deployment-569544576b-wtfz5, memex-portal-deployment-569544576b-cqlnw, memex-portal-deployment-867dc45877-lgskl, memex-portal-deployment-867dc45877-h8tjc, memex-portal-deployment-867dc45877-fmcss, memex-portal-deployment-58c7bb4b9d-97pjf, memex-portal-deployment-5b6c4f89f8-rpfmq, memex-portal-deployment-58c7bb4b9d-z2xcv, memex-portal-deployment-5b6c4f89f8-hpcbf, memex-portal-deployment-5b6c4f89f8-97sbj, memex-portal-deployment-5b6c4f89f8-kz4rh, memex-portal-deployment-6f64bf4b56-gxmrr, memex-portal-deployment-6f64bf4b56-2mzrf, memex-portal-deployment-6f64bf4b56-xqw96, memex-portal-deployment-858f57dc6-8zlfh, memex-portal-deployment-547649b6f8-wmt6n, memex-portal-deployment-547649b6f8-48jrz, memex-portal-deployment-858f57dc6-wns9j, memex-portal-deployment-7b67cb5577-rctvc, memex-portal-deployment-7d8787bcc7-4htmx, memex-portal-deployment-7544fdd47b-pbrvx, memex-portal-deployment-6dc5679f77-nd67w, memex-portal-deployment-6dc5679f77-vf2qx, memex-portal-deployment-858f57dc6-bkkfv, memex-portal-deployment-858f57dc6-ff5kn, memex-portal-deployment-858f57dc6-6wggn, memex-portal-deployment-6fd666bc95-s79r4, memex-portal-deployment-6fd666bc95-f7brs, memex-portal-deployment-66ff6fc5f9-nwl9f, memex-portal-deployment-66ff6fc5f9-mnw9n, memex-portal-deployment-79b5fd5f6b-7wdn4, memex-portal-deployment-79b5fd5f6b-jg4td, memex-portal-deployment-cf988c6f5-gtqpl, memex-portal-deployment-7b5f547cb7-cp2wv, memex-portal-deployment-7b5f547cb7-ps2nz, memex-portal-deployment-6c7b9889c7-rkr8g, memex-portal-deployment-6c7b9889c7-72fmv, memex-portal-deployment-cf5dc996f-pgjtz, memex-portal-deployment-cf5dc996f-gk2bv, memex-portal-deployment-6c46b84848-hnh9v, memex-portal-deployment-6c46b84848-c9mm5, memex-portal-deployment-6bfbc8ff4b-rz9p4, memex-portal-deployment-6bfbc8ff4b-vvc82, memex-portal-deployment-86779d495b-ksxqq, memex-portal-deployment-86779d495b-qrnh6, memex-portal-deployment-595d8b9995-j5f9p, memex-portal-deployment-595d8b9995-xntj7, memex-portal-deployment-7d95465fbb-s7gds, memex-portal-deployment-7d95465fbb-qrhvg, memex-portal-deployment-5df4ccb89-z8w2k, memex-portal-deployment-5df4ccb89-sprw2, memex-portal-deployment-5df4ccb89-txn2l, memex-portal-deployment-d999f9888-sfpph, memex-portal-deployment-dbc8997cd-q2n4s, memex-portal-deployment-66cfbf759-svxh9, memex-portal-deployment-75597864c8-wwfnf, memex-portal-deployment-9bf79b774-tk88b, memex-portal-deployment-67dbc7bc9c-lswwx, memex-portal-deployment-9bf79b774-qlks7, memex-portal-deployment-9bf79b774-jwrng, memex-portal-deployment-85859db964-mgxj7, memex-portal-deployment-77cfc55bfc-5ztnt, memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-54f4c96b6b-pn9z5, memex-portal-deployment-77cfc55bfc-tgtr4, memex-portal-deployment-77cfc55bfc-gxz25, memex-portal-deployment-5b9cbf46dd-5nqbz, memex-portal-deployment-5b9cbf46dd-zn7nf, memex-portal-deployment-79ddd44d77-k8qlc, memex-portal-deployment-6c74f8458-w46gf, memex-portal-deployment-bf77c847-tctvp, memex-portal-deployment-bf77c847-dmpfd, memex-portal-deployment-bf77c847-m5l4r, memex-portal-deployment-5bf869698f-5qgvz, memex-portal-deployment-574d54c98d-cdd47, memex-portal-deployment-747c879d8c-ph572, memex-portal-deployment-8ff67f8c7-gcf24, memex-portal-deployment-7fd9668b69-sjm5v, memex-portal-deployment-bfc5dcf6f-7z88q, memex-portal-deployment-6c7d6b6b44-klrt7, memex-portal-deployment-7fd9668b69-j4fdv, memex-portal-deployment-7fd9668b69-4qtkx, memex-portal-deployment-854cd8cd6d-8bc2q, memex-portal-deployment-854cd8cd6d-r4cn5, memex-portal-deployment-7fd9668b69-lzm7l, memex-portal-deployment-7fd9668b69-v72dx, memex-portal-deployment-6546c779d4-4ngqv, memex-portal-deployment-7fd9668b69-4zzzs, memex-portal-deployment-7fd9668b69-jnwlv, memex-portal-deployment-7fd9668b69-kvwgv, memex-portal-deployment-7cb6684584-ztqz8, memex-portal-deployment-54787585bc-qfwxx, memex-portal-deployment-6c967c7c4c-2k4xb, memex-portal-deployment-6b879996c7-dstvb, memex-portal-deployment-6b879996c7-lk6fd, memex-portal-deployment-6b879996c7-cmnfg, memex-portal-deployment-6c967c7c4c-kdff5, memex-portal-deployment-bbf878dfb-gknl9, memex-portal-deployment-5564495d46-f8xhs, memex-portal-deployment-86899455c7-z9nm2, memex-portal-deployment-6cd5d8f887-lvmpf, memex-portal-deployment-6cd5d8f887-2kcwk, memex-portal-deployment-58dbb6d98d-8msww, memex-portal-deployment-58dbb6d98d-lzqrr, memex-portal-deployment-7978987466-kmrpk, memex-portal-deployment-85f5bcf79d-n6lj2, memex-portal-deployment-85f5bcf79d-cgwtr, memex-portal-deployment-69956b6dbc-ndnxt, memex-portal-deployment-69956b6dbc-gx6z6, memex-portal-deployment-69956b6dbc-s246c, memex-portal-deployment-69956b6dbc-gj8m2, memex-portal-deployment-69956b6dbc-v5r29, memex-portal-deployment-65f8c68995-c7z8l, memex-portal-deployment-65f8c68995-z27tq, memex-portal-deployment-5f69f6b6bd-74xjv, memex-portal-deployment-5f69f6b6bd-z7tb6, memex-portal-deployment-69956b6dbc-ntksf, memex-portal-deployment-69956b6dbc-mczlv, memex-portal-deployment-6b7c4f94db-4sgtk, memex-portal-deployment-5485b778c9-mqz2v, memex-portal-deployment-75b545bbd-nssvp, memex-portal-deployment-6b7c4f94db-zzvfq, memex-portal-deployment-75b545bbd-gwrc6, memex-portal-deployment-77494bdbcc-7ndrn, memex-portal-deployment-77494bdbcc-5djgr, memex-portal-deployment-77494bdbcc-rg99v, memex-portal-deployment-77494bdbcc-6sv8l, memex-portal-deployment-77494bdbcc-bphfb, memex-portal-deployment-77494bdbcc-kkjld, memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-86dbb4d55c-jj7gc, memex-portal-deployment-857546649-rb7qd, memex-portal-deployment-857546649-xx9j4, memex-portal-deployment-857546649-vf2q6, memex-portal-deployment-c54d5847c-bf27p, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-5dd79dc746-4bhs7, memex-portal-deployment-5dd79dc746-45vzd, memex-portal-deployment-5dd79dc746-6vmxd, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-777677694-cc6zz, memex-portal-deployment-777677694-hjkc8, memex-portal-deployment-777677694-d4j8c, memex-portal-deployment-786d86cf54-895wl, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-76cf776847-5hflx, memex-portal-deployment-76cf776847-2x9rq

Recent log lines

2026-09-23 21:13:45Z memex-portal-deployment-76cf776847-5hflx fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'plugin-registry-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''
2026-09-23 21:14:12Z memex-portal-deployment-76cf776847-5hflx fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: 'plugin-registry-standard//Standard-TotalRequestTimeout', Operation Key: '', Result: ''
2026-09-24 05:43:17Z memex-portal-deployment-76cf776847-2x9rq fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: '-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''

_{Re-addressed by the current identity function: this incident inherited 9ca334c1e8dad9ca (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Still occurring: 36 more since the last update (4119 total).

Last seen: 2026-09-24 06:15:11Z
Pods: memex-portal-deployment-6c99c9f796-mh5cs, memex-portal-deployment-84855f7679-z4rsp, memex-portal-deployment-575dc48bfb-wq7s8, memex-portal-deployment-6bd9c46f8b-z9t6r, memex-portal-deployment-85ddbf6c7f-p4t4b, memex-portal-deployment-59479dc7d9-kmwvr, memex-portal-deployment-748bfcc86b-2m6vg, memex-portal-deployment-6fc79755d4-r96vd, memex-portal-deployment-57885dbb86-76zsg, memex-portal-deployment-6d9cdd447b-tm4ck, memex-portal-deployment-6cd98575f6-gr5ds, memex-portal-deployment-784494885b-c5sz5, memex-portal-deployment-6854bb96cb-z6bn9, memex-portal-deployment-6c9bb5d95c-jb9dq, memex-portal-deployment-69f9fc445-6fws2, memex-portal-deployment-c9584c6dd-dms29, memex-portal-deployment-cc58dc7c6-ckll8, memex-portal-deployment-67967894c4-7fvvg, memex-portal-deployment-c594497fc-dsgds, memex-portal-deployment-86bd794cb5-4rs66, memex-portal-deployment-8489f66759-zlbcg, memex-portal-deployment-745d7588ff-4mrxf, memex-portal-deployment-797df55dfb-cn4p8, memex-portal-deployment-97c56b569-tjnmj, memex-portal-deployment-bf88ffcbb-t6nhv, memex-portal-deployment-6cc6f7997b-99xkc, memex-portal-deployment-7799dd87b9-vshbj, memex-portal-deployment-6fcf6bbc47-c2nw9, memex-portal-deployment-6fcf6bbc47-8qvz6, memex-portal-deployment-79766fd4f6-w66fq, memex-portal-deployment-648c756f74-smlsb, memex-portal-deployment-745b99b846-hrk46, memex-portal-deployment-84799488cb-cb7q4, memex-portal-deployment-78cf8bd9f5-klqft, memex-portal-deployment-f96d46ddf-zn9nl, memex-portal-deployment-8669cb7675-xksqg, memex-portal-deployment-5d6f77b68c-srtqg, memex-portal-deployment-5b7b597fb5-s6xf9, memex-portal-deployment-596d7ff9bf-vtsp6, memex-portal-deployment-7c47bc9c85-xhspn, memex-portal-deployment-79756bcc4f-mb2c5, memex-portal-deployment-65f87d4594-5vsw8, memex-portal-deployment-696cc645c6-gfw6n, memex-portal-deployment-6c5bd56bf6-kqk6s, memex-portal-deployment-747d45bbf7-djm8g, memex-portal-deployment-c89ddfbb6-txnd5, memex-portal-deployment-6dfbd755f6-c5dr6, memex-portal-deployment-66cc6466bf-frwpk, memex-portal-deployment-68cd75c658-8qt9s, memex-portal-deployment-f47f8cdd-rx8xj, memex-portal-deployment-7cb5549c57-8c9gr, memex-portal-deployment-89d7df89f-lrbc4, memex-portal-deployment-89d7df89f-k9fkr, memex-portal-deployment-764d5fbf-nn6nj, memex-portal-deployment-76b967b5fb-pkw5p, memex-portal-deployment-559ddb8f8-dbpx6, memex-portal-deployment-98785f547-76qjj, memex-portal-deployment-59ccff47c5-xw4zx, memex-portal-deployment-d7dd9796b-rc4bh, memex-portal-deployment-d7dd9796b-66g44, memex-portal-deployment-588c4cbb68-jm8cz, memex-portal-deployment-55f6c4c4c5-wcccn, memex-portal-deployment-58cf8c86f7-p95sf, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-7897bfd494-x8rfw, memex-portal-deployment-7897bfd494-dl9wb, memex-portal-deployment-7d6b864b54-w4xnl, memex-portal-deployment-7d6b864b54-gr6n6, memex-portal-deployment-688b496b7-2sdt8, memex-portal-deployment-688b496b7-brp4j, memex-portal-deployment-999c9c79d-8ccj2, memex-portal-deployment-999c9c79d-g9wps, memex-portal-deployment-6dc7cf965d-8f5t6, memex-portal-deployment-64d9857f9d-bmrnj, memex-portal-deployment-64d9857f9d-v7hmx, memex-portal-deployment-df7ddfdf4-xb9bk, memex-portal-deployment-6d58c699f8-bp5bs, memex-portal-deployment-6d58c699f8-9t675, memex-portal-deployment-775748f44d-7xlkm, memex-portal-deployment-7969b8cfff-fqnk9, memex-portal-deployment-76cbbdd5f-9mtkj, memex-portal-deployment-76cbbdd5f-82mvq, memex-portal-deployment-569544576b-q6nvd, memex-portal-deployment-569544576b-89r5k, memex-portal-deployment-569544576b-9h52b, memex-portal-deployment-569544576b-wtfz5, memex-portal-deployment-569544576b-cqlnw, memex-portal-deployment-867dc45877-lgskl, memex-portal-deployment-867dc45877-h8tjc, memex-portal-deployment-867dc45877-fmcss, memex-portal-deployment-58c7bb4b9d-97pjf, memex-portal-deployment-5b6c4f89f8-rpfmq, memex-portal-deployment-58c7bb4b9d-z2xcv, memex-portal-deployment-5b6c4f89f8-hpcbf, memex-portal-deployment-5b6c4f89f8-97sbj, memex-portal-deployment-5b6c4f89f8-kz4rh, memex-portal-deployment-6f64bf4b56-gxmrr, memex-portal-deployment-6f64bf4b56-2mzrf, memex-portal-deployment-6f64bf4b56-xqw96, memex-portal-deployment-858f57dc6-8zlfh, memex-portal-deployment-547649b6f8-wmt6n, memex-portal-deployment-547649b6f8-48jrz, memex-portal-deployment-858f57dc6-wns9j, memex-portal-deployment-7b67cb5577-rctvc, memex-portal-deployment-7d8787bcc7-4htmx, memex-portal-deployment-7544fdd47b-pbrvx, memex-portal-deployment-6dc5679f77-nd67w, memex-portal-deployment-6dc5679f77-vf2qx, memex-portal-deployment-858f57dc6-bkkfv, memex-portal-deployment-858f57dc6-ff5kn, memex-portal-deployment-858f57dc6-6wggn, memex-portal-deployment-6fd666bc95-s79r4, memex-portal-deployment-6fd666bc95-f7brs, memex-portal-deployment-66ff6fc5f9-nwl9f, memex-portal-deployment-66ff6fc5f9-mnw9n, memex-portal-deployment-79b5fd5f6b-7wdn4, memex-portal-deployment-79b5fd5f6b-jg4td, memex-portal-deployment-cf988c6f5-gtqpl, memex-portal-deployment-7b5f547cb7-cp2wv, memex-portal-deployment-7b5f547cb7-ps2nz, memex-portal-deployment-6c7b9889c7-rkr8g, memex-portal-deployment-6c7b9889c7-72fmv, memex-portal-deployment-cf5dc996f-pgjtz, memex-portal-deployment-cf5dc996f-gk2bv, memex-portal-deployment-6c46b84848-hnh9v, memex-portal-deployment-6c46b84848-c9mm5, memex-portal-deployment-6bfbc8ff4b-rz9p4, memex-portal-deployment-6bfbc8ff4b-vvc82, memex-portal-deployment-86779d495b-ksxqq, memex-portal-deployment-86779d495b-qrnh6, memex-portal-deployment-595d8b9995-j5f9p, memex-portal-deployment-595d8b9995-xntj7, memex-portal-deployment-7d95465fbb-s7gds, memex-portal-deployment-7d95465fbb-qrhvg, memex-portal-deployment-5df4ccb89-z8w2k, memex-portal-deployment-5df4ccb89-sprw2, memex-portal-deployment-5df4ccb89-txn2l, memex-portal-deployment-d999f9888-sfpph, memex-portal-deployment-dbc8997cd-q2n4s, memex-portal-deployment-66cfbf759-svxh9, memex-portal-deployment-75597864c8-wwfnf, memex-portal-deployment-9bf79b774-tk88b, memex-portal-deployment-67dbc7bc9c-lswwx, memex-portal-deployment-9bf79b774-qlks7, memex-portal-deployment-9bf79b774-jwrng, memex-portal-deployment-85859db964-mgxj7, memex-portal-deployment-77cfc55bfc-5ztnt, memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-54f4c96b6b-pn9z5, memex-portal-deployment-77cfc55bfc-tgtr4, memex-portal-deployment-77cfc55bfc-gxz25, memex-portal-deployment-5b9cbf46dd-5nqbz, memex-portal-deployment-5b9cbf46dd-zn7nf, memex-portal-deployment-79ddd44d77-k8qlc, memex-portal-deployment-6c74f8458-w46gf, memex-portal-deployment-bf77c847-tctvp, memex-portal-deployment-bf77c847-dmpfd, memex-portal-deployment-bf77c847-m5l4r, memex-portal-deployment-5bf869698f-5qgvz, memex-portal-deployment-574d54c98d-cdd47, memex-portal-deployment-747c879d8c-ph572, memex-portal-deployment-8ff67f8c7-gcf24, memex-portal-deployment-7fd9668b69-sjm5v, memex-portal-deployment-bfc5dcf6f-7z88q, memex-portal-deployment-6c7d6b6b44-klrt7, memex-portal-deployment-7fd9668b69-j4fdv, memex-portal-deployment-7fd9668b69-4qtkx, memex-portal-deployment-854cd8cd6d-8bc2q, memex-portal-deployment-854cd8cd6d-r4cn5, memex-portal-deployment-7fd9668b69-lzm7l, memex-portal-deployment-7fd9668b69-v72dx, memex-portal-deployment-6546c779d4-4ngqv, memex-portal-deployment-7fd9668b69-4zzzs, memex-portal-deployment-7fd9668b69-jnwlv, memex-portal-deployment-7fd9668b69-kvwgv, memex-portal-deployment-7cb6684584-ztqz8, memex-portal-deployment-54787585bc-qfwxx, memex-portal-deployment-6c967c7c4c-2k4xb, memex-portal-deployment-6b879996c7-dstvb, memex-portal-deployment-6b879996c7-lk6fd, memex-portal-deployment-6b879996c7-cmnfg, memex-portal-deployment-6c967c7c4c-kdff5, memex-portal-deployment-bbf878dfb-gknl9, memex-portal-deployment-5564495d46-f8xhs, memex-portal-deployment-86899455c7-z9nm2, memex-portal-deployment-6cd5d8f887-lvmpf, memex-portal-deployment-6cd5d8f887-2kcwk, memex-portal-deployment-58dbb6d98d-8msww, memex-portal-deployment-58dbb6d98d-lzqrr, memex-portal-deployment-7978987466-kmrpk, memex-portal-deployment-85f5bcf79d-n6lj2, memex-portal-deployment-85f5bcf79d-cgwtr, memex-portal-deployment-69956b6dbc-ndnxt, memex-portal-deployment-69956b6dbc-gx6z6, memex-portal-deployment-69956b6dbc-s246c, memex-portal-deployment-69956b6dbc-gj8m2, memex-portal-deployment-69956b6dbc-v5r29, memex-portal-deployment-65f8c68995-c7z8l, memex-portal-deployment-65f8c68995-z27tq, memex-portal-deployment-5f69f6b6bd-74xjv, memex-portal-deployment-5f69f6b6bd-z7tb6, memex-portal-deployment-69956b6dbc-ntksf, memex-portal-deployment-69956b6dbc-mczlv, memex-portal-deployment-6b7c4f94db-4sgtk, memex-portal-deployment-5485b778c9-mqz2v, memex-portal-deployment-75b545bbd-nssvp, memex-portal-deployment-6b7c4f94db-zzvfq, memex-portal-deployment-75b545bbd-gwrc6, memex-portal-deployment-77494bdbcc-7ndrn, memex-portal-deployment-77494bdbcc-5djgr, memex-portal-deployment-77494bdbcc-rg99v, memex-portal-deployment-77494bdbcc-6sv8l, memex-portal-deployment-77494bdbcc-bphfb, memex-portal-deployment-77494bdbcc-kkjld, memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-86dbb4d55c-jj7gc, memex-portal-deployment-857546649-rb7qd, memex-portal-deployment-857546649-xx9j4, memex-portal-deployment-857546649-vf2q6, memex-portal-deployment-c54d5847c-bf27p, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-5dd79dc746-4bhs7, memex-portal-deployment-5dd79dc746-45vzd, memex-portal-deployment-5dd79dc746-6vmxd, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-777677694-cc6zz, memex-portal-deployment-777677694-hjkc8, memex-portal-deployment-777677694-d4j8c, memex-portal-deployment-786d86cf54-895wl, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-76cf776847-5hflx, memex-portal-deployment-76cf776847-2x9rq

Recent log lines

2026-09-24 06:14:51Z memex-portal-deployment-76cf776847-2x9rq fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: '-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''
2026-09-24 06:15:03Z memex-portal-deployment-76cf776847-2x9rq fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: '-standard//Standard-AttemptTimeout', Operation Key: '', Result: ''
2026-09-24 06:15:11Z memex-portal-deployment-76cf776847-2x9rq fail: Polly[0]
      Resilience event occurred. EventName: 'OnTimeout', Source: '-standard//Standard-TotalRequestTimeout', Operation Key: '', Result: ''

_{Re-addressed by the current identity function: this incident inherited 9ca334c1e8dad9ca (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

2026-09-24: who `-standard//…` is. It is not an unnamed client. The bundle pipeline is nearly quiet.

This comment corrects the 09-23 16:48Z comment and settles its open question. I measured with read-only Logs InstanceActions on memex.meshweaver.cloud. Its operator reads the cluster's Loki, so the memex namespace (the control instance) was readable too.

1. `-standard` is the shared defaults pipeline, not an unnamed client

ConfigureHttpClientDefaults hands out a builder with no name. AddStandardResilienceHandler names its pipeline PipelineNameHelper.GetName(builder.Name, "standard"), which comes out as {builder.Name}-standard (I decompiled Microsoft.Extensions.Http.Resilience 10.10.0 to check). As a result:

Every client that does not re-register itself logs as -standard, whether it has a name or not.
Nothing selects an instance, so the registry key is HttpKey("-standard", ""). That is ONE pipeline with ONE circuit breaker, shared by all of those clients.

A sweep of core and Plugins found no CreateClient() without a name. The callers are ordinary named clients: self-update-handover, deployment-report, module-published-broadcast, OpenGraphPreview, the OCI listers, WebSearchPlugin, TeamsClient, and others.

2. The caller behind today's samples is `SelfUpdateHandover.Post`

Ops/logs-memexcloud-20260924-timeout-stack2 (pod memex-portal-deployment-589ff8f895-vn49b, 100 min): every Polly.Timeout.TimeoutRejectedException is followed by ResilienceHandler.SendAsync frames and then Memex.Portal.Shared.SelfUpdate.SelfUpdateHandover.<>c__DisplayClass37_0.<<Pos…. This holds at 12:33:06, 12:34:44, 12:52:31, 13:43:15 and 13:43:32Z.
Ops/logs-memexcloud-20260924-2x9rq-stack: the pod the bot quoted at 05:43 and 06:15Z, …76cf776847-2x9rq, is a memex-cloud replica. Its stacks end in the same frame, at 06:24:22, 06:25:05, 06:25:25, 06:29:53 and 06:30:11Z.
The target is memex-cloud's signed release/restart hand-over to https://memex.systemorph.com/api/hooks/Hosting/PlatformBuilds.

3. The target stops answering because the control instance is restarting

I compared two reads:

Ops/logs-memexcloud-20260924-totaltimeouts-1440: 157 lines, each a -standard//Standard-TotalRequestTimeout or a [SelfUpdate] … hand-over … FAILED; the next check announces it again.
Ops/logs-memex-20260924-boots-600: the control instance's own Now listening on and Application is shutting down lines.

The large bursts line up with control replicas starting:

memex-cloud failures	control replicas starting
05:50–06:30	06:29:58 and 06:30:32; no graceful shutdown logged before them
07:10–07:20	07:07:40 and 07:22:13
08:20	08:22:27
09:20–09:30	09:21:30 and 09:32:16
12:20–12:50	12:23:27 and 12:47:51
13:40	13:43:32 and 13:43:53; these attempts got 503 in 1.3 ms, an ingress with no ready backend

The small buckets (07:50, 08:40–08:50, 13:10) have no boot beside them, and I have not explained them. The boot read cannot see a replica that was killed without a shutdown line, or one that was alive but stalled.

4. The pipeline this issue was filed for is almost quiet

memex-cloud, 360 min: plugin-registry-bundles-standard|plugin-registry-standard|did NOT complete after matched 0 lines.
memex (control), 360 min: one line, Bundle for OgCard@1.1.6 over HTTP from https://memex.meshweaver.cloud did NOT complete after 119999 ms … the registry … did not begin a response within 120 s (11:57:13Z, pod …56fbdcd48f-7cd4h). That is the registry-side time-to-first-byte tracked in The plugin registry's bundle index hangs for authenticated callers — 180s, 0 bytes received, while /api/version answers in 0.12s #4963.
On 09-19 the same bundle pipeline logged 375 in about 12 h.

What changed

fix(resilience): every HttpClient names its own pipeline instance — '-standard//' named nobody (#4528) #5664: an outermost handler stamps the client's name onto each request, and the defaults' standard handler selects its pipeline instance by that name (SelectPipelineBy). Lines now read -standard/self-update-handover/Standard-AttemptTimeout, and each client gets its own breaker. The key is the client name, not the host: the defaults also fetch URLs taken from user data, and the pipeline registry never evicts. A test reads the identity from Polly's own log line. Its negative control reproduces '-standard//Standard-Retry'. The PR adds the doc page Doc/Architecture/EveryHttpClientNamesItsPipeline.
Systemorph/MeshWeaver.Plugins#2359: the fingerprint splits. LogLineParser.Normalize now keeps {pipeline:<pipeline>/<instance>} and drops the strategy, so an attempt timeout and the total timeout it rolls up into stay one incident. Five Polly sources become five identities. Against the parser on main, all five fold into 1.

What remains

The control instance restarts about hourly and stays unresponsive for minutes after it starts listening. This is the real reason the hand-over gets no answer. No client budget fixes it, and raising the hand-over's timeout would only make each failure slower. The hand-over already behaves as designed: it fails, and the next check announces again.
After fix(#2229): create-then-point-read of an activity can NotFound-race the router #2359 is deployed, this issue's counter should carry only plugin-registry-bundles-standard. The occasional remaining failure is The plugin registry's bundle index hangs for authenticated callers — 180s, 0 bytes received, while /api/version answers in 0.12s #4963's registry time-to-first-byte. This issue should then close on a week of that pipeline staying quiet, not on the shared counter.
Not established: why the control replicas restart (crash, roll, or liveness kill). I did not read their exit reasons.

Triage 2026-09-25: fixed on this pipeline. The instrument fix is waiting on a re-queue, then deploy and an observation window.

The bundle transfer's budget measured its SIZE, and recorded nothing (#4528) #4549 (streaming, a silence bound) and Bundle transfers fail on SILENCE at a named stage — the client owns every clock (#4528) #4977 are live on both portals. As of the 09-24 comment the bundle pipeline itself was almost quiet: 0 lines on memex-cloud in 360 min, and 1 on memex, which is registry time-to-first-byte and is tracked as The plugin registry's bundle index hangs for authenticated callers — 180s, 0 bytes received, while /api/version answers in 0.12s #4963.
Plugins#2359 (the fingerprint split per Polly pipeline) merged 2026-09-24 15:08Z.
fix(resilience): every HttpClient names its own pipeline instance — '-standard//' named nobody (#4528) #5664 (each HttpClient names its own pipeline instance) is open. Its CI is green, but it is labelled queue-rejected. Both ejections are attributed to other PRs, and the 09-25 08:05Z comment says it is ready for a person to re-queue (gh pr merge 5664 --auto). Per the merge-queue rules that is a person's act, and I did not take it.

Remaining: re-queue #5664, deploy both halves (blocked behind the V58 wall, #4764), then close on a week of plugin-registry-bundles-standard staying quiet under its own fingerprint.

2026-09-26 status: every named fix is deployed; the bundles pipeline's own fingerprint now shows a DIFFERENT failure (registry 500s), fixed in #5769; the registry's serve cost is measured and cut in #5768

Deployed (checked with git merge-base --is-ancestor against /api/version on both portals — memex.systemorph.com 45e7f94e4e, memex.meshweaver.cloud 4c8530d7dd): #4549 (32f0523e2) and #5664 (92006e8ef5) are on both. So the bundles pipeline now reports under its own name.

What that name shows today — Admin/_LogIncident/613ea204988e9e23 (→ #5764, closed as a duplicate of #3876): memex pods 85f748bdd7-qvqf7/-nk4wg, plugin-registry-bundles-standard//Standard-Retry, attempt 3, Result: '500', 6.6–17.3 s per attempt, 17:38:40Z → 17:50:16Z. Not OnTimeout. The 500 is the registry's: FileNotFoundException in the per-package bundle route when the assembly store evicts a version between resolving and opening it (#3876's recurrence, same minutes). Fix: #5769 (503 + Retry-After: 30, pinned on the wire).

The 7–17 s per attempt is the registry's own serve cost, now measured stage by stage inside the registry (details on #4963): one derivation of the module-activation list over the share is 7.3–8.7 s, and the index runs it twice per request. #5768 makes the index read a maintained snapshot. The per-package download still derives it twice (~16 s on this registry) — named in Doc/Architecture/PluginRegistryIndexCost as the next candidate; not changed here, because the download must resolve a version another replica just published.

Remains open (sev:H closes on post-roll verification, not on a merge): after #5768/#5769 roll, a quiet week on plugin-registry-bundles-standard under its own fingerprints (OnTimeout and 5xx exhaustion alike). Not established: whether any OnTimeout on this pipeline has occurred since the split — I found no split incident for plugin-registry-bundles-standard//Standard-AttemptTimeout by search, which is a floor, not proof of absence.

2026-09-30 ~16:55Z: closing condition NOT met; the bundles pipeline now exhausts on 503s

#5768 (index snapshot) and #5769 (503 + Retry-After: 30 on the serve race) merged 2026-09-26 and are ancestors of e6cd43a64d38, which both portals run.

Admin/_LogIncident/613ea204988e9e23 on the control instance is the bundles pipeline under its own fingerprint. It has kept advancing: 29 occurrences from 2026-09-26 17:38Z to 2026-09-30 13:50:26Z, across 14 memex (control) pods. All 10 retained samples read the same way:

Execution attempt. Source: 'plugin-registry-bundles-standard//Standard-Retry', Result: '503', Handled: 'True', Attempt: '3', Execution Time: 6.2–7.5 s

Samples are dated 09-28 00:18Z, 02:13–02:15Z, 04:14Z, 20:50–21:00Z, and 09-30 13:43Z and 13:50Z.

What changed: the result went from 500 to 503, so #5769 is doing what it says. But attempt 3 is the last retry of the standard pipeline, and the standard retry honours Retry-After. So the registry answered 503 on four attempts spread over roughly 90 s or more. That does not fit a one-off eviction race.

The root cause is not established from here. The registry route has three 503 branches (PluginBundleEndpoints.cs): the serve race (TransientWhenServedBytesMoved, #3876), a stalled catalogue read (UnavailableOnAStalledRead, #5345), and the republish window (BeingRepublished, #3401). Each logs a distinct Warning on the registry side, and Warnings do not become incidents. Telling them apart needs a Logs read on the registry instance (memex-cloud) around 2026-09-30 13:43–13:51Z, with query Plugin bundles:. That is an InstanceAction. I did not issue it.

Ask (maintainer or operator): run { "requestedAction": "Logs", "deployment": "Deployments/memex-cloud", "query": "Plugin bundles:", "sinceMinutes": … } covering 13:40–13:55Z today. The branch it names is the next fix. Keep this issue open.

Reopened by the fleet's bug-fix pool hand-over (maintainer request 2026-09-30: start a bug-fix thread for every open sev:H bug). This bug never got a bug-fix thread (it was recorded before the intake handed classified bugs to one, or never recorded); the reopened event is the intake's supported path to start it on the EU bug-fix pool. The close/reopen changes nothing else.

2026-09-30: why the 503 persisted, and the fix (#5928, merged as `dead2513fd`)

Which 503 branch this was. The per-package bundle route has three 503 branches: instance-key auth unavailable (Retry-After: 5, bounded at about 10 s), a stalled catalogue read (Retry-After: 5), and the serve race TransientWhenServedBytesMoved (Retry-After: 30). The evidence points to the serve race, and it was not a race:

Per-attempt time. Each attempt took 6.2–7.5 s. That is the download's authoritative activation read followed by a fast failure. An auth 503 would land at about 10 s.
The same file failed again and again. In Admin/_LogIncident/c9af8fba2318492d on memex-cloud, MyAi_Panel/v751-… failed 4× in 30 s, and SocialMedia_Profile/v1413-… and LinkedIn_TileMigration/v643-… failed 2× each.

Root cause. FileSystemAssemblyStore.TryGetAssemblyPath returned the newest dll name the directory listed. On the Azure Files share, a dll that was evicted while another replica still holds it open is delete-pending. It stays in the listing and returns ENOENT on every open until that handle closes. So the lookup reported a hit on unreadable bytes for hours, and the route's 503 asked consumers to retry a condition that no retry could outlast.

Fix.

The lookup now proves each candidate with an open. An unopenable name is skipped and logged as a Warning (Assembly cache: … cannot be opened). An openable sibling of the same version is used if there is one; otherwise it is a miss. On a miss the route counts the type and the consumer compiles it, which is the designed path.
Put's first-write-wins probe uses the same check.
A publication of identical bytes no longer lands on its own dead content-hash name. It steps to a salted-hash name.
Four tests on a dangling symlink, each red before the fix; 1147/1147 pass after. No retry or budget was changed.

Not established. Whether the share-side mechanism is delete-pending or a stale SMB directory cache (the fix covers both). Also why the record's LastCompiledVersion names a version that eviction has already removed.

Closing condition (unchanged). After this rolls to memex-cloud, 613ea204988e9e23 stops advancing. The confirming read is a Logs action on Deployments/memex-cloud with query Assembly cache:, which should show the skips where the 503s used to be. I did not issue it because it has not been approved. No recycle is needed.

2026-10-07: what is still owed, measured. The shape and the budgets are live; the measurement was invisible (fix: Plugins#3093)

Read-only, through governed Logs InstanceActions on the control instance.

1. The roll is done. Both portals run builds that contain #4549, #5664, #5768 and #5769 (09-26/09-30 notes). Both run Plugins publication 01f3db7c.

2. The inverted budgets are resolved in code. RegistryUpdateReconciler.PerPackageAdoptBudget = PluginBundleClient.TransferStallBudget + 1 min (src/MeshWeaver.PluginCatalog/RegistryUpdateReconciler.cs:1042). The client's silence bound fires first, and every refusal names its stage. Neither number was raised.

3. The byte/rate measurement existed in code but never reached Loki. That is the root cause of "still owed".

PluginBundleClient logs each completed transfer (…: N byte(s) over HTTP from R in T ms (K KiB/s), PluginBundleClient.cs:870) at Information, and each refusal (:951) at Warning.
The portal host's "MeshWeaver": "Warning" catch-all keeps only the refusals.
Measured: Ops/Actions/logs-4528-memex-stalls-0430-1040-20261007 (370 min, uncut) has 14 refusals on memex from 08:35:53Z to 09:12:54Z.
Ops/Actions/logs-4528-memex-rate-lines-0830-0920-20261007 and …-cloud-rate-lines2-… have 0 KiB/s or Bundle index from lines in the same windows, although both instances adopted bundles.
Fix: Systemorph/MeshWeaver.Plugins#3093 pins the TYPE to Information in both image hosts, with its cost stated. ShippedLoggingLevelsGuard holds the pin and has a scope control. Its negative control fails as expected.

4. What the refusals say, and what that rules out. All 14 refusals, plus memex-cloud's own at 10:20:53Z, are NoResponse with 0 bytes after 120 000 ms. No transfer was cut mid-body and none was over-size. So nothing measured today points at the archive size. What the instrument reads is the registry never beginning a response. That is #4963, where today's evidence is posted (it clusters in roll windows, and the registry side logs nothing for those requests).

Close condition for this issue: after a portal rolls with #3093's host config, a Logs read for KiB/s returns completed-transfer lines beside any refusal. The remaining zero-byte stall belongs to #4963.

🤖 Generated with Claude Code

Verified 2026-10-07 ~14:30Z — closing

The closing condition from the 11:00Z comment holds: "after a portal rolls with #3093's host config, a Logs read for KiB/s returns completed-transfer lines beside any refusal."

Deployed. Both portals run image 3.0.0-ci.10148, built from core 57a6e5fd and Plugins f60a811f. f60a811f is the merge of Plugins#3093 (main-cd #10148 log: PLUGINS_SHA: f60a811f…). memex replicas booted 13:12–13:47Z and memex-cloud replicas 13:09–13:12Z, so the windows below are fully post-roll.
memex, governed read Ops/Actions/logs-4528-sweep2-memex-kibs-1300-1425-20261007: query KiB/s|Bundle index from|did NOT complete after, 13:00–14:25Z, 89 lines read, not truncated. Completed transfers are now visible, for example Bundle for OpenAI@1.6.8: 14567010 byte(s) over HTTP from https://memex.meshweaver.cloud in 14128 ms (1006 KiB/s) (pod …79bfb7dc8b-ddhc7, 14:03:42Z). Index lines read 26636 byte(s) in 1371–58305 ms. Refusals appear beside them, for example Bundle for Copilot@1.3.38 … did NOT complete after 120002 ms — 0 of an undeclared number of byte(s) … the registry … did not begin a response within 120 s (14:07:51Z).
memex-cloud, Ops/Actions/logs-4528-sweep2-cloud-kibs-1300-1425-20261007: same query and window, 7 lines, not truncated. Five Bundle index … 16031 byte(s) in 1202–19411 ms lines and two Bundle index … did NOT complete refusals.
The measurement now answers the question. Completed bundles up to about 16.6 MB finish in well under the 120 s budget (≈1 MiB/s). Every refusal in the window is again 0 bytes, with no response begun. That is the registry-side stall tracked in The plugin registry's bundle index hangs for authenticated callers — 180s, 0 bytes received, while /api/version answers in 0.12s #4963, as the 11:00Z comment said. No refusal was cut mid-body.

The remaining zero-byte stall lives on #4963.

Bundle transfers: the 120 s budget bounded SIZE, not responsiveness — #4549 fixed the shape; the roll, the byte/rate measurement and the inverted 3 min/5 min budgets are still owed #4528

Description

What is failing

The measurement

Why it matters beyond the latency

Where to look

What this is NOT

Not investigated here

Activity

rbuergi commented on Sep 17, 2026

rbuergi commented on Sep 17, 2026

2026-09-17 — merged, not deployed, and by the fix's own doc not yet settled

Still owed, in order

rbuergi commented on Sep 19, 2026

🚨 The roll this issue's close condition waits on HAPPENED — and the defect did not stop

The consumer is on a set carrying #4549, on every replica, and has been for ~10.7 h

And plugin-registry-bundles is still timing out, right now

The cadence is the reconciler's, exactly as the 09-17 comment established

What this does to the three owed items

rbuergi commented on Sep 19, 2026

Follow-up, same session: the population is BOTH production deployments, and the incident now reports HERE

1. Attribution completed — the second pod generation is memex-cloud

2. Fifteen minutes later the shape has advanced 22 more

3. The incident node now points here, not at #4222

rbuergi commented on Sep 19, 2026

A second, independent consumer reproduces this from outside the portal — and it localises the stall to behind authentication

Where it is not

Why this raises the priority rather than just adding detail

systemorph commented on Sep 19, 2026

systemorph commented on Sep 20, 2026

systemorph commented on Sep 20, 2026

systemorph commented on Sep 21, 2026

rbuergi commented on Sep 21, 2026

Not verified in production — leaving open. I could inspect the code and the shipped configuration, and nothing more.

What IS established: the fix is rolled

What IS established: the shipped configuration

What is NOT established: everything timing-shaped — i.e. the whole point of the issue

Also relevant to whoever picks this up

The reading that would close this

33 remaining items

systemorph commented on Sep 23, 2026

systemorph commented on Sep 24, 2026

systemorph commented on Sep 24, 2026

systemorph commented on Sep 24, 2026

rbuergi commented on Sep 24, 2026

2026-09-24: who -standard//… is. It is not an unnamed client. The bundle pipeline is nearly quiet.

1. -standard is the shared defaults pipeline, not an unnamed client

2. The caller behind today's samples is SelfUpdateHandover.Post

3. The target stops answering because the control instance is restarting

4. The pipeline this issue was filed for is almost quiet

What changed

What remains

rbuergi commented on Sep 25, 2026

Triage 2026-09-25: fixed on this pipeline. The instrument fix is waiting on a re-queue, then deploy and an observation window.

rbuergi commented on Sep 26, 2026

2026-09-26 status: every named fix is deployed; the bundles pipeline's own fingerprint now shows a DIFFERENT failure (registry 500s), fixed in #5769; the registry's serve cost is measured and cut in #5768

rbuergi commented on Sep 30, 2026

2026-09-30 ~16:55Z: closing condition NOT met; the bundles pipeline now exhausts on 503s

rbuergi commented on Sep 30, 2026

rbuergi commented on Sep 30, 2026

2026-09-30: why the 503 persisted, and the fix (#5928, merged as dead2513fd)

rbuergi commented on Oct 7, 2026

2026-10-07: what is still owed, measured. The shape and the budgets are live; the measurement was invisible (fix: Plugins#3093)

rbuergi commented on Oct 7, 2026

Verified 2026-10-07 ~14:30Z — closing

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

And `plugin-registry-bundles` is still timing out, right now

2026-09-24: who `-standard//…` is. It is not an unnamed client. The bundle pipeline is nearly quiet.

1. `-standard` is the shared defaults pipeline, not an unnamed client

2. The caller behind today's samples is `SelfUpdateHandover.Post`

2026-09-30: why the 503 persisted, and the fix (#5928, merged as `dead2513fd`)