Skip to content

Cloud registry: instance-key resolution answers 503 for hours — a verified key with a seeded grant cannot fetch anything #2915

New issue
New issue
Closed
#3034
Closed
Cloud registry: instance-key resolution answers 503 for hours — a verified key with a seeded grant cannot fetch anything#2915
#3034
Labels
bugSomething isn't working

Description

@sierragolflima
sierragolflima
opened on Aug 31, 2026

What is failing

GET https://memex.meshweaver.cloud/api/plugins/bundles/index.json with a freshly minted instance key answers:

HTTP 503
{"error":"Instance-key resolution is temporarily unavailable — retry shortly.
 This is NOT a statement about your key or your grant."}

— continuously from ~15:45Z to at least 18:08Z on 2026-08-31 (~2.5 hours). GET /api/plugins with the same key answers the same way.

The key and grant are provably correct

  • The instance sglauser/MeshWeaverInstance/sglauser-local-3 was registered self-service at 15:44:49Z.
  • SHA-256 of the presented raw key equals the instance record's keyHash exactly (7064c344067c…).
  • Admin/_PluginGrant/sglauser-local-3 exists: {"source":"Plugins","packageId":"*"}, no tier cap, not revoked, written by system-security three seconds after registration (the default-grant seed).

So all three inputs of the authenticator's read exist and are consistent — the 503 means the resolution read reaches no verdict, which is exactly the InstanceAuthResponses unavailable path (503 + Retry-After, never cached).

It is not key-specific

Unauthenticated point reads on the same portal failed the same way earlier today: get Admin/Coupons/STAFF via the MCP surface returned "Unavailable — this read reached no verdict, so it is UNKNOWN whether this node exists … read of 'Admin/Coupons/STAFF' reached no verdict within 10s" — twice, ~13:20Z. Listing queries (search) answer normally throughout; point reads are what fail.

Ruled out

  • Warm-grain persistence: after the initial failures, the path received zero requests for two full hours (18:08–20:08 CEST), then a single probe — still 503. This is not a failing grain waiting to idle out.
  • Client transport: the body is the registry's own JSON error, not a proxy page; the same client fetched other cloud endpoints normally.
  • Caller error: the hash match above; and the error text itself disclaims the key and grant.

Why it matters

This is the first end-to-end run of the §17 consumer path with a live key — the run #2417's closing comment noted nobody had performed ("Nobody has run a §17 install end-to-end with a live registry key and watched the five packs land. … If a §17 install turns out not to land them, that is a new defect with a different cause — please file it fresh"). It fails on first contact, at the registry, before any of the landing machinery is exercised. Until resolution recovers, no consumer — including a fresh memex-local registry install — can fetch anything from this registry.

How to reproduce

  1. On memex.meshweaver.cloud: Settings ▸ Security ▸ Instances → register an instance (self-service); copy the mwi_ key.
  2. curl -H "Authorization: Bearer mwi_…" https://memex.meshweaver.cloud/api/plugins/bundles/index.json
  3. Observe 503 with the body above, on every attempt.

What I did not do

  • Nothing was mutated on the cloud beyond the self-service registrations themselves: sglauser-local (its key was lost to a copy error — may be revoked) and sglauser-local-3 (in use). No restarts, no repairs, no writes.
  • Did not diagnose server-side — no access to the deployment's logs; the storm-breaker/point-read shape is an inference from the symptom pattern, not a verified cause.
  • Note the deployment predates a26219457 (plan-on-the-record), so the failing read set is the pre-plan one: index node → instance node → grant node.

Activity

  1. sierragolflima
    added
    bugSomething isn't working
    on Aug 31, 2026

  2. rbuergi commented on Aug 31, 2026

    @rbuergi
    rbuergi
    on Aug 31, 2026
    Contributor

    RCA, measured on the live portal (2026-08-31 ~20:0xZ)

    Every 503 is one exception, 36 times in the last 2 h:

    Token validation via ApiToken/4d645d0cb14c faulted — validation UNAVAILABLE (retryable)
MeshWeaver.Messaging.DeliveryFailureException: No node found at 'ApiToken/4d645d0cb14c'.
Closest ancestor is 'ApiToken' (remainder='4d645d0cb14c').

    The key's INDEX NODE does not exist. The design is two nodes per key (ApiTokenViaCreateNodeTest documents it): the user-scoped token ({user}/ApiToken/{hashPrefix}) plus the O(1) lookup index (ApiToken/{hashPrefix}). For this key NEITHER exists anywhere searchable (path:*4d645d0c* → 0 on the whole mesh), while a neighbouring token (rbuergi/ApiToken/748c9264…) exists and receives deliveries — so the validation PATH is healthy; the DATA for this key never landed.

    The key was minted in exactly the window where this repo measured two write-lane defect classes live in production (#2899: writes acknowledged + versioned + invisible; the stream-reply loss behind #2901): a mint whose index write was dropped produces a key that verifies nowhere, forever, with a "retryable" 503 that never becomes true. Suspect surface: the mint/rotation changes of the last 3 days (#2802 AdoptKeyHash rotation, #2811 licence-on-fetch, #2814 consent registration) — whichever path minted this key must be checked for (a) writing BOTH nodes, (b) verifying the write landed (read-back, not acknowledgement — the #2899 lesson).

    Operational impact right now: SocialMedia's REQUIRED test-repos check fails on this key on every run — it is the last blocker in the container-build drain.

    Repair: re-mint through the current flow and read BOTH nodes back before handing the key out; then rotate the consuming repos' REGISTRY_KEY secret. The durable fix is mint-time read-back verification.

  3. rbuergi commented on Aug 31, 2026

    @rbuergi
    rbuergi
    on Aug 31, 2026
    Contributor

    The mint path reproduces the half-mint LIVE — one-minute repro, 2026-08-31 ~20:2xZ

    Following the documented two-node mint shape (ApiTokenViaCreateNodeTest) with a fresh probe token on memex.meshweaver.cloud:

    1. create rbuergi/ApiToken/4f4d0501f6a3 → Created (acknowledged)
    2. create ApiToken/4f4d0501f6a3 (the index) → Created (acknowledged)
    3. Bearer call with the probe token → 401 (treated as invalid)
    4. read-back ApiToken/4f4d0501f6a3 → Not found
    5. read-back rbuergi/ApiToken/4f4d0501f6a3 → Unavailable — no verdict within 10 s (the A freshly uploaded content file 503s for minutes (10s timeout per request) before it starts serving — /api/content settle window #2901 per-request flap, verbatim)

    So the mint machinery cannot currently produce a consistent key on this portal: acknowledgments do not imply landings (#2899's class, now demonstrated in the ApiToken partition itself), and reads flap (#2901). Re-minting is NOT a usable repair until the write/read lanes are fixed — any freshly minted key has coin-toss integrity. This blocks SocialMedia's required test-repos check and therefore the last leg of the container-build drain.

    Priority order that unblocks everything: the reply-loss fix (#2901 → the streams-retirement program) and read-back-verified multi-node writes (#2899), then one clean re-mint + secret rotation. Probe leftovers are labeled mint-path-probe, safe to revoke/delete wherever their halves actually landed.

  4. rbuergi commented on Sep 1, 2026

    @rbuergi
    rbuergi
    on Sep 1, 2026
    Contributor

    Re-measured 2026-09-01 (~14:1xZ) — the 503 is permanent, and the cause is measurable from the MCP surface

    The reporter's inference ("mint-time writes are not read-back-verified") is confirmed, and the failing read is now named exactly. The instance node is durably versioned and invisible to every read seam.

    Seam Query Answer
    version store get_versions sglauser/MeshWeaverInstance/sglauser-local-3 v1, 2026-08-31T15:44:49Z, system-security, MeshWeaverInstance, name "sglauser local 3"
    point read get @sglauser/MeshWeaverInstance/sglauser-local-3 Not found
    query index (partition) search namespace:sglauser scope:descendants 3 nodes — all Markdown from April; no instance
    query index (subtree) search namespace:sglauser/MeshWeaverInstance scope:descendants 0
    query index (by type) search nodeType:MeshWeaverInstance 22 — every one under rbuergi/…, none under sglauser
    children listing get @sglauser/* the same 3 Markdown nodes
    sibling write get @Admin/_PluginGrant/sglauser-local-3 readable — v1, Active, {"source":"Plugins","packageId":"*"}, written by system-security 3 s later

    So the registration wrote two nodes. The one into Admin landed visibly. The one into the sglauser user partition landed durably but unreadably — and it is the one instance-key resolution has to read. InstanceAuthResponses' unavailable path is answering honestly: the read reaches no verdict. What is wrong is the word temporarily. Nothing will change on retry, because the node will never be readable.

    Ruled out (measured, not inferred)

    🚨 This is the same defect as #2899, and it falsifies #2899's stated root cause

    #2899 attributes "acknowledged, versioned, invisible" to install-materialization shadowing the durable layer for a Store-plugin partition. sglauser is a plain user partition: no Store/Plugin root, no installPaths, no install at all. Same outcome. The mechanism is therefore not plugin-specific.

    (#2899 re-measured on systemorph the same hour and unchanged: get_versions AgenticEngineering/Introduction → v1 with state: Active and full content via get_version; get @AgenticEngineering/Introduction → Not found; search namespace:AgenticEngineering scope:descendants → 2.)

    The ask this settles

    Registration must read the instance node back through the seam the resolver will use before handing the key to the user, and fail loudly when that read comes back empty.

    A registration the user can see fail and retry is strictly better than a credential that 503s forever with a message telling them to retry. The same rule covers the API-token index and the install-ledger entry.

    The remaining unknown — why the node row is unreachable while its version row is not — needs the partition schemas inspected on the portal; it cannot be settled from MCP. Both instances are still live and reproduce on demand, so the evidence has not decayed.

    Nothing was mutated. All queries above are read-only.

  5. rbuergi commented on Sep 1, 2026

    @rbuergi
    rbuergi
    on Sep 1, 2026
    Contributor

    The creation window was still in Loki — and it names a mechanism the read seams could not

    The 14:23Z comment closed with the one thing it could not settle: "why the node row is unreachable while its version row is not — needs the partition schemas inspected on the portal; it cannot be settled from MCP." I now have cluster access. The Postgres half is still unobtainable read-only (see below), but the log half turned out to be the more informative one, and the registration is inside Loki's 31 d retention.

    What the portal logged at the moment of registration

    15:44:49Z  s9sr9  Node created at sglauser/MeshWeaverInstance/sglauser-local-3 by system-security
15:44:49Z  s9sr9  [ROUTE] NotFound: No node found at 'Admin/_PluginGrant/sglauser-local-3'…
15:44:49Z  s9sr9  Node created at Admin/_PluginGrant/sglauser-local-3 by system-security

    Both writes were made, in the same second, on the same pod. So this is not a half-mint in the sense of one write never being issued — the registration did what it was supposed to do.

    What failed, 19 minutes later

    16:04:14Z  bwgqp  Response did not arrive on time in '00:00:30' for message:
                  'Request [S10.244.2.211:11111 messagehub/sglauser/MeshWeaverInstance/sglauser-local-3]
                        -> [S10.244.4.53:11111 sys.svc.dir.mem/10.244.4.53:11111@…]
                  Orleans.GrainDirectory.IDhtGrainDirectory…'
16:04:15Z  bwgqp  Failed to address message … -> [ messagehub/sglauser/MeshWeaverInstance/sglauser-local-3]
                  at Orleans.Runtime.ActivationData.ActivateAsync(…)

    The node's per-node hub could not activate, because the call it makes to the Orleans grain directory (sys.svc.dir.mem / IDhtGrainDirectory) timed out. That is what GetMeshNode(...) timed out after 10.0s is downstream of, and therefore what every Instance-key resolution UNAVAILABLE and every 503 in this issue is downstream of.

    🚨 This is the same degradation as #2906, in the same window, on the same silos

    Three details line up and I do not think it is coincidence:
    the registration ran on pod …-s9sr9
    …-s9sr9 is also the pod that, 2 min 46 s earlier, logged #2906's ClientDirectory 30 s timeout 15:42:03Z
    the directory silo timing out here (S10.244.4.53) is the same silo that was the publisher in #2906's failing message
    a VMSS node-pool operation on aks-silos-10171001-vmss Started 15:28:44Z → Succeeded 15:40:29Z
    the target silo in #2906 was scheduler-stalled, ramping to 439 long-turn warnings / 10 min at 15:40

    This credential was minted into a four-minute window in which the cluster's Orleans directory subsystem was demonstrably not answering. #2906 and this issue were filed as unrelated; they are two symptoms of one degradation.

    What this does and does not change

    Does not change: the outcome, or the ask. The node is still unreadable 28 h later, the key still 503s forever, and the fix this issue converged on — registration must read the instance node back through the seam the resolver will use, and fail loudly — is strengthened, not weakened. A registration performed during a directory brown-out is exactly the case a read-back catches and an acknowledgement does not.

    Does change: the mechanism to look for. The 14:23Z framing is "the write landed durably but unreadably". The logs say the write landed and then the hub that owns the node could not be activated. Those are different repairs — the first needs the row/index reconciled, the second needs nothing repaired at all once the directory is healthy. I want to be careful not to overclaim: I have not shown these are the same thing, and the 14:23Z measurement (version row present, both read seams absent, 17 h later) is real and is not explained by a transient directory fault. Both can be true — a directory brown-out during the write, and a row that never became visible.

    One methodological caveat that bears on the earlier evidence: a parallel read sweep on memex today found that Not found is returned for nodes that resolve on the very next call — three of four Hosting/Skill/* nodes read Not found on attempt 1 and succeeded on attempt 2. So on this portal a single point-read negative is a weaker signal than it appears. The get_versions-vs-search disagreement in the 14:23Z comment is unaffected (the query index is a separate seam and was checked repeatedly), but any future "the point read says absent" should be stated with an attempt count.

    Not obtainable read-only

    The partition-schema question stands unanswered. Postgres is not in-cluster (kubectl get pods -A finds no postgres/pgbouncer; the servers are memexaks-pg and memexpostgres-… as Azure flexible servers), so reading sglauser.mesh_nodes means obtaining a connection string from a live Deployment's env or Key Vault. I deliberately did not: reading a secret's value to audit it is how the audit becomes the leak, and no measurement here was worth it. Reporting it as unobtainable rather than as absent.

    Verdict: keep — still 100% reproducible, and now with a named upstream

    Nothing measured today argues for closing. What would close it: the mint path performing a read-back through the resolver's own seam and failing loudly, per this issue's own ask — plus, independently, the directory degradation tracked on #2938 (which is live and worse today).

    (Read-only: the Loki query API and kubectl get. Nothing mutated, no secret read.)

  6. rbuergi commented on Sep 2, 2026

    @rbuergi
    rbuergi
    on Sep 2, 2026
    Contributor

    Same cause as #2938. PR #3034 is open (CI running, not merged); it closes both.

    Your diagnosis was client-side only — you said so explicitly — so here is the server-side half, measured on memex-cloud.

    Instance-key resolution is a GetMeshNode point read: a GetDataRequest to the MeshWeaverInstance/<id> per-node hub, whose reply has to travel back to the requester's pod hub. #2938's logs show that pairing in one window: Instance-key resolution UNAVAILABLE … Reader: Hub portal/nodeops-<A> on the requesting pod, and Directed delivery to pod hub 'portal/nodeops-<A>' was refused on the pod that owns the instance hub. The request arrives; the reply cannot get back.

    The reason it cannot get back: a pod-hub claim publishes an address→silo mapping into Orleans' grain directory, that directory is re-partitioned on every cluster membership change, and the claim was asserted exactly once. A mapping lost in that reshuffle was never re-made — and [PreferLocalPlacement] then guarantees that every subsequent delivery creates a throw-away activation on the caller's silo and refuses there, forever, while the refusal is NACKed to the message's sender and never to the owner.

    Your own evidence discriminates for this and against a key/grant problem. You reported that point reads reached no verdict while listing queries (search) answered normally throughout. That is exactly what this mechanism predicts: a listing is served on the pod that took the request, while a point read needs a cross-silo reply to come back to that pod's hub. It also explains the two properties you ruled out by hand — it is not warm-grain persistence (nothing decays; the mapping is simply absent) and it is not key-specific (your unauthenticated get Admin/Coupons/STAFF failed the same way).

    The endpoint's own behaviour was correct the whole time: 503 + Retry-After, caching nothing, and explicitly refusing to infer anything about your key. The defect was upstream of it.

    The fix re-asserts the claim on every cluster membership change — the event that can invalidate it — which is the move Orleans' own ClientDirectory already makes with its client routing table. Full RCA committed as Doc/Architecture/PodHubClaimReassertion.

    One thing to re-check once #3034 is deployed, since it would falsify the link if it failed: register an instance, present its key to /api/plugins/bundles/index.json, and fetch a blob — repeatedly, so it lands on several replicas. A single probe cannot falsify a per-replica fault, which is the trap the earlier "fast 401 = all clear" reading fell into.

  7. rbuergi
    closed this as completedin #3034on Sep 2, 2026
  8. rbuergi
    added a commit that references this issue on Sep 2, 2026

    docs(#2901): the /api/content 503 is the pod-hub claim loss, measured…

    d2cd673
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions