A storage stall told a signed-in user their account did not exist by rbuergi · Pull Request #970 · Systemorph/MeshWeaver

rbuergi · 2026-08-08T21:23:03Z

Addresses #637 — the half that PR #947 did not reach.

What #947 already fixed (verified on current `main`, not rebuilt here)

PR #947 landed this morning and fixes the API-token leg end-to-end: a tri-state
TokenValidationStatus (Valid / Invalid / Unavailable) at the ApiTokenService seam,
ApiTokenAuthenticationHandler answering 503 + Retry-After via a HandleChallengeAsync
override, plus the UserContextMiddleware bearer branch, SignalRConnectionRegistry,
GrpcConnectionRegistry, ApiTokenNodeType's thrown-fault branch, ValidateTokenResponse.IsUnavailable
and TokenValidationUnavailableException. I re-read each of those on main before writing anything;
none of it is duplicated here. This PR reuses that work — it shares #947's RetryAfterSeconds
constant and routes into its existing 503 challenge path.

What was still broken

The identical collapse survived on the identity / authorization leg — which is what a browser
user experiences, and is arguably worse than the 401:

OnboardingMiddleware.FindUserByEmail fell back to null on timeout. null means "not
onboarded", so a storage stall redirected a fully signed-in user to the sign-up form. The old
log line said it out loud: "falling back to null (will redirect to /onboarding)". Signing in
again cannot fix a storage stall — and the user is being invited to create a second account.
OnboardingMiddleware.LoadUserRoles defaulted to an empty role set on timeout or fault,
indistinguishable from "this user genuinely has no grants". A blip silently stripped a user's
privileges and every screen they opened answered "Access denied" — an availability failure
reported as an authorization failure.
ApiTokenAuthenticationHandler wrapped its role enrichment in a bare catch { }, so the same
stall authenticated an API caller with a silently reduced role set. Every later request was then
refused misleadingly.

One mistake made twice: "we could not find out" and "we found out, and the answer is no" were
literally the same value, so no caller could branch on it.

The fix — the distinction lives where the timeout is known

IdentityReadOutcome<T> is either Resolved (the answer, possibly a definitive null/empty) or
Unavailable (no answer was reached). IdentityRead.Bounded is the single place a budget is
applied to an identity read, so it is the single place that knows a read gave up. Nothing sniffs an
exception message — a text classifier would drift the moment someone rewords a banner.

OnboardingMiddleware answers 503 + Retry-After for Unavailable, with localized plain-text
copy that says the sign-in survived and that re-authenticating will not help. A resolved verdict
is untouched: an unknown user still goes to /onboarding, a user with no grants still gets an empty
role set.
ApiTokenAuthenticationHandler routes an unavailable role read into A degraded silo answered 401 — indistinguishable from a forged token #947's existing 503 challenge
rather than authenticating an under-privileged principal. Sharing RetryAfterSeconds closes one of
the drift gaps A degraded silo answered 401 — indistinguishable from a forged token #947 listed as an honest gap.
Keys added to both strings.en.json and strings.de.json, resolved through
AccessService.Localize off AccessContext.Locale — never ambient CultureInfo.

Explicitly not the fix: no widened timeout, no retry or watchdog to make the stall rarer, no
catch-and-default. The 20 s budget is unchanged; only what it reports changed.

Tests — and proof they are not vacuous

test/MeshWeaver.Auth.Test/IdentityUnavailableNotDeniedTests.cs, 8 tests, all green (full project
139/139; LocalizationTest 33/33).

They assert the classification under an induced stall (Observable.Never / Observable.Throw —
deterministic, no sleeps), not merely that some error appears. I verified the teeth directly: reverting
Bounded's timeout and fault legs to the old Resolved(null) turns
StalledRead_IsUnavailable_NotADefinitiveNegative and FaultedRead_IsUnavailable_NotADefinitiveNegative
red, and the other six stay green — correctly, since those pin the definitive legs and the response
contract, which that revert does not change.

The definitive legs run against a real mesh (MonolithMeshTestBase): an unknown email must resolve
to a definitive absence and a grant-less user to an empty role set, so the fix cannot over-reach into
"everything is retryable".

Honest gaps

The pipeline→503 mapping and the response shape are forward guards, not before/after proofs.
IdentityUnavailable_Is503WithRetryAfter_NeverARedirectOr401 drives the real response writer over a
real HttpContext (asserting 503, Retry-After, and no Location header — the redirect is the
defect), but it calls a method that did not exist before, so it could not have failed against the old
code. Driving OnboardingMiddleware.InvokeAsync end-to-end would need a stallable workspace query
behind a constructed PortalApplication; I judged that harness disproportionate here.
This is a behaviour change for the browser path: a request that used to bounce to /onboarding
during a storage stall now returns 503. That is the intended contract, but it is not
backwards-compatible for anything that relied on the bounce.
Same for API-token role enrichment: a wedged AccessAssignment query now 503s the request instead
of authenticating with the token's mint-time roles. Deliberate, and consistent with the call A degraded silo answered 401 — indistinguishable from a forged token #947 made
— but a real change. In practice it is rare: workspace.GetQuery caches per user per process, so only
the first resolution can hit it.
Three sites of the same defect class remain, out of scope here and better as their own change:
AccessControlPipeline reports a permission-fold fault as Access denied; MeshOperations.FetchNode
reports a read timeout as Not found; UserIdentityCache treats a cold cache as "user unknown".
Fixing those needs a tri-state through the permission fold and a new ErrorType on the message bus.

Verification

Release + -warnaserror, one project per invocation, all 0 Warning(s) / 0 Error(s):
Memex.Portal.Shared, MeshWeaver.Messaging.Hub, MeshWeaver.Documentation, MeshWeaver.Auth.Test.
Branch merged with current main before push.

🤖 Generated with Claude Code

PR #947 fixed the API-token leg of #637: a validation read timeout answers 503 + Retry-After instead of 401. The identical collapse survived on the identity/authorization leg, where it is what a BROWSER user experiences. OnboardingMiddleware.FindUserByEmail fell back to null on timeout — and null means "not onboarded", so a storage stall redirected a fully signed-in user to the SIGN-UP form. Signing in again cannot fix a storage stall. LoadUserRoles defaulted to an EMPTY role set on timeout or fault, indistinguishable from "this user has no grants", so a blip silently stripped a user's privileges and every screen they opened answered "Access denied". Both are one mistake made twice: "we could not find out" and "we found out, and the answer is no" were the same value, so no caller could branch on it. The distinction now lives at the read, where the timeout is actually known — IdentityReadOutcome<T> is either Resolved (the answer, possibly a definitive null) or Unavailable (no answer). Nothing sniffs an exception message. - OnboardingMiddleware answers 503 + Retry-After with localized, plain-text copy that says the sign-in survived and that re-authenticating will not help. A resolved verdict is untouched: an unknown user still goes to onboarding. - ApiTokenAuthenticationHandler's bare `catch { }` around role enrichment is gone. An unavailable role read takes the SAME retryable 503 path as an unavailable token validation instead of authenticating an under-privileged principal whose every later request is denied misleadingly. It shares #947's RetryAfterSeconds constant, closing one of that PR's noted drift gaps. - Localization keys in strings.en.json AND strings.de.json, resolved off AccessContext.Locale — never ambient CultureInfo. Tests pin the CLASSIFICATION under an induced stall, not merely that an error appears. Verified non-vacuous: reverting Bounded's timeout leg to the old Resolved(null) turns the two defect-pinning tests red. The definitive legs run against a real mesh so the fix cannot over-reach into "everything is retryable". Refs #637 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ailable-not-denied

Copilot

Pull request overview

This PR extends the “Unavailable ≠ invalid” distinction from API-token validation (PR #947) to the browser identity/authorization path, so storage/query stalls no longer present as “account does not exist” or “access denied”. It introduces a two-shape identity-read outcome (Resolved vs Unavailable), maps “Unavailable” to a retryable 503 + Retry-After, and adds localization + tests to pin the classification and HTTP contract.

Changes:

Add IdentityReadOutcome<T> + IdentityRead.Bounded(...) to classify bounded identity reads as either Resolved (including definitive null/empty) or Unavailable (timeout/fault).
Update OnboardingMiddleware to treat identity/role read unavailability as an availability failure (503 + Retry-After) rather than redirecting to onboarding or silently stripping roles.
Add localized user-facing strings and new tests covering classification, HTTP response shape, and definitive “absent” legs against a real mesh.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/MeshWeaver.Auth.Test/IdentityUnavailableNotDeniedTests.cs	Adds tests that pin identity-read classification, 503 response shape (incl. `Retry-After`), localization, and definitive negative legs.
src/MeshWeaver.Messaging.Hub/Localization/strings.en.json	Adds English localization keys for the identity-unavailable 503 body.
src/MeshWeaver.Messaging.Hub/Localization/strings.de.json	Adds German localization keys for the identity-unavailable 503 body.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-08-identity-unavailable-not-denied.md	Documents the behavior change and the new “unavailable vs negative verdict” distinction.
memex/Memex.Portal.Shared/Authentication/UserRoleResolver.cs	Changes role resolution to return an `IdentityReadOutcome` so callers can distinguish empty roles from unavailable reads.
memex/Memex.Portal.Shared/Authentication/OnboardingMiddleware.cs	Implements the new Unavailable outcome and writes a localized 503 + `Retry-After` instead of redirecting/denying.
memex/Memex.Portal.Shared/Authentication/IdentityReadOutcome.cs	Introduces the new outcome type and the single bounded-read classifier (`IdentityRead.Bounded`).
memex/Memex.Portal.Shared/Authentication/ApiTokenAuthenticationHandler.cs	Routes unavailable DB role enrichment into the existing retryable 503 challenge path instead of authenticating a reduced-role principal.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-08-08T21:39:51Z