Skip to content

StreamConfiguration<T>.GetHashCode recurses through SynchronizationStream<T>, corrupting UserIdentityCache lookups #2163

Closed

Description

@meshweaver-cloud

What is failing

MeshWeaver.Data.Serialization.StreamConfiguration<T>.GetHashCode() and MeshWeaver.Data.Serialization.SynchronizationStream<T>.GetHashCode() call into each other via the default ObjectEqualityComparer<T> — the stack trace shows the cycle ObjectEqualityComparer<T>.GetHashCode → StreamConfiguration<T>.GetHashCode → SynchronizationStream<T>.GetHashCode → ObjectEqualityComparer<T>.GetHashCode → … repeating without a base case. The failure surfaced in MeshWeaver.Blazor.Infrastructure.UserIdentityCache.As<User>, where the cache returned a NodeTypeDefinition under a key it thought identified a User ("…is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible").

Probable cause

High confidence on the mechanism, medium confidence on which of the two GetHashCode overrides is the wrong one. StreamConfiguration<T> almost certainly composes a hash that includes its owning SynchronizationStream<T> (or vice-versa) and delegates through EqualityComparer<T>.Default.GetHashCode(field) — where T closes the loop. The UserIdentityCache message is the downstream symptom: with a broken GetHashCode, distinct stream/configuration instances collide in the cache's dictionary and As<T> finds an entry of the wrong CLR type, which it then reports as "not convertible". Only the log line was captured (no exceptionType), so the recursion was caught by the logging call before it ran the pod out of stack — but the next hit could StackOverflow the process.

Impact

Low volume so far: 1 occurrence, one pod (memex-portal-deployment-7cfbf545d9-gtlhr), namespace memex, seen once at 2026-08-24T17:48:16Z. But the defect is in a generic hash contract used by the data-synchronization layer and the Blazor identity cache — any traffic pattern that puts two StreamConfiguration<T> / SynchronizationStream<T> instances into the same dictionary can either return the wrong-typed value silently or overflow the stack and kill the pod. Treat as latent-severe even though the counter is small.

Where to look

  • MeshWeaver.Data.Serialization.StreamConfiguration<T>.GetHashCode() — top frame; check what fields it hashes and whether any of them route back to the owning SynchronizationStream<T>.
  • MeshWeaver.Data.Serialization.SynchronizationStream<T>.GetHashCode() — the other half of the cycle; if this hashes its Configuration (a StreamConfiguration<T>), that's the loop.
  • Fix is almost certainly to remove the back-reference from one side of the hash (hash by a stable identity — id/name — not by the paired object), or to override neither and fall back to reference equality.
  • Downstream: MeshWeaver.Blazor.Infrastructure.UserIdentityCache.As<T> — its "not convertible" branch is doing the right thing (it noticed the type mismatch instead of casting blindly); no change needed there, but a defensive log at insert-time that flags a key collision would have shortened this triage.

Evidence

Fingerprint 6e643370636ed34a
Category MeshWeaver.Blazor.Infrastructure.UserIdentityCache
Severity Error
Top frame MeshWeaver.Data.Serialization.StreamConfiguration1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()`
Namespace memex
Pods memex-portal-deployment-7cfbf545d9-gtlhr
Occurrences 2
First seen 2026-08-24 17:48:16Z
Last seen 2026-08-24 17:48:16Z
Recent log lines
2026-08-24 17:48:16Z memex-portal-deployment-7cfbf545d9-gtlhr fail: MeshWeaver.Blazor.Infrastructure.UserIdentityCache[0]
      As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
   at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
   at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
   at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
   at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, …[truncated]
2026-08-24 17:48:16Z memex-portal-deployment-7cfbf545d9-gtlhr fail: MeshWeaver.Blazor.Infrastructure.UserIdentityCache[0]
      As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
   at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
   at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
   at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
   at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
   at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, …[truncated]

Opened automatically from Admin/_LogIncident/6e643370636ed34a. Recurrences are folded into this issue rather than opening new ones.

Activity

  1. systemorph commented on Sep 17, 2026

    @systemorph
    Contributor

    Reopened — this fault is happening again.

    Still occurring: 3 more since the last update (5 total).

    • Last seen: 2026-08-25 01:14:17Z
    • Pods: memex-portal-deployment-7cfbf545d9-gtlhr, memex-portal-deployment-7cfbf545d9-jk5qc, memex-portal-deployment-58c7bb4b9d-97pjf
    Recent log lines
    2026-08-24 18:09:17Z memex-portal-deployment-7cfbf545d9-jk5qc fail: MeshWeaver.Blazor.Infrastructure.UserIdentityCache[0]
          As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
       at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
       at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
       at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
       at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, …[truncated]
    2026-08-24 18:09:17Z memex-portal-deployment-7cfbf545d9-jk5qc fail: MeshWeaver.Blazor.Infrastructure.UserIdentityCache[0]
          As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
       at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
       at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
       at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
       at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, …[truncated]
    2026-08-25 01:14:17Z memex-portal-deployment-58c7bb4b9d-97pjf fail: MeshWeaver.Blazor.Infrastructure.UserIdentityCache[0]
          As<User> for User: value is MeshWeaver.Graph.Configuration.NodeTypeDefinition (MeshWeaver.Graph), not convertible
       at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
       at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
       at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
       at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
       at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, …[truncated]
    
  2. rbuergi commented on Sep 17, 2026

    @rbuergi
    Contributor

    Re-closed — the reopen above was on stale evidence (#4577).

    The recurrence bot reopened this at 2026-09-17T07:51:17Z under the headline "this fault is happening again", but the occurrence its own comment cites is OLDER than the close it reverted:

    the bot's Last seen: the close it reverted
    2026-08-25 01:14:17Z 2026-08-25T18:39:21Z

    So the counter moved, not the clock — the fault's newest occurrence predates the close, which is the one thing "still happening" has to mean. Root cause and fix: LogIncidentFiler.Reopen reopened on the sole condition that the issue was closed, while the comment-debt clock pays a final count for incidents whose fault has stopped (the fix, Systemorph/MeshWeaver.Plugins#2014, is written but NOT yet merged — it makes a reopen require incident.LastSeen > issue.ClosedAt).

    This restores the close exactly as it stood; nothing is asserted about the defect itself. If the fault happens again, the next occurrence postdates this close and reopens the issue on evidence that means what it says.

    One of 53 issues whose reopen was measured stale (129 bot reopens on 2026-09-17, 125 of them inside the 07:51-07:53Z sweep; 76 postdate their close and were left open). Measured and re-closed 2026-09-17 by Claude Opus 5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions