Repository navigation
Infinite recursion between StreamConfiguration<T>.GetHashCode() and SynchronizationStream<T>.GetHashCode() #2172
Description
What is failing
MeshWeaver.Data.Serialization.StreamConfiguration<T>.GetHashCode() and MeshWeaver.Data.Serialization.SynchronizationStream<T>.GetHashCode() call each other through the default ObjectEqualityComparer<T> and never terminate. Any code path that hashes a stream key blows the stack.
Probable cause
High confidence. The captured stack is a tight, repeating cycle:
ObjectEqualityComparer<T>.GetHashCode(T)
→ StreamConfiguration<T>.GetHashCode()
→ SynchronizationStream<T>.GetHashCode()
→ ObjectEqualityComparer<T>.GetHashCode(T) // back to the top
The shape says each type’s GetHashCode() delegates to a field/property whose own hash resolves via the default comparer back into the other type — most likely StreamConfiguration hashes its owning SynchronizationStream and SynchronizationStream hashes its StreamConfiguration (or one of them falls back to this through the default comparer). Fix is to break the cycle: hash on identity/immutable key fields (Owner, Reference, stream id) rather than delegating to the peer object, or drop one of the two overrides entirely.
Impact
Low so far but load-bearing risk. 4 occurrences over ~4 ms on a single pod (memex-portal-deployment-…-jk5qc) at 18:09:17 UTC — a single request storm, not an ongoing outage. However, StackOverflowException from recursive GetHashCode is unrecoverable and will crash the process the moment a hot path (routing table lookup, dictionary keyed on a stream) hits it under real traffic. Treat as a latent crash bug, not a nuisance.
Where to look
The log category is MeshWeaver.Hosting.Orleans.RoutingGrain — misleading. That’s just where the invocation originates (routing a message to a portal/… hub whose stream key it hashes). The defect is entirely in the MeshWeaver.Data.Serialization assembly:
MeshWeaver.Data.Serialization.StreamConfiguration<T>.GetHashCode()MeshWeaver.Data.Serialization.SynchronizationStream<T>.GetHashCode()
Start from either override and follow what it delegates to — the two ends of the cycle are in these two files.
No prior incident found on this fingerprint or on the StreamConfiguration<T> / SynchronizationStream<T> pair.
Evidence
| Fingerprint | 52c67215dc2ff138 |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Error |
| Top frame | MeshWeaver.Data.Serialization.StreamConfiguration1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()` |
| Namespace | memex |
| Pods | memex-portal-deployment-7cfbf545d9-jk5qc |
| Occurrences | 4 |
| First seen | 2026-08-24 18:09:17Z |
| Last seen | 2026-08-24 18:09:17Z |
Recent log lines
2026-08-24 18:09:17Z memex-portal-deployment-7cfbf545d9-jk5qc fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Cannot deliver the NotFound DeliveryFailure for hV-ypLdXVEesO8W4wBzPIA to sender rsalzmann: that hub has no live subscriber on its stream, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: Stream-routed delivery to 'portal/oBl6_KD2S8YTnpqvjdmjNWGzkoRFKrnLUC6KUNOdA8g' has no live subscriber: no silo in this cluster is currently serving that hub, so the message could not be delivered.
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Ge…[truncated]
2026-08-24 18:09:17Z memex-portal-deployment-7cfbf545d9-jk5qc fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Stream-routed delivery to 'portal/oBl6_KD2S8YTnpqvjdmjNWGzkoRFKrnLUC6KUNOdA8g' has no live subscriber: no silo in this cluster is currently serving that hub, so the message could not be delivered. Message RawJson (ZxYwup-rVE6CJcRB04HOjQ) from rsalzmann was NOT posted — a publish to a subscriber-less stream succeeds and discards, which is why this had to be checked rather than observed. Surfacing DeliveryFailure to the sender.
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at…[truncated]
2026-08-24 18:09:17Z memex-portal-deployment-7cfbf545d9-jk5qc fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Cannot deliver the NotFound DeliveryFailure for zQpV38tzR0eAjMFK8s1nEA to sender rsalzmann: that hub has no live subscriber on its stream, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: Stream-routed delivery to 'portal/oBl6_KD2S8YTnpqvjdmjNWGzkoRFKrnLUC6KUNOdA8g' has no live subscriber: no silo in this cluster is currently serving that hub, so the message could not be delivered.
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
…[truncated]
2026-08-24 18:09:17Z memex-portal-deployment-7cfbf545d9-jk5qc fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Cannot deliver the NotFound DeliveryFailure for XTYlYjYse0aTZUSk1IWHAg to sender rsalzmann: that hub has no live subscriber on its stream, and a NACK has no NACK of its own. The sender will wait out its own request budget — the original failure was: Stream-routed delivery to 'portal/oBl6_KD2S8YTnpqvjdmjNWGzkoRFKrnLUC6KUNOdA8g' has no live subscriber: no silo in this cluster is currently serving that hub, so the message could not be delivered.
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at MeshWeaver.Data.Serialization.SynchronizationStream`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode()
at System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetHashCode(System.__Canon)
at MeshWeaver.Data.Serialization.StreamConfiguration`1[[System.__Canon, System.Private.CoreLib, Version=10.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Ge…[truncated]
Opened automatically from Admin/_LogIncident/52c67215dc2ff138. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Aug 25, 2026
Fixed by #2239. Root cause: these types are
records, so the compiler synthesized structuralGetHashCodeover every field — and the field graph contains a cycle.StreamConfiguration<T>'s primary-ctor propertyStreamis a back-pointer to the owningSynchronizationStream<T>, andSynchronizationStream's own hash walks its privateChangeItem<TStream>? current, descendingChangeItem → EntityStore → InstanceCollection. Both recorded production stacks were entered by hashing a STREAM — cycle 1 fed cycle 2. Fix: a live/identity object gets reference identity (ReferenceEquals+RuntimeHelpers.GetHashCode); the containers hash keys+count, never the arbitrary instance values. Proven red-before/green-after by reverse-applying the patch (cycle 1 crashed the test host with SIGABRT). Also fixed a latentInvalidOperationExceptionfromEntityStore.GetHashCode()'s seedless.Aggregate()on an empty store.systemorph commentedon Sep 17, 2026
Reopened — this fault is happening again.
Still occurring: 2 more since the last update (6 total).
memex-portal-deployment-7cfbf545d9-jk5qcRecent log lines
rbuergi commentedon Sep 17, 2026
Re-closed — the reopen above was on stale evidence (#4577).
The recurrence bot reopened this at
2026-09-17T07:51:24Zunder the headline "this fault is happening again", but the occurrence its own comment cites is OLDER than the close it reverted:Last seen:2026-08-24 18:09:17Z2026-08-25T18:39:43ZSo the counter moved, not the clock — the fault's newest occurrence predates the close, which is the one thing "still happening" has to mean. Root cause and fix:
LogIncidentFiler.Reopenreopened on the sole condition that the issue was closed, while the comment-debt clock pays a final count for incidents whose fault has stopped (the fix, Systemorph/MeshWeaver.Plugins#2014, is written but NOT yet merged — it makes a reopen requireincident.LastSeen > issue.ClosedAt).This restores the close exactly as it stood; nothing is asserted about the defect itself. If the fault happens again, the next occurrence postdates this close and reopens the issue on evidence that means what it says.
One of 53 issues whose reopen was measured stale (129 bot reopens on 2026-09-17, 125 of them inside the 07:51-07:53Z sweep; 76 postdate their close and were left open). Measured and re-closed 2026-09-17 by Claude Opus 5.