Fixed #1131 - Race condition between SuspendNotifications() and .ResumeNotifications()#1132
Open
JakenVeina wants to merge 3 commits into
Open
Fixed #1131 - Race condition between SuspendNotifications() and .ResumeNotifications()#1132JakenVeina wants to merge 3 commits into
SuspendNotifications() and .ResumeNotifications()#1132JakenVeina wants to merge 3 commits into
Conversation
…synchronize updates to internal state for tracking notification suspensions, but `.ResumeNotifications()` did not use any synchronization when updating that same state, allowing for corruption of state if the two methods happen to overlap, concurrently. Also fixed that the test for this behavior was intermittent, specifically that it tended to falsely pass when running in parallel with many other tests. The test has now been moved to its own `IntegrationTests` fixture, to guarantee it is never parallelized with other tests. Resolves #1131.
Member
|
The lock that you're using is the same lock the DQ uses. However, doing the Resume while holding the lock will cause the changes to be emitted while holding the lock. Let me see if I can suggest an alternative. |
…thout holding it ResumeNotifications() was wrapped in lock(_locker), which via lock reentrancy held the cache lock across the DeliveryQueue drain, delivering accumulated changes to subscribers while the lock was held. Restore off-lock delivery and close the suspend/resume state-divergence race a different way: - SuspendNotifications() no longer wraps the resume callback in the cache lock; queued changes drain outside the lock, as the DeliveryQueue intends. - ResumeNotifications() re-checks the suspend count under the lock before emitting the resume signal, so a concurrent SuspendNotifications() in the decrement/emit window wins and the count and the subject cannot diverge. - SuspensionTracker.SuspendNotifications() gates the suspended signal on the subject's own value rather than the count, keeping it monotonic while a resume's signal is still in flight. Tests: - Add StaleResumeSignalIsSuppressedByConcurrentReSuspend: deterministic proof that a connection made during a racing re-suspend does not activate on a stale resume signal (fails without the count re-check). - Add ResumeDeliversPendingChangesWithoutHoldingTheLock: proves a blocked subscriber does not stall a concurrent lock-requiring operation during resume delivery. - Remove ResumeSignalUnderLockPreventsStaleSnapshotFromReSuspend: redundant coverage that only passed by timing out a deadlock.
JakenVeina
commented
Jul 11, 2026
| public sealed class IntegrationTests | ||
| : IntegrationTestFixtureBase | ||
| { | ||
| [Fact] |
Collaborator
Author
There was a problem hiding this comment.
Any particular reason to move this?
| { | ||
| using var readLock = _notifications.AcquireReadLock(); | ||
| _suspensionTracker.Value.EmitResumeNotification(); | ||
| if (!_suspensionTracker.Value.AreNotificationsSuspended) |
Collaborator
Author
There was a problem hiding this comment.
So, we're basically saying "state corruption is cool, if we just guard against the impact"? I don't think this is sound the way it's currently implemented. In particular, .SuspendNotifications() does not acquire any locks from _notifications, so even with this lock here, it's possible that a concurrent .SuspendNotifications() can swap _suspensionTracker.Value.AreNotificationsSuspended from false to true in the time between this check, and the call to .EmitResumeNotification().
| // separate steps, so the count can briefly reach zero before the 'false' signal | ||
| // is emitted. Gating on the subject keeps the signal monotonic and avoids a | ||
| // redundant 'true' when a suspend interleaves in that window. | ||
| if (!_areNotificationsSuspended.IsDisposed && !_areNotificationsSuspended.Value) |
Collaborator
Author
There was a problem hiding this comment.
Would Interlocked.Increment() and Interlocked.Decrement() not make more sense for tracking _notifySuspendCount?
Otherwise, does _notifySuspendCount even have a purpose anymore?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixed that
ObservableCache.SuspendNotifications()uses alockto synchronize updates to internal state for tracking notification suspensions, but.ResumeNotifications()did not use any synchronization when updating that same state, allowing for corruption of state if the two methods happen to overlap, concurrently.Also fixed that the test for this behavior was intermittent, specifically that it tended to falsely pass when running in parallel with many other tests. The test has now been moved to its own
IntegrationTestsfixture, to guarantee it is never parallelized with other tests.Resolves #1131.
@dwcullop I suspect this may not be a proper fix, in the context of
DeliveryQueue<>. I.E. this solution involves using a basiclock()when perhaps the proper solution should be using the locking mechanisms uponDeliveryQueue<>. But I don't have the familiarity with it. For now, I just duplicated the existinglock(_locker)upon.SuspendNotifications()over to.ResumeNotifications(). If you'd rather take a crack at this yourself instead, go for it.The test changes we should definitely keep, though.