Repository navigation
GH-3954: stop assigning durability agents to nodes that cannot run them - #3963
Merged
Merged
Conversation
A node started with Durability.DurabilityAgentEnabled = false never registers the durability agent family, so it threw "Unrecognized agent scheme 'wolverinedb'" the moment the leader handed it one. The leader re-issued the identical assignment every five minutes forever, no durability agent ran anywhere for that store, and owner_id = 0 outgoing envelopes were never recovered -- silently, with every queue table reading zero. The reporter found a nine day backlog this way. The issue's premise turned out to be wrong, and following it would have made things worse. It states that the incapable node's Capabilities correctly omit the scheme, so "the leader has the data". It does not. WolverineNode.Capabilities is populated only from _agentFamilies.Values.OfType<IStaticAgentFamily>(), and MessageStoreCollection is an IAgentFamily and NOT an IStaticAgentFamily -- so no wolverinedb:// Uri has ever appeared in Capabilities on ANY node, capable or not. The suggested per-agent match (Node.Capabilities.Contains(agent.Uri)) would therefore have found zero capable nodes everywhere and stranded the durability agent cluster-wide. Per-agent matching is wrong here for a second reason: Capabilities is a startup snapshot, while this family's agent list GROWS at runtime as tenant databases are added, so per-agent matching would strand every later-added tenant's agent. Whether a node can run these agents is genuinely a per-NODE property -- the setting turns the whole family on or off -- so that is what gets published: MessageStoreCollection.DurabilityCapabilityUri, written into Capabilities at startup when the family is actually registered, and consulted by a new capability-aware overload of DistributeEvenlyWithAffinity. Only the two store/durability paths call that method, so the change is scoped to this family exactly as intended. Deliberately NOT gated on AllNodesHaveSameCapabilities, which returns true trivially both when there is one node (_nodes.Skip(1) is empty) and when every node is equally incapable -- precisely the two configurations reported, so a fix modelled on the blue/green path would have changed neither. Also handled: the GH-3785 affinity preference follows another family's placements and may name a node that cannot host durability agents. An incapable preference is now discarded rather than honoured; co-location is an optimisation, capability is not. When no node in the cluster is capable, nothing is assigned and a warning names the condition and the setting. The original failure was silent in both directions, so leaving the agents unassigned without saying why would only trade one quiet failure for another. Verified: 9 new AssignmentGrid tests, confirmed to fail 5/9 against the ungated overload and pass 9/9 with the gate. Plus two end-to-end tests asserting a normal Balanced Postgres host publishes the marker AND still gets its durability agent -- the regression that would matter most if the marker ever stopped being published. Existing durability_modes suite still 7/7. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JG8Un6iNeyXECKJk3jo5uC
_runtime.Stores is null on a runtime with no message store, so reaching through it to get the scheme threw a NullReferenceException at startup. The setting is the same condition -- the constructor registers the family if and only if the flag is set -- and it is null safe. Caught by released_agent_failover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JG8Un6iNeyXECKJk3jo5uC
Member
Author
|
Regression sweep after the change (including the
Plus the new tests: 9 |
This was referenced Aug 16, 2026
This was referenced Aug 24, 2026
Closed
This was referenced Aug 28, 2026
This was referenced Sep 10, 2026
This was referenced Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3954.
A node started with
Durability.DurabilityAgentEnabled = falsenever registers the durability agent family, so it threwUnrecognized agent scheme 'wolverinedb'the moment the leader handed it one. The leader re-issued the identical assignment every five minutes forever, no durability agent ran anywhere for that store, andowner_id = 0outgoing envelopes were never recovered — silently, with every queue table reading zero. The reporter found a nine-day backlog this way.The issue states that the incapable node's capabilities correctly omit the scheme, so "the leader has the data to avoid the node."
It does not.
WolverineNode.Capabilitiesis populated only from_agentFamilies.Values.OfType<IStaticAgentFamily>()(NodeAgentController.StartLocalProcessing.cs:10), andMessageStoreCollectionis declared: IAgentFamily, IAsyncDisposable— notIStaticAgentFamily. There are exactly two sites that add toCapabilities, both fed by that same filter.So no
wolverinedb://Uri has ever appeared inCapabilitieson any node, capable or not. The suggested fix — per-agentNode.Capabilities.Contains(agent.Uri)— would have found zero capable nodes everywhere and stranded the durability agent cluster-wide.Per-agent matching is wrong here for a second, independent reason:
Capabilitiesis a startup snapshot, while this family's agent list grows at runtime as tenant databases are added. Per-agent matching would strand every later-added tenant database's durability agent.What this does instead
Whether a node can run these agents is genuinely a per-node property — the setting turns the whole family on or off — so that is what gets published:
MessageStoreCollection.DurabilityCapabilityUri, written intoCapabilitiesat startup when the family is actually registeredDistributeEvenlyWithAffinityOnly
MessageStoreCollectionandMultiTenantedMessageDatabasecall that method, so the change is scoped to the store/durability family exactly as intended —DistributeEvenlyWithBlueGreenSemanticsandDistributeByGroupAffinityare untouched.It is deliberately not gated on
AllNodesHaveSameCapabilities, which returns true trivially both when there is a single node (_nodes.Skip(1)is empty) and when every node is equally incapable — precisely the two configurations in the report. A fix modelled on the blue/green path would have changed neither.Two more cases handled:
Verification
AssignmentGridtests. Confirmed 5/9 fail against the ungated overload and 9/9 pass with the gate, so the gate is demonstrably what's doing the work (a plain stash can't serve as the red baseline here, since the tests reference new API).durability_modessuite still 7/7, including its Balanced-mode running-agent count.🤖 Generated with Claude Code
https://claude.ai/code/session_01JG8Un6iNeyXECKJk3jo5uC