Skip to content

GH-3954: stop assigning durability agents to nodes that cannot run them - #3963

Merged
jeremydmiller merged 2 commits into
mainfrom
gh-3954-durability-agent-capability
Aug 16, 2026
Merged

jeremydmiller merged 2 commits into
mainfrom
gh-3954-durability-agent-capability

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #3954.

A node started with Durability.DurabilityAgentEnabled = false never registers the durability agent family, so it threw Unrecognized agent scheme 'wolverinedb' the moment the leader handed it one. The leader re-issued the identical assignment every five minutes forever, no durability agent ran anywhere for that store, and owner_id = 0 outgoing envelopes were never recovered — silently, with every queue table reading zero. The reporter found a nine-day backlog this way.

⚠️ The issue's premise is wrong, and following it would have made things worse

The issue states that the incapable node's capabilities correctly omit the scheme, so "the leader has the data to avoid the node."

It does not. WolverineNode.Capabilities is populated only from _agentFamilies.Values.OfType<IStaticAgentFamily>() (NodeAgentController.StartLocalProcessing.cs:10), and MessageStoreCollection is declared : IAgentFamily, IAsyncDisposable — not IStaticAgentFamily. There are exactly two sites that add to Capabilities, both fed by that same filter.

So no wolverinedb:// Uri has ever appeared in Capabilities on any node, capable or not. The suggested fix — per-agent Node.Capabilities.Contains(agent.Uri) — would have found zero capable nodes everywhere and stranded the durability agent cluster-wide.

Per-agent matching is wrong here for a second, independent reason: Capabilities is a startup snapshot, while this family's agent list grows at runtime as tenant databases are added. Per-agent matching would strand every later-added tenant database's durability agent.

What this does instead

Whether a node can run these agents is genuinely a per-node property — the setting turns the whole family on or off — so that is what gets published:

  • MessageStoreCollection.DurabilityCapabilityUri, written into Capabilities at startup when the family is actually registered
  • consulted by a new capability-aware overload of DistributeEvenlyWithAffinity

Only MessageStoreCollection and MultiTenantedMessageDatabase call that method, so the change is scoped to the store/durability family exactly as intended — DistributeEvenlyWithBlueGreenSemantics and DistributeByGroupAffinity are untouched.

It is deliberately not gated on AllNodesHaveSameCapabilities, which returns true trivially both when there is a single node (_nodes.Skip(1) is empty) and when every node is equally incapable — precisely the two configurations in the report. A fix modelled on the blue/green path would have changed neither.

Two more cases handled:

Verification

  • 9 new AssignmentGrid tests. Confirmed 5/9 fail against the ungated overload and 9/9 pass with the gate, so the gate is demonstrably what's doing the work (a plain stash can't serve as the red baseline here, since the tests reference new API).
  • 2 new end-to-end tests asserting a normal Balanced Postgres host publishes the marker and still gets its durability agent — the regression that would matter most if the marker ever stopped being published.
  • Existing durability_modes suite still 7/7, including its Balanced-mode running-agent count.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JG8Un6iNeyXECKJk3jo5uC

jeremydmiller and others added 2 commits August 16, 2026 12:27
A node started with Durability.DurabilityAgentEnabled = false never registers the
durability agent family, so it threw "Unrecognized agent scheme 'wolverinedb'" the
moment the leader handed it one. The leader re-issued the identical assignment every
five minutes forever, no durability agent ran anywhere for that store, and owner_id = 0
outgoing envelopes were never recovered -- silently, with every queue table reading
zero. The reporter found a nine day backlog this way.

The issue's premise turned out to be wrong, and following it would have made things
worse. It states that the incapable node's Capabilities correctly omit the scheme, so
"the leader has the data". It does not. WolverineNode.Capabilities is populated only
from _agentFamilies.Values.OfType<IStaticAgentFamily>(), and MessageStoreCollection is
an IAgentFamily and NOT an IStaticAgentFamily -- so no wolverinedb:// Uri has ever
appeared in Capabilities on ANY node, capable or not. The suggested per-agent match
(Node.Capabilities.Contains(agent.Uri)) would therefore have found zero capable nodes
everywhere and stranded the durability agent cluster-wide.

Per-agent matching is wrong here for a second reason: Capabilities is a startup
snapshot, while this family's agent list GROWS at runtime as tenant databases are
added, so per-agent matching would strand every later-added tenant's agent.

Whether a node can run these agents is genuinely a per-NODE property -- the setting
turns the whole family on or off -- so that is what gets published:
MessageStoreCollection.DurabilityCapabilityUri, written into Capabilities at startup
when the family is actually registered, and consulted by a new capability-aware
overload of DistributeEvenlyWithAffinity. Only the two store/durability paths call
that method, so the change is scoped to this family exactly as intended.

Deliberately NOT gated on AllNodesHaveSameCapabilities, which returns true trivially
both when there is one node (_nodes.Skip(1) is empty) and when every node is equally
incapable -- precisely the two configurations reported, so a fix modelled on the
blue/green path would have changed neither.

Also handled: the GH-3785 affinity preference follows another family's placements and
may name a node that cannot host durability agents. An incapable preference is now
discarded rather than honoured; co-location is an optimisation, capability is not.

When no node in the cluster is capable, nothing is assigned and a warning names the
condition and the setting. The original failure was silent in both directions, so
leaving the agents unassigned without saying why would only trade one quiet failure
for another.

Verified: 9 new AssignmentGrid tests, confirmed to fail 5/9 against the ungated
overload and pass 9/9 with the gate. Plus two end-to-end tests asserting a normal
Balanced Postgres host publishes the marker AND still gets its durability agent --
the regression that would matter most if the marker ever stopped being published.
Existing durability_modes suite still 7/7.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG8Un6iNeyXECKJk3jo5uC
_runtime.Stores is null on a runtime with no message store, so reaching through it to
get the scheme threw a NullReferenceException at startup. The setting is the same
condition -- the constructor registers the family if and only if the flag is set --
and it is null safe. Caught by released_agent_failover.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG8Un6iNeyXECKJk3jo5uC
@jeremydmiller

Copy link
Copy Markdown
Member Author

Regression sweep after the change (including the released_agent_failover NRE fix in the second commit):

Suite Result
CoreTests (full) 2406 passed, 2 skipped, 0 failed
PersistenceTests (full) 92 / 92
PostgresqlTests multi-tenant durability 3 / 3
MartenTests.Distribution 49 / 49

Plus the new tests: 9 AssignmentGrid unit tests (5/9 confirmed failing against the ungated overload) and 2 end-to-end capability-publication tests.

@jeremydmiller
jeremydmiller merged commit e879f06 into main Aug 16, 2026
38 checks passed
This was referenced Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant