Repository navigation
Description
Problem
Once ControlConnection's reconnection handler is parked on its backoff schedule, the only clock that can bring the control connection back is reconnection_policy's own schedule. No event re-arms it early -- not a host coming back up, not the idle heartbeat. With the defaults (ExponentialReconnectionPolicy(1.0, 600.0), cassandra/cluster.py:774) the retry interval climbs 1s, 2s, 4s ... 512s and then plateaus at 600s, so roughly 17 minutes into an outage the driver is only probing once every 10 minutes. A cluster that becomes reachable one second after a failed attempt keeps its control connection down for up to max_reconnection_delay.
Concretely:
ControlConnection.on_up()ispass(cassandra/cluster.py:4700-4701). A host coming back up never touches the parked handler.ControlConnection.return_connection()(cassandra/cluster.py:4731-4733) is driven by the idle heartbeat and callsreconnect()once peridle_heartbeat_interval(default 30s,cassandra/cluster.py:935) for as long as the connection stays defunct.reconnect()now returns early whenself._reconnection_handler is not None(cassandra/cluster.py:4058-4061), so this path no longer advances anything.on_add()only refreshes metadata;on_down()/on_remove()call the samereconnect()and hit the same guard.
Relationship to #1024
This is a deliberate consequence of #1024, not an accident, and it is the correct trade for the bug that PR fixes -- but the second-order cost was never decided on.
Before #1024, each heartbeat-driven reconnect() cancelled the in-flight handler and rebuilt its schedule from base_delay, which meant the control connection was effectively retried every ~30s forever. That was also the bug: a burst of errors, or a merely idle defunct connection, reset the backoff on every pass so it never grew, which is the reconnect storm #295 reports. #1024 closed that by making reconnect() respect a handler that is already retrying. The backoff now genuinely grows -- and with nothing else able to re-arm it, growth is unbounded up to max_delay.
So the driver traded a storm for a stall. Worst-case control-connection recovery latency went from ~idle_heartbeat_interval to ~max_reconnection_delay: 30s to 10 minutes at defaults. The control connection does eventually recover (the default policy has max_attempts=None, so the handler retries forever); this is a latency regression, not a permanent loss.
Why it was not fixed in #1024
Every candidate fix picks a different point on that same trade-off, and picking wrongly reopens #295:
- Make
ControlConnection.on_up()re-arm. Cancel the parked handler and callreconnect()whenself._connectionisNoneor defunct. Most targeted -- the node being back is exactly the signal worth reacting to -- but it fires once per host coming back, so a whole-cluster restart produces the burst control-connection: reconnect when down handling is skipped #1024 set out to collapse unless the re-arm is itself deduplicated. - Clamp the control connection's own schedule below
reconnection_policy.max_delay. Keeps the fix off the event paths entirely, but invents a second implicit config surface: the control connection would stop honouring thereconnection_policythe user configured, with no knob saying so. - Keep a heartbeat-driven floor. Let
return_connection()advance a handler that has been parked longer than some bound. Cheapest to reason about, but partially undoes control-connection: reconnect when down handling is skipped #1024's stated acceptance criterion that the schedule no longer resets once peridle_heartbeat_interval.
This needs an author/maintainer decision about which cadence the control connection is supposed to promise, which is why #1024 left it out rather than guessing.
Minimum acceptable outcome
Even if the behaviour is judged acceptable as-is, the CHANGELOG should state it: after #1024 the control connection's retry cadence follows reconnection_policy up to max_reconnection_delay instead of being re-driven every idle_heartbeat_interval. Operators who relied on the old ~30s floor need to know to lower max_reconnection_delay.
Related
- control-connection: reconnect when down handling is skipped #1024 -- the PR that introduced this trade-off
- After a node restarts, driver reconnects multiple times, causing queries to fail #295 -- the reconnect storm any fix here must not reintroduce
- Session might not reconnect if all nodes are restarted at once #230 -- session not reconnecting after all nodes restart at once; different diagnosis (refresh failure), same symptom class
Activity
dkropachev commentedon Nov 12, 2026
The minimum acceptable outcome described here was included in merged PR #1024. Its CHANGELOG entry explicitly states that control-connection retries follow
reconnection_policyup tomax_reconnection_delay, instead of restarting everyidle_heartbeat_interval, and advises operators to lower the maximum delay when they need a shorter recovery bound. That guidance is also present on current master.Closing as completed for that documented outcome. The active-backoff behavior itself is unchanged; finite-schedule exhaustion remains separately tracked in #1040.