Repository navigation
Description
Problem
PR #1024 makes heartbeat returns stop re-arming a finite control-connection reconnection schedule once that schedule is exhausted for the same defunct connection.
python-driver/cassandra/cluster.py
Lines 4119 to 4124 in 72db15c
| if (exhausted and not reconnect_requested and | |
| self._failed_connection is not None and | |
| control_connection._connection is | |
| self._failed_connection): | |
| control_connection._reconnection_exhausted_connection = \ | |
| self._failed_connection |
The guard remains until a successful connection is installed. This is consistent with the public ReconnectionPolicy contract: a finite iterable means reconnection attempts cease after exhaustion.
python-driver/cassandra/policies.py
Lines 873 to 878 in 72db15c
| def new_schedule(self): | |
| """ | |
| This should return a finite or infinite iterable of delays (each as a | |
| floating point number of seconds) in-between each failed reconnection | |
| attempt. Note that if the iterable is finite, reconnection attempts | |
| will cease once the iterable is exhausted. |
The operational result is terminal automatic recovery for that failed control connection after finite exhaustion. Later heartbeat returns cannot start another recovery attempt.
python-driver/cassandra/cluster.py
Lines 5358 to 5365 in 72db15c
| def return_connection(self, connection): | |
| if connection is self._connection and (connection.is_defunct or connection.is_closed): | |
| with self._reconnection_lock: | |
| if connection is self._reconnection_exhausted_connection: | |
| log.debug("[control connection] Reconnection schedule is " | |
| "exhausted for the defunct connection") | |
| return | |
| self.reconnect() |
Scope
This differs from #1029. That issue covers recovery latency while an infinite or still-active handler is backing off. This issue covers the state after a finite handler has stopped.
Design constraints
- Silently restarting after a cooldown turns
max_attempts=Ninto unlimited batches of N attempts. - A custom policy exposes an arbitrary iterable, so the driver cannot derive a universal maximum policy delay.
- Re-enabling every idle heartbeat risks restoring the reconnect storm tracked by After a node restarts, driver reconnects multiple times, causing queries to fail #295.
- The default infinite reconnection policy is unaffected by finite exhaustion.
Expected outcome
Define the control-connection recovery contract after finite schedule exhaustion. If automatic recovery should resume, expose that behavior explicitly through policy or configuration semantics rather than implicitly redefining every finite schedule.
Acceptance criteria
- Terminal versus periodic recovery is chosen and documented.
- Built-in and custom finite policies have deterministic behavior.
- Periodic recovery, if supported, does not reset active backoff or create one attempt per heartbeat.
- Unit tests cover exhaustion, repeated heartbeat returns, successful recovery, and shutdown.
- Operator-facing documentation explains how to select terminal or continuing recovery.
Related
- control-connection: reconnect when down handling is skipped #1024 introduced and documents the current guard.
- Nothing re-arms a backing-off control connection: recovery latency grows from
idle_heartbeat_intervaltomax_reconnection_delay#1029 tracks active-handler recovery latency. - After a node restarts, driver reconnects multiple times, causing queries to fail #295 tracks reconnect storms caused by repeated schedule resets.
Activity