Repository navigation
Open
Labels
bugSomething isn't workingSomething isn't working
Description
python-driver re-sends a statement with is_idempotent=False to the next host in three cases where it may already have run, so a write such as a counter update can be applied twice:
- SERVER_ERROR: the default
RetryPolicy.on_request_errorreturnsRETRY_NEXT_HOSTwithout checkingis_idempotent. - Connection lost with the request in flight: the same policy method gives the same answer.
- Failed push (
ConnectionSendError, "a push failed after request bytes might have been queued", from response-future: release request ids after send failure #872):_query()returnsNoneandsend_request()moves to the next host without asking the retry policy, so evenFallthroughRetryPolicydoesn't stop it.
java-driver 4.x and the Rust driver check idempotence before asking the retry policy: a request that may have run and isn't marked idempotent is raised. Doing the same in ResponseFuture, and sending ConnectionSendError down that path, fixes all three. This change behaviour for applications that rely on these retries without marking statements idempotent. Case 3 can be fixed on its own first, since nobody relies on their retry policy being bypassed.
Reproductions
Fails on master (ab7906c):
import unittest
from unittest.mock import Mock
from cassandra import ConsistencyLevel
from cassandra.cluster import ControlConnection, ControlConnectionQueryFallback, ResponseFuture, Session
from cassandra.connection import Connection, ConnectionShutdown, DefaultEndPoint
from cassandra.policies import FallthroughRetryPolicy, RetryPolicy
from cassandra.protocol import QueryMessage, ServerError
from cassandra.query import SimpleStatement
UPDATE = SimpleStatement("UPDATE ks.counters SET n = n + 1 WHERE k = 0", is_idempotent=False)
def make_session():
session = Mock(spec=Session)
session.keyspace = None
session.is_shutdown = False
session.cluster.allow_control_connection_query_fallback = ControlConnectionQueryFallback.Disabled
session.cluster.control_connection = ControlConnection(
session.cluster, timeout=1, schema_event_refresh_window=0,
topology_event_refresh_window=0, status_event_refresh_window=0)
session.cluster._default_load_balancing_policy.make_query_plan.return_value = ['ip1', 'ip2']
session._pools.get.return_value.is_shutdown = False
return session
def make_future(session, retry_policy):
message = QueryMessage(query=UPDATE.query_string, consistency_level=ConsistencyLevel.ONE)
return ResponseFuture(session, message, UPDATE, 1, retry_policy=retry_policy)
class NonIdempotentResendTest(unittest.TestCase):
def test_server_error_or_lost_connection_is_not_resent(self):
for case, error in (("SERVER_ERROR", Mock(spec=ServerError, info={})),
("lost connection", ConnectionShutdown("Connection to ip1 was closed"))):
with self.subTest(case):
session = make_session()
session._pools.get.return_value.borrow_connection.return_value = (Mock(spec=Connection), 1)
rf = make_future(session, RetryPolicy())
rf.send_request()
rf._set_result('ip1', None, None, error)
session.cluster.scheduler.schedule_with_shutdown.assert_not_called()
def test_failed_push_is_not_resent(self):
session = make_session()
# The push to ip1 fails after the request may have been queued, which
# send_msg reports as ConnectionSendError.
first = Connection(DefaultEndPoint('127.0.0.1'))
first.push = Mock(side_effect=OSError('failed to wake the reactor'))
first.close = Mock()
second = Connection(DefaultEndPoint('127.0.0.2'))
second.push = Mock()
connections = iter([first, second])
def borrow_connection(**kwargs):
connection = next(connections)
with connection.lock:
connection.in_flight += 1
return connection, connection.get_request_id()
session._pools.get.return_value.borrow_connection.side_effect = borrow_connection
make_future(session, FallthroughRetryPolicy()).send_request()
second.push.assert_not_called()Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
Rajatendu1 commentedon Nov 17, 2026
I have a small local patch for case 3 on current master (882b21b), limited to pooled application sends. After a ConnectionSendError, it finishes connection/stream cleanup, then calls on_request_error and uses the existing retry-decision handler instead of immediately trying the next host. It preserves RETHROW, IGNORE, same-host/next-host retry and delay. Internal PREPARE attempts and errors before push keep their existing fallback paths.
The 13 regression/control cases cover FallthroughRetryPolicy with both idempotency flags, custom decisions, scheduled host choice, cleanup ordering and policy exceptions. On the original source, 11 fail and two controls pass. The candidate passes all 13. The related cluster/query/response-future/connection/policy suites go from 685 passed to 698 passed, with the same 34 libev-dependent skips, on Linux Python 3.10 (pure Python) and 3.13 (Cython-compiled cluster module, other modules pure Python). No server integration or full reactor/platform matrix was run. These tests prove the unwanted second send attempt, not a live duplicate write.
Would a focused PR for this pooled-send policy bypass be welcome? It would leave the wider default-policy idempotency changes in cases 1 and 2, and the separate control-connection fallback send path, out of scope rather than claim to fix the whole issue.
Prepared with AI assistance; the reported tests were run locally.
response-future: preserve page binding and cover error claims
response-future: preserve failed-push attempt context
response-future: respect retry policy after failed pushes
response-future: respect retry policy after failed pushes
response-future: respect retry policy after failed pushes
response-future: respect retry policy after failed pushes
response-future: coalesce retries after connection failure
response-future: preserve page binding and cover error claims
response-future: preserve failed-push attempt context
response-future: retain context through control fallback
response-future: isolate speculative work and retain retry context
response-future: coalesce retries after connection failure