Skip to content

Writes not marked idempotent are re-sent to another host after errors where they may have run #1168

Open

Description

python-driver re-sends a statement with is_idempotent=False to the next host in three cases where it may already have run, so a write such as a counter update can be applied twice:

  1. SERVER_ERROR: the default RetryPolicy.on_request_error returns RETRY_NEXT_HOST without checking is_idempotent.
  2. Connection lost with the request in flight: the same policy method gives the same answer.
  3. Failed push (ConnectionSendError, "a push failed after request bytes might have been queued", from response-future: release request ids after send failure #872): _query() returns None and send_request() moves to the next host without asking the retry policy, so even FallthroughRetryPolicy doesn't stop it.

java-driver 4.x and the Rust driver check idempotence before asking the retry policy: a request that may have run and isn't marked idempotent is raised. Doing the same in ResponseFuture, and sending ConnectionSendError down that path, fixes all three. This change behaviour for applications that rely on these retries without marking statements idempotent. Case 3 can be fixed on its own first, since nobody relies on their retry policy being bypassed.

Reproductions

Fails on master (ab7906c):

import unittest
from unittest.mock import Mock

from cassandra import ConsistencyLevel
from cassandra.cluster import ControlConnection, ControlConnectionQueryFallback, ResponseFuture, Session
from cassandra.connection import Connection, ConnectionShutdown, DefaultEndPoint
from cassandra.policies import FallthroughRetryPolicy, RetryPolicy
from cassandra.protocol import QueryMessage, ServerError
from cassandra.query import SimpleStatement

UPDATE = SimpleStatement("UPDATE ks.counters SET n = n + 1 WHERE k = 0", is_idempotent=False)


def make_session():
    session = Mock(spec=Session)
    session.keyspace = None
    session.is_shutdown = False
    session.cluster.allow_control_connection_query_fallback = ControlConnectionQueryFallback.Disabled
    session.cluster.control_connection = ControlConnection(
        session.cluster, timeout=1, schema_event_refresh_window=0,
        topology_event_refresh_window=0, status_event_refresh_window=0)
    session.cluster._default_load_balancing_policy.make_query_plan.return_value = ['ip1', 'ip2']
    session._pools.get.return_value.is_shutdown = False
    return session


def make_future(session, retry_policy):
    message = QueryMessage(query=UPDATE.query_string, consistency_level=ConsistencyLevel.ONE)
    return ResponseFuture(session, message, UPDATE, 1, retry_policy=retry_policy)


class NonIdempotentResendTest(unittest.TestCase):

    def test_server_error_or_lost_connection_is_not_resent(self):
        for case, error in (("SERVER_ERROR", Mock(spec=ServerError, info={})),
                            ("lost connection", ConnectionShutdown("Connection to ip1 was closed"))):
            with self.subTest(case):
                session = make_session()
                session._pools.get.return_value.borrow_connection.return_value = (Mock(spec=Connection), 1)
                rf = make_future(session, RetryPolicy())
                rf.send_request()
                rf._set_result('ip1', None, None, error)
                session.cluster.scheduler.schedule_with_shutdown.assert_not_called()

    def test_failed_push_is_not_resent(self):
        session = make_session()
        # The push to ip1 fails after the request may have been queued, which
        # send_msg reports as ConnectionSendError.
        first = Connection(DefaultEndPoint('127.0.0.1'))
        first.push = Mock(side_effect=OSError('failed to wake the reactor'))
        first.close = Mock()
        second = Connection(DefaultEndPoint('127.0.0.2'))
        second.push = Mock()
        connections = iter([first, second])

        def borrow_connection(**kwargs):
            connection = next(connections)
            with connection.lock:
                connection.in_flight += 1
                return connection, connection.get_request_id()

        session._pools.get.return_value.borrow_connection.side_effect = borrow_connection
        make_future(session, FallthroughRetryPolicy()).send_request()
        second.push.assert_not_called()

Activity

  1. added theissue type on Nov 12, 2026
  2. Rajatendu1 commented on Nov 17, 2026

    @Rajatendu1

    I have a small local patch for case 3 on current master (882b21b), limited to pooled application sends. After a ConnectionSendError, it finishes connection/stream cleanup, then calls on_request_error and uses the existing retry-decision handler instead of immediately trying the next host. It preserves RETHROW, IGNORE, same-host/next-host retry and delay. Internal PREPARE attempts and errors before push keep their existing fallback paths.

    The 13 regression/control cases cover FallthroughRetryPolicy with both idempotency flags, custom decisions, scheduled host choice, cleanup ordering and policy exceptions. On the original source, 11 fail and two controls pass. The candidate passes all 13. The related cluster/query/response-future/connection/policy suites go from 685 passed to 698 passed, with the same 34 libev-dependent skips, on Linux Python 3.10 (pure Python) and 3.13 (Cython-compiled cluster module, other modules pure Python). No server integration or full reactor/platform matrix was run. These tests prove the unwanted second send attempt, not a live duplicate write.

    Would a focused PR for this pooled-send policy bypass be welcome? It would leave the wider default-policy idempotency changes in cases 1 and 2, and the separate control-connection fallback send path, out of scope rather than claim to fix the whole issue.

    Prepared with AI assistance; the reported tests were run locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions