Skip to content

Fresh pool publication can retain a closed connection and suppress recovery #1292

Open

Description

@dkropachev

Problem

A fresh pool can be installed successfully after its only connection has closed, and failure handling can then remain permanently suppressed. This affects the fresh-pool path used for ADD/UP handling, rather than renewal of an already-shutdown pool.

Verified on master at 27b00056de1434dc8554325e37c9c7242dc5b785 with the pure-Python driver and mocked transport.

The failure requires this interleaving:

  1. A new pool initializes while another operation changes the session keyspace.
  2. Before publication, the candidate runs its internal asynchronous USE to synchronize with the session.
  3. The successful USE response races with a socket close. The real pool callback returns the connection while the candidate is still unpublished.
  4. return_connection() sets signaled_error=True, then discounts failure handling because this pool is not yet installed.
  5. Session publication sees no usable connection, but rejects that state only when renewing an already-shutdown pool. The fresh candidate is installed and creation returns True.

Subsequent closed-socket heartbeat handling returns early because the error is already marked signaled. Pool reconciliation also retains the pool because is_shutdown=False. No host reconnector is started. The affected session can remain unable to query that host even after the node is healthy.

Relevant code:

Expected behavior

When a connection is required, a candidate with no usable connection must not be published as a successful pool. The failure must retain a recovery path; returning a connection before publication must not suppress all later failure handling.

Reproduction

Run the following script from the pinned source checkout with the development dependencies and pure-Python driver. It uses the existing unit fixture and real pool publication/keyspace-completion logic; the transport double schedules the disconnect at the response callback boundary.

from threading import Lock
from unittest.mock import Mock, patch
from cassandra.connection import Connection
from cassandra.protocol_features import ProtocolFeatures
from cassandra.pool import HostConnection
from tests.unit.util import HashableMock
from tests.unit.test_pool_metadata_ownership import pool_session

generator = pool_session.__wrapped__()
session, host = next(generator)
conn = HashableMock(spec=Connection, lock=Lock(), in_flight=0, max_request_id=100,
    is_closed=False, is_defunct=False, signaled_error=False, orphaned_threshold_reached=False,
    orphaned_request_ids=set(), keyspace=None, features=ProtocolFeatures(shard_id=0))
session.cluster.connection_factory = Mock(return_value=conn)
session.cluster.signal_connection_failure = Mock()
original_sync = HostConnection._sync_keyspace

def change_session_keyspace(pool, candidate, *args, **kwargs):
    result = original_sync(pool, candidate, *args, **kwargs)
    # Another thread's USE changes Session.keyspace during new pool initialization.
    session.keyspace='ks'
    return result

def successful_use_then_disconnect(keyspace, callback):
    # Real set_keyspace_async reserves a stream and handles SET_KEYSPACE,
    # then its callback can see a disconnect admitted by another thread.
    conn.in_flight += 1
    conn.keyspace=keyspace
    conn.is_closed=True
    callback(conn, None)
conn.set_keyspace_async.side_effect=successful_use_then_disconnect
try:
    with patch.object(HostConnection, '_sync_keyspace', change_session_keyspace):
        result = session.add_or_renew_pool(host, True).result(5)
    pool = session._pools[host]
    print('creation result:', result)
    print('internal USE calls:', conn.set_keyspace_async.call_count)
    print('installed pool shutdown:', pool.is_shutdown)
    print('installed open_count:', pool.open_count)
    print('transport already signaled_error:', conn.signaled_error)
    print('failure signals:', session.cluster.signal_connection_failure.call_count)
    # Subsequent heartbeat sees closed socket, but return_connection returns early.
    pool.return_connection(conn)
    print('after heartbeat shutdown:', pool.is_shutdown)
    print('reconnection handler:', host._reconnection_handler)
    print('reconcile futures:', session.update_created_pools())
    print('after reconcile open_count:', session._pools[host].open_count)
finally:
    try: next(generator)
    except StopIteration: pass

Observed output:

creation result: True
internal USE calls: 1
installed pool shutdown: False
installed open_count: 0
transport already signaled_error: True
failure signals: 0
after heartbeat shutdown: False
reconnection handler: None
reconcile futures: set()
after reconcile open_count: 0

Activity

  1. added
    bugSomething isn't working
    release-blockerMust be resolved before the next release.
    on Nov 19, 2026
  2. 31 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingrelease-blockerMust be resolved before the next release.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions