Repository navigation
Description
Problem
A fresh pool can be installed successfully after its only connection has closed, and failure handling can then remain permanently suppressed. This affects the fresh-pool path used for ADD/UP handling, rather than renewal of an already-shutdown pool.
Verified on master at 27b00056de1434dc8554325e37c9c7242dc5b785 with the pure-Python driver and mocked transport.
The failure requires this interleaving:
- A new pool initializes while another operation changes the session keyspace.
- Before publication, the candidate runs its internal asynchronous
USEto synchronize with the session. - The successful
USEresponse races with a socket close. The real pool callback returns the connection while the candidate is still unpublished. return_connection()setssignaled_error=True, then discounts failure handling because this pool is not yet installed.- Session publication sees no usable connection, but rejects that state only when renewing an already-shutdown pool. The fresh candidate is installed and creation returns
True.
Subsequent closed-socket heartbeat handling returns early because the error is already marked signaled. Pool reconciliation also retains the pool because is_shutdown=False. No host reconnector is started. The affected session can remain unable to query that host even after the node is healthy.
Relevant code:
- Publication only rejects unusable candidates when renewing a retired pool.
- Connection failure is marked signaled before unpublished-pool ownership is checked.
- Internal keyspace completion returns the connection.
- Heartbeat returns an already-closed connection without enabling shutdown-on-error.
- Reconciliation retains every nonshutdown pool.
Expected behavior
When a connection is required, a candidate with no usable connection must not be published as a successful pool. The failure must retain a recovery path; returning a connection before publication must not suppress all later failure handling.
Reproduction
Run the following script from the pinned source checkout with the development dependencies and pure-Python driver. It uses the existing unit fixture and real pool publication/keyspace-completion logic; the transport double schedules the disconnect at the response callback boundary.
from threading import Lock
from unittest.mock import Mock, patch
from cassandra.connection import Connection
from cassandra.protocol_features import ProtocolFeatures
from cassandra.pool import HostConnection
from tests.unit.util import HashableMock
from tests.unit.test_pool_metadata_ownership import pool_session
generator = pool_session.__wrapped__()
session, host = next(generator)
conn = HashableMock(spec=Connection, lock=Lock(), in_flight=0, max_request_id=100,
is_closed=False, is_defunct=False, signaled_error=False, orphaned_threshold_reached=False,
orphaned_request_ids=set(), keyspace=None, features=ProtocolFeatures(shard_id=0))
session.cluster.connection_factory = Mock(return_value=conn)
session.cluster.signal_connection_failure = Mock()
original_sync = HostConnection._sync_keyspace
def change_session_keyspace(pool, candidate, *args, **kwargs):
result = original_sync(pool, candidate, *args, **kwargs)
# Another thread's USE changes Session.keyspace during new pool initialization.
session.keyspace='ks'
return result
def successful_use_then_disconnect(keyspace, callback):
# Real set_keyspace_async reserves a stream and handles SET_KEYSPACE,
# then its callback can see a disconnect admitted by another thread.
conn.in_flight += 1
conn.keyspace=keyspace
conn.is_closed=True
callback(conn, None)
conn.set_keyspace_async.side_effect=successful_use_then_disconnect
try:
with patch.object(HostConnection, '_sync_keyspace', change_session_keyspace):
result = session.add_or_renew_pool(host, True).result(5)
pool = session._pools[host]
print('creation result:', result)
print('internal USE calls:', conn.set_keyspace_async.call_count)
print('installed pool shutdown:', pool.is_shutdown)
print('installed open_count:', pool.open_count)
print('transport already signaled_error:', conn.signaled_error)
print('failure signals:', session.cluster.signal_connection_failure.call_count)
# Subsequent heartbeat sees closed socket, but return_connection returns early.
pool.return_connection(conn)
print('after heartbeat shutdown:', pool.is_shutdown)
print('reconnection handler:', host._reconnection_handler)
print('reconcile futures:', session.update_created_pools())
print('after reconcile open_count:', session._pools[host].open_count)
finally:
try: next(generator)
except StopIteration: passObserved output:
creation result: True
internal USE calls: 1
installed pool shutdown: False
installed open_count: 0
transport already signaled_error: True
failure signals: 0
after heartbeat shutdown: False
reconnection handler: None
reconcile futures: set()
after reconcile open_count: 0
Activity
cluster: keep fresh pool recovery tied to owning work
cluster: keep fresh pool recovery tied to owning work
pool: reject unusable fresh candidates before publication
pool: retry fresh creation when peer pools discount DOWN
cluster: finish pending ADD after session pool recovery
cluster: fence recovered ADD at readiness publication
pool: retry fresh creation when peer pools discount DOWN
Preserve pending ADD slots across reconciliation and shutdown
Complete recovered ADD slots that become ignored
Test ignored ADD readiness after an unresolved intent renewal
Bind recovered ADD readiness to its owning completion
pool: retain cleanup work across ignored intent renewal
pool: retry fresh creation when peer pools discount DOWN
31 remaining items
pool: retain ADD recovery after reconciliation policy errors
pool: preserve ADD recovery authority through publication
pool: retry failed fresh slots and complete their ADD
pool: preserve ADD completion across reconciliation and shutdown
pool: preserve detached cleanup through ignored intent handoff
pool: retain ADD recovery after reconciliation policy errors
pool: preserve ADD recovery authority through publication
pool: retry failed fresh slots and complete their ADD
pool: preserve ADD completion across reconciliation and shutdown
pool: preserve detached cleanup through ignored intent handoff
pool: retain ADD recovery after reconciliation policy errors
pool: preserve ADD recovery authority through publication
pool: preserve ADD recovery authority through publication
pool: retain ADD recovery after reconciliation policy errors
pool: preserve ADD recovery authority through publication