Skip to content

add the manager thread, test harness and first migrated writer #1257

Open

Description

@dkropachev

Parent: #382
Depends on: #1256

LifecycleCoordinator delivers notifications on submitting or worker threads. There is no dedicated place to run management handlers.

Add the manager and notifier described in #1256 and move the first real writer onto them, so the mechanism lands together with a user. The first writer is host health: up, down, preparation, addition readiness and the per-host reconnector, because they share is_up and cannot be split without two writers.

  • Give LifecycleCoordinator a dedicated daemon manager thread and a notifier thread per Cluster. They start on the first connect(), and register with _clusters_for_shutdown. LifecycleCoordinator._lock becomes the result lock and _closed its closed flag. Remove draining on the submitting thread.
  • Add per-key desired state with a dirty bit, attempt ids for worker results, and an enqueue/close lock so a result refused after close is closed by its worker.
  • connect() claims a CONNECTING state under Cluster._lock and releases the lock before any wait on the manager; a second connect() during CONNECTING waits on the same initial future.
  • Add the waiting rules and leaf-lock list, the driver-owned future type and the ResponseFuture completion rule, closing on the notifier, the inbox with posted flags and the result lock, the manager timer heap, the failed-submission rule, weak Session references, and the exit and fork rules from define responsibilities for cluster and pool management #1256.
  • Assert in tests that the manager never waits and executor tasks never wait on managed work.
  • Add a deterministic test harness: a manager and notifier that tests step by hand, a fake executor and fake timers. Later stages use it.
  • Move host up, down, preparation and addition readiness (on_add, _finalize_add, _notify_pending_add, failed-endpoint-add retry) to the manager. Reactor, scheduler and worker threads only enqueue UP, DOWN and REMOVE observations.
  • on_down and signal_connection_failure no longer return a dispatch status; the "not dispatched, so _recover_pool" fallback moves into the manager's DOWN handler. The DOWN handler accepts a pool failure only if session._pools.get(host) is failed_pool, a lock-free read.
  • The host reconnector becomes a manager attempt driven by reconnection_policy.new_schedule() on the manager timer heap. A reconnector success is an UP observation carrying its attempt id.
  • The query admission check in ResponseFuture._query becomes the reads metadata.get_host_by_host_id(host.host_id) is host and host.is_up is not False, without host.lock. A send that races removal fails when the pool closes, as a send on a closing pool does today.
  • add_execution_profile posts its pool reconcile to the manager (a user thread may wait on it), so update_created_pools runs only on the manager and reads host key state directly.
  • Pool futures from add_or_renew_pool post a pool-result message to the host key through add_done_callback, until make the cluster manager control Session pool changes #1259.
  • Until make the cluster manager update topology and host status #1258, topology application stays on workers. transaction() becomes a plain with self._lock adapter, dispatch_current reduces to is_current(host), and publish_host, publish_replacements, _retire and _notify_replacements are edited to stop using the deleted fences. _retire no longer calls set_down() or touches reconnector state; it posts a REMOVE observation and the REMOVE handler sets is_up. LifecycleCoordinator.submit becomes an adapter that posts a closure to the manager, so _notify_host_removed, _notify_location_changed and Session.on_down's continuation run on the manager. retire, remove, _notify_addition and _notify_replacements stay on topology workers under transaction() until make the cluster manager update topology and host status #1258; their cluster.on_add calls post to the manager. _apply_node_list_and_token_map and _notify_replacements post a retry to the host key instead of scheduling retry_failed_endpoint_add.
  • ControlConnection.on_add/on_remove submit their topology refresh to the executor instead of running it inline, so the manager never waits on it.
  • ControlConnection._signal_error posts DOWN and always calls reconnect(); _reconnect_pending deduplicates it until make the cluster manager control reconnects and refresh completion #1261.
  • _shutdown_detached_pool hands the pool to the notifier instead of closing it inline.
  • Interim shutdown adapter until finish shutdown across managers and background work #1262: on the manager, Cluster.shutdown() and Session.shutdown() hand the request to the notifier and return; on the notifier, today's _shutdown(wait_for_executor=False) runs inline.
  • _notify_host_removed posts its listener loop to the notifier.
  • A pool-creation failure carries its PoolFailureReservation, and the DOWN handler checks it with is_current() until make the cluster manager control Session pool changes #1259 turns creation failures into (Session, host) attempt results.
  • Host.is_currently_reconnecting stays as a public method that reads a plain Host._reconnecting flag written by the manager.
  • The DOWN discount check (_discount_down_events) runs in the manager's DOWN handler and ignores the pool or connection the DOWN carries and every pool of that Session.
  • Management timers for this stage (reconnector backoff, failed-endpoint-add retry, recovery retry) move to the manager's timer heap.
  • Measure bootstrap time against host count and recovery latency with concurrent DDL in flight, and decide whether management I/O needs its own pool instead of Cluster.executor.
  • Log each applied message at debug level with its key, attempt id and outcome (applied, stale, merged).

Deleted in this stage, together with their lines in LifecycleCoordinator.close():

  • Coordinator: drain(), _draining, _events, _local, delivering, submit_async, collect_futures, submit_current, the event decorator, and the lifecycle branch of run_in_executor.
  • Health fences: Host._health_generation, _readiness_generation, _down_event_generation, _currently_handling_node_up, _currently_handling_node_down, _lifecycle_preparation, and observe_health, begin_preparation/finish_preparation/cancel_preparation, begin_up/finish_up, begin_down/dispatch_down/finish_down/down_is_current/supersede_down, submit_readiness/readiness_is_current, mark_ready.
  • Addition fences: _addition_token, host._lifecycle_pending, _endpoint_add_retry_attempts, _endpoint_add_retry_callback_errors, _pending_host_addition, _pending_host_addition_notification, _pending_host_addition_callback, _endpoint_add_retry_pending, and reserve_addition/begin_addition/claim_addition/addition_is_current/remember_addition/release_addition/abandon_addition/claim_addition_notification.
  • Reconnector: _HostReconnectionHandler, Host._reconnection_handler, Host.get_and_set_reconnection_handler, _clear_reconnection_handler, _release_slot, and Cluster._restart_reconnector.
  • Handlers replaced by manager handlers: on_down_potentially_blocking, _start_reconnector, _on_up, _on_up_prepared, _on_up_preparation_failed, _on_up_pools_completed, _cleanup_failed_on_up_handling, _is_current_host, _on_add, _on_add_prepared (including pools_completed) and _on_add_preparation_failed.
  • Recovery: Session._recover_pool and its closures (replaced by the DOWN handler and the manager timer).
  • Failed-endpoint-add retry: retry_failed_endpoint_add, _retry_failed_endpoint_add and _schedule_failed_endpoint_add_retry (replaced by the manager timer).

Kept as named adapters: LifecycleCoordinator.submit until #1259, the executor refresh in ControlConnection.on_add/on_remove, transaction(), is_current, dispatch_current, publish_host, publish_replacements, _retire and _notify_replacements until #1258; the pool-result add_done_callback until #1259.

Test handler failures, concurrent submissions, UP/DOWN flapping, overlapping additions, failed startup, a listener that calls connect() and refresh_nodes(), shutdown, interpreter exit, fork before and after connect(), and late results. Verify cancellation and shutdown through concurrent.futures.wait() as well as individual futures.

Done when host health and addition have one writer, the code listed above is deleted, and the harness is used by the tests above.

Activity

  1. changed the title [-]Add a serial cluster manager runtime with explicit completion semantics[/-] [+]add management execution and operation completion[/+] on Nov 17, 2026
  2. changed the title [-]add management execution and operation completion[/-] [+]add the manager thread, test harness and first migrated writer[/+] on Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions