Repository navigation
Description
Parent: #382
Depends on: #1256
LifecycleCoordinator delivers notifications on submitting or worker threads. There is no dedicated place to run management handlers.
Add the manager and notifier described in #1256 and move the first real writer onto them, so the mechanism lands together with a user. The first writer is host health: up, down, preparation, addition readiness and the per-host reconnector, because they share is_up and cannot be split without two writers.
- Give
LifecycleCoordinatora dedicated daemon manager thread and a notifier thread perCluster. They start on the firstconnect(), and register with_clusters_for_shutdown.LifecycleCoordinator._lockbecomes the result lock and_closedits closed flag. Remove draining on the submitting thread. - Add per-key desired state with a dirty bit, attempt ids for worker results, and an enqueue/close lock so a result refused after close is closed by its worker.
connect()claims a CONNECTING state underCluster._lockand releases the lock before any wait on the manager; a secondconnect()during CONNECTING waits on the same initial future.- Add the waiting rules and leaf-lock list, the driver-owned future type and the
ResponseFuturecompletion rule, closing on the notifier, the inbox withpostedflags and the result lock, the manager timer heap, the failed-submission rule, weak Session references, and the exit and fork rules from define responsibilities for cluster and pool management #1256. - Assert in tests that the manager never waits and executor tasks never wait on managed work.
- Add a deterministic test harness: a manager and notifier that tests step by hand, a fake executor and fake timers. Later stages use it.
- Move host up, down, preparation and addition readiness (
on_add,_finalize_add,_notify_pending_add, failed-endpoint-add retry) to the manager. Reactor, scheduler and worker threads only enqueue UP, DOWN and REMOVE observations. on_downandsignal_connection_failureno longer return a dispatch status; the "not dispatched, so_recover_pool" fallback moves into the manager's DOWN handler. The DOWN handler accepts a pool failure only ifsession._pools.get(host) is failed_pool, a lock-free read.- The host reconnector becomes a manager attempt driven by
reconnection_policy.new_schedule()on the manager timer heap. A reconnector success is an UP observation carrying its attempt id. - The query admission check in
ResponseFuture._querybecomes the readsmetadata.get_host_by_host_id(host.host_id) is hostandhost.is_up is not False, withouthost.lock. A send that races removal fails when the pool closes, as a send on a closing pool does today. add_execution_profileposts its pool reconcile to the manager (a user thread may wait on it), soupdate_created_poolsruns only on the manager and reads host key state directly.- Pool futures from
add_or_renew_poolpost a pool-result message to the host key throughadd_done_callback, until make the cluster manager control Session pool changes #1259. - Until make the cluster manager update topology and host status #1258, topology application stays on workers.
transaction()becomes a plainwith self._lockadapter,dispatch_currentreduces tois_current(host), andpublish_host,publish_replacements,_retireand_notify_replacementsare edited to stop using the deleted fences._retireno longer callsset_down()or touches reconnector state; it posts a REMOVE observation and the REMOVE handler setsis_up.LifecycleCoordinator.submitbecomes an adapter that posts a closure to the manager, so_notify_host_removed,_notify_location_changedandSession.on_down's continuation run on the manager.retire,remove,_notify_additionand_notify_replacementsstay on topology workers undertransaction()until make the cluster manager update topology and host status #1258; theircluster.on_addcalls post to the manager._apply_node_list_and_token_mapand_notify_replacementspost a retry to the host key instead of schedulingretry_failed_endpoint_add. ControlConnection.on_add/on_removesubmit their topology refresh to the executor instead of running it inline, so the manager never waits on it.ControlConnection._signal_errorposts DOWN and always callsreconnect();_reconnect_pendingdeduplicates it until make the cluster manager control reconnects and refresh completion #1261._shutdown_detached_poolhands the pool to the notifier instead of closing it inline.- Interim shutdown adapter until finish shutdown across managers and background work #1262: on the manager,
Cluster.shutdown()andSession.shutdown()hand the request to the notifier and return; on the notifier, today's_shutdown(wait_for_executor=False)runs inline. _notify_host_removedposts its listener loop to the notifier.- A pool-creation failure carries its
PoolFailureReservation, and the DOWN handler checks it withis_current()until make the cluster manager control Session pool changes #1259 turns creation failures into (Session, host) attempt results. Host.is_currently_reconnectingstays as a public method that reads a plainHost._reconnectingflag written by the manager.- The DOWN discount check (
_discount_down_events) runs in the manager's DOWN handler and ignores the pool or connection the DOWN carries and every pool of that Session. - Management timers for this stage (reconnector backoff, failed-endpoint-add retry, recovery retry) move to the manager's timer heap.
- Measure bootstrap time against host count and recovery latency with concurrent DDL in flight, and decide whether management I/O needs its own pool instead of
Cluster.executor. - Log each applied message at debug level with its key, attempt id and outcome (applied, stale, merged).
Deleted in this stage, together with their lines in LifecycleCoordinator.close():
- Coordinator:
drain(),_draining,_events,_local,delivering,submit_async,collect_futures,submit_current, theeventdecorator, and the lifecycle branch ofrun_in_executor. - Health fences:
Host._health_generation,_readiness_generation,_down_event_generation,_currently_handling_node_up,_currently_handling_node_down,_lifecycle_preparation, andobserve_health,begin_preparation/finish_preparation/cancel_preparation,begin_up/finish_up,begin_down/dispatch_down/finish_down/down_is_current/supersede_down,submit_readiness/readiness_is_current,mark_ready. - Addition fences:
_addition_token,host._lifecycle_pending,_endpoint_add_retry_attempts,_endpoint_add_retry_callback_errors,_pending_host_addition,_pending_host_addition_notification,_pending_host_addition_callback,_endpoint_add_retry_pending, andreserve_addition/begin_addition/claim_addition/addition_is_current/remember_addition/release_addition/abandon_addition/claim_addition_notification. - Reconnector:
_HostReconnectionHandler,Host._reconnection_handler,Host.get_and_set_reconnection_handler,_clear_reconnection_handler,_release_slot, andCluster._restart_reconnector. - Handlers replaced by manager handlers:
on_down_potentially_blocking,_start_reconnector,_on_up,_on_up_prepared,_on_up_preparation_failed,_on_up_pools_completed,_cleanup_failed_on_up_handling,_is_current_host,_on_add,_on_add_prepared(includingpools_completed) and_on_add_preparation_failed. - Recovery:
Session._recover_pooland its closures (replaced by the DOWN handler and the manager timer). - Failed-endpoint-add retry:
retry_failed_endpoint_add,_retry_failed_endpoint_addand_schedule_failed_endpoint_add_retry(replaced by the manager timer).
Kept as named adapters: LifecycleCoordinator.submit until #1259, the executor refresh in ControlConnection.on_add/on_remove, transaction(), is_current, dispatch_current, publish_host, publish_replacements, _retire and _notify_replacements until #1258; the pool-result add_done_callback until #1259.
Test handler failures, concurrent submissions, UP/DOWN flapping, overlapping additions, failed startup, a listener that calls connect() and refresh_nodes(), shutdown, interpreter exit, fork before and after connect(), and late results. Verify cancellation and shutdown through concurrent.futures.wait() as well as individual futures.
Done when host health and addition have one writer, the code listed above is deleted, and the harness is used by the tests above.
Activity
docs: define cluster and pool manager ownership
docs: define cluster and pool manager ownership
docs: define cluster and pool manager ownership
docs: define cluster and pool manager ownership
management: add bounded native runtime
management: add bounded native runtime
management: add bounded native runtime
management: add bounded native runtime
management: add bounded native runtime
docs: define cluster and pool manager ownership
management: add bounded native runtime
management: add bounded native runtime
docs: define cluster and pool manager ownership
docs: define cluster and pool manager ownership
docs: define cluster and pool manager ownership
[-]Add a serial cluster manager runtime with explicit completion semantics[/-][+]add management execution and operation completion[/+]management: support staged lifecycle admission
lifecycle: route management through staged adapters
docs: stage cluster and pool manager ownership evaluation
docs: align shutdown ownership and validation guidance
management: support staged lifecycle admission
lifecycle: route management through staged adapters
lifecycle: route management through staged adapters
management: support staged lifecycle admission
lifecycle: route management through staged adapters
[-]add management execution and operation completion[/-][+]add the manager thread, test harness and first migrated writer[/+]