Search before asking
Description
This is the second implementation phase of AIP-5: Dynamic Resource Allocation for Optimizer, building on the configuration foundation of #4253. It adds demand-driven scale-up for dynamic-allocation-enabled groups. Scale-down lands in later phases, and everything remains opt-in: groups without dynamic-allocation.enabled = true are unaffected.
Scope:
-
dynamic-allocation.executor-parallelism (default 1). The scaling unit is one homogeneous K-thread optimizer instance (the Spark executor model); this property configures K. Validation: 1 ≤ executor-parallelism ≤ max-parallelism — a single instance already exceeding the cap could never be created, which would leave an enabled group as a silent no-op.
-
A dedicated scale keeper. The existing group keeper checks groups at optimizer-group.min-parallelism-check-interval × consecutive attempts (minutes), which would make the DRA backlog timeouts (seconds) unreachable. A new OptimizerScaleKeeper (same AbstractKeeper infrastructure: HA leader gating, delay queue, lifecycle) evaluates each DRA group at its own sustained-backlog-timeout. It owns both the floor and the demand scaling of DRA groups; the legacy keeper skips them but keeps watching, resuming floor duty if DRA is disabled later. Groups are picked up on create, startup load, and update (covering enabling DRA on an existing group at runtime).
-
Scale-up decision. Per group and evaluation round:
- Floor:
min-parallelism deficits are satisfied immediately (no timing gate), in K-thread instance units.
- Immediate demand: when
busy + serviceable PLANNED > effective threads, scale out with an exponential ramp (1, 2, 4, 8 instances) clamped to the actual need — Spark ExecutorAllocationManager semantics, including resetting the ramp when the clamp binds or demand clears.
- Future demand: pending tables while every thread is busy add a single probe instance. This also covers the zero-optimizer cold start: planning is only driven by
pollTask, so with no optimizer nothing polls, planning never runs, and PLANNED tasks never materialize — pending tables are the only observable demand signal.
- Demand must persist for
scheduler-backlog-timeout before the first scale-out; later rounds are spaced by sustained-backlog-timeout, so a trickle drained between rounds never accumulates toward one. The max-parallelism cap always wins.
-
Quota-mode-aware serviceable demand. A table with a proportional quota (self-optimizing.quota ≤ 1) counts its whole PLANNED backlog (scaling raises its limit, which is ceil(quota × availableCore)); a table with an absolute quota (> 1) counts only free slots, since its fixed limit cannot be raised by scaling.
-
Boot-window accounting. Requested-but-unregistered capacity counts toward effective threads, so a booting pod is not re-requested every round. Registration is optimizer-driven and heartbeat expiry only starts after it, so these entries carry their own boot deadline (3 minutes); a request that never registers is pruned instead of freezing scale-up, and a synchronously failed request is retried on a later round.
-
Planning-bound guard. Idle threads while tables wait as PENDING and no PLANNED tasks materialize means the bottleneck is the serialized planning (optimizer.max-planning-parallelism), not thread capacity — scaling out would only add idle threads. The keeper surfaces this as an edge-triggered warning naming the config to raise, and does not scale.
The AIP-5 page will be revised alongside this phase: the scaling-granularity section currently refers to execution-parallel, which is not a group-level property, and will document dynamic-allocation.executor-parallelism instead.
Out of scope for this phase (later phases): per-optimizer idle tracking, scale-down + graceful drain, and the new gauges/counters (the planning-bound condition is log-only here; metrics land in the observability phase).
Parent issue
AIP-5: #4191
Are you willing to submit PR?
Code of Conduct
Search before asking
Description
This is the second implementation phase of AIP-5: Dynamic Resource Allocation for Optimizer, building on the configuration foundation of #4253. It adds demand-driven scale-up for dynamic-allocation-enabled groups. Scale-down lands in later phases, and everything remains opt-in: groups without
dynamic-allocation.enabled = trueare unaffected.Scope:
dynamic-allocation.executor-parallelism(default1). The scaling unit is one homogeneous K-thread optimizer instance (the Spark executor model); this property configures K. Validation:1 ≤ executor-parallelism ≤ max-parallelism— a single instance already exceeding the cap could never be created, which would leave an enabled group as a silent no-op.A dedicated scale keeper. The existing group keeper checks groups at
optimizer-group.min-parallelism-check-interval × consecutive attempts(minutes), which would make the DRA backlog timeouts (seconds) unreachable. A newOptimizerScaleKeeper(sameAbstractKeeperinfrastructure: HA leader gating, delay queue, lifecycle) evaluates each DRA group at its ownsustained-backlog-timeout. It owns both the floor and the demand scaling of DRA groups; the legacy keeper skips them but keeps watching, resuming floor duty if DRA is disabled later. Groups are picked up on create, startup load, and update (covering enabling DRA on an existing group at runtime).Scale-up decision. Per group and evaluation round:
min-parallelismdeficits are satisfied immediately (no timing gate), in K-thread instance units.busy + serviceable PLANNED > effective threads, scale out with an exponential ramp (1, 2, 4, 8 instances) clamped to the actual need — SparkExecutorAllocationManagersemantics, including resetting the ramp when the clamp binds or demand clears.pollTask, so with no optimizer nothing polls, planning never runs, and PLANNED tasks never materialize — pending tables are the only observable demand signal.scheduler-backlog-timeoutbefore the first scale-out; later rounds are spaced bysustained-backlog-timeout, so a trickle drained between rounds never accumulates toward one. Themax-parallelismcap always wins.Quota-mode-aware serviceable demand. A table with a proportional quota (
self-optimizing.quota ≤ 1) counts its whole PLANNED backlog (scaling raises its limit, which isceil(quota × availableCore)); a table with an absolute quota (> 1) counts only free slots, since its fixed limit cannot be raised by scaling.Boot-window accounting. Requested-but-unregistered capacity counts toward effective threads, so a booting pod is not re-requested every round. Registration is optimizer-driven and heartbeat expiry only starts after it, so these entries carry their own boot deadline (3 minutes); a request that never registers is pruned instead of freezing scale-up, and a synchronously failed request is retried on a later round.
Planning-bound guard. Idle threads while tables wait as PENDING and no PLANNED tasks materialize means the bottleneck is the serialized planning (
optimizer.max-planning-parallelism), not thread capacity — scaling out would only add idle threads. The keeper surfaces this as an edge-triggered warning naming the config to raise, and does not scale.The AIP-5 page will be revised alongside this phase: the scaling-granularity section currently refers to
execution-parallel, which is not a group-level property, and will documentdynamic-allocation.executor-parallelisminstead.Out of scope for this phase (later phases): per-optimizer idle tracking, scale-down + graceful drain, and the new gauges/counters (the planning-bound condition is log-only here; metrics land in the observability phase).
Parent issue
AIP-5: #4191
Are you willing to submit PR?
Code of Conduct