Repository navigation
fix(datafusion): size the memory pool by the effective partition count - #9183
Merged
Merged
Conversation
mem_pool_size() multiplied the 150MB per-partition default by target_partition.unwrap_or(1), but when target_partition is None the session config is not overridden and DataFusion falls back to its default target_partitions, which is the available parallelism (number of CPU cores). The FairSpillPool was therefore sized for one partition while sort-heavy plans (e.g. BTree scalar index training) spawned one ExternalSorterMerge consumer per core, each reserving ~40MB unspillable against a 150MB pool. On multi-core machines this caused load-dependent, flaky ResourcesExhausted failures under concurrent create_index workloads (observed in the frag_reuse clone tests, which had been worked around with serial_test). Size the pool by the effective partition count instead: when target_partition is None, multiply the per-partition default by datafusion_common's get_available_parallelism(), the same source DataFusion uses for its target_partitions default. The env-parse-error fallback branch uses the same value, so the two fallbacks agree. Explicit mem_pool_size (option or LANCE_MEM_POOL_SIZE env var) still sets the total pool size, unscaled, and target_partition: Some(n) still yields 150MB * n. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DgAshYD7wVzPVdjXRuWPPs
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The pool now follows the same effective partition count as the DataFusion session while preserving explicit memory and partition overrides, closing the observed fair-share exhaustion without changing plan parallelism.
lance-gatekeeper
Bot
added
the
K-approved
ChilePiquin
approved these changes
Sep 14, 2026
wjones127
approved these changes
Sep 17, 2026
2 of 3 tasks
Xuanwo
pushed a commit
that referenced
this pull request
Sep 30, 2026
) ## Root cause The original CI failure ran before #9183 sized the default memory pool by the effective DataFusion partition count. New production evidence on #9402 exposed a remaining failure: concurrent scalar BTree trainings use spilling sorts, but `execute_plan` reused a cached `SessionContext` and its single `FairSpillPool`. Each `ExternalSorterMerge` can need up to 40 MiB of non-spillable memory when it re-reserves after a spill. Several trainings can therefore exhaust the shared pool even when each sort fits by itself. ## Fix Give each spilling caller a fresh session and bounded pool; continue caching non-spilling sessions. Keep the named 40 MiB merge reservation limit and the partition headroom regression guard. The new regression holds three spilling contexts with the same 150 MiB configuration at once and reserves 100 MiB in each; the old shared-pool behavior fails on the second reservation. Each spilling execution retains its configured memory-pool and temporary-disk budgets, but concurrent executions can consume multiples of `LANCE_MEM_POOL_SIZE` and `LANCE_MAX_TEMP_DIRECTORY_SIZE`. On resource-constrained hosts, limit concurrent index builds or size those per-execution budgets accordingly. ## Validation - `cargo test -p lance-datafusion` (183 unit tests and 5 doctests passed; 3 existing ignored doctests) - `cargo test -p lance --test scalar_index_spill` (passed) - `cargo test -p lance --features slow_tests --test integration_tests query::primitives::test_query_integer` (8 passed, including `case_7_uint32`) - `cargo fmt --all -- --check` - `cargo clippy --all --tests --benches -- -D warnings` - `RUSTDOCFLAGS='-D warnings' cargo doc --workspace --no-deps` Fixes #9402 <!-- lance-gatekeeper-fix:v1 agent=7ba7a17aff9f983820ad859ecc276c6e generation=1 --> --------- Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LanceExecutionOptions::mem_pool_size()multiplies the 150MB per-partition default bytarget_partition.unwrap_or(1), but whentarget_partitionisNonethe session it serves does not run with 1 partition:new_session_contextskipswith_target_partitions, so DataFusion falls back to its own default,get_available_parallelism()(all CPU cores).The result is a
FairSpillPoolsized for one partition feeding a plan that runs one partition per core. A sort-heavy plan (BTree scalar index training, for example) spawns oneExternalSorterMergeper partition, each reserving ~40MB that cannot spill; on a 12-core machine that is ~480MB of demand against a 150MB pool. The failure is load-dependent and flaky:We hit this repeatedly in
lancefrag_reuse clone tests whenever severalcreate_indexcalls run concurrently, and worked around it there withserial_test. Any concurrent index-training workload can hit it.Fix
mem_pool_size()now multiplies by the effective partition count:target_partition.unwrap_or_else(get_available_parallelism), using the exact samedatafusion_common::utils::get_available_parallelismthat DataFusion's session default uses, so the pool and the session can no longer disagree. Behavior is unchanged for every explicit configuration:target_partition: Some(n)still yields 150MB * nmem_pool_sizeoption still sets the total pool, unscaledLANCE_MEM_POOL_SIZEenv value still sets the total pool, unscaledValidation
lance-datafusion144 + 5 tests green, including the sizing test updated to derive its expectedNonevalue from the same parallelism source (no hardcoded core count); fmt and clippy clean;cargo check -p lanceclean.🤖 Generated with Claude Code
https://claude.ai/code/session_01DgAshYD7wVzPVdjXRuWPPs