fix(index): keep the shuffler scratch directory alive for the whole build

[LuciferYang]

Problem

prepare_vector_segment_build creates the shuffle scratch directory as a local TempStdDir and returns only the shuffler, which carries just the directory path. The guard drops when the helper returns, so the directory is removed before the build writes into it. The shuffle writer recreates it, and from then on nothing owns it. Every vector index build leaks one scratch directory, and each one holds a shuffled copy of the vector column.

Fixes #8992.

What this changes

The helper returns the guard alongside the shuffler, and both callers hold it until the build finishes. The incremental build path and the optimize path already keep their guards in the enclosing function scope and are untouched.

#6980 fixed the same thing for the legacy IVF_HNSW partition staging directory.

Test plan

test_shuffle_scratch_dir_is_not_leaked runs two real index builds in a child process with TMPDIR pointed at an isolated directory, then asserts no scratch directory survives. Negative control: with the guard bound as bare _ at one call site, the test reports leaked 2 scratch directories, one per build. It builds through create_index, so it covers the local path; the distributed path needs precomputed centroids and is not exercised by it.

test_prepare_vector_segment_build_keeps_scratch_dir_alive covers the helper's own contract: it drives the returned shuffler, asserts the shuffle output lands under the returned guard, then drops the guard and asserts the output goes with it.

End to end, one index build with TMPDIR pointed at an empty directory:

	left behind
abcd800cd	.tmpXXXXXX/ with shuffle_data.lance, shuffle_data.spill, shuffle_offsets.lance, shuffle_offsets.spill
this branch	nothing

• cargo test -p lance --lib 3382 passed, 4 ignored
• cargo clippy --all --tests --benches -- -D warnings clean
• cargo fmt --all --check clean

[lance-gatekeeper]

✅ Gate recommendation: approve.

The change restores scratch-directory ownership at the helper boundary and keeps the guard alive through both local and distributed builds, preserving cleanup on success, error, and cancellation. The scoped guard is the appropriate minimal fix; moving ownership into the shuffler and reader types would broaden the contract without improving these caller lifetimes.