Skip to content

perf(#8333): index ranges served by a scan or in physical order run in the parallel scan's workers - #8602

Merged
lvca merged 5 commits into
mainfrom
fix/8333-index-range-parallel-scan
Sep 28, 2026
Merged

lvca merged 5 commits into
mainfrom
fix/8333-index-range-parallel-scan

Conversation

@lvca

@lvca lvca commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Follow-up of #8333 after #8524 (reopened on the reporter's comment): with scans parallel, an index on the filtered property made a non-selective range query slower than the same query without the index.

What was wrong

  • Full-scan branch. The adaptive index fetch did switch to a scan of the type, but the aggregation is planned in parallel only over a type scan directly upstream, and EXTRACT VALUE FROM INDEX ENTRY / FILTER ITEMS BY TYPE sat in between: projections and aggregation ran on the query thread.
  • Physical-order branch. The records were loaded one lookupByRID() at a time on the query thread. Each lookup takes the database read lock, checks permissions, bumps a shared AtomicLong and copies the record content, so even spread over workers it stopped scaling past ~4 threads.
  • Threshold. A fixed 25% ignored that the scan is split across W workers while the index entries are always read by one thread (the LSM cursor alone costs ~0.09 µs per entry).

Fix

  • ParallelAggregationSource (type scan, filtered type scan, index fetch): the parallel aggregation reads it through the row filters an index fetch leaves behind it (FILTER ITEMS BY TYPE, FILTER ITEMS WHERE for the conditions the index does not answer). The workers evaluate those filters on their own copies of the AST. The full-scan branch hands the whole pipeline to the parallel scan of the type (question 1 of the comment).
  • The physical-order branch cuts the sorted record addresses into slices, and the workers of a parallel scan load one slice each. A range too large to hold is loaded chunk after chunk, with the partial aggregations carried from one round to the next so the groups keep the sequential order. Rows re-sorted by an ORDER BY are loaded by the workers too. PROFILE shows served by physical order (N entries matched), loaded in parallel.
  • LocalBucket.iterator(positions, from, to): a positions mode of BucketIterator. It reads every page once and builds its records in batches under one read lock, exactly as the page scan does (placeholders, multi-page records, deletions, transaction pages and read events are handled by the same code, now extracted into readSlot()). The sequential SQL path and OpenCypher's NodeIndexRangeScan use it too.
  • arcadedb.queryIndexMaxSelectivity (question 2 of the comment) is now the share for a scan on one thread, default 0.6, divided by (1 + W) / 2 for a scan on W workers: 60% of the type sequential, 24% on 4 workers, 6% on 18. That formula fits the crossovers measured on 2M rows in cache and 4M rows under maxPageRAM=256.

Behavior change: a record deleted between the index read and the load is now skipped silently, as a scan skips it, instead of logging a WARNING per record.

Numbers

Reporter's IndexVsParallelScan repro, 2M rows, producer pool capped at 4 threads, same machine, p50 of 7:

filter no index index, main index, this PR
96% (Q1) 211 ms 643 ms 247 ms
14% (Q6) 98 ms 131 ms 59 ms
1.2% 84 ms 9 ms 4 ms

The 96% case still pays for reading the index entries up to the threshold before switching to the scan (about 40-50 ms here). Deciding without walking the entries needs a range-cardinality estimate from the LSM index, which is a separate change.

Tests

  • Issue8333IndexRangeParallelTest (new): both branches aggregate in the workers, residual conditions and the type check included; groups come out in the sequential order; chunked ranges aggregate round after round; ORDER BY rows are loaded in parallel; the sub-type check; inside a transaction the load stays sequential and sees its own changes; the worker-scaled threshold. Every answer is compared with the same query run sequentially and without the index.
  • BucketPositionsIteratorTest (new): positions and slices, one page lookup per page, duplicates, deleted/moved/multi-page records, positions past the bucket, transaction changes.
  • Issue8333CypherIndexRangeSelectivityTest: the wide-range case moved from 50% to 70% of the label, since the label scan is sequential and gives way at 60% now.
  • Existing Issue8333IndexRangeSelectivityTest and Issue8523ParallelScanAggregationTest, plus the engine unit-test lane.

Docs: settings.adoc and concepts/indexes.adoc updated in arcadedb-docs.

…n the parallel scan's workers

After #8524 made scans parallel, an index on the filtered property made a non-selective range slower than the
query without it: the adaptive index fetch served the rows through steps the parallel aggregation could not see
through, so the projections and the aggregation ran on the query thread, and the physical-order branch loaded its
records one lookupByRID() at a time on that thread too.

- The parallel aggregation now reads a ParallelAggregationSource through the row filters an index fetch leaves
  behind it (FILTER ITEMS BY TYPE, the conditions the index does not answer), which the workers evaluate on their
  own copies. The full-scan branch hands the whole pipeline to the parallel scan of the type.
- The physical-order branch cuts the sorted record addresses in slices that the workers of a parallel scan load,
  chunk after chunk for a range too large to hold, the partial aggregations carrying on from round to round so the
  groups keep the sequential order. Rows re-sorted by an ORDER BY are loaded by the workers too.
- Records are loaded through LocalBucket.iterator(positions, from, to), a positions mode of BucketIterator: every
  page read once and its records built in batches under one read lock, as a scan builds them, instead of a read
  lock, a permission check, a shared counter and a content copy per record (the load did not scale past ~4 workers).
  The sequential SQL path and OpenCypher's NodeIndexRangeScan use it as well.
- arcadedb.queryIndexMaxSelectivity is now the share for a scan on one thread (default 0.6), divided by
  (1 + W) / 2 for a scan on W workers, since the index entries are read by one thread whatever the parallelism:
  24% of the type on 4 workers, 6% on 18, as measured in and out of the page cache.
@lvca lvca added this to the 26.10.1 milestone Sep 28, 2026
@lvca lvca self-assigned this Sep 28, 2026
@coderabbitai

coderabbitai Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: a3d6738d-1bc1-49ef-bf79-1e08211071eb

📥 Commits

Reviewing files that changed from the base of the PR and between d895a59 and f6da4a2.

📒 Files selected for processing (18)
  • engine/src/main/java/com/arcadedb/GlobalConfiguration.java
  • engine/src/main/java/com/arcadedb/engine/BucketIterator.java
  • engine/src/main/java/com/arcadedb/engine/LocalBucket.java
  • engine/src/main/java/com/arcadedb/query/opencypher/executor/operators/NodeIndexRangeScan.java
  • engine/src/main/java/com/arcadedb/query/sql/executor/AggregateProjectionCalculationStep.java
  • engine/src/main/java/com/arcadedb/query/sql/executor/FetchFromTypeExecutionStep.java
  • engine/src/main/java/com/arcadedb/query/sql/executor/FetchFromTypeWithFilterStep.java
  • engine/src/main/java/com/arcadedb/query/sql/executor/FilterByTypeStep.java
  • engine/src/main/java/com/arcadedb/query/sql/executor/FilterStep.java
  • engine/src/main/java/com/arcadedb/query/sql/executor/GetValueFromIndexEntryStep.java
  • engine/src/main/java/com/arcadedb/query/sql/executor/ParallelAggregationSource.java
  • engine/src/main/java/com/arcadedb/query/sql/executor/ParallelTypeScan.java
  • engine/src/main/java/com/arcadedb/query/sql/executor/PhysicalOrderRidBuffer.java
  • engine/src/main/java/com/arcadedb/query/sql/executor/PhysicalOrderRidFetcher.java
  • engine/src/main/java/com/arcadedb/query/sql/executor/PhysicalOrderSliceStep.java
  • engine/src/test/java/com/arcadedb/engine/BucketPositionsIteratorTest.java
  • engine/src/test/java/com/arcadedb/query/opencypher/Issue8333CypherIndexRangeSelectivityTest.java
  • engine/src/test/java/com/arcadedb/query/sql/executor/Issue8333IndexRangeParallelTest.java

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 1 remain after this review.


📝 Walkthrough

Walkthrough

The changes add position-based bucket iteration and parallel loading of physical-order index ranges. Query execution can use those scans in parallel aggregation, including across multiple scan rounds and supported type and WHERE filters.

Changes

Index-Range Aggregation

Layer / File(s) Summary
Bucket-position iteration
engine/src/main/java/com/arcadedb/engine/BucketIterator.java, engine/src/main/java/com/arcadedb/engine/LocalBucket.java, engine/src/test/java/com/arcadedb/engine/BucketPositionsIteratorTest.java
BucketIterator and LocalBucket accept sorted position ranges. Tests cover ordering, slices, record changes, and transaction-local updates.
Physical-order range loading
engine/src/main/java/com/arcadedb/GlobalConfiguration.java, engine/src/main/java/com/arcadedb/query/sql/executor/PhysicalOrderRidBuffer.java, engine/src/main/java/com/arcadedb/query/sql/executor/PhysicalOrderRidFetcher.java, engine/src/main/java/com/arcadedb/query/sql/executor/PhysicalOrderSliceStep.java, engine/src/main/java/com/arcadedb/query/sql/executor/ParallelTypeScan.java, engine/src/main/java/com/arcadedb/query/opencypher/executor/operators/NodeIndexRangeScan.java
The fetcher and buffer expose physical-order bucket-position slices and pass-through entries. Slice steps load records, while scan thresholds and work-unit sizing account for workers. The selectivity default changes to 0.6f.
Parallel scan integration
engine/src/main/java/com/arcadedb/query/sql/executor/ParallelAggregationSource.java, engine/src/main/java/com/arcadedb/query/sql/executor/FetchFromType*Step.java, engine/src/main/java/com/arcadedb/query/sql/executor/Filter*Step.java, engine/src/main/java/com/arcadedb/query/sql/executor/GetValueFromIndexEntryStep.java
Query steps expose parallel scan planning. GetValueFromIndexEntryStep plans physical-order slices and scan rounds, and returns fetched records directly.
Parallel aggregation and validation
engine/src/main/java/com/arcadedb/query/sql/executor/AggregateProjectionCalculationStep.java, engine/src/test/java/com/arcadedb/query/sql/executor/Issue8333IndexRangeParallelTest.java, engine/src/test/java/com/arcadedb/query/opencypher/Issue8333CypherIndexRangeSelectivityTest.java
Aggregation discovers eligible scan sources, applies captured type and WHERE filters in workers, and aggregates successive scan rounds. Tests compare execution results and plans across indexed, sequential, and scan paths.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant AggregateProjectionCalculationStep
  participant GetValueFromIndexEntryStep
  participant PhysicalOrderRidFetcher
  participant ParallelTypeScan
  participant PhysicalOrderSliceStep
  participant LocalBucket
  participant BucketIterator
  AggregateProjectionCalculationStep->>GetValueFromIndexEntryStep: request parallel scan plan
  GetValueFromIndexEntryStep->>PhysicalOrderRidFetcher: obtain sorted bucket-position slices
  GetValueFromIndexEntryStep->>ParallelTypeScan: plan scan units
  ParallelTypeScan->>PhysicalOrderSliceStep: provide physical-order work units
  PhysicalOrderSliceStep->>LocalBucket: request positional iterator
  LocalBucket->>BucketIterator: create iterator for position range
  BucketIterator-->>PhysicalOrderSliceStep: return records
  PhysicalOrderSliceStep-->>AggregateProjectionCalculationStep: supply rows for aggregation
Loading

Suggested reviewers: robfrank

Merge Risk: ⚪ Minimal · up to f6da4

This change speeds up aggregation over index ranges by running parallel scans and loading records in physical order. It also raises the default index-selectivity threshold, so more queries will switch to a scan. The added tests compare these results with sequential execution, and no concrete defect remains at this commit.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to f6da4

Record-read permission checks appear to remain in place, but the new multi-round path leaves uncertainty about resource cleanup when a query is interrupted.

Retained concerns

  • Medium · reliability · inferred: An interruption while filling a later index-range chunk leaves its source open until the enclosing execution step is closed. The new parallel aggregation rounds expand reliance on that outer cleanup path, whose execution after failure was not established; repeated aborted queries could impair resource containment.
Security review details

Security Blast Radius

  • inferred — Queries eligible for physical-order loading can now use multiple record-loading workers and successive aggregation rounds. The examined paths do not establish a new cross-database or elevated-privilege route.

Trust Boundaries and Controls

  • observed — Parallel scan workers capture and install the requesting user in their thread-local database context, then remove that context during cleanup. Positional bucket iteration performs the read-permission check under that context; the previous content-loading lookup also delegated permission enforcement to the bucket.

Resilience and Maintainability Implications

  • inferred — Interrupted chunk filling is a conditional availability exposure rather than a demonstrated leak: an active index source remains dependent on outer step closure, and closure after every exceptional or cancelled query was not proven.

Hardening Proposals

  • proposed — Make ownership of the active chunk source explicit on cancellation and exceptional exits, or establish that every enclosing query failure closes the execution step.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.41% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 123 functions across 18 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely describes the main performance change: serving index ranges through scans or physical-order loading in parallel workers.
Description check ✅ Passed The description provides a detailed summary, motivation, related issue, implementation details, performance data, tests, and documentation notes. It does not use every template heading and omits the c…
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@mergify

mergify Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

@codacy-production

codacy-production Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 94 complexity

Metric Results
Complexity 94

View in Codacy

🟢 Coverage 91.07% diff coverage · -6.07% coverage variation

Metric Results
Coverage variation ✅ -6.07% coverage variation
Diff coverage ✅ 91.07% diff coverage

View coverage diff in Codacy

Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (d895a59) 195302 163962 83.95%
Head commit (f6da4a2) 228066 (+32764) 177635 (+13673) 77.89% (-6.07%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#8602) 336 306 91.07%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@claude

claude Bot commented Sep 28, 2026

Copy link
Copy Markdown

Reviewed the diff (index range parallel loading, BucketIterator positions mode, and the parallel aggregation pushdown). This is a well-documented, carefully reasoned change with strong test coverage for the scenarios it describes. Two issues below, one of them a real correctness gap.

Bugs

1. PartialAggregation.matches() evaluates residual WHERE conditions without setting $current on the worker context — engine/src/main/java/com/arcadedb/query/sql/executor/AggregateProjectionCalculationStep.java:474-488

Every other place in the engine that evaluates a WHERE clause against a row sets context.setVariable("current", row) immediately before calling matchesFilters:

  • FilterStep.java:75 (context.setVariable("current", nextItem); right before whereClause.matchesFilters(...))
  • ScanWithFilterStep.java:112 and :118

FilterStep's comment even documents why: "Set $current before WHERE evaluation so method calls (e.g. replace()) resolve correctly" — this was fixed once already for a real bug.

The new PartialAggregation.matches() reimplements the same subtype check and WHERE evaluation for the parallel-aggregation workers, but never sets $current on workerContext before calling condition.matchesFilters(row, context). Any residual condition pushed down into this path (the conditions an index fetch or scan doesn't already answer) that relies on $current — a method call, a subquery referencing the current record, etc. — will evaluate against a worker context where $current is stale or unset, while the exact same query run sequentially (or below the parallel-safety threshold) evaluates it correctly. This is a silent correctness divergence between the parallel and sequential paths, not a crash, so it would show up as wrong rows/aggregates rather than a test failure — and none of the residual conditions in the new Issue8333IndexRangeParallelTest cases exercise $current, so the gap isn't currently caught.

Fix is presumably a one-liner: context.setVariable("current", row); before the conditions loop in matches().

2. Silent row loss if ParallelTypeScan.isAllowed() flips mid-query during a chunked physical-order load — GetValueFromIndexEntryStep.java:186 and :255

nextParallelAggregationRound (line 186) and the inline round-advance in parallelPhysicalOrderResultSet (line 255) both call fetcher.nextChunk() first — which unconditionally loads the next chunk's data into the buffer — and only then call planRound(context, false), which re-checks ParallelTypeScan.isAllowed(database) (line 210) for every round, not just the first. QUERY_PARALLEL_SCAN is a SCOPE.DATABASE runtime setting that can be toggled while a query is mid-flight (e.g. a long-lived streamed ResultSet being drained across several syncPull calls). If it flips to disallowed between rounds, planRound returns null, and both call sites treat null exactly like "no more rows": the loop in aggregateInParallel (AggregateProjectionCalculationStep.java while (scan != null)) and the hasNext() loop in parallelPhysicalOrderResultSet both stop, discarding the chunk that was already loaded into the buffer instead of falling back to serving it sequentially. The query completes successfully with fewer rows / an under-counted aggregate, with no exception or log. Narrow trigger, but worth a fallback-to-sequential instead of a silent stop, given the failure mode is silently wrong data rather than a visible error.

Minor

  • GetValueFromIndexEntryStep.planRound(context, true) calls fetcher.slices(entriesPerUnit, null) once to count slices for the "too small to share" check, then unitsOfCurrentChunk calls fetcher.slices(entriesPerUnit, consumer) again to build them. Not a real hot-path cost (the cost is proportional to slice/unit count, not matched entries), so this is a nit rather than a performance concern — could still just early-return the built list when it turns out too small instead of computing twice.

Other observations (no action needed)

  • The BucketIterator refactor (readSlot() extraction) is a clean, faithful lift of the existing scan logic into a shared method used by both the page scan and the new positions mode — the exception handling and corruption-skip comments all still apply correctly to both callers.
  • The (unit << 32 | ordinal) position-packing math used to keep group-first-seen ordering stable across aggregation rounds (roundPosition = units << 32 combined via +) is safe: the low 32 bits of roundPosition are always zero and ordinal never sets bit 32, so addition and bitwise-OR are equivalent here.
  • Threshold/config changes (QUERY_INDEX_MAX_SELECTIVITY default 0.25 → 0.6, divided by (1+W)/2) are clearly explained and backed by the numbers in the PR description.
  • Test coverage is thorough for the documented scenarios (both branches, chunking, ORDER BY, sub-types, in-transaction fallback, worker-scaled threshold) — the two gaps above are specifically the kind of edge case that wouldn't be caught by scenario-based tests like these.

🤖 Generated with Claude Code

… parallel to its last chunk

Review follow-up: every round after the first re-checked whether parallel scans were allowed, after the next chunk
had been loaded, and read a refusal as no entry left, so parallel scans disabled mid-query silently dropped the
remaining chunks. The first round now decides for the whole execution. Also covers a residual condition reading
$current in the workers.
@claude

claude Bot commented Sep 28, 2026

Copy link
Copy Markdown

Review

This is a well-documented, carefully-reasoned fix for the #8333/#8524 regression (an index on a filtered column made a non-selective query slower than no index at all, once scans went parallel). I traced through BucketIterator's new positions mode, PhysicalOrderRidFetcher/PhysicalOrderRidBuffer's chunk/slice reuse across rounds, ParallelTypeScan's row-mode vs. aggregate-mode exhaustion guarantees, and AggregateProjectionCalculationStep's multi-round partial-aggregation merge/position-encoding logic. The core concurrency invariants hold up:

  • The unitIndex << 32 | ordinal position encoding used for sequential-order group tracking composes correctly across rounds (roundPosition = units << 32 cleanly offsets the unit index because the low 32 bits are always zero).
  • The apparent risk of a data race on the shared positions[] arrays when a chunk is replaced (fetcher.nextChunk() reusing PhysicalOrderRidBuffer's arrays) turns out to be safe: row-mode's pull() only reports exhaustion once every unit's channel has drained its END_OF_UNIT marker, which a producer sends only after it's done reading the arrays, and ParallelTypeScan.aggregate() is synchronous/blocking per round, so nextChunk() never runs concurrently with a worker still touching the previous chunk's slices.
  • parallelPhysicalOrderResultSet's "page stopped short -> round exhausted" assumption holds because the page's dispatched cap and the outer fetched counter move in lockstep, so hitting the page-size cap and hitting genuine round exhaustion cannot be confused, given the outer while (fetched < nRecords ...) guard.

Findings

  1. GlobalConfiguration.java:909 - QUERY_INDEX_MAX_SELECTIVITY's default moved from 0.25f to 0.6f. This is called out in the PR description and is deliberate, but note it is a global change to index-vs-scan plan selection, not scoped only to the new parallel paths: for a sequential scan (no parallelism), a range matching between 25% and 60% of a type now stays on the index/physical-order path where it previously fell back to a full scan. Only the two Issue8333* tests in this PR were adjusted for the new threshold - worth double-checking no other existing test (or downstream consumer) asserts plan choice/row order in that selectivity band, since it will silently pick a different strategy now.

  2. NodeIndexRangeScan.java (fetchAdaptively, around line 317-320) - the catch (RecordNotFoundException e) around addVertex(((Identifiable) entry).asVertex()) looks dead now: fetcher.nextRecord() no longer returns a bare RID that needs a fresh lookupByRID() at this call site (a gone record is now skipped inside BucketIterator itself, or the entry is already a resolved pass-through value). The comment "Deleted since the index answered: nothing to match" no longer describes a reachable path. Not a bug, but worth removing or re-verifying so the next reader is not misled about what this guards against.

  3. GetValueFromIndexEntryStep.planRound() - the "first chunk too small to share" check (fetcher.slices(entriesPerUnit, null) < 2) only counts bucket-position slices and ignores the pass-through unit that unitsOfCurrentChunk() appends separately. A first chunk with exactly one position slice and non-empty pass-through entries would actually produce 2 units via unitsOfCurrentChunk(), but planRound bails out to sequential execution anyway, missing an available parallelization opportunity. Minor, perf-only, not a correctness issue.

Other notes

  • Test coverage is solid: BucketPositionsIteratorTest exercises the new positions-mode iterator directly (page-cache hit counting to verify the "one lookup per page, not per record" claim, duplicates, deletions/placeholders/multi-page records, out-of-range positions, transaction visibility), and Issue8333IndexRangeParallelTest cross-checks both the scan and physical-order branches against sequential execution and against the no-index baseline, including chunked rounds and sub-type filtering.
  • The BucketIterator scan-loop refactor into readSlot() is a faithful mechanical extraction (continue -> return) with no behavior change found.
  • Docs (settings.adoc, concepts/indexes.adoc) are updated in the companion arcadedb-docs PR per the description.

Solid numbers in the writeup (643ms -> 247ms on the 96%-selectivity case, 9ms -> 4ms on the 1.2% case), and the explanation of why the old threshold/approach broke down (index entries always read by one thread regardless of scan parallelism) is clear and testable.

@claude

claude Bot commented Sep 28, 2026

Copy link
Copy Markdown

Review

I read through the full diff and cross-checked the dependent call sites (FilterByTypeStep, NodeIndexRangeScan, ParallelTypeScan, AggregateProjectionCalculationStep, PhysicalOrderRidFetcher). I was not able to run mvn verify in this environment (sandboxed, no approval for the build), so this is a static review, not a verified-green run.

Overall

This is a well-scoped, carefully engineered fix. A few things stood out positively:

  • The BucketIterator positions-mode refactor (readSlot() extraction) preserves the original scan semantics exactly: the exception handling (RecordNotFoundException, SerializationException, corrupted slot/placeholder handling, multi-page/placeholder resolution) is moved into readSlot() unchanged, and the outer try/finally in fetchNext() still does the currentRecordInPage bookkeeping. Good, low-risk refactor.
  • The parallel-worker row filter in AggregateProjectionCalculationStep.PartialAggregation.matches() mirrors FilterByTypeStep's sequential logic exactly (isElement() -> getType() -> isSubTypeOf(), null type rejected), so the "type check runs in the workers" behavior should be equivalent to running it on the query thread.
  • The "once parallel, always parallel" fix in the last commit (deciding on the first chunk only, via parallelDecided) correctly addresses the mid-query QUERY_PARALLEL_SCAN toggle bug, and it's covered by disablingParallelScansMidQueryLosesNoChunk.
  • ParallelTypeScan.ofUnits() deliberately skips the isAllowed()/units.size() < 2 guard that plan() applies for continuation rounds; that's intentional per the docstring ("a round of an execution already parallel must not fall back to nothing"), consistent with the fix above.
  • Test coverage is thorough: both branches (scan fallback and physical order), chunked rounds, ORDER BY re-sort, sub-type filtering, in-transaction fallback to sequential, and the worker-scaled threshold formula are all exercised, and every case is cross-checked against the same query run sequentially and without the index.

Minor observations / questions (non-blocking)

  1. GetValueFromIndexEntryStep.chooseStrategy picks the scan threshold using ParallelTypeScan.plannedWorkers(db, filterBucketIds), which predicts the parallelism of a full type scan. But the actual physical-order-chunked load can still end up sequential if the first chunk is too small to split (planRound's fetcher.slices(entriesPerUnit, null) < 2 check). In that edge case the threshold is computed assuming more speedup than the query actually gets, so the index could be dropped for a scan slightly earlier than ideal. Probably fine in practice (self-correcting, and the description already frames the formula as an empirical fit), but worth a sanity check on a range that's borderline-sized for splitting.

  2. QUERY_INDEX_MAX_SELECTIVITY default changes from 0.25f to 0.6f and its meaning changes (now "share for a scan on one thread," divided by workers). This is a behavior-changing default worth calling out in release notes, for anyone who benchmarked against the old default.

  3. Docs: the PR body mentions settings.adoc / concepts/indexes.adoc are updated in arcadedb-docs separately - worth confirming that companion PR merges alongside this one so the shipped default doesn't go undocumented for a release.

  4. Small style nit: AggregateProjectionCalculationStep.containsSame() does an O(n) identity scan per partial; n is bounded by worker count so this is negligible, just noting it in case the pattern gets reused somewhere hotter later.

Security / performance

No security concerns (no new user-input handling; all changes are internal engine data-structure/scheduling code). Performance-wise this is the point of the PR, and the reported numbers (GetValueFromIndexEntryStep / ParallelTypeScan) look like a real improvement for the reported regression; the reasoning about the index read being single-threaded while the scan splits across workers is sound, and the crossover formula is documented with its empirical basis.

Verdict

No blocking issues found in review. Nice work on the thread-safety reasoning (shared long[] position slices safely partitioned across workers, mutated between rounds only after every worker of a round has finished) and on keeping the parallel path's output byte-for-byte/order-for-order identical to the sequential path.

…rd addresses as a unit; comment on what the Cypher catch still guards
@claude

claude Bot commented Sep 28, 2026

Copy link
Copy Markdown

Review

This is a well-engineered, carefully-reasoned performance fix. I read through the full diff (GlobalConfiguration, BucketIterator/LocalBucket, ParallelAggregationSource/AggregateProjectionCalculationStep, GetValueFromIndexEntryStep, ParallelTypeScan, PhysicalOrderRidBuffer/PhysicalOrderRidFetcher/PhysicalOrderSliceStep, NodeIndexRangeScan, and both new test classes) and traced the trickiest correctness properties by hand rather than trusting the comments at face value:

Verified correct:

  • Row ordering across parallel workers. ParallelTypeScan.aggregate() encodes (unitIndex << 32) | ordinal per row, and AggregateProjectionCalculationStep adds a roundPosition = priorUnits << 32 offset per chunk round. I checked that unitsOfCurrentChunk() builds units in the same bucket/position order that PhysicalOrderRidFetcher.nextRecord() would traverse sequentially (both go through PhysicalOrderRidBuffer.slices()), and that pass-through entries are appended last in both paths. So "groups come out in the sequential order" holds, including across chunked rounds, not just within one.
  • QUERY_INDEX_MAX_SELECTIVITY / worker-scaled threshold formula. share = min(1, maxSelectivity * 2 / (1 + max(1, scanWorkers))) reproduces exactly the 60% / 24% / 6% figures in the docstring and the test's scanThreshold assertions for 1/4/18 workers, and correctly clamps to 100% when the setting is pushed above 1.
  • SqlAstInspector.isParallelSafe(preProjection, projection, groupBy, conditions) – the new 4th argument (a List<WhereClause>) works with the existing varargs+reflection signature without any change needed to that file, since allNodesMatch already walks Collection generically. Good reuse, no silent gap in the parallel-safety check for the residual conditions now evaluated in workers.
  • Buffer/array reuse safety. PhysicalOrderRidBuffer's backing arrays are shared by slices "until nextChunk()" per the javadoc; nextChunk() is only ever called after a round's ParallelTypeScan has fully closed (parallelRound.close() before fetcher.nextChunk() in parallelPhysicalOrderResultSet, and similarly via nextParallelAggregationRound), so there's no window where a worker could still be reading a slice while the next chunk overwrites it.
  • The "once parallel, always" fix (4th commit) is correct: parallelDecided/parallelRounds are set once from the first chunk and later chunks skip the isAllowed/size-threshold re-check, so a mid-query config flip (as disablingParallelScansMidQueryLosesNoChunk tests) can no longer be misread as "no entries left."
  • BucketIterator's new positions-mode cleanly reuses readSlot() (extracted, not duplicated) for placeholders/multi-page/transaction-local records; the "gone" cases (dropped page, deleted slot) mirror how a page scan already treats them.

Minor / non-blocking observations:

  • GetValueFromIndexEntryStep has picked up a lot of responsibility now (strategy choice, scan/physical-order/chunked-parallel dispatch, round planning, pass-through handling). It's well-commented and each piece is small, but if this area grows further it might be worth splitting the "parallel round" bookkeeping (planRound/parallelPhysicalOrderResultSet/nextParallelAggregationRound) into its own small class.
  • Behavior change worth calling out explicitly (even though it's arguably an improvement): the old load(rid, context) path logged a WARNING for a record deleted since the index read it; the new BucketIterator positions path skips such records silently, consistent with how a plain scan treats them. Good for log noise under concurrent deletes, but it's a visible behavior change from main that isn't mentioned in the PR description.
  • AggregateProjectionCalculationStep.containsSame() is a linear scan over partials, called once per round — bounded by worker count, so it's not a real perf concern, just noting it in case partials ever grows unbounded in a future change.

Test coverage: Strong. Issue8333IndexRangeParallelTest covers both branches (scan fallback and physical-order), chunked rounds, ORDER BY re-sort, sub-type filtering, in-transaction fallback to sequential, and the mid-query parallel-scan-disabled edge case, every one diffed against the same query run sequentially and without the index. BucketPositionsIteratorTest covers duplicates, deleted/moved/multi-page records, positions past the bucket, and transaction-local visibility, plus a cache-hit-count assertion (not wall-clock) to pin the "one page read, not N lookups" claim. No new tests needed a benchmark/slow/vector tag by the size of their fixtures (20k and 3k rows respectively).

Style/process: No System.out left in, no em dashes, no hardcoded ports (n/a here), no new dependencies. I couldn't run mvn -o -pl engine -am verify in this environment, so please make sure the full engine test module (plus opencypher) is green before merging, particularly around the two modified Issue8333*SelectivityTest classes whose thresholds moved.

Overall: I did not find any correctness bugs. This looks ready pending green CI.

@codecov

codecov Bot commented Sep 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.44048% with 59 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.66%. Comparing base (d895a59) to head (f6da4a2).
⚠️ Report is 21 commits behind head on main.

Files with missing lines Patch % Lines
...query/sql/executor/GetValueFromIndexEntryStep.java 71.66% 9 Missing and 8 partials ⚠️
...edb/query/sql/executor/PhysicalOrderSliceStep.java 52.77% 11 Missing and 6 partials ⚠️
...l/executor/AggregateProjectionCalculationStep.java 86.48% 4 Missing and 6 partials ⚠️
.../main/java/com/arcadedb/engine/BucketIterator.java 92.10% 4 Missing and 2 partials ⚠️
.../arcadedb/query/sql/executor/ParallelTypeScan.java 87.80% 2 Missing and 3 partials ⚠️
...db/query/sql/executor/PhysicalOrderRidFetcher.java 86.66% 0 Missing and 4 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #8602      +/-   ##
============================================
+ Coverage     75.64%   75.66%   +0.01%     
+ Complexity     3339     3337       -2     
============================================
  Files          2040     2042       +2     
  Lines        195302   195521     +219     
  Branches      41041    41093      +52     
============================================
+ Hits         147741   147943     +202     
+ Misses        31382    31374       -8     
- Partials      16179    16204      +25     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lvca
lvca merged commit bc519db into main Sep 28, 2026
37 checks passed
@lvca
lvca deleted the fix/8333-index-range-parallel-scan branch September 28, 2026 21:35
tae898 added a commit to humemai/arcadedb-embedded-python that referenced this pull request Sep 29, 2026
…longer matches the scripts, one list of owed work (DECISIONS #123)

A read-only audit of every doc under benchmarks/ against main at 7142db4,
each finding checked before it was applied.

CAMPAIGN: the routine's derive_budgets step names --rows (its default is
September's freeze); the results asset is published AFTER the switch (it
bundles the live page); staging is smoke -> laptop skeleton -> chain with
the six gates; no queue script sources campaign_env.sh; monitoring is
mini_watch.sh; the duplicated qOB paragraph goes; stage state lives in
STATUS.txt and HANDOFF, not here; 17 generated stages plus qOL and qOM;
STRICT_WORKLOADS is three pairs; forty-nine operations; DECISIONS #42 not
the deleted #41; five repin-prep SHAs orphaned by the rebase replaced by
their rebased commits; the one-line-of-development rule states the
repin-prep exception; the 2026-09-19 stage table and its contradictory
"publish at the previous pin" clause are replaced by what the stages
measured. Section 7: items 5 (the fsync facts, formerly only in HANDOFF),
9 (the six fixes of 2026-09-28 and the 10M re-verification HANDOFF alone
carried), 15 (COMPACT TIMESERIES TYPE), 17 (PR ArcadeData#8602), and new items 18-21
gather the owed work that lived in BUGS F66, PAGE-SPEC, FAIRNESS, and
PROTOCOL.

PROTOCOL/PAGE-SPEC/FAIRNESS/COMPARATORS/READING-RESULTS/PUBLISHING/
RESULTS-MAP/README and the SciPy suite's docs: nothing is withheld any more
(ArcadeData#7610 released 2026-09-18; WITHHELD_CELLS and KNOWN_DISAGREEMENTS empty,
the mechanism described once in PUBLISHING); the l_shipdate row states
INDEX_DECISIONS; Q6/Q9 excluded at sf1full; October's l2olap is four
cells; budgets per #106; gates that need paper.tex retagged REMEMBERED;
the duplicated planned/retired pair and document-table opener go; laptop
durability ratios replaced by the durability table; a Kuzu bullet deleted;
three orphaned table rows moved back; F43 -> F42; ArcadeData#7611 filed and fixed in
the pin; the jvm payload check runs in the bindings tests; the landing
pulls sparse_mp; per-lane result files are September's.

Code the docs promised: publish_results_asset reads GATE_STATUS.txt where
make_paper_tables writes it (the published manifest said gates: null); the
preview inventory's header names the preview payload; the paper-era
tables_summary.md crib (read by nothing) and its writer are removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QHcrtLcfHdr5UhbCdnpM77
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant