Repository navigation
perf(#8333): index ranges served by a scan or in physical order run in the parallel scan's workers - #8602
Conversation
…n the parallel scan's workers After #8524 made scans parallel, an index on the filtered property made a non-selective range slower than the query without it: the adaptive index fetch served the rows through steps the parallel aggregation could not see through, so the projections and the aggregation ran on the query thread, and the physical-order branch loaded its records one lookupByRID() at a time on that thread too. - The parallel aggregation now reads a ParallelAggregationSource through the row filters an index fetch leaves behind it (FILTER ITEMS BY TYPE, the conditions the index does not answer), which the workers evaluate on their own copies. The full-scan branch hands the whole pipeline to the parallel scan of the type. - The physical-order branch cuts the sorted record addresses in slices that the workers of a parallel scan load, chunk after chunk for a range too large to hold, the partial aggregations carrying on from round to round so the groups keep the sequential order. Rows re-sorted by an ORDER BY are loaded by the workers too. - Records are loaded through LocalBucket.iterator(positions, from, to), a positions mode of BucketIterator: every page read once and its records built in batches under one read lock, as a scan builds them, instead of a read lock, a permission check, a shared counter and a content copy per record (the load did not scale past ~4 workers). The sequential SQL path and OpenCypher's NodeIndexRangeScan use it as well. - arcadedb.queryIndexMaxSelectivity is now the share for a scan on one thread (default 0.6), divided by (1 + W) / 2 for a scan on W workers, since the index entries are read by one thread whatever the parallelism: 24% of the type on 4 workers, 6% on 18, as measured in and out of the page cache.
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (18)
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 1 remain after this review. 📝 WalkthroughWalkthroughThe changes add position-based bucket iteration and parallel loading of physical-order index ranges. Query execution can use those scans in parallel aggregation, including across multiple scan rounds and supported type and WHERE filters. ChangesIndex-Range Aggregation
Priority: ➖ Normal Estimated code review effort: 4 (Complex) | ~45 minutes Change: Bug fix Sequence Diagram(s)sequenceDiagram
participant AggregateProjectionCalculationStep
participant GetValueFromIndexEntryStep
participant PhysicalOrderRidFetcher
participant ParallelTypeScan
participant PhysicalOrderSliceStep
participant LocalBucket
participant BucketIterator
AggregateProjectionCalculationStep->>GetValueFromIndexEntryStep: request parallel scan plan
GetValueFromIndexEntryStep->>PhysicalOrderRidFetcher: obtain sorted bucket-position slices
GetValueFromIndexEntryStep->>ParallelTypeScan: plan scan units
ParallelTypeScan->>PhysicalOrderSliceStep: provide physical-order work units
PhysicalOrderSliceStep->>LocalBucket: request positional iterator
LocalBucket->>BucketIterator: create iterator for position range
BucketIterator-->>PhysicalOrderSliceStep: return records
PhysicalOrderSliceStep-->>AggregateProjectionCalculationStep: supply rows for aggregation
Suggested reviewers: Merge Risk: ⚪ Minimal · up to This change speeds up aggregation over index ranges by running parallel scans and loading records in physical order. It also raises the default index-selectivity threshold, so more queries will switch to a scan. The added tests compare these results with sequential execution, and no concrete defect remains at this commit. Security Architecture ReviewSecurity architecture risk: 🟡 Moderate · up to Record-read permission checks appear to remain in place, but the new multi-round path leaves uncertainty about resource cleanup when a query is interrupted. Retained concerns
Security review detailsSecurity Blast Radius
Trust Boundaries and Controls
Resilience and Maintainability Implications
Hardening Proposals
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again. |
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 94 |
🟢 Coverage 91.07% diff coverage · -6.07% coverage variation
Metric Results Coverage variation ✅ -6.07% coverage variation Diff coverage ✅ 91.07% diff coverage Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (d895a59) 195302 163962 83.95% Head commit (f6da4a2) 228066 (+32764) 177635 (+13673) 77.89% (-6.07%) Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#8602) 336 306 91.07% Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
|
Reviewed the diff (index range parallel loading, Bugs1. Every other place in the engine that evaluates a
The new Fix is presumably a one-liner: 2. Silent row loss if
Minor
Other observations (no action needed)
🤖 Generated with Claude Code |
… parallel to its last chunk Review follow-up: every round after the first re-checked whether parallel scans were allowed, after the next chunk had been loaded, and read a refusal as no entry left, so parallel scans disabled mid-query silently dropped the remaining chunks. The first round now decides for the whole execution. Also covers a residual condition reading $current in the workers.
|
Review This is a well-documented, carefully-reasoned fix for the #8333/#8524 regression (an index on a filtered column made a non-selective query slower than no index at all, once scans went parallel). I traced through
Findings
Other notes
Solid numbers in the writeup (643ms -> 247ms on the 96%-selectivity case, 9ms -> 4ms on the 1.2% case), and the explanation of why the old threshold/approach broke down (index entries always read by one thread regardless of scan parallelism) is clear and testable. |
ReviewI read through the full diff and cross-checked the dependent call sites ( OverallThis is a well-scoped, carefully engineered fix. A few things stood out positively:
Minor observations / questions (non-blocking)
Security / performanceNo security concerns (no new user-input handling; all changes are internal engine data-structure/scheduling code). Performance-wise this is the point of the PR, and the reported numbers ( VerdictNo blocking issues found in review. Nice work on the thread-safety reasoning (shared |
…rd addresses as a unit; comment on what the Cypher catch still guards
ReviewThis is a well-engineered, carefully-reasoned performance fix. I read through the full diff ( Verified correct:
Minor / non-blocking observations:
Test coverage: Strong. Style/process: No Overall: I did not find any correctness bugs. This looks ready pending green CI. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #8602 +/- ##
============================================
+ Coverage 75.64% 75.66% +0.01%
+ Complexity 3339 3337 -2
============================================
Files 2040 2042 +2
Lines 195302 195521 +219
Branches 41041 41093 +52
============================================
+ Hits 147741 147943 +202
+ Misses 31382 31374 -8
- Partials 16179 16204 +25 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…longer matches the scripts, one list of owed work (DECISIONS #123) A read-only audit of every doc under benchmarks/ against main at 7142db4, each finding checked before it was applied. CAMPAIGN: the routine's derive_budgets step names --rows (its default is September's freeze); the results asset is published AFTER the switch (it bundles the live page); staging is smoke -> laptop skeleton -> chain with the six gates; no queue script sources campaign_env.sh; monitoring is mini_watch.sh; the duplicated qOB paragraph goes; stage state lives in STATUS.txt and HANDOFF, not here; 17 generated stages plus qOL and qOM; STRICT_WORKLOADS is three pairs; forty-nine operations; DECISIONS #42 not the deleted #41; five repin-prep SHAs orphaned by the rebase replaced by their rebased commits; the one-line-of-development rule states the repin-prep exception; the 2026-09-19 stage table and its contradictory "publish at the previous pin" clause are replaced by what the stages measured. Section 7: items 5 (the fsync facts, formerly only in HANDOFF), 9 (the six fixes of 2026-09-28 and the 10M re-verification HANDOFF alone carried), 15 (COMPACT TIMESERIES TYPE), 17 (PR ArcadeData#8602), and new items 18-21 gather the owed work that lived in BUGS F66, PAGE-SPEC, FAIRNESS, and PROTOCOL. PROTOCOL/PAGE-SPEC/FAIRNESS/COMPARATORS/READING-RESULTS/PUBLISHING/ RESULTS-MAP/README and the SciPy suite's docs: nothing is withheld any more (ArcadeData#7610 released 2026-09-18; WITHHELD_CELLS and KNOWN_DISAGREEMENTS empty, the mechanism described once in PUBLISHING); the l_shipdate row states INDEX_DECISIONS; Q6/Q9 excluded at sf1full; October's l2olap is four cells; budgets per #106; gates that need paper.tex retagged REMEMBERED; the duplicated planned/retired pair and document-table opener go; laptop durability ratios replaced by the durability table; a Kuzu bullet deleted; three orphaned table rows moved back; F43 -> F42; ArcadeData#7611 filed and fixed in the pin; the jvm payload check runs in the bindings tests; the landing pulls sparse_mp; per-lane result files are September's. Code the docs promised: publish_results_asset reads GATE_STATUS.txt where make_paper_tables writes it (the published manifest said gates: null); the preview inventory's header names the preview payload; the paper-era tables_summary.md crib (read by nothing) and its writer are removed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QHcrtLcfHdr5UhbCdnpM77
Follow-up of #8333 after #8524 (reopened on the reporter's comment): with scans parallel, an index on the filtered property made a non-selective range query slower than the same query without the index.
What was wrong
EXTRACT VALUE FROM INDEX ENTRY/FILTER ITEMS BY TYPEsat in between: projections and aggregation ran on the query thread.lookupByRID()at a time on the query thread. Each lookup takes the database read lock, checks permissions, bumps a sharedAtomicLongand copies the record content, so even spread over workers it stopped scaling past ~4 threads.Fix
ParallelAggregationSource(type scan, filtered type scan, index fetch): the parallel aggregation reads it through the row filters an index fetch leaves behind it (FILTER ITEMS BY TYPE,FILTER ITEMS WHEREfor the conditions the index does not answer). The workers evaluate those filters on their own copies of the AST. The full-scan branch hands the whole pipeline to the parallel scan of the type (question 1 of the comment).ORDER BYare loaded by the workers too.PROFILEshowsserved by physical order (N entries matched), loaded in parallel.LocalBucket.iterator(positions, from, to): a positions mode ofBucketIterator. It reads every page once and builds its records in batches under one read lock, exactly as the page scan does (placeholders, multi-page records, deletions, transaction pages and read events are handled by the same code, now extracted intoreadSlot()). The sequential SQL path and OpenCypher'sNodeIndexRangeScanuse it too.arcadedb.queryIndexMaxSelectivity(question 2 of the comment) is now the share for a scan on one thread, default 0.6, divided by(1 + W) / 2for a scan on W workers: 60% of the type sequential, 24% on 4 workers, 6% on 18. That formula fits the crossovers measured on 2M rows in cache and 4M rows undermaxPageRAM=256.Behavior change: a record deleted between the index read and the load is now skipped silently, as a scan skips it, instead of logging a WARNING per record.
Numbers
Reporter's
IndexVsParallelScanrepro, 2M rows, producer pool capped at 4 threads, same machine, p50 of 7:mainThe 96% case still pays for reading the index entries up to the threshold before switching to the scan (about 40-50 ms here). Deciding without walking the entries needs a range-cardinality estimate from the LSM index, which is a separate change.
Tests
Issue8333IndexRangeParallelTest(new): both branches aggregate in the workers, residual conditions and the type check included; groups come out in the sequential order; chunked ranges aggregate round after round;ORDER BYrows are loaded in parallel; the sub-type check; inside a transaction the load stays sequential and sees its own changes; the worker-scaled threshold. Every answer is compared with the same query run sequentially and without the index.BucketPositionsIteratorTest(new): positions and slices, one page lookup per page, duplicates, deleted/moved/multi-page records, positions past the bucket, transaction changes.Issue8333CypherIndexRangeSelectivityTest: the wide-range case moved from 50% to 70% of the label, since the label scan is sequential and gives way at 60% now.Issue8333IndexRangeSelectivityTestandIssue8523ParallelScanAggregationTest, plus the engine unit-test lane.Docs:
settings.adocandconcepts/indexes.adocupdated in arcadedb-docs.