Skip to content

fix(#9575): appends no longer arm a full-bucket free-space scan - #9901

Merged
lvca merged 1 commit into
mainfrom
worktree-arcadedb-9575-gather-rescan
Oct 13, 2026
Merged

lvca merged 1 commit into
mainfrom
worktree-arcadedb-9575-gather-rescan

Conversation

@lvca

@lvca lvca commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Follow-up to #9575 / #9581, reported in discussion #9559.

After #9581 the reporter re-ran the 209M vertex import with a 24 GB heap. Memory was fine (GC 1%, heap 6 to 15 GB) but the vertex pass still decayed from 60,000 vertices/s to ~600/s, roughly as 1/N from about 9M vertices on. #9581 fixed the memory side; this is the storage side.

Cause

LocalBucket.updatePageStatistics counted EVERY write as a change to the bucket's free-space statistics, appends included. In an append-only load the map of reusable pages is empty each time an appended page fills, so the next allocation calls gatherPageStatistics(). Once its 5 s throttle is over, that call reads every page of the bucket looking for space an append cannot have freed.

The scan grows with the file. At 1.3M pages it is ~6 minutes of disk reads (~270 us a page uncached), against ~5 s of useful work (~300k vertices) between two scans. That matches the reporter's log exactly: ~300k vertices every 6 minutes at 88M vertices.

Change

  • updatePageStatistics arms the next gather only when a write gives space back or measures a page (delta >= 0). Inserts and enlargements take space, so they cannot leave a page a scan would list, and the pages they fill are already tracked by the incremental map update.
  • gatherPageStatistics() settles "nothing to consume" with a plain read of the counter before the atomic reset. That is the common answer on every allocation made while the map is empty.
  • expireGatherThrottleForTesting() (package-private) so a test does not sleep through the 5 s throttle.

Verification

  • Issue9575AppendOnlyLoadDoesNotRescanBucketTest counts scans, no clock. Appending in five rounds with the throttle expired before each: 6 scans before the fix, 1 after. A second test pins the other side: a delete still arms a gather, and the pages it freed are reused instead of growing the file.
  • Connected suites pass (Issue8660*, Issue6339, Issue6358, Issue6396, Issue8401, Issue9483, Issue9694, Issue7121, Issue6163, SchemaReloadDoesNotEagerScanBuckets, RecordRecycling, RandomDelete), and the whole com.arcadedb.engine package: 1173 tests, 0 failures.
  • 15M vertices of ~1 KB (ids above 2^31), 8 GB heap: gatherPageStatistics was on the importer thread's stack in 5 of 40 samples before the fix and in 0 of 30 after; the vertex pass went from 63.9 s to 53.6 s.

Caveats

  • The decay itself needs a file larger than the OS cache, so it did not reproduce on a machine that caches the whole file (8 us a page cached, 270 us uncached). The reporter's rerun is the real confirmation.
  • The edge pass is not measured. Relocating a vertex frees space on its old page (delta > 0), which can still arm scans. A scan armed by a delete is still O(pages).

Summary by CodeRabbit

  • Bug Fixes
    • Append-only workloads no longer trigger unnecessary bucket scans, improving performance as data grows.
    • Scans still run when deleted space can be reused, helping limit additional bucket growth.

A GraphImporter load of 209M vertices kept 1% GC and a healthy heap, yet its
vertex pass fell from 60,000 to 600 vertices a second (discussion #9559, after
the memory fixes of #9581).

LocalBucket.updatePageStatistics counted every write as a change to the
free-space statistics, appends included. In an append-only load the map of
reusable pages is empty each time an appended page fills, so the next
allocation calls gatherPageStatistics(). Once its 5 s throttle is over the
scan reads EVERY page of the bucket looking for space an append cannot have
freed. The scan grows with the file (about 6 minutes of disk reads at 1.3M
pages, against 5 s of useful work between two scans), so the load got slower
the more it had written.

A write now arms the next gather only when it gives space back or measures a
page (delta >= 0). gatherPageStatistics() also settles "nothing to consume"
with a plain read before the atomic reset, which is the common answer on every
allocation made while the map is empty.

Test: Issue9575AppendOnlyLoadDoesNotRescanBucketTest counts scans (no clock,
the throttle is expired through a test hook): 1 scan became 6 before the fix.
A second test pins that a delete still arms a gather and its pages are reused.
On a 15M-vertex import the scan left the main thread's stack (5 of 40 samples
before, 0 of 30 after) and the vertex pass went from 63.9 s to 53.6 s with
the file fully cached; the decay itself needs a file larger than the OS cache.
@mergify

mergify Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@claude

claude Bot commented Oct 13, 2026

Copy link
Copy Markdown

Review of #9901

The fix is small and targeted, and the diagnosis in the description is convincing. I found no blocking issues.

What looks good

  • Gating changesFromLastStats on delta >= 0 matches the documented delta convention (negative means space taken, positive means space given back, 0 means a measurement). The delta == 0 cases (compressed page accounting, updatePageStatistics(..., 0) at the commit-time measurement paths) still arm a gather, which is the safe side.
  • The early plain read in gatherPageStatistics() is cheap and correct. It sits after the throttle check and is skipped on firstRun and resuming. Increments that land between the read and getAndSet are carried into the next cycle, so fix(#4952,#4957,#4958,#4960): pool discipline, TimeSeries threading, storage/WAL and LSM low-severity cleanups (2026-07 audit) #5063 is not regressed.
  • The tests count scans instead of timing them, which follows the CLAUDE.md guidance on wall-clock assertions. expireGatherThrottleForTesting() is package-private and keeps the first-run state.
  • Both directions are covered: appends do not scan, and deletes still arm a gather and get their pages reused.

Points to consider (non-blocking)

  1. Behavior change for in-place shrinks and relocations. Anything that now relies on a scan to discover pages that were partially filled by a failed or rolled-back insert will no longer arm one. Pages that are partially filled and then abandoned (a rollback after an insert took space) are not tracked by the incremental map, since the rollback never calls updatePageStatistics. If rollback leaves reusable space, it was previously found by accident through the counter. I think the rolled-back page image is simply discarded, so nothing is lost, but it is worth confirming.
  2. Edge pass. As the description says, the edge pass is unmeasured, and a scan armed by a relocation or delete is still O(pages). A follow-up could bound or incrementalize the scan (for example, start from a cursor, or cap pages per gather). That would fix the decay for mixed workloads too.
  3. Test sizing. appendingNeverTriggersAFullBucketScan writes 120k records of about 1 KB. That is roughly 120 MB per run, so consider @Tag("slow") if it takes noticeable time in CI. Also, pagesAfterFirstRun * 3 assumes the page size is the default, so it would be more robust to assert growth relative to a fixed record count.
  4. Test style. deletedSpaceIsStillFoundByTheNextGather has a + 5 page slack. It is fine, but a one-line comment on why 5 would help the next reader.

Overall this looks good to merge once the reporter confirms the fix on the uncached large-file case.

@coderabbitai

coderabbitai Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: cae64748-a6a1-41d4-9e57-108e4fc2610f
📥 Commits

Reviewing files that changed from the base of the PR and between af98f36 and 2050660.

📒 Files selected for processing (2)
  • engine/src/main/java/com/arcadedb/engine/LocalBucket.java
  • engine/src/test/java/com/arcadedb/engine/Issue9575AppendOnlyLoadDoesNotRescanBucketTest.java

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review.

📝 Walkthrough

LocalBucket now skips page-statistics scans when no free-space changes have accumulated, except on a first run or when resuming a truncated scan. Regression tests cover append-only writes and reuse of space after deletions.

Changes: Bucket statistics scan scheduling

Layer / File(s) Summary
Statistics scan scheduling
engine/src/main/java/com/arcadedb/engine/LocalBucket.java
gatherPageStatistics() skips scans when no qualifying changes have accumulated, and updatePageStatistics() counts only changes with a non-negative delta. A package-private test hook expires the gather throttle after a prior scan.
Append-only and space-reuse regression tests
engine/src/test/java/com/arcadedb/engine/Issue9575AppendOnlyLoadDoesNotRescanBucketTest.java
Tests check that repeated appends do not increase gather scans and that appends after deletions trigger a gather while limiting page growth. Helpers commit and restart transactions after each 2,000 records.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix

Suggested reviewers: robfrank

Merge Risk: ⚪ Minimal · up to 20506

This change avoids repeated whole-bucket free-space scans during append-only imports, while deletes still free space for reuse. No actionable merge-blocking risk was identified.

Pre-merge checks · Passed 4 · Failed 1

Failed Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks (4 passed)
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main fix: preventing unnecessary full-bucket free-space scans during appends.
Description check Passed The description provides detailed motivation, cause, implementation changes, related issues, verification results, and caveats. It does not use the template headings and omits the checklist, but it is…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@lvca lvca self-assigned this Oct 13, 2026
@lvca lvca added this to the 26.11.1 milestone Oct 13, 2026
@lvca
lvca merged commit 38ebe31 into main Oct 13, 2026
@lvca
lvca deleted the worktree-arcadedb-9575-gather-rescan branch October 13, 2026 12:03
oc007us pushed a commit to oc007us/arcadedb-grpc that referenced this pull request Oct 17, 2026
…x reads in the GraphImporter edge pass

Discussion ArcadeData#9559 (third report, after ArcadeData#9901): a 209M-vertex GraphImporter load still stalled.

Vertex pass: records larger than a page are written as multi-page chunks that are not declared
hole-free, so every commit measured their pages and armed a free-space gather, which read the whole
vertex bucket on the writer's thread. The stalls grew with the file (3 min at 35M vertices, 20 at 209M).
LocalBucket now reads at most 1,024 pages per gather and resumes from its cursor on the next one; an
open lap continues after the 5 s throttle without needing a new change, and the ArcadeData#8660 map-full resume
stays unthrottled.

Edge pass: every 500k-edge GraphBatch flush read the record of each distinct endpoint to find its edge
list, one random read per vertex (about 15 min per flush on the reporter's disk), although the importer
had created those vertices without edges. GraphBatch.newEdgeWithHints() lets a caller declare that an
endpoint has no edges yet in that direction; GraphImporter tracks, per vertex and direction, whether it
has already handed an edge to a batch, and hints only the first one. getVertexHeadReads() exposes the
count on both.

Refs ArcadeData#9575
robfrank pushed a commit that referenced this pull request Nov 5, 2026
…x reads in the GraphImporter edge pass

Discussion #9559 (third report, after #9901): a 209M-vertex GraphImporter load still stalled.

Vertex pass: records larger than a page are written as multi-page chunks that are not declared
hole-free, so every commit measured their pages and armed a free-space gather, which read the whole
vertex bucket on the writer's thread. The stalls grew with the file (3 min at 35M vertices, 20 at 209M).
LocalBucket now reads at most 1,024 pages per gather and resumes from its cursor on the next one; an
open lap continues after the 5 s throttle without needing a new change, and the #8660 map-full resume
stays unthrottled.

Edge pass: every 500k-edge GraphBatch flush read the record of each distinct endpoint to find its edge
list, one random read per vertex (about 15 min per flush on the reporter's disk), although the importer
had created those vertices without edges. GraphBatch.newEdgeWithHints() lets a caller declare that an
endpoint has no edges yet in that direction; GraphImporter tracks, per vertex and direction, whether it
has already handed an edge to a batch, and hints only the first one. getVertexHeadReads() exposes the
count on both.

Refs #9575

(cherry picked from commit 4790ace)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant