Repository navigation
fix(#9575): appends no longer arm a full-bucket free-space scan - #9901
Conversation
A GraphImporter load of 209M vertices kept 1% GC and a healthy heap, yet its vertex pass fell from 60,000 to 600 vertices a second (discussion #9559, after the memory fixes of #9581). LocalBucket.updatePageStatistics counted every write as a change to the free-space statistics, appends included. In an append-only load the map of reusable pages is empty each time an appended page fills, so the next allocation calls gatherPageStatistics(). Once its 5 s throttle is over the scan reads EVERY page of the bucket looking for space an append cannot have freed. The scan grows with the file (about 6 minutes of disk reads at 1.3M pages, against 5 s of useful work between two scans), so the load got slower the more it had written. A write now arms the next gather only when it gives space back or measures a page (delta >= 0). gatherPageStatistics() also settles "nothing to consume" with a plain read before the atomic reset, which is the common answer on every allocation made while the map is empty. Test: Issue9575AppendOnlyLoadDoesNotRescanBucketTest counts scans (no clock, the throttle is expired through a test hook): 1 scan became 6 before the fix. A second test pins that a delete still arms a gather and its pages are reused. On a 15M-vertex import the scan left the main thread's stack (5 of 40 samples before, 0 of 30 after) and the vertex pass went from 63.9 s to 53.6 s with the file fully cached; the decay itself needs a file larger than the OS cache.
|
This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again. |
Up to standards ✅🟢 Issues
|
|
Review of #9901 The fix is small and targeted, and the diagnosis in the description is convincing. I found no blocking issues. What looks good
Points to consider (non-blocking)
Overall this looks good to merge once the reporter confirms the fix on the uncached large-file case. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
…x reads in the GraphImporter edge pass Discussion ArcadeData#9559 (third report, after ArcadeData#9901): a 209M-vertex GraphImporter load still stalled. Vertex pass: records larger than a page are written as multi-page chunks that are not declared hole-free, so every commit measured their pages and armed a free-space gather, which read the whole vertex bucket on the writer's thread. The stalls grew with the file (3 min at 35M vertices, 20 at 209M). LocalBucket now reads at most 1,024 pages per gather and resumes from its cursor on the next one; an open lap continues after the 5 s throttle without needing a new change, and the ArcadeData#8660 map-full resume stays unthrottled. Edge pass: every 500k-edge GraphBatch flush read the record of each distinct endpoint to find its edge list, one random read per vertex (about 15 min per flush on the reporter's disk), although the importer had created those vertices without edges. GraphBatch.newEdgeWithHints() lets a caller declare that an endpoint has no edges yet in that direction; GraphImporter tracks, per vertex and direction, whether it has already handed an edge to a batch, and hints only the first one. getVertexHeadReads() exposes the count on both. Refs ArcadeData#9575
…x reads in the GraphImporter edge pass Discussion #9559 (third report, after #9901): a 209M-vertex GraphImporter load still stalled. Vertex pass: records larger than a page are written as multi-page chunks that are not declared hole-free, so every commit measured their pages and armed a free-space gather, which read the whole vertex bucket on the writer's thread. The stalls grew with the file (3 min at 35M vertices, 20 at 209M). LocalBucket now reads at most 1,024 pages per gather and resumes from its cursor on the next one; an open lap continues after the 5 s throttle without needing a new change, and the #8660 map-full resume stays unthrottled. Edge pass: every 500k-edge GraphBatch flush read the record of each distinct endpoint to find its edge list, one random read per vertex (about 15 min per flush on the reporter's disk), although the importer had created those vertices without edges. GraphBatch.newEdgeWithHints() lets a caller declare that an endpoint has no edges yet in that direction; GraphImporter tracks, per vertex and direction, whether it has already handed an edge to a batch, and hints only the first one. getVertexHeadReads() exposes the count on both. Refs #9575 (cherry picked from commit 4790ace)
Follow-up to #9575 / #9581, reported in discussion #9559.
After #9581 the reporter re-ran the 209M vertex import with a 24 GB heap. Memory was fine (GC 1%, heap 6 to 15 GB) but the vertex pass still decayed from 60,000 vertices/s to ~600/s, roughly as 1/N from about 9M vertices on. #9581 fixed the memory side; this is the storage side.
Cause
LocalBucket.updatePageStatisticscounted EVERY write as a change to the bucket's free-space statistics, appends included. In an append-only load the map of reusable pages is empty each time an appended page fills, so the next allocation callsgatherPageStatistics(). Once its 5 s throttle is over, that call reads every page of the bucket looking for space an append cannot have freed.The scan grows with the file. At 1.3M pages it is ~6 minutes of disk reads (~270 us a page uncached), against ~5 s of useful work (~300k vertices) between two scans. That matches the reporter's log exactly: ~300k vertices every 6 minutes at 88M vertices.
Change
updatePageStatisticsarms the next gather only when a write gives space back or measures a page (delta >= 0). Inserts and enlargements take space, so they cannot leave a page a scan would list, and the pages they fill are already tracked by the incremental map update.gatherPageStatistics()settles "nothing to consume" with a plain read of the counter before the atomic reset. That is the common answer on every allocation made while the map is empty.expireGatherThrottleForTesting()(package-private) so a test does not sleep through the 5 s throttle.Verification
Issue9575AppendOnlyLoadDoesNotRescanBucketTestcounts scans, no clock. Appending in five rounds with the throttle expired before each: 6 scans before the fix, 1 after. A second test pins the other side: a delete still arms a gather, and the pages it freed are reused instead of growing the file.Issue8660*,Issue6339,Issue6358,Issue6396,Issue8401,Issue9483,Issue9694,Issue7121,Issue6163,SchemaReloadDoesNotEagerScanBuckets,RecordRecycling,RandomDelete), and the wholecom.arcadedb.enginepackage: 1173 tests, 0 failures.gatherPageStatisticswas on the importer thread's stack in 5 of 40 samples before the fix and in 0 of 30 after; the vertex pass went from 63.9 s to 53.6 s.Caveats
delta > 0), which can still arm scans. A scan armed by a delete is still O(pages).Summary by CodeRabbit