Repository navigation
fix: batch the import activity log per phase — the O(n²) CPU burner throttling memex-cloud - #1341
Conversation
…n memex-cloud `NodeTypeCompilationActivity.AppendLog` does one `stream.Update` per message, and `StaticRepoImporter` called it PER ITEM at four sites (kept-on-conflict, failed upsert, kept-from-prune, pruned). Every `stream.Update` runs `CreateSingleObjectPatch`, which `SerializeToNode`s the ENTIRE node content (O(n)) and diffs all n elements (O(n)) — on every write. Over n appends that is O(n²) CPU and allocation. Live attribution on memex-cloud: 5,239 writes / ~719 MB serialised on ONE import activity (141 kB content); 230,674 writes and ~7.1 GB across 25,579 activity nodes; no non-import node in the top 20. Downstream: ~7.8 cores demanded against a 6-core quota, so the process sat at 51% of CFS periods throttled, dragging turns, timers and Dequeue together (thread-pool delay #1284 → routing pile-up #1172). The patch OUTPUT was never the problem — `PatchExtensions.DiffArray` is element-wise, so an append emits a single O(1) `add` op. The cost is on the INPUT side of the patcher, which is why swapping the patch library (#1292) moved nothing. Fix: append once per PHASE instead of once per item. - New `NodeTypeCompilationActivity.AppendLogs(...)` writes N messages in ONE Update (`Messages.AddRange`); `AppendLog` is now a one-message call into it, with a doc comment spelling out why it must never be called in a loop. - `StaticRepoImporter.Run` carries each per-file line along with its per-file result tuple (as `WrittenPaths` already did) and appends the whole upsert phase in one write; the prune phase likewise appends its kept-from-prune + pruned lines in one write once the deletes have settled. - `Complete` was not the right vehicle on its own: it writes the TERMINAL status, so folding the phase lines into it would defer all progress past the prune, content sync and manifest write. The phase-level append sits alongside it. Nothing is lost from the audit trail — every kept and pruned path is still named on the activity log, and still lands before the terminal status. The accepted cost, stated by the owner, is per-phase rather than per-file live progress. Measured by the new test (`StaticRepoImportActivityWriteCountTest`), which counts the attempt activity node's own revision counter (`MeshNode.Version`), never log text: items in the phase │ activity writes before │ after ───────────────────────────────────────────────────── 4 │ 8 │ 5 16 │ 20 │ 5 i.e. writes grew 1:1 with the item count and are now a constant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR reduces CPU and allocation pressure during large static-repo imports by batching activity-log writes per phase, avoiding the previous “one stream.Update per item” pattern that caused O(n²) re-serialization/diff work on the growing activity node content.
Changes:
- Add
NodeTypeCompilationActivity.AppendLogs(...)to append many log messages in a singlestream.Update, and routeAppendLogthrough it. - Refactor
StaticRepoImporterto accumulate per-item log lines during upsert/prune and emit them once per phase viaAppendLogs. - Add an integration-style regression test that asserts the import activity’s write count stays constant as the number of “kept local change” items increases, using
MeshNode.Versionas the write counter.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| test/MeshWeaver.Graph.Test/StaticRepoImportActivityWriteCountTest.cs | Adds a regression guard ensuring activity-log writes don’t scale with per-item conflict logging. |
| src/MeshWeaver.Graph/StaticRepoImporter.cs | Carries per-item log lines along with per-item results and writes them once per phase to avoid O(n²) activity-node patch input cost. |
| src/MeshWeaver.Graph/Configuration/NodeTypeCompilationActivity.cs | Introduces AppendLogs (batched append) and routes single-message appends through it. |
| src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-12-imports-no-longer-burn-cpu-on-their-own-progress-log.md | Adds a “What’s New” entry describing the user-visible impact and tradeoff (phase-level progress). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Correction: "the patch output was never the problem" holds for the own-stream path onlyRecording this here because it is the sentence a future reader will rely on, and it is true for Surfaced while root-causing #1351 ( The two patch paths differOwn-stream — RFC 6902 JSON Patch. As this PR says: Cross-hub — RFC 7396 JSON merge patch. So a single append to an N-element collection in This matters because cross-hub is the common case for activity writes: the writer is rarely the A closed door, so nobody re-opens it
Where this goesNot a regression in this PR — batching was the right fix for the CPU term and it holds. The message-size |
…rites fix: batch the indexing and import progress logs — the O(n²) shape #1341 left behind
The live production CPU burner
memex-cloudsits at 51% of CFS periods throttled. The dominant term is the static-repoimporter writing its own progress log once per item.
NodeTypeCompilationActivity.AppendLogdoes onestream.Updateper message, andStaticRepoImportercalled it per pruned / kept / failed node at four sites. Eachstream.Updateruns
CreateSingleObjectPatch, whichSerializeToNodes the entire node content (O(n)) and diffsall n elements (O(n)) — on every write. Over n appends that is O(n²) CPU and allocation.
Live attribution, by node path:
AgenticEngineering/_Activity/import-69f6248b…UWDeepfield/_Activity/import-af545232…MeshWeaver/_Activity/import-1d1a65…230,674 writes / ~7.1 GB serialised across 25,579 activity nodes; no non-import node in the top
20. Downstream: ~7.8 cores demanded against a 6-core quota → CFS throttling → turns, timers and
Dequeueall slip together (thread-pool delay #1284, routing pile-up #1172). No deadlock.The patch output was never the problem.
PatchExtensions.DiffArrayis element-wise, so anappend walks the equal prefix and emits a single O(1)
addop. The cost is on the input side ofthe patcher — which is exactly why replacing the patch library in #1292 (1.2× faster diff, 17–19%
fewer allocations) did not move the needle: the dominant cost is re-serialising a growing document.
The fix — batch per phase
NodeTypeCompilationActivity.AppendLogs(...)writes N messages in oneUpdate(
Messages.AddRange).AppendLogbecomes the one-message call into it, and its doc comment nowstates the cost model so the per-item shape does not come back.
StaticRepoImporter.Runcarries each per-file line along with its per-file result tuple —the same trick
WrittenPathsalready used — and appends the whole upsert phase in one write.The prune phase likewise appends its kept-from-prune + pruned lines in one write once the
deletes have settled.
Completewas not the right vehicle on its own. It writes the terminal status, so foldingthe phase lines into it would defer every line past the prune, content-sync and manifest-write
phases — no live progress at all. The phase-level append sits alongside it.
Nothing leaves the audit trail: every kept and every pruned path is still named on the activity
log (issue #604's requirement), and still lands before the terminal status — the per-path write
queue is serial, so a reader that sees a non-Running status has already seen every line.
Accepted cost (owner's call): per-item live progress during a long import is replaced by
per-phase progress. The satellite-node design that would keep per-item progress alters the
ActivityLogcontract, its readers and the GUI, and was deliberately deferred as a separate change.No band-aids: no quota change, no scheduler hop, no recursion-depth bound, no throttle/debounce on
the writes. The work itself is gone, not moved.
Verification — before / after write counts
New test
StaticRepoImportActivityWriteCountTestruns a real import into a fresh partition, thenre-imports the same paths under a two-way policy so every node takes the "kept local change" branch
— a phase that writes no content at all, so the attempt-activity node's revisions are exactly
the import's own logging. It asserts on
MeshNode.Version(the node's own "+1 per realmodification" counter), never on log text, so a reworded message cannot satisfy it.
Writes grew 1:1 with the item count; they are now a constant (opening line, root line, one phase
line, terminal summary). Fail-before/pass-after was verified by reverting the
src/diff, rebuildingand re-running: RED with
4 -> 8; 16 -> 20, GREEN with4 -> 5; 16 -> 5.Extrapolating the measured production shape: the 5,239-write / 719 MB activity becomes a handful of
writes, and the ~7.1 GB of serialisation across the 25,579 activity nodes collapses by the same
order.
Test runs (local, this branch)
MeshWeaver.Graph.Test— 1060 passed, 0 failed (includes the new test)MeshWeaver.GitSync.Test— 160 passed, 0 failed (covers the prune/preserve paths and the"activity log NAMES the pruned node" assertion of GitHub sync: git→mesh import silently FullReplace-prunes mesh-only nodes (no paths logged, no opt-out for bidirectional Spaces) #604)
MeshWeaver.Documentation.Test—WhatsNewEntryIntegrityTest+DocumentationLinkIntegrityTestgreendotnet build -c Release -warnaserrorclean forMeshWeaver.Graph,MeshWeaver.Graph.Test,MeshWeaver.DocumentationWhat's New entry:
Category: Fix— users notice imports being far cheaper.🤖 Generated with Claude Code