Repository navigation
fix: take a per-repository write lock before build, update and postprocess - #1126
Open
snowshadow wants to merge 1 commit into
Open
snowshadow wants to merge 1 commit into
snowshadow wants to merge 1 commit into
Conversation
…ocess A build spans many commits and many seconds, and nothing excluded a second one. The only cross-process protection was SQLite's busy timeout, which is not reached until the first write — so every duplicate run paid for a full parse, and its own process pool, before discovering that an older run had already done that work. `postprocess` had the same exposure in the other direction: it rebuilds flows, communities and FTS from whatever nodes are stored, so running it against a graph another process is still writing leaves derived tables describing a graph that no longer exists. That is the report in tirth8205#1073. `build`, `update` and `postprocess` now take an exclusive `flock` on `<data dir>/write.lock` for the whole write window. A caller that cannot take it returns `status: contended` with the holder's PID instead of repeating the work; the CLI prints which PID owns it and exits `EX_TEMPFAIL` (75) — non-zero, so a hook cannot mistake a skip for success, but distinct from the `1` a real failure uses. `--wait` (`wait_for_lock`) blocks and then runs for callers that need the work done rather than skipped. Measured on the reporter's machine, the failure mode this removes: an agent hook firing `code-review-graph update --skip-flows` on every tool call had ten concurrent runs of one repository, ~90 Python workers on 18 cores, load average 44, and 63 of 64 GB of RAM in use. Design notes: - The lock is named from the store's own `db_path`, not from a second path resolution, so the file it guards is always the database about to be written, including under `CRG_DATA_DIR` and `--data-dir`. - The name and location are fixed (`write.lock` in the data directory) so external tooling can exclude itself with `flock -n .code-review-graph/write.lock ...`, as the issue asked. - `flock`, not `lockf`: the lock belongs to the open file description, so a probe from a second descriptor cannot drop a lock this process holds, and the kernel releases it on `SIGKILL` — a killed or timed-out build never wedges the repository. - Two repositories never block each other, and a lock file that cannot be opened degrades to the old behaviour rather than failing the command. - Skipping is the default because the caller that overlaps is usually a hook whose work the in-flight run already covers. `watch` writes through `incremental_update` directly, and `embed`/`forget` are separate writers; they are the remaining entry points and are left for a follow-up rather than widened here. Tests (tests/test_concurrency_and_daemon.py, -m concurrency): a second update stands down and leaves the graph untouched, the skipped run names the holder, a finished build leaves the lock free, a SIGKILLed holder does not wedge the repository, `--wait` blocks and then does the work, two repositories do not block each other, the tool entry point reports contention instead of parsing, postprocess stands down for a running build, and a burst of eight concurrent updates builds the graph exactly once. Eight of the nine fail without the fix. Refs tirth8205#1073
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
build,updateandpostprocessnow take an exclusive lock for the whole write window. A caller that cannot take it stands down withstatus: contendedand the holder's PID instead of repeating the work.This is the mechanism behind #1073. The report there is about correctness — one writer post-processing a graph another writer has only partly written — and it is also a CPU problem, which is how I hit it:
What changed
code_review_graph/locking.py(new):WriteLock, an advisory exclusiveflockon<data dir>/write.lock.build_or_update_graphtakes it before the first parse;run_postprocesstakes it before rebuilding flows/communities/FTS. Both share one helper so they cannot drift.{"status": "contended", "skipped": true, "holder_pid": n, "lock_path": ...}with zeroed counts, so callers that unpack the existing keys keep working.build,update,postprocess): printsSkipped: another code-review-graph ... is already running ... (PID n)and exitsEX_TEMPFAIL(75).--waitblocks instead of skipping.wait_for_lockonbuild_or_update_graph,run_postprocess, and their MCP tools' docs.Why the choices
Exit 75, not 0 or 1. A skip is not success — a caller must not believe a graph was refreshed when it was not — but it is not the failure
1reports either (bad config, corrupt database).EX_TEMPFAILis exactly this condition, so a hook script can branch on it without matching strings, and|| truehooks are unaffected.A skip by default,
--waiton request. The caller that overlaps is nearly always a hook, and the in-flight run covers the same working tree, so skipping is free there. A human runningupdateby hand usually wants the wait.Lock named from
store.db_path, not a second path resolution. The file guards the database that is actually about to be written, including underCRG_DATA_DIRand--data-dir; two resolutions of "the" repo root can disagree.Fixed name and location (
<data dir>/write.lock). As the issue asked: external tooling can exclude itself withflock -n .code-review-graph/write.lock ...and both sides exclude each other. One database, one lock.flock, notlockf. The lock belongs to the open file description, so a probe from a second descriptor cannot silently drop a lock this process holds, and the kernel releases it onSIGKILL— a build killed by a hook timeout never wedges the repository.Failure to open the lock file degrades to the old behaviour rather than failing the command; Windows (no
flock) does the same, so behaviour there is unchanged.Scope
This covers the two entry points that own the long parse plus the stage that rewrites derived data. Deliberately left out, because they need a different treatment and I did not want to widen the diff:
watchwrites throughincremental_updatedirectly. Handing it a non-blocking lock is wrong — a skipped batch would silently drop file-change events — so it needs "block or retry this batch", which is its own change.embedandforgetare single short writes that SQLite already serialises transaction by transaction, with no long parse to duplicate.Happy to extend this PR to cover them, or to file follow-ups if you prefer to land this first.
Tests
tests/test_concurrency_and_daemon.py(-m concurrency), 10 new tests:test_the_lock_lives_in_the_data_directorywrite.lockpath, soflock -ninterop holdstest_second_update_skips_while_the_first_holds_the_locktest_the_skipped_run_names_the_process_that_owns_the_worktest_a_finished_build_leaves_the_lock_freetest_a_killed_holder_does_not_wedge_the_repositorytest_wait_blocks_until_the_holder_finishes--waitwaits, then does the worktest_builds_of_different_repositories_do_not_block_each_othertest_the_tool_entry_point_reports_contention_instead_of_parsingcontendedtest_postprocess_stands_down_for_a_running_buildtest_a_burst_of_concurrent_updates_builds_the_graph_once8 of the 9 behaviour tests fail on
staging(verified by revertingtools/build.pyand re-running); the burst test is the regression for the reported symptom.Two existing tests were updated for the new contract:
test_concurrent_builds_do_not_corrupt_or_lose_filesnow expects a loser to exit 75 with the skip message rather than die on the busy timeout (the timeout path itself is still covered by the_LockHeldtests, which hold the SQLite lock without holding ours), and the exact-kwargs assertions intests/test_cli.pyandtests/test_embedding_refresh.pygainedwait_for_lock=False.Verification
Two tests fail in my sandbox both before and after this change, for environmental reasons:
test_released_shapes.py::test_recorded_hook_blocks_match_the_tags(the clone has no release tags) andtest_e2e_real_repo.py::test_hybrid_search_ranks_impact_radius_nodes_first(no embedding model available). Confirmed by stashing the branch and re-running them onstaging.Docs
CHANGELOG.mdunder Unreleased,docs/COMMANDS.md(--wait), and a newdocs/TROUBLESHOOTING.mdsection covering the skip message, the exit code and theflockinterop.Refs #1073