fix(relay-admin): make thread deletions atomic and fence expired action leases under row lock - #7853
Conversation
Admin delete decremented events.reply_count, which thread summaries never read, ignored the root, and decremented again on every action against an already-deleted target. Run the canonical NIP-29 delete body inside the lease-fenced admin transaction so parent/root thread_metadata move exactly once, and refresh the live thread summary after commit like DELETE_EVENT. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
🔐 Codex Security Review
Review SummaryOverall Risk: NONE
FindingsNo concrete security, correctness, or reliability findings were identified. Notes
Generated by Codex Security Review | |
…unused counter helpers The final marker UPDATE in each admin mutation action (ban, timeout, kick, delete, record_failure) compared action_lease_expires_at against now(), which PostgreSQL fixes at transaction start. If the lease expired while the domain write was in progress, the fence still saw the old timestamp and committed. Switch the final ownership fence in all five actions to clock_timestamp() so it reads wall-clock time at the moment of the UPDATE. The pre-entry ownership SELECT that short-circuits before any domain write is unchanged (now() is correct there: it runs at statement start with nothing waiting behind it). Add admin_delete_with_lease_expiring_during_write_changes_nothing to reproduce the gap deterministically: a per-row trigger on events delays the target UPDATE by 1 s; the lease is set to expire after 500 ms. Under the old now() predicate the transaction committed; with clock_timestamp() it rolls back. Rename admin_delete_with_lost_lease_changes_nothing to admin_delete_with_already_expired_lease_changes_nothing to accurately describe what it proves (pre-entry rejection of an already-expired lease). Remove increment_reply_count, decrement_reply_count, and the Db::decrement_reply_count wrapper from thread.rs. No production callers exist outside the forwarding wrappers; both helpers perform unguarded autocommit writes that bypass the deletion-transition/atomicity invariant enforced by the canonical paths. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
wesbillman
left a comment
There was a problem hiding this comment.
Carl, an automated reviewer, commenting via Wes’s GitHub account.
Review clear: no blocking code, product, or security defects found. Reviewed HEAD dc6c535da566c5c392cf9f33a023a6b904eae8eb against BASE d01e5f82058463709a22e93bb4cd795da5f53e10.
Source review covered canonical deletion/counter atomicity and idempotency, tenant scope, removed-helper callers, all five final lease fences and ownership/recovery paths, and canonical best-effort live-summary integration. One non-blocking timing-test robustness note is inline.
Validation: existing PostgreSQL CI passed 476 tests, including all four new regressions, on GitHub’s merge of this head into the pinned base. This review ran no code or tests.
CI is not fully green: Desktop Smoke E2E (4) fails the middle-page scroll coverage assertion. That unchanged spec runs against an in-browser mock bridge without the Rust relay/database, so it does not exercise this change. The desktop gate still needs separate disposition; this is not a merge approval.
| .expect("install delay trigger"); | ||
|
|
||
| let lease_until = Utc::now() + chrono::Duration::milliseconds(500); | ||
| let (action_id, token) = enforcing_action(&pool, community_id, lease_until).await; |
There was a problem hiding this comment.
Non-blocking test robustness: the 500ms lease starts before enforcing_action completes its database setup. If setup consumes that interval, execute_delete_with_marker exits at the unchanged pre-entry ownership check, and the assertions still pass without reaching the final clock_timestamp() fence. Please ensure the lease is live after setup (and fail explicitly if that precondition is lost), rather than allowing slow setup to silently turn this into another already-expired-lease test. No production fencing defect found; this is a weakness in the regression test's evidence.
clock_timestamp() in the marker UPDATE predicate can be evaluated by Postgres before it waits to lock the target row. If another transaction holds the action row and releases it unchanged after the lease expires, the update commits — the expired owner persists its mutation, marker, or failure state contrary to the guarantee. Fix: for every admin-action path (ban, timeout, kick, delete, and record_failure) acquire the action row with SELECT ... FOR UPDATE inside the same transaction before the domain write. The wall-clock expiry check in the marker UPDATE then runs under that lock, not before it. The token/ state/marker CAS and the single-transaction structure are unchanged. record_failure previously used a single-statement execute(pool). It now begins a transaction, acquires the row lock with an explicit SELECT FOR UPDATE / expiry check, then updates state; this brings it in line with the four mutation functions. Also fix the in-flight-expiry regression test: - assert the lease is live immediately after setup so a slow run fails the precondition instead of passing through early rejection - assert the stored step_marker is NULL as well as the return value and event state - correct the comment arithmetic (1 s is *longer* than 500 ms) Add admin_delete_with_lease_expiring_during_row_lock_wait_changes_nothing to exercise the new lock-first shape: an external transaction locks the action row, the worker blocks at its FOR UPDATE, the test asserts the lease is still live at that point, waits for DB-clock expiry, releases the unchanged row, and asserts the worker returns false with the event, both counters, and step_marker unchanged. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…lock test comment admin_delete_with_lease_expiring_during_write_changes_nothing: replace the timed pg_sleep trigger with a lock-based approach. The target event row is locked externally; the worker acquires the action-row lock first (via its existing SELECT FOR UPDATE), then blocks at the UPDATE events domain write. pg_stat_activity + pg_blocking_pids confirms the worker is observably blocked while the lease is live before expiry is asserted; a slow setup fails the named precondition instead of passing through early rejection. Releases the event row unchanged via rollback, then asserts rejection, event, counters, and stored marker. The reverted-now() + 650 ms delay mutation that previously produced a false green now produces RED on this test as well (both expiry tests RED). admin_delete_with_lease_expiring_during_row_lock_wait_changes_nothing: update the doc comment — the worker blocks at the SELECT FOR UPDATE, not the marker UPDATE. Broaden the pg_stat_activity observer to accept either the FOR UPDATE or the marker UPDATE query, so lock-removal fails on committed behavior rather than on SQL shape. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…amp() change The previous commit's mutation-restore step used a blanket sed substitution that accidentally changed the four pre-entry ownership SELECT EXISTS checks (ban:468, timeout:556, kick:656, delete:746) from now() to clock_timestamp(). Those checks are early-rejection optimizations that are correct with now(); only the final fences require clock_timestamp(). This commit reverts them. Production code is now byte-identical to c5aaacc; this commit touches only the mod postgres_tests section. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
wesbillman
left a comment
There was a problem hiding this comment.
Carl, an automated reviewer, commenting via Wes’s GitHub account.
Review clear: no blocking code, product, or security defects found in the corrective change. HEAD 16cb762f692aa0a289bdd6fd40d024b177c9fe51, BASE d01e5f82058463709a22e93bb4cd795da5f53e10.
Follow-up to the prior review: checked all five lock-first/final-clock fences, ownership changes during waits, deletion/counter rollback, and post-commit summary integration. The new observed-lock/DB-clock regressions address the earlier timing-test concern. Lease validity is checked under the action-row lock, not at physical COMMIT completion.
Validation: existing PostgreSQL CI passed 477/477 on merge c98b1710d0ebedd869af1b787b7daf54e4293944, containing this head but using base 99c2acf90cfbb1cb2d3a8bd900c0ec1642e20540. This is not standalone-head or pinned-base certification. This source-only review ran no code or tests.
Non-blocking housekeeping: name the two new pg_stat_activity regressions with cluster_global_, per crates/buzz-db/TESTING.md:40–43. Both queries filter to their database and exact blocker PID; no concrete cross-test failure was established, so this is not a correctness blocker.
Broader CI was red in the snapshot, with relay/integration E2E failures and some checks still running. Their cause was not established here; those gates still need disposition. This is not merge approval.
…ead-counters Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com> * origin/main: chore(release): release Buzz Desktop version 0.5.25 (#7867) fix(ci): consume the published MinIO image (#7870) fix(mobile): converge sidebar managers on relay head with resume re-read (#7806) fix(ci): bootstrap the reusable MinIO image in GHCR (#7869) Discover alternate Buzz ACP commands (#6948) fix(hooks): surface nextest failures and stale pnpm deps in pre-push (#7850) feat(acp): run one prepared task from a file or stdin (#7851) Fix mobile heart and warning emoji with native font fallback (#7842) chore(mesh): upgrade MeshLLM to 0.76.2 (#7559) Signed-off-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz>
Both inspect cluster-wide PG state, so nextest must serialize them. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
wesbillman
left a comment
There was a problem hiding this comment.
Carl, an automated reviewer, commenting via Wes’s GitHub account.
CLEAR: no actionable regression found since the prior clear review.
- The branch-specific delta renames two PostgreSQL lease-expiry tests into the existing
cluster_global_serial group, resolving the prior housekeeping note. The production deletion/counter, lease-fence, and live-summary paths are unchanged. Main-merge integration was reviewed separately. - Source-only review with an independent DB lane; no builds, tests, or PR code executed. Existing exact-head PostgreSQL CI reports success. The broader snapshot had 50 successful checks, 26 skipped and two desktop smoke checks still running, so this is not an all-green claim.
- No code fixes requested. Required CI remains a separate merge gate; unchanged unrelated surfaces were excluded.
Head: 7b1edfe150ce3049ac3811cfd102bbe68b282bfa. Base: 02753722a7dd06560402a5b92491b048968c1a63.
…ction * origin/main: (21 commits) docs(vision): add /buzz/v1 read endpoints to the protocol contract (#7879) 🤖 fix(justfile): point just staging at the current staging relay (#7881) fix(relay-admin): make thread deletions atomic and fence expired action leases under row lock (#7853) feat(desktop): relay admin console for the /api/admin/v1 operator surface (#4768) fix(mobile): keep retired sections manager out of successor cache (#7873) Select one feature flag provider at compile time (#7677) chore(release): release Buzz Desktop version 0.5.25 (#7867) fix(ci): consume the published MinIO image (#7870) fix(mobile): converge sidebar managers on relay head with resume re-read (#7806) fix(ci): bootstrap the reusable MinIO image in GHCR (#7869) Discover alternate Buzz ACP commands (#6948) fix(hooks): surface nextest failures and stale pnpm deps in pre-push (#7850) feat(acp): run one prepared task from a file or stdin (#7851) Fix mobile heart and warning emoji with native font fallback (#7842) chore(mesh): upgrade MeshLLM to 0.76.2 (#7559) feat(agents): humanize uncurated Databricks model ids with a label grammar (#7844) fix: route databricks claude fqns to anthropic messages (#7829) feat(relay): add opt-in newest-first thread windows (#7823) refactor: move agent Git bootstrap into ACP harness (#7819) Use worker snapshots for relay storage metrics (#7845) ... Signed-off-by: Tom Brow <tomb@block.xyz>
…rcement * origin/main: docs: specify durable data backfills (#7326) docs(vision): add /buzz/v1 read endpoints to the protocol contract (#7879) 🤖 fix(justfile): point just staging at the current staging relay (#7881) fix(relay-admin): make thread deletions atomic and fence expired action leases under row lock (#7853) feat(desktop): relay admin console for the /api/admin/v1 operator surface (#4768) fix(mobile): keep retired sections manager out of successor cache (#7873) Select one feature flag provider at compile time (#7677) chore(release): release Buzz Desktop version 0.5.25 (#7867) Signed-off-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz>
…ivery * origin/main: feat(relay): enforce NIP-FI assertion+NIP-98 pairing on HTTP ingress (#7264) fix(ci): run the admin disabled-mode DB test in the PostgreSQL lane (#7900) 🤖 docs: add pre-PR checklist and review guidance to AGENTS.md (#7897) docs: specify durable data backfills (#7326) docs(vision): add /buzz/v1 read endpoints to the protocol contract (#7879) 🤖 fix(justfile): point just staging at the current staging relay (#7881) fix(relay-admin): make thread deletions atomic and fence expired action leases under row lock (#7853) feat(desktop): relay admin console for the /api/admin/v1 operator surface (#4768) fix(mobile): keep retired sections manager out of successor cache (#7873) Signed-off-by: Tom Brow <tomb@block.xyz>
What
Two fixes to
crates/buzz-db/src/store/relay_admin_actions.rsand one cleanup incrates/buzz-db/src/store/thread.rs.1. Thread counter atomicity
The admin delete action now calls the canonical
soft_delete_event_and_update_thread_in_txbody shared with NIP-29 and NIP-09 event-ID deletion. The delete, parentreply_countdecrement, rootdescendant_countdecrement, and step marker commit or roll back together in one transaction. A second action against an already-deleted target is a no-op and does not double-decrement. After a committed delete,emit_live_thread_summaryrefreshes the live thread count the same wayDELETE_EVENTdoes.2. Lock-first lease expiry fence (all five admin-action paths)
On main, the final marker
UPDATEin every admin-action path (execute_ban_with_marker,execute_timeout_with_marker,execute_kick_with_marker,execute_delete_with_marker,record_failure) checksclock_timestamp()in theWHEREclause. PostgreSQL can evaluate that predicate before waiting to lock the target row. If another transaction holds the action row and releases it unchanged after the lease expires, the update commits under the stale timestamp.On this branch, each path acquires the action row with
SELECT … FOR UPDATEinside the same transaction before any domain write. The wall-clock expiry check in the markerUPDATEruns after that lock is held, not before. The pre-entry ownershipSELECT EXISTSchecks remain — they are early-rejection optimizations that are safe because the correct final fence rolls back any provisional work.record_failureon main uses a single-statementexecute(pool). On this branch it begins a transaction, acquires the row lock withSELECT … FOR UPDATEand an explicit expiry check, then writesstate = 'failed'— matching the four mutation functions.Lease validity is checked at the protected ownership check under the action-row lock, not at physical
COMMITcompletion.execute_ban_with_markerSELECT … FOR UPDATEadded after pre-entry check, before domain writeexecute_timeout_with_markerexecute_kick_with_markerexecute_delete_with_markerrecord_failureSELECT … FOR UPDATE+ clock check before state update3. Unused standalone counter helpers removed
increment_reply_count,decrement_reply_count(both inthread.rs), andDb::decrement_reply_counthave no production callers outside their forwarding wrappers. Both perform unguarded autocommit writes that bypass the deletion-transition/atomicity invariant. Removed to prevent future misuse.