Skip to content

fix(relay-admin): make thread deletions atomic and fence expired action leases under row lock - #7853

Merged
wpfleger96 merged 7 commits into
mainfrom
hayt/admin-delete-thread-counters
Sep 24, 2026
Merged

wpfleger96 merged 7 commits into
mainfrom
hayt/admin-delete-thread-counters

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

What

Two fixes to crates/buzz-db/src/store/relay_admin_actions.rs and one cleanup in crates/buzz-db/src/store/thread.rs.

1. Thread counter atomicity

The admin delete action now calls the canonical soft_delete_event_and_update_thread_in_tx body shared with NIP-29 and NIP-09 event-ID deletion. The delete, parent reply_count decrement, root descendant_count decrement, and step marker commit or roll back together in one transaction. A second action against an already-deleted target is a no-op and does not double-decrement. After a committed delete, emit_live_thread_summary refreshes the live thread count the same way DELETE_EVENT does.

2. Lock-first lease expiry fence (all five admin-action paths)

On main, the final marker UPDATE in every admin-action path (execute_ban_with_marker, execute_timeout_with_marker, execute_kick_with_marker, execute_delete_with_marker, record_failure) checks clock_timestamp() in the WHERE clause. PostgreSQL can evaluate that predicate before waiting to lock the target row. If another transaction holds the action row and releases it unchanged after the lease expires, the update commits under the stale timestamp.

On this branch, each path acquires the action row with SELECT … FOR UPDATE inside the same transaction before any domain write. The wall-clock expiry check in the marker UPDATE runs after that lock is held, not before. The pre-entry ownership SELECT EXISTS checks remain — they are early-rejection optimizations that are safe because the correct final fence rolls back any provisional work.

record_failure on main uses a single-statement execute(pool). On this branch it begins a transaction, acquires the row lock with SELECT … FOR UPDATE and an explicit expiry check, then writes state = 'failed' — matching the four mutation functions.

Lease validity is checked at the protected ownership check under the action-row lock, not at physical COMMIT completion.

Function Change
execute_ban_with_marker SELECT … FOR UPDATE added after pre-entry check, before domain write
execute_timeout_with_marker same
execute_kick_with_marker same
execute_delete_with_marker same
record_failure wrapped in transaction; SELECT … FOR UPDATE + clock check before state update

3. Unused standalone counter helpers removed

increment_reply_count, decrement_reply_count (both in thread.rs), and Db::decrement_reply_count have no production callers outside their forwarding wrappers. Both perform unguarded autocommit writes that bypass the deletion-transition/atomicity invariant. Removed to prevent future misuse.

Admin delete decremented events.reply_count, which thread summaries never
read, ignored the root, and decremented again on every action against an
already-deleted target. Run the canonical NIP-29 delete body inside the
lease-fenced admin transaction so parent/root thread_metadata move exactly
once, and refresh the live thread summary after commit like DELETE_EVENT.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 requested a review from a team as a code owner September 23, 2026 23:04
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

🔐 Codex Security Review

Note: This is an automated, security-focused review generated by Codex.
Use it as a supplement to human review; false positives are possible.

Scope

  • Exact PR diff: 8f6b66f9ff6b2e42cd713506805763269375b473...7b1edfe150ce3049ac3811cfd102bbe68b282bfa
  • Model: gpt-5.6-sol

💡 Click "edited" above to see earlier reviews for this PR.


Review Summary

Overall Risk: NONE

No concrete security, correctness, or reliability regressions were found in the authorized PR range.

Findings

No concrete security, correctness, or reliability findings were identified.

Notes

  • No additional limitations were reported.

Generated by Codex Security Review |
Requested by: @wpfleger96 |
Workflow run

…unused counter helpers

The final marker UPDATE in each admin mutation action (ban, timeout, kick,
delete, record_failure) compared action_lease_expires_at against now(), which
PostgreSQL fixes at transaction start. If the lease expired while the domain
write was in progress, the fence still saw the old timestamp and committed.

Switch the final ownership fence in all five actions to clock_timestamp() so
it reads wall-clock time at the moment of the UPDATE. The pre-entry ownership
SELECT that short-circuits before any domain write is unchanged (now() is
correct there: it runs at statement start with nothing waiting behind it).

Add admin_delete_with_lease_expiring_during_write_changes_nothing to reproduce
the gap deterministically: a per-row trigger on events delays the target UPDATE
by 1 s; the lease is set to expire after 500 ms. Under the old now() predicate
the transaction committed; with clock_timestamp() it rolls back.

Rename admin_delete_with_lost_lease_changes_nothing to
admin_delete_with_already_expired_lease_changes_nothing to accurately describe
what it proves (pre-entry rejection of an already-expired lease).

Remove increment_reply_count, decrement_reply_count, and the Db::decrement_reply_count
wrapper from thread.rs. No production callers exist outside the forwarding
wrappers; both helpers perform unguarded autocommit writes that bypass the
deletion-transition/atomicity invariant enforced by the canonical paths.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@github-actions github-actions Bot removed the codex-security-review-current The posted Codex security review matches its recorded range. label Sep 24, 2026
@wpfleger96 wpfleger96 changed the title 🤖 fix(relay): update thread_metadata counters on admin delete fix(relay-admin): use clock_timestamp() at final lease fence; remove unused counter helpers Sep 24, 2026
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Sep 24, 2026

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carl, an automated reviewer, commenting via Wes’s GitHub account.

Review clear: no blocking code, product, or security defects found. Reviewed HEAD dc6c535da566c5c392cf9f33a023a6b904eae8eb against BASE d01e5f82058463709a22e93bb4cd795da5f53e10.

Source review covered canonical deletion/counter atomicity and idempotency, tenant scope, removed-helper callers, all five final lease fences and ownership/recovery paths, and canonical best-effort live-summary integration. One non-blocking timing-test robustness note is inline.

Validation: existing PostgreSQL CI passed 476 tests, including all four new regressions, on GitHub’s merge of this head into the pinned base. This review ran no code or tests.

CI is not fully green: Desktop Smoke E2E (4) fails the middle-page scroll coverage assertion. That unchanged spec runs against an in-browser mock bridge without the Rust relay/database, so it does not exercise this change. The desktop gate still needs separate disposition; this is not a merge approval.

.expect("install delay trigger");

let lease_until = Utc::now() + chrono::Duration::milliseconds(500);
let (action_id, token) = enforcing_action(&pool, community_id, lease_until).await;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking test robustness: the 500ms lease starts before enforcing_action completes its database setup. If setup consumes that interval, execute_delete_with_marker exits at the unchanged pre-entry ownership check, and the assertions still pass without reaching the final clock_timestamp() fence. Please ensure the lease is live after setup (and fail explicitly if that precondition is lost), rather than allowing slow setup to silently turn this into another already-expired-lease test. No production fencing defect found; this is a weakness in the regression test's evidence.

clock_timestamp() in the marker UPDATE predicate can be evaluated by
Postgres before it waits to lock the target row. If another transaction
holds the action row and releases it unchanged after the lease expires,
the update commits — the expired owner persists its mutation, marker, or
failure state contrary to the guarantee.

Fix: for every admin-action path (ban, timeout, kick, delete, and
record_failure) acquire the action row with SELECT ... FOR UPDATE inside
the same transaction before the domain write. The wall-clock expiry check
in the marker UPDATE then runs under that lock, not before it. The token/
state/marker CAS and the single-transaction structure are unchanged.

record_failure previously used a single-statement execute(pool). It now
begins a transaction, acquires the row lock with an explicit SELECT FOR
UPDATE / expiry check, then updates state; this brings it in line with the
four mutation functions.

Also fix the in-flight-expiry regression test:
- assert the lease is live immediately after setup so a slow run fails
  the precondition instead of passing through early rejection
- assert the stored step_marker is NULL as well as the return value and
  event state
- correct the comment arithmetic (1 s is *longer* than 500 ms)

Add admin_delete_with_lease_expiring_during_row_lock_wait_changes_nothing
to exercise the new lock-first shape: an external transaction locks the
action row, the worker blocks at its FOR UPDATE, the test asserts the
lease is still live at that point, waits for DB-clock expiry, releases
the unchanged row, and asserts the worker returns false with the event,
both counters, and step_marker unchanged.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@github-actions github-actions Bot removed the codex-security-review-current The posted Codex security review matches its recorded range. label Sep 24, 2026
@wpfleger96 wpfleger96 changed the title fix(relay-admin): use clock_timestamp() at final lease fence; remove unused counter helpers fix(relay-admin): make thread deletions atomic and fence expired action leases under row lock Sep 24, 2026
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Sep 24, 2026
…lock test comment

admin_delete_with_lease_expiring_during_write_changes_nothing: replace the
timed pg_sleep trigger with a lock-based approach. The target event row is
locked externally; the worker acquires the action-row lock first (via its
existing SELECT FOR UPDATE), then blocks at the UPDATE events domain write.
pg_stat_activity + pg_blocking_pids confirms the worker is observably
blocked while the lease is live before expiry is asserted; a slow setup
fails the named precondition instead of passing through early rejection.
Releases the event row unchanged via rollback, then asserts rejection,
event, counters, and stored marker.

The reverted-now() + 650 ms delay mutation that previously produced a
false green now produces RED on this test as well (both expiry tests RED).

admin_delete_with_lease_expiring_during_row_lock_wait_changes_nothing:
update the doc comment — the worker blocks at the SELECT FOR UPDATE, not
the marker UPDATE. Broaden the pg_stat_activity observer to accept either
the FOR UPDATE or the marker UPDATE query, so lock-removal fails on
committed behavior rather than on SQL shape.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@github-actions github-actions Bot removed the codex-security-review-current The posted Codex security review matches its recorded range. label Sep 24, 2026
…amp() change

The previous commit's mutation-restore step used a blanket sed substitution
that accidentally changed the four pre-entry ownership SELECT EXISTS checks
(ban:468, timeout:556, kick:656, delete:746) from now() to clock_timestamp().
Those checks are early-rejection optimizations that are correct with now();
only the final fences require clock_timestamp(). This commit reverts them.

Production code is now byte-identical to c5aaacc; this commit touches only
the mod postgres_tests section.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Sep 24, 2026
@github-actions github-actions Bot added codex-security-review-current The posted Codex security review matches its recorded range. and removed codex-security-review-current The posted Codex security review matches its recorded range. labels Sep 24, 2026

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carl, an automated reviewer, commenting via Wes’s GitHub account.

Review clear: no blocking code, product, or security defects found in the corrective change. HEAD 16cb762f692aa0a289bdd6fd40d024b177c9fe51, BASE d01e5f82058463709a22e93bb4cd795da5f53e10.

Follow-up to the prior review: checked all five lock-first/final-clock fences, ownership changes during waits, deletion/counter rollback, and post-commit summary integration. The new observed-lock/DB-clock regressions address the earlier timing-test concern. Lease validity is checked under the action-row lock, not at physical COMMIT completion.

Validation: existing PostgreSQL CI passed 477/477 on merge c98b1710d0ebedd869af1b787b7daf54e4293944, containing this head but using base 99c2acf90cfbb1cb2d3a8bd900c0ec1642e20540. This is not standalone-head or pinned-base certification. This source-only review ran no code or tests.

Non-blocking housekeeping: name the two new pg_stat_activity regressions with cluster_global_, per crates/buzz-db/TESTING.md:40–43. Both queries filter to their database and exact blocker PID; no concrete cross-test failure was established, so this is not a correctness blocker.

Broader CI was red in the snapshot, with relay/integration E2E failures and some checks still running. Their cause was not established here; those gates still need disposition. This is not merge approval.

…ead-counters

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>

* origin/main:
  chore(release): release Buzz Desktop version 0.5.25 (#7867)
  fix(ci): consume the published MinIO image (#7870)
  fix(mobile): converge sidebar managers on relay head with resume re-read (#7806)
  fix(ci): bootstrap the reusable MinIO image in GHCR (#7869)
  Discover alternate Buzz ACP commands (#6948)
  fix(hooks): surface nextest failures and stale pnpm deps in pre-push (#7850)
  feat(acp): run one prepared task from a file or stdin (#7851)
  Fix mobile heart and warning emoji with native font fallback (#7842)
  chore(mesh): upgrade MeshLLM to 0.76.2 (#7559)

Signed-off-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz>
@github-actions github-actions Bot removed the codex-security-review-current The posted Codex security review matches its recorded range. label Sep 24, 2026
Both inspect cluster-wide PG state, so nextest must serialize them.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Sep 24, 2026
@wpfleger96
wpfleger96 enabled auto-merge (squash) September 24, 2026 21:48

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carl, an automated reviewer, commenting via Wes’s GitHub account.

CLEAR: no actionable regression found since the prior clear review.

  • The branch-specific delta renames two PostgreSQL lease-expiry tests into the existing cluster_global_ serial group, resolving the prior housekeeping note. The production deletion/counter, lease-fence, and live-summary paths are unchanged. Main-merge integration was reviewed separately.
  • Source-only review with an independent DB lane; no builds, tests, or PR code executed. Existing exact-head PostgreSQL CI reports success. The broader snapshot had 50 successful checks, 26 skipped and two desktop smoke checks still running, so this is not an all-green claim.
  • No code fixes requested. Required CI remains a separate merge gate; unchanged unrelated surfaces were excluded.

Head: 7b1edfe150ce3049ac3811cfd102bbe68b282bfa. Base: 02753722a7dd06560402a5b92491b048968c1a63.

@wpfleger96
wpfleger96 merged commit 676e8c4 into main Sep 24, 2026
139 of 143 checks passed
@wpfleger96
wpfleger96 deleted the hayt/admin-delete-thread-counters branch September 24, 2026 21:55
brow added a commit that referenced this pull request Sep 25, 2026
…ction

* origin/main: (21 commits)
  docs(vision): add /buzz/v1 read endpoints to the protocol contract (#7879)
  🤖 fix(justfile): point just staging at the current staging relay (#7881)
  fix(relay-admin): make thread deletions atomic and fence expired action leases under row lock (#7853)
  feat(desktop): relay admin console for the /api/admin/v1 operator surface (#4768)
  fix(mobile): keep retired sections manager out of successor cache (#7873)
  Select one feature flag provider at compile time (#7677)
  chore(release): release Buzz Desktop version 0.5.25 (#7867)
  fix(ci): consume the published MinIO image (#7870)
  fix(mobile): converge sidebar managers on relay head with resume re-read (#7806)
  fix(ci): bootstrap the reusable MinIO image in GHCR (#7869)
  Discover alternate Buzz ACP commands (#6948)
  fix(hooks): surface nextest failures and stale pnpm deps in pre-push (#7850)
  feat(acp): run one prepared task from a file or stdin (#7851)
  Fix mobile heart and warning emoji with native font fallback (#7842)
  chore(mesh): upgrade MeshLLM to 0.76.2 (#7559)
  feat(agents): humanize uncurated Databricks model ids with a label grammar (#7844)
  fix: route databricks claude fqns to anthropic messages (#7829)
  feat(relay): add opt-in newest-first thread windows (#7823)
  refactor: move agent Git bootstrap into ACP harness (#7819)
  Use worker snapshots for relay storage metrics (#7845)
  ...

Signed-off-by: Tom Brow <tomb@block.xyz>
wpfleger96 pushed a commit that referenced this pull request Sep 25, 2026
…rcement

* origin/main:
  docs: specify durable data backfills (#7326)
  docs(vision): add /buzz/v1 read endpoints to the protocol contract (#7879)
  🤖 fix(justfile): point just staging at the current staging relay (#7881)
  fix(relay-admin): make thread deletions atomic and fence expired action leases under row lock (#7853)
  feat(desktop): relay admin console for the /api/admin/v1 operator surface (#4768)
  fix(mobile): keep retired sections manager out of successor cache (#7873)
  Select one feature flag provider at compile time (#7677)
  chore(release): release Buzz Desktop version 0.5.25 (#7867)

Signed-off-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz>
brow added a commit that referenced this pull request Sep 25, 2026
…ivery

* origin/main:
  feat(relay): enforce NIP-FI assertion+NIP-98 pairing on HTTP ingress (#7264)
  fix(ci): run the admin disabled-mode DB test in the PostgreSQL lane (#7900)
  🤖 docs: add pre-PR checklist and review guidance to AGENTS.md (#7897)
  docs: specify durable data backfills (#7326)
  docs(vision): add /buzz/v1 read endpoints to the protocol contract (#7879)
  🤖 fix(justfile): point just staging at the current staging relay (#7881)
  fix(relay-admin): make thread deletions atomic and fence expired action leases under row lock (#7853)
  feat(desktop): relay admin console for the /api/admin/v1 operator surface (#4768)
  fix(mobile): keep retired sections manager out of successor cache (#7873)

Signed-off-by: Tom Brow <tomb@block.xyz>

This branch was successfully deployed

1 active deployment
codex-review — 7b1edfe1 Deployed Sep 24, 2026 by wpfleger96 via Run Codex Security Review #5576
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

codex-security-review-current The posted Codex security review matches its recorded range.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants