Stage 8: Delivery worker — Background sending across pods

[bakhterets]

Implement the background worker that delivers queued notifications across pods.

Files: internal/notification/worker.go

Tasks:

• Implement hot path: in-process publisher signal after DB commit to send immediately.
• Implement safety ticker: low-frequency scan (e.g., 5m) for due rows and stale locks.
• Recover stuck rows: if status='processing' and locked_at is older than lease timeout, recover to pending (or failed if max attempts exhausted).
• Claim batch: SELECT ... FOR UPDATE SKIP LOCKED where status='pending' and next_attempt_at <= now() and mark as processing with locked_by/locked_at and increment attempts.
• Commit claim transaction, then send (never hold DB lock during SMTP).
• Per-row recover() isolation: each send runs in panic guard so one failure does not crash worker.
• Update outbox on result: sent → status='sent'; failure with retries left → status='pending' with next_attempt_at = now() + backoff(); failure with no retries → status='failed'.
• Backoff strategy: exponential with cap (e.g., 5m → 10m → 20m → 40m → max 2h).
• Bounded batch size (e.g., 50 rows) and concurrency. Ensure lease_timeout > SMTP_timeout + send time.
• Graceful shutdown: stop claiming new rows, finish in-flight sends, wait for graceful deadline.
• Verify: unit tests for backoff math, lease recovery, panic isolation, FOR UPDATE SKIP LOCKED no double-claim, signal path and ticker path.

Deliverable: Background worker reliably delivers queued notifications with lease management and retry logic.