Retry failed YouTube WebSub subscriptions promptly after a successful redeploy #1692

Problem

A successful site redeploy can inherit a long YouTube WebSub retry delay from the previous deployment. The failure count and RetryAt are persisted in Redis, and the first tick of a newly started leader still honors that deadline. After deploying a potential fix, we can therefore wait for the remainder of the capped, approximately one-hour backoff before exercising the changed code.

This is a retry-eligibility problem, not a reason to reset the distributed leadership lease or discard a valid Google subscription lease. Persisted backoff protects against retry storms and should remain the normal behavior.

Follow-up to #1685; preserve the distributed backoff protections introduced in #1674.

Production evidence

Read-only observations on September 17, 2026, between 19:50:30 and 19:50:56 UTC:

• The YouTube leadership key's TTL was extended twice during the observation, so a missing leadership lease did not explain the delay at that time.
• Subscription state had 43 accumulated failures, compared with an earlier log reporting attempt 39. Attempts had continued; this was not evidence that the worker had stopped.
• There was no pending subscription request, active topic, or recent verification confirmation.
• The persisted next retry deadline was 20:06:54.739 UTC, approximately 16 minutes after inspection.

Earlier real hub requests returned HTTP 503. The latest failure's HTTP status was not available in the retrieved logs. A deployment-aware retry would improve recovery and diagnosis; it is not an established fix for Google's 503 responses.

Proposed behavior

Allow one coordinated, expedited subscription attempt after a new successful deployment, when a failed subscription or overdue renewal would otherwise be held behind an older retry deadline.

• Run the attempt through the existing leader and subscription-reservation mechanisms, on the first eligible worker tick after the deployment is ready. Do not introduce an independent subscription sender.
• Claim the expedited attempt atomically in shared state. Use a reliable deployment/recovery identifier, not a random process or leader identifier. Concurrent replicas and overlapping old/new revisions must not repeatedly unlock attempts.
• Preserve valid subscription leases and pending verification requests. Do not flush Redis, delete leadership keys, weaken callback validation, or treat HTTP acceptance as successful verification.
• Keep failure history. If the expedited attempt fails, resume normal persisted backoff rather than resetting the counter and starting a fresh rapid retry sequence.
• Define how planned configuration restarts differ from crash loops, scale-out, and ordinary leadership changes. Repeated restarts must not become a way to generate unbounded attempts.
• Account for explicit provider retry constraints when deciding whether an expedited attempt is eligible. Do not increase normal YouTube Data API discovery polling or its request budget.

Acceptance criteria

• A new ready deployment can make one eligible recovery attempt without waiting out the previous deployment's full retry delay.
• The attempt budget is shared across replicas and remains bounded during rolling deployment, leader handoff, and repeated process restarts.
• Healthy subscriptions, valid pending callbacks, leadership ownership, and unrelated Redis state are preserved.
• Failure returns to normal persisted backoff; only a valid verification callback establishes or renews the subscription.
• Safe diagnostics explain whether startup recovery was allowed or deferred, including the reason and next eligible time, without logging tokens or raw subscription state.
• Deterministic tests cover future retry deadlines, concurrent replicas, overlapping revisions, crash loops, leadership loss, pending verification across restart, healthy active leases, and failed expedited attempts.
• Operational documentation explains deployment/restart behavior and expected recovery timing.

Relevant code

• src/statichost/StaticHost/Live/YouTube/YouTubeWebSubService.cs: RunLeaderAsync, TickAsync.
• src/statichost/StaticHost/Live/YouTube/YouTubeWebSubSubscriptionState.cs: YouTubeWebSubSubscriptionTransitions.TryBegin, MarkRequestFailed.
• src/statichost/StaticHost/Live/YouTube/RedisYouTubeWebSubSubscriptionState.cs: persisted state and optimistic concurrency.
• src/statichost/StaticHost/Live/LiveStatusCoordination.cs: distributed leadership.
• src/statichost/StaticHost/Live/README.md: operational guidance.

Non-goals

No production state reset as part of filing this issue. Do not change the hub/topic URLs, remove verification safeguards, or claim that redeployment itself repairs the external hub failure.