buzz-relay: git conformance probe hangs startup forever when a backend connection stalls (no request timeout, no deadline)

[webdevtodayjason]

Symptom

Fresh self-hosted deployment from the deploy/compose bundle (image ghcr.io/block/buzz:main, the bundle's pinned MinIO RELEASE.2025-09-07T16-13-09Z). The relay boots, starts the A3 conformance probe, logs two transport drops, and then goes silent forever:

14:34:29 INFO  running git object-store conformance probe (A3 gate) race_width=32 race_rounds=3
14:34:30 WARN  transport drop (pre-classification: socket/send failure) phase=if_match_race round=0 racer=14
14:34:33 WARN  transport drop (pre-classification: socket/send failure) phase=if_none_match_race round=0 racer=6
(no further log output — observed for 10+ minutes before intervention)

The probe runs before the health/main listeners bind, so:

• /_readiness and the relay port refuse connections,
• the container healthcheck loops unhealthy indefinitely,
• nothing restarts it — the process is alive, just pending on a future that will never resolve.

Net effect: the documented self-hosted quickstart bricks whenever a single backend connection stalls, with no error, no timeout, and no diagnostic beyond silence. BUZZ_GIT_CONFORMANCE_PROBE=false is the only way out, and nothing tells the operator that.

Root cause

Three things compound:

1. The shared rust-s3 bucket client (GitStore::new → Bucket::new) sets no request timeout — a connection that is accepted but never answered pends indefinitely.
2. Both race phases collect racers with futures_util::future::join_all, which waits for every racer — one stalled socket parks the whole phase.
3. run_conformance_probe has no overall deadline, and main.rs awaits it before binding any listener.

The probe's existing drop-and-floor classification correctly handles racers that fail with S3Error::{Reqwest, Http, Io} — but a stalled connection never produces an error to classify. Under 32-way concurrent PUTs to the same key, MinIO occasionally does exactly this (note the adjacent socket/send failures in the same rounds).

Deterministic repro (no MinIO required)

Point a GitStore at a TCP listener that accepts connections and never responds, then call run_conformance_probe — it never returns. The linked PR adds exactly this as a regression test ("tarpit").

Proposed fix

See the linked PR: per-operation timeout (racer elapse = transport drop under the existing drop-and-floor rule, so A3 semantics are unchanged) plus a whole-probe deadline that fails closed with phase deadline and actionable guidance. Both env-overridable (BUZZ_GIT_PROBE_OP_TIMEOUT_SECS, BUZZ_GIT_PROBE_DEADLINE_SECS, matching the existing BUZZ_GIT_PROBE_WRITERS/_ROUNDS naming).

Related but distinct: #2470 is the probe failing on GCS (a classified, visible failure) and #2511 improves failure messaging + adds a Helm disable toggle — both assume the probe produces an outcome. This issue is the case where it produces nothing at all.

Environment

• Relay: ghcr.io/block/buzz:main (2026-07-24), single-node deploy/compose on Docker 29.x
• Object store: MinIO RELEASE.2025-09-07T16-13-09Z (the bundle's pinned image), same host
• Probe config: defaults (width 32, rounds 3)

[webdevtodayjason]

Fix opened: the PR linked above bounds every probe operation (racer timeout = transport drop under the existing drop-and-floor rule) and adds a whole-probe deadline, with tarpit regression tests that reproduce this exact hang.

[jsilets]

+1, same symptom on Railway with MinIO in the same project. With BUZZ_GIT_CONFORMANCE_PROBE=true the relay logs "running git object-store conformance probe (A3 gate)" and then nothing, restart-looping while the deployment reports healthy. BUZZ_GIT_CONFORMANCE_PROBE=false fixes startup. A bounded probe as in #2724 would have saved a debugging session, since the hang produces zero error output.

[talfirevic]

Fresh reproduction on a current revision, plus a confirmed workaround and one detail that I think changes the severity of this issue.

Stack: deploy/compose bundle unmodified except for a reverse-proxy override, bundled MinIO RELEASE.2025-09-07T16-13-09Z, Postgres 17, Redis 7. Relay image ghcr.io/block/buzz@sha256:e0115067… (revision 5bf78671).

The hang is intermittent, not backend-specific

This is the part worth flagging: the title frames this as "when a backend connection stalls", which reads like a misbehaving or exotic S3 backend. On my deployment it is the bundled MinIO from the documented quickstart, and the same stack had already completed the probe successfully three times before hanging on the fourth:

boot	revision	probe result
1	2b873cf	passed in ~7s
2	02f640bc	passed in ~7s
3	02f640bc	passed in ~7s
4	5bf78671	hung indefinitely

Same MinIO container throughout (up 36h, healthy, untouched across all four relay restarts). So this looks like a race that any self-hoster can hit on any restart, rather than a property of a particular backend.

The transport-drop WARNs do not distinguish the two cases. A successful boot logs them too:

INFO running git object-store conformance probe (A3 gate) race_width=32 race_rounds=3
WARN transport drop (pre-classification: socket/send failure) phase=if_none_match_race round=2 racer=17
...
INFO git object-store backend admitted: A3 conformance probe passed race_width=32 race_rounds=3 transport_drops=3
INFO Health probe listener started port=8080

The hanging boot is identical up to the drops, then simply stops:

08:49:44 INFO running git object-store conformance probe (A3 gate) race_width=32 race_rounds=3
08:49:46 WARN transport drop (pre-classification: socket/send failure) phase=if_match_race round=1 racer=7
(nothing further)

So transport_drops on its own is not a useful signal — a passing run had three of them.

Impact

Everything before the probe succeeds — migrations complete, partitions reconcile, owner bootstraps, Redis/Postgres/MinIO/search/media all report ready. Then the health listener never binds:

bash: connect: Connection refused
bash: line 1: /dev/tcp/127.0.0.1/8080: Connection refused

14 consecutive healthcheck failures, container unhealthy forever, process alive. Behind a reverse proxy this surfaces as a hard 502 to users. docker compose up -d --wait reports the unhealthy container but still exits 0, so an upgrade script that doesn't explicitly assert health will report success on a bricked relay.

Workaround (confirmed)

Setting BUZZ_GIT_CONFORMANCE_PROBE=false and restarting the identical image boots cleanly and stays healthy — relay responding 200, NIP-11 served, WebSocket upgrade 101, zero ERROR lines. That isolates the fault to the probe rather than the build, and unblocks anyone currently stuck.

Caveat, stated plainly: this skips a real correctness gate on S3 CAS semantics rather than fixing anything, and I can't prove the probe would have passed on that boot had it not hung.

A bounded deadline on the probe with a loud failure — or letting the health listener bind before the probe runs, so a stalled gate is diagnosable instead of silent — would turn this from "quickstart bricks with no output" into an ordinary error.

[ppounder]

Another independent hit of this today, on the same revision 5bf78671 as the comment above — bundled deploy/compose stack, bundled MinIO RELEASE.2025-09-07T16-13-09Z, unmodified apart from a reverse proxy. Same signature: probe starts, one transport-drop WARN, then silence; everything before it (migrations, partitions, owner bootstrap, Redis/Postgres/MinIO/search/media) reports ready, and the listener never binds.

Two things I can add that I don't think are in the thread yet.

restart: unless-stopped is not a mitigation. The process stays alive and never exits, so Docker's restart policy never triggers. The container sits unhealthy indefinitely with nothing to recover it. Operators reading the bundle's restart policy as a safety net (I did) don't get one — external health-polling is the only thing that recovers this.

The unattended case is where it hurts. Mine was triggered by a scheduled overnight update, so the relay was down ~9 hours before anyone looked. As noted above, docker compose up -d --wait exits 0 on the unhealthy container, so the update reported success. Intermittent + silent + exit-0 means the failure is invisible precisely when no one is watching; interactive repros understate the severity.

Restarting the identical image brought it up in ~7s, consistent with the intermittency reported above.

FWIW #2724 looks review-ready (mergeable, checks green) and has been open since 24 Jul while reports keep landing — 4 reporters now. Happy to test it against this deployment if that's useful.