Skip to content

daemon: timeout WebSocket serial open_port pending attaches #977

Description

@zackees

Context

During the FastLED dual-board stress run, the daemon repeatedly logged that it was staying alive because it had open serial sessions plus pending serial attaches:

Daemon staying alive: 2 open serial sessions, 2 pending serial attaches

Issue #976 tracks the operator-facing recovery gap: fbuild daemon locks and clear-locks need to expose and recover stale serial sessions. There is still a narrower self-healing gap in the daemon: the WebSocket serial attach flow can remain counted as a pending attach while awaiting open_port(...) without a wall-clock deadline.

The current WebSocket path has a bounded first-frame handshake:

  • crates/fbuild-daemon/src/handlers/websockets.rs uses WS_ATTACH_HANDSHAKE_TIMEOUT around socket.recv().

But after parsing SerialClientMessage::Attach, the same handler awaits:

  • ctx.serial_manager.open_port(&port, baud_rate, &client_id, ...)

without the timeout pattern already used by the HTTP monitor/deploy paths:

  • crates/fbuild-daemon/src/handlers/operations/monitor.rs wraps open_port with SERIAL_OPEN_PORT_TIMEOUT_SECS.
  • crates/fbuild-daemon/src/handlers/operations/deploy.rs wraps post-deploy monitor open_port with POST_DEPLOY_OPEN_TIMEOUT.

If USB re-enumeration, a Windows serial driver, or the serial manager path stops making progress, the WebSocket handler can keep pending_serial_attaches positive and make the daemon look wedged until the process is restarted or external clients are killed.

Proposal

Apply the same hard-deadline pattern to the WebSocket serial attach open_port await.

Suggested behavior:

  • Add a constant such as WS_SERIAL_OPEN_PORT_TIMEOUT, likely 30 seconds to match the HTTP monitor/deploy paths.
  • Wrap the WebSocket attach open_port(...) await in tokio::time::timeout(...).
  • On timeout, send a SerialServerMessage::Error frame explaining that open_port(<port>) exceeded the deadline, then close the WebSocket.
  • Ensure the pending attach guard is dropped on every timeout/error return so DaemonContext::busy_reason() does not keep reporting a stale pending attach.
  • Add a regression test using a delayed/hung serial manager or injectable open future so the timeout path can be verified without real hardware.

Acceptance criteria

  • A WebSocket serial attach whose open_port(...) future does not complete returns an error and closes within a bounded time.
  • After the timeout path exits, pending_serial_attaches is decremented and the daemon can become idle/self-evict normally.
  • The WebSocket attach path's timeout behavior is consistent with the HTTP monitor/deploy open_port timeout behavior.
  • Regression coverage proves the pending attach count does not remain stuck after an open_port timeout.

Related issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    Triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions