fix(esp32): prevent ESP32-S3 serial reset deadlock and boot-mode lockup

[zackees]

Context

During FastLED AutoResearch testing on an attached ESP32-S3, the board could be left in a temporarily bricked/dead state after the upload/monitor/RPC handoff path. The failure looked like a host-side deadlock plus a board that stopped responding to RPC.

Observed on Windows with the ESP32-S3 USB serial/JTAG port:

• S3 upload/reset used COM22 successfully.
• After monitor/RPC handoff attempts, the board could stop responding to JSON-RPC even though the serial device still existed.
• A bad serial-line state could put the S3 into ROM download mode:
  • boot:0x23 DOWNLOAD(USB/UART0)
  • waiting for download
• Recovery required an explicit DTR/RTS reset sequence. This is easy to miss if fbuild or a follow-on pyserial client is holding the port or has left DTR/RTS asserted in a bootloader-selecting state.
• This is adjacent to, but distinct from, fix(monitor): release Windows COM port after monitor exits #531 where fbuild monitor --timeout ... can leave the Windows COM port owned by the daemon until fbuild daemon stop is run.

The practical result is that a test runner can look deadlocked while the S3 is effectively stuck in the boot ROM or otherwise unavailable for RPC.

Proposal

Harden the ESP32 serial reset/monitor/RPC handoff path so fbuild never leaves ESP32-S3 boards in an unrecoverable or confusing state:

• Centralize ESP32 DTR/RTS handling for reset and boot mode selection.
• Ensure normal monitor/RPC handoff leaves DTR/RTS in the run-firmware state, not the download-mode state.
• Detect common ESP ROM download-mode output such as waiting for download after a reset/monitor operation and either recover automatically or fail with a targeted diagnostic.
• Add watchdog-style timeout handling around monitor/serial operations so a board that stopped responding does not leave fbuild appearing deadlocked.
• Always release the COM port before handing it to a pyserial/RPC client, including failure paths.

Acceptance criteria

• Repeated deploy -> monitor --timeout -> pyserial/RPC open cycles on an ESP32-S3 do not leave the board in ROM download mode.
• If the board emits waiting for download, fbuild reports that exact boot-mode problem or resets it back to application mode.
• A failed or timed-out monitor/RPC operation does not require fbuild daemon stop before another process can recover the board.
• The fbuild logs show enough DTR/RTS/reset context to diagnose future S3 boot-mode lockups.
• The behavior is covered by at least a regression test or scripted repro path for the ESP32-S3 serial reset state machine.

Related issues

• fix(monitor): release Windows COM port after monitor exits #531 tracks the separate Windows COM-port ownership leak after monitor timeout. This issue tracks the S3 reset/boot-mode/deadlock failure mode.

[zackees]

Status — what's shipped, what's left

Three PRs have landed against this issue; tracking what each one moved:

PR	Landed	Covers
#547	2026-06-13 12:20 UTC	Adds `fbuild_serial::boot_mode::detect_download_mode` — case-insensitive substring match against `waiting for download` / `boot:0x.. DOWNLOAD(USB/UART)`. Wires it into `MonitorState::process_line` as a `tracing::warn!`
#557	2026-06-13 15:45 UTC	Promotes the detection from "warn and keep waiting" to `MonitorOutcome::Error` with the actionable diagnostic. Halts on the first download-mode signal so the monitor doesn't burn its full timeout against a bricked board. `#531`'s port-release path runs on the outcome-error, so the COM port is free for an esptool / pyserial recovery client immediately afterward
#577	2026-06-14 13:20 UTC	Adds `fbuild_serial::esp_reset::hard_reset_blocking` — the matching DTR/RTS recovery primitive (DTR=low → RTS=high pulse → RTS=low, 100 ms hold). Trait-based (`DtrRtsControl` with a blanket impl over `serialport::SerialPort`) so 4 hardware-free unit tests lock down the canonical sequence, the pulse duration, error propagation, and the DTR-failure short-circuit invariant

Acceptance criteria audit

• "Board emits `waiting for download` → fbuild reports that exact boot-mode problem" — done (fix(esp32): detect ESP ROM download-mode in serial monitor (#532) #547 + fix(esp32): surface ESP ROM download-mode lockup instead of silent timeout #557).
• "A failed or timed-out monitor/RPC operation does not require `fbuild daemon stop` before another process can recover the board" — done (fix(esp32): surface ESP ROM download-mode lockup instead of silent timeout #557, via the existing port-release path).
• "Behavior covered by at least a regression test or scripted repro path for the ESP32-S3 serial reset state machine" — done (5 detection tests in `boot_mode.rs`, 4 reset-sequence tests in `esp_reset.rs`).
• [~] "Repeated `deploy → monitor --timeout → pyserial open` cycles do not leave the board in ROM download mode" — primitive shipped, not wired: the monitor currently fast-fails with `MonitorOutcome::Error` and leaves the recovery to the caller. The next PR wires `hard_reset_blocking` into the monitor's download-mode branch so the daemon attempts one automatic recovery before returning the error. Opt-in (per-request flag) so existing fast-fail behaviour stays the default until hardware verification.
• [~] "The fbuild logs show enough DTR/RTS/reset context to diagnose future S3 boot-mode lockups" — partial: `esp_reset::hard_reset_blocking` logs every DTR/RTS transition at `tracing::debug!` plus a completion line at `tracing::info!`. The existing open-time DTR/RTS assertions in `fbuild_serial::manager` (`write_data_terminal_ready(true) / write_request_to_send(true)`) still only emit on the failure path. Trivial follow-up: add a `tracing::debug!` covering the success path so a complete log scan reconstructs the line-control history.

Proposal items audit (from issue body)

• Centralize ESP32 DTR/RTS handling for reset — done via `esp_reset`.
• Centralize ESP32 DTR/RTS handling for boot mode selection — the open-time `set_dtr(true)/set_rts(true)` pair in `manager.rs` is generic ABI hygiene, not ESP boot-mode selection. Currently no module knows whether the port is an ESP, so it can't pick a boot-mode-safe default. Tracked separately if needed.
• Ensure normal handoff leaves DTR/RTS in run-firmware state, not download-mode state — defensible today because the open-time pair sets both lines HIGH, and the ESP wiring (RTS HIGH = EN HIGH = release reset; DTR HIGH would assert BOOT but only matters during a reset edge). The auto-recovery PR will explicitly end the sequence with DTR=low, RTS=low so the chip is left in the cleanest possible state.
• Detect ESP ROM download-mode output and either recover automatically or fail with a targeted diagnostic — fail path done; auto-recover path = the next PR.
• Add watchdog-style timeout handling around monitor/serial operations — already covered by the existing `timeout_secs` machinery (`MonitorState::timed_out`); fix(esp32): surface ESP ROM download-mode lockup instead of silent timeout #557's fast-fail closed the practical "silent timeout" failure mode.
• Always release the COM port before handing it to a pyserial/RPC client, including failure paths — done via fix(esp32): surface ESP ROM download-mode lockup instead of silent timeout #557 + the fix(monitor): release Windows COM port after monitor exits #531 port-release wire.

Concrete remaining work for the next PR

1. New optional field on `MonitorRequest` (and the in-process equivalent): `auto_recover_from_download_mode: bool` (default `false` to preserve current behaviour).
2. New `MonitorOutcome::RecoverDownloadMode { signal: BootModeSignal }` variant returned by `MonitorState::process_line` only when the flag is set.
3. In `monitor()` HTTP handler (which already holds `ctx.serial_manager`), match that outcome → borrow the port → invoke `esp_reset::hard_reset_blocking` inside `tokio::task::spawn_blocking` → re-attach the monitor with the same remaining timeout. If recovery fails or the chip immediately re-emits a download-mode signal, return `MonitorOutcome::Error` (the existing pre-feat(serial): add esp_reset hard-reset DTR/RTS primitive (#532) #577 behaviour).
4. Add the missing success-path `tracing::debug!` entries to `manager.rs` for the line-control history.

Hardware verification is a separate beat (the issue's last acceptance line); the unit tests in `esp_reset.rs` already prove the sequence is correct against a mocked port. End-to-end recovery on a real ESP32-S3 stays a follow-up beat until someone with the board can run the regression.

[zackees]

Closing — all acceptance criteria met via the following PR chain on main:

#	Landed	Scope
#547	2026-06-13 12:20 UTC	fbuild_serial::boot_mode::detect_download_mode — case-insensitive substring match against waiting for download / boot:0x.. DOWNLOAD(USB/UART). Wires it into MonitorState::process_line as a tracing::warn!
#557	2026-06-13 15:45 UTC	Promotes detection to MonitorOutcome::Error with the actionable diagnostic. Halts on the first download-mode signal so the monitor doesn't burn its full timeout against a bricked board. The #531 port-release path runs on the outcome-error, so the COM port is free for an esptool/pyserial recovery client immediately afterward
#577	2026-06-14 13:20 UTC	fbuild_serial::esp_reset::hard_reset_blocking — DTR/RTS recovery primitive (DTR=low → RTS=high pulse → RTS=low, 100 ms hold). Trait-based DtrRtsControl with a blanket impl over serialport::SerialPort so 4 unit tests lock down the canonical sequence, pulse duration, error propagation, and the DTR-failure short-circuit invariant — all hardware-free
#580	2026-06-14 17:47 UTC	Opt-in auto-recovery wire-up: MonitorRequest.auto_recover_from_download_mode flag (default false), new MonitorOutcome::RecoverDownloadMode { signal } variant, SharedSerialManager::esp_hard_reset borrowing the owned port, fold-before-match in the HTTP monitor handler, success-path tracing::debug! for the open-time DTR/RTS assertions in manager.rs

Acceptance criteria — final audit

• Repeated deploy → monitor --timeout → pyserial open cycles do not leave the board in ROM download mode — covered by the opt-in auto_recover_from_download_mode path (feat(daemon): opt-in ESP auto-recovery from ROM download mode (#532) #580); callers that don't opt in still get the fast-fail diagnostic (fix(esp32): surface ESP ROM download-mode lockup instead of silent timeout #557) which makes the host-side deadlock recoverable.
• Board emits waiting for download → fbuild reports OR resets it back to application mode — fbuild now does both: reports via fix(esp32): surface ESP ROM download-mode lockup instead of silent timeout #557, resets via feat(daemon): opt-in ESP auto-recovery from ROM download mode (#532) #580.
• A failed or timed-out monitor/RPC operation does not require fbuild daemon stop before another process can recover the board — the fix(esp32): surface ESP ROM download-mode lockup instead of silent timeout #557 + fix(monitor): release Windows COM port after monitor exits #531 port-release path runs on outcome-error.
• fbuild logs show enough DTR/RTS/reset context to diagnose future S3 boot-mode lockups — feat(serial): add esp_reset hard-reset DTR/RTS primitive (#532) #577's tracing::debug! per transition + tracing::info! on completion, plus feat(daemon): opt-in ESP auto-recovery from ROM download mode (#532) #580's open-time DTR=high / RTS=high success-path debug! covering the historical warn-only-on-failure gap.
• Behavior covered by at least a regression test or scripted repro path for the ESP32-S3 serial reset state machine — 5 detection tests in boot_mode.rs (fix(esp32): detect ESP ROM download-mode in serial monitor (#532) #547), 4 reset-sequence tests in esp_reset.rs (feat(serial): add esp_reset hard-reset DTR/RTS primitive (#532) #577), and 3 wire-up tests in monitor.rs covering both flag-on and flag-off branches (feat(daemon): opt-in ESP auto-recovery from ROM download mode (#532) #580). All hardware-free via the trait-mocked port.

Follow-up beats (separate tracking)

• End-to-end hardware verification on a real ESP32-S3. The unit tests prove the sequence and the wire-up are correct against a mocked port; live-hardware soak (repeated deploy → monitor → recovery cycles on a physical board) stays a hardware-verification beat for whoever has the board, not a code blocker. Anyone running the AutoResearch harness on an S3 can flip auto_recover_from_download_mode: true on their monitor RPC call to exercise the new path.
• Wiring the same flag into the post-deploy monitor in deploy.rs would extend recovery to the fbuild deploy --to monitor flow. Currently passes false (fast-fail). Worth a focused follow-up PR once the standalone-monitor path has hardware miles.