investigate(rp2040): harden hub-path BOOTSEL mass-storage diagnostics

[zackees]

Parent: FastLED/FastLED#3648. Related stock-ROM work: #1040 and #1049.

Context

The attached RP2040 has a repeatable physical USB-topology differential on Windows:

• Direct motherboard USB: fbuild deploy and AutoResearch RPC smoke pass.
• USB hub: the same deployment path does not work.

Direct-port evidence, captured with fbuild only:

FBUILD_DEV_MODE=1 UV_NO_SYNC=1 bash autoresearch rp2040 --rpc-smoke --timeout 120s --skip-lint --clean

wrote G:\NEW.UF2
deploy succeeded (full flash); FBUILD_DEPLOY_PORT=COM12
RESULT: RPC smoke PASS

The known working runtime identity is COM12, 2E8A:000A, serial 5303284720C4641C. The daemon records its expected subsequent-deploy transition as COM12 -> BOOTSEL G: -> COM12.

Earlier hub-path observations included Windows storage errors 1006/1392 and Disk 51/153 events. RAM-target UF2 transfers previously ejected/re-enumerated through the same ROM path, while flash-target transfer was unconfirmed. Those observations do not identify a QSPI component fault. The current physical A/B establishes that the hub path is a causal condition, but not whether its mechanism is power/reset timing, USB enumeration, or Windows removable-storage interaction.

fbuild already gives generic direct-port/hub recovery advice for Windows error 121 and direct-port guidance for error 1006 in crates/fbuild-deploy/src/rp2040.rs. It has no topology-specific diagnostic or acceptance record for this observed differential.

Proposal

Add an fbuild-owned diagnostic/hardening slice for RP2040 BOOTSEL mass-storage failures:

1. Preserve fbuild as the only deploy interface; do not add PlatformIO or direct flash-tool workarounds.
2. Capture and emit the available transport context around an RP2040 transfer failure (selected BOOTSEL volume, transfer/eject outcome, Windows error, runtime identity, and USB topology when the platform can supply it).
3. Make the actionable failure distinguish a hub-path reproduction from an unclassified transfer failure without claiming a QSPI root cause.
4. Add deterministic tests for the rendered diagnostics and retain the existing typed error guidance.
5. Add physical A/B evidence for the same board/cable: hub failure signature and direct-motherboard success signature, recording serial/VID:PID, command, and outcome.

Acceptance criteria

• A hub-path failure reports the observed transfer/eject symptom and a direct-motherboard retry recommendation, without asserting a flash-component cause.
• The diagnostic includes all available identity/transport facts and explicitly labels unavailable topology information rather than guessing.
• Focused fbuild tests cover the hub-specific actionable message plus the existing 1006/121/1392 branches.
• A same-board physical A/B log records hub failure and direct-port fbuild-only success, including board serial, VID:PID, command, and decisive output.
• No PlatformIO, raw serial, or direct flashing tool is introduced.

Open questions

• Does the hub change power/reset timing, USB enumeration, or Windows MSC/filter behavior?
• Can fbuild obtain stable Windows USB topology for the selected MSC device without requiring an additional driver?

Related issues

• feat(rp2040): zero-touch deployment to an as-received Pico via ROM UF2 #1040 — stock ROM UF2 first-deployment acceptance.
• feat(rp2040): deploy parity with Arduino-Pico, PlatformIO, and official picotool #1049 — RP2040 deploy parity and attached-board diagnostic acceptance.
• meta(rp2040): transport, channel-driver implementation, and postmortem closeout FastLED#3648 — parent burn-down.

[zackees]

Investigation: why the hub path fails — mechanism triage + web findings

TL;DR: There is a documented silicon erratum, RP2040-E5 ("USB device fails to exit RESET state on busy USB bus"), that specifically predicts the observed hub-vs-direct differential: it affects the mask-ROM BOOTSEL code, which cannot be patched, while runtime firmware built with the Pico SDK carries a software workaround. A device behind a hub shares its full-speed segment with sibling traffic and may never see the ~800 µs of bus idle the USB controller needs after a bus reset; a device alone on a direct root port always does. The erratum applies to B0/B1 steppings and was fixed by a mask change in B2 — so the single most decisive fact to add to the A/B log is this board's chip stepping. It is not the only candidate (VBUS sag during QSPI flash programming fits the RAM-vs-flash observation already recorded here), so below is a triage of the three mechanism families from the open question, plus an answer to the topology-detection question.

1. USB enumeration: RP2040-E5 (strongest topology-specific candidate)

• The erratum: after a host-initiated bus reset, the RP2040 USB device controller requires ~800 µs of continuous idle (J-state) before it transitions to CONNECTED. If the bus stays busy, the device never exits RESET, never answers, and does not enumerate (rp-rs/rp-hal#99).
• Why the hub matters: the RP2040 is a full-speed-only device. Behind a hub it shares its downstream FS segment with hub-forwarded traffic — the Pico SDK's own workaround notes that after reset the bus is idle "but on a hub that broadcasts packets for other devices this isn't the case". Common siblings (USB-serial adapters, mice, keyboards) are polled every 30–50 µs, enough to keep the segment busy indefinitely. On a direct motherboard root port the RP2040 is the only device on its segment, so post-reset idle is guaranteed and the erratum can never trigger.
• Why BOOTSEL specifically: the workaround (forcing an artificial J-state via GPIO15 debug muxing for ~800 µs) is software, shipped in pico-sdk/TinyUSB as PICO_RP2040_USB_DEVICE_ENUMERATION_FIX (raspberrypi/pico-sdk#759). Runtime firmware gets it; the boot ROM's mass-storage code is "the first stage bootloader that is encoded in the (hardened) ROM" and does not contain it (detailed writeup). That exactly matches the recorded asymmetry: runtime COM12 (2E8A:000A) behaves over the hub, while the ROM BOOTSEL path (2E8A:0003 → G:) is the part that fails.
• Stepping caveat: E5 applies to B0/B1; a Raspberry Pi engineer confirmed "B2 did fix a h/w USB issue (errata) with a small mask change… there were some minimal bootrom changes too" (forum thread). If this board is B2, E5 is ruled out (or at least heavily weakened — some post-B2 hub-visibility reports exist) and mechanisms 2/3 move up. The stepping is available through fbuild's already-managed picotool fallback (ROM version maps V1→B0, V2→B1, V3→B2); no new tool is needed.

2. Power/reset timing: VBUS sag during QSPI flash programming

Flash-target UF2 writes drive erase/program cycles on the QSPI flash — the highest peak-current phase of the whole deploy. A bus-powered hub, or a hub with aggressive per-port current limiting, can sag VBUS mid-transfer and brown out or USB-glitch the board, which the host sees as a mid-copy device error.

This is the mechanism that best fits an observation already in this issue: RAM-target UF2 transfers previously ejected/re-enumerated through the same hub path, while flash-target transfer was unconfirmed. RAM targets never touch QSPI and draw less peak current. If a repeat A/B shows RAM-UF2-over-hub passing while flash-UF2-over-hub fails within the same enumeration, that is strong evidence for power rather than enumeration — without implicating the flash component itself.

3. Windows MSC / transport behavior

• The event signature is downstream, not a root cause. Event 153 is the storage class driver retrying a request the miniport timed out; Event 51 is a failed paging write (Microsoft: Event ID 51). Repeated timeouts escalate to a USB port reset by the host stack — and a mid-copy port reset is exactly where mechanism 1 bites on B0/B1 silicon: the ROM device fails to re-enumerate on the still-busy hub segment, the open handle to G: is invalidated, and the copy surfaces error 1006 (ERROR_FILE_INVALID, 0x3EE — other Pico users report it verbatim as 0x800703EE during UF2 copies, e.g. this thread) or 1392.
• The hub's transaction translator is only in the path on the hub leg. Because the RP2040 is FS-only, behind a USB 2.0/3.x hub every MSC bulk transfer becomes split transactions through the hub's TT; cheap single-TT hubs are a known source of bulk stalls/retries under load. A direct root port has no hub TT between host and device. Hub make/model and single- vs multi-TT are worth recording in the A/B log.
• Pure Windows-filter explanations (indexer/AV touching the synthetic FAT) are topology-independent and cannot alone explain the differential, though the slower hub path widens race windows against the ROM's synthetic-FAT semantics.

Chained story consistent with all recorded evidence

hub adds TT scheduling + shared VBUS + shared FS segment → mid-copy transient (bulk timeout or power dip) → storport retries (Event 153) / failed write (Event 51) → host resets the port → on B0/B1 silicon the ROM cannot re-enumerate on the busy segment (E5, no workaround possible in mask ROM) → volume handle invalidated → error 1006/1392 with no observed eject transition — which is precisely the branch crates/fbuild-deploy/src/rp2040.rs already distinguishes from the accepted late-error-after-eject case.

Open question 2: topology without a driver — yes, user mode is sufficient

Windows exposes everything needed with inbox drivers, no admin, no new driver:

• Walk the devnode chain with CfgMgr32: volume → disk PDO (USBSTOR\…) → CM_Get_Parent → USB device node (USB\VID_2E8A&PID_0003\…) → next parent. If the parent's instance ID is USB\ROOT_HUB30\… the board is on a root port (direct); if it is another USB\VID_xxxx&PID_xxxx device, that is an external hub — keep walking to count tiers.
• DEVPKEY_Device_LocationInfo on the USB devnode yields the classic Port_#0004.Hub_#0003 string for the acceptance record.
• Hub characteristics (bus- vs self-powered, multi-TT, per-port status) come from user-mode hub IOCTLs (IOCTL_USB_GET_NODE_INFORMATION, IOCTL_USB_GET_NODE_CONNECTION_INFORMATION_EX) — the same path Microsoft's USBView sample uses.
• All reachable from Rust via windows-rs; "explicitly labels unavailable topology information" then only applies to non-Windows hosts or genuinely absent properties.

Suggested additions to the physical A/B matrix

Experiment	Discriminates
Chip stepping via managed picotool (B0/B1 vs B2)	whether E5 applies at all
Hub with the board as the only downstream device	busy-bus (E5) vs hub-intrinsic (TT/power)
Same hub, self-powered vs bus-powered	VBUS sag
RAM-target vs flash-target UF2 over the same hub session	QSPI current draw vs transport
Second hub model (multi-TT, different vendor)	TT quality
Direct-port control run	baseline (already captured: COM12 pass)

Sources

• rp-rs/rp-hal#99 — USB enumeration errata (E5 mechanics, 30–50 µs sibling polling)
• raspberrypi/pico-sdk#759 — E5 fix is opt-in SDK code (PICO_RP2040_USB_DEVICE_ENUMERATION_FIX)
• Raspberry Pi Forums — RP2040 revisions: B2 fixed the USB erratum via mask change
• 3DP-Build-Log — RP2040 USB bug writeup (ROM first-stage bootloader affected, unpatchable)
• Raspberry Pi Forums — E5 workaround enablement
• Raspberry Pi Forums — UF2 copy failing with 0x800703EE (= error 1006)
• Microsoft — Information about Event ID 51

[zackees]

Resolved by merged PR #1083 (harden(rp2040): hub-path deploy robustness — retries, picotool fallback, topology diagnostics, tunable budgets).

Acceptance criteria check:

• Hub-path failures now report transfer/eject symptom + direct-motherboard retry guidance without asserting a QSPI/flash-component cause (typed error-5, error-121/1006/1392 branches retained and extended).
• Diagnostics include CfgMgr32-derived USB topology (root-port vs hub-tier depth, location string) and explicitly label unavailable facts (hub power mode, sibling count) instead of guessing.
• 36 new deterministic tests cover retry policy, topology classification, watchdog, CDC downgrade, timeout parsing, and attribution (fbuild-deploy: 225 passed / 0 failed).
• Physical A/B on the reference board (2E8A:000A, serial 5303284720C4641C, COM12, Windows 10, over the same USB hub previously producing errors 1006/1392 and Disk 51/153 events): three consecutive fbuild deploy tests/platform/rp2040 -e rp2040 -p COM12 runs succeeded end-to-end — full flash via G:\NEW.UF2, runtime CDC re-enumeration, sketch serial output confirmed (Hello from RP2040 Pico!). This exceeds the original acceptance bar (hub failure + direct-port success) by demonstrating hub-path success after hardening.
• No PlatformIO, raw serial, or direct flashing tool introduced; fbuild remains the sole deploy interface.

Sibling issue #1082 (broader hardening-plan inventory) is already closed by the same PR.