Skip to content

feat(online-data): ingest per-board VID:PID tables from upstream vendor manifests #722

Description

@zackees

Goal

Extend the online-data USB-VID aggregator (the pipeline introduced in #718 / #719) with per-board VID:PID coverage by ingesting structured board-manifest data from upstream vendors. Today the aggregator carries:

  • Vendor names (via the merged usb-vid.json — text DBs + the curated vendor_names_inlined.py overlay covering 253 newer VIDs).
  • PlatformIO board catalog (pio-boards.json — names + MCU family but no VID:PID).
  • MCU → VID heuristics (mcu_to_vid.json — confidence-scored, not per-board).

What's still missing for the headline VID:PID → board lookup: a precise per-board VID:PID table that says e.g. "0x303A:0x4002 is an ESP32-S3" rather than "any ESP32-S3-family board on VID 0x303A."

Authoritative public sources (in priority order)

Tier Source Format Coverage Type
1 raspberrypi/usb-pidReadme.md Markdown table VID 0x2E8A PID allocations (Pico, Pimoroni, Cytron, WIZnet, TI USB-TMC etc.) PID allocation registry
1 espressif/usb-pidsallocated-pids.txt + allocated-pids-espressif-devboards.txt Plain text (VVVV:PPPP description) VID 0x303A customer & devboard PIDs (TinyS2, LILYGO, ATMegaZero, ESP32-S3-DevKitC, ESP32-P4 etc.) PID allocation registry
2 arduino/ArduinoCore-avrboards.txt boardname.vid.N=0xVVVV / pid.N=0xPPPP Leonardo, Micro, Esplora, Yún (VID 0x2341, 0x2A03) Board-manifest
2 arduino/ArduinoCore-samdboards.txt Same Zero, MKR family, Nano 33 IoT (VID 0x03EB, 0x2341) Board-manifest
2 espressif/arduino-esp32boards.txt Same ESP32-S2/S3/C3 family + partner boards (UM TinyS2, LOLIN S2 Mini, etc.) Board-manifest
2 adafruit/Adafruit_nRF52_Arduinoboards.txt Same Feather nRF52840 Express/Sense, ItsyBitsy, Circuit Playground Bluefruit, CLUE (VID 0x239A) Board-manifest
2 SiliconLabs/arduinoboards.txt Same Nano Matter, Seeed XIAO MG24 (VID 0x2341, 0x2886) Board-manifest (official)
3 stm32duino/Arduino_Core_STM32boards.txt Same Nucleo, Discovery, Eval, generic STM32 (VID 0x0483, PIDs 0x5740 / 0x3744 / 0x3748 / 0x374B / 0x3753) Board-manifest (community, not ST-official)
4 NordicSemiconductor/nrf-udev71-nrf.rules udev rules VID 0x1915 (all Nordic devices) — vendor-level only, no per-product Auxiliary
4 NordicSemiconductor/nrf-device-setup-js (archived) → README Prose Single explicit pair: 0x1915:0x521F PCA10059 bootloader Auxiliary, archived 2022-01-10

Gaps confirmed by the research

  • NXP (MCUXpresso org): no public PID allocation file. Per-example USB descriptors in nxp-mcuxpresso/mcux-sdk-middleware-usb only.
  • Microchip (Harmony USB): same shape — per-example USB descriptors, no centralized registry.
  • TI: no maintained list. energia/Energia is the closest but explicitly unmaintained.
  • STM32: no ST-owned official list. stm32duino/Arduino_Core_STM32 is community-run.

Proposed implementation

Add a new fetch+parse step per source under online-data-tools/, modeled on the existing fetch_gowdy_supplement.py + overlay_usb_vid.py pattern:

  1. fetch_rpi_pid_registry.py — parses raspberrypi/usb-pid/Readme.md markdown table → {vidpid: product_name} JSON. Tier-1 (most authoritative).
  2. fetch_espressif_pid_registry.py — parses espressif/usb-pids/allocated-pids*.txt (two files) → same shape. Tier-1.
  3. fetch_boards_txt.py — generic parser for Arduino-style boards.txt. Walks boardname.vid.N + pid.N pairs and emits {vidpid: f"{board_friendly_name}"}. Used for the 6 tier-2/3 board-manifest repos.
  4. overlay_per_board_pids.py — merges all per-board sources into a NEW online-data/data/usb-board-pids.json keyed by 8-hex-digit VVVVPPPP, with {vendor, board, source} per entry. Doesn't disturb the existing usb-vid.json.
  5. SQLite consumerbuild_sqlite.py populates vidpid from BOTH the existing usb_product rows AND the new per-board source, with the per-board source winning on conflict (it's the authoritative consumer-facing name).
  6. Workflow wiring — add the new fetches to update-data.yml with continue-on-error: true (same fault-tolerance pattern as the existing tier-1/2/3 sources).

Output schema sketch

// online-data/data/usb-board-pids.json
{
  "2e8a0003": {"vendor": "Raspberry Pi", "board": "RP2040 Boot",            "source": "raspberrypi/usb-pid"},
  "2e8a000b": {"vendor": "Raspberry Pi", "board": "Pico CircuitPython",     "source": "raspberrypi/usb-pid"},
  "303a8001": {"vendor": "Unexpected Maker", "board": "TinyS2 Arduino",     "source": "espressif/usb-pids"},
  "303a7012": {"vendor": "Espressif",    "board": "ESP32-P4 Function EV",   "source": "espressif/usb-pids"},
  "23418057": {"vendor": "Arduino",      "board": "Nano 33 IoT",            "source": "arduino/ArduinoCore-samd"},
  "239a8029": {"vendor": "Adafruit",     "board": "Feather nRF52840 Express","source":"adafruit/Adafruit_nRF52_Arduino"},
  "23410072": {"vendor": "Arduino",      "board": "Nano Matter",            "source": "SiliconLabs/arduino"}
}

Acceptance criteria

  • All 9 sources fetched + parsed nightly by update-data.yml.
  • online-data/data/usb-board-pids.json committed with at least 200 per-board entries on first successful run.
  • manifest.json advertises the new dataset (alongside usb-vid, pio-boards, vendor_boards).
  • The www SQLite vidpid table is populated from this new source, so a canned-query vidpid='2e8a000b' returns "Pico CircuitPython" not just "Pico SDK CDC UART".
  • fbuild-core::usb::resolver::pretty(0x303A, 0x4002) resolves through the overlay (tier-2) and returns "Unexpected Maker TinyS2 Arduino" instead of the synthetic "Device 0x4002" placeholder.
  • Tests cover each parser (small fixture per format), the overlay merge, and the SQLite ingestion.
  • Gaps (NXP / Microchip / TI / non-Espressif Nordic) documented in online-data-tools/README.md.

Out of scope (defer to follow-ups)

  • Scraping vendor SDK examples for embedded USB descriptors (NXP / Microchip / TI). Would require navigating each vendor's GitHub forest and is a much larger investment.
  • Cross-vendor PID-collision detection (e.g. when an OEM ships under multiple VIDs). Add as usb-board-pid-conflicts.json once the basic ingestion is live.
  • Adding the new dataset to the embedded usb-vendors.tar.zst archive in fbuild. The archive is intentionally small (vendor-only); per-PID data stays in the runtime overlay + the www SQLite-over-HTTP backend.

Sub-issue checklist

Per-vendor ingest sub-issues. This meta auto-closes when all are CLOSED.

Tier-1 PID registries (most authoritative)

Tier-2 board-manifests (Arduino-core boards.txt)

End-to-end verification (blocked on all of #723#739)

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    Triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions