Skip to content

Runtime Ethernet PHY (RMII + W5500) + firmware-variant collapse + catalog provisioning #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ewowi merged 11 commits into from
Jun 15, 2026
Merged

Runtime Ethernet PHY (RMII + W5500) + firmware-variant collapse + catalog provisioning #20

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
3139716
Catalog-driven module provisioning + 16 MB partition dedup
ewowi Jun 13, 2026
ee45102
Add LED drivers explicitly per board + nest catalog controls + loopba…
ewowi Jun 13, 2026
c892466
Generalize TX-power cap naming + vendor-consistent board names/images
ewowi Jun 14, 2026
e15b6f5
Add board capabilities + LED pins + a boards.json validator
ewowi Jun 14, 2026
d08612e
Generate firmwares.json + dedup release assets; expand & compress boa…
ewowi Jun 14, 2026
30669d8
Normalise detected chip to family + CI hardening + review fixes
ewowi Jun 14, 2026
c348350
Make the picture picker the installer; add device pin-profiles; close…
ewowi Jun 15, 2026
1161e77
Runtime Ethernet PHY config (RMII + W5500) + firmware-variant collapse
ewowi Jun 15, 2026
e86544a
Add Pin control type; eth pins as number inputs; process CodeRabbit r…
ewowi Jun 15, 2026
0bfa226
Document ControlType::Pin in the control-type reference
ewowi Jun 15, 2026
f35b747
Process Opus reviewer findings (pre-merge)
ewowi Jun 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 81 additions & 18 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,13 @@ jobs:
# (latest) and workflow_dispatch with a non-version tag.
runs-on: ubuntu-latest
steps:
# persist-credentials: false on every checkout that doesn't push — keeps the
# GITHUB_TOKEN out of .git/config where later steps (incl. third-party Docker
# actions) could read it. The `release` job is the one exception: its
# "Re-create latest" step does `git push -f`, so it keeps the credential.
- uses: actions/checkout@v4
with:
persist-credentials: false
- uses: astral-sh/setup-uv@v3
- name: Verify tag matches library.json version
if: startsWith(github.ref, 'refs/tags/v') && !contains(github.ref_name, '-') || (github.event_name == 'workflow_dispatch' && startsWith(inputs.tag, 'v'))
Expand All @@ -66,15 +72,36 @@ jobs:
TAG: ${{ inputs.tag || github.ref_name }}
run: uv run scripts/build/verify_version.py --tag "$TAG"

build-esp32:
# The shipping firmware list, read from the generated docs/install/firmwares.json
# (projected from build_esp32.py's FIRMWARES dict, drift-guarded by
# check_firmwares.py). Emitted as a JSON array so build-esp32's matrix can
# fromJSON() it — GitHub matrices can't read a file at parse time, so a job
# output is the standard bridge. This is the ONLY firmware list in CI now.
firmwares:
needs: verify-version
runs-on: ubuntu-latest
outputs:
list: ${{ steps.gen.outputs.list }}
steps:
- uses: actions/checkout@v4

@coderabbitai coderabbitai Bot Jun 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify unpinned and credential-persisting checkout usage in this workflow.
rg -n 'uses:\s*actions/checkout@' .github/workflows/release.yml -A3
rg -n 'persist-credentials:\s*false' .github/workflows/release.yml

Repository: MoonModules/projectMM

Length of output: 1291

Harden checkout in the new firmwares job.

The checkout at line 80 uses a mutable tag (@v4) and retains default credential persistence. Pin to a commit SHA and disable persisted credentials to reduce supply-chain and token-exposure risk.

🧰 Tools
🪛 zizmor (1.25.2)

[warning] 80-80: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false

(artipacked)

[error] 80-80: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/release.yml at line 80, The actions/checkout action at
line 80 in the `firmwares` job uses a mutable version tag (`@v4`) and retains
default credential persistence. Replace the mutable tag with a pinned commit SHA
and add the parameter persist-credentials: false to the checkout action
configuration to reduce supply-chain and token-exposure risk.

Source: Linters/SAST tools

with:
persist-credentials: false
- id: gen
run: |
set -euo pipefail
echo "list=$(jq -c '[.firmwares[] | select(.ships) | .name]' docs/install/firmwares.json)" >> "$GITHUB_OUTPUT"

build-esp32:
needs: [verify-version, firmwares]
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
firmware: [esp32, esp32-16mb, esp32-eth, esp32-eth-wifi, esp32s3-n16r8, esp32s3-n8r8, esp32p4-eth]
firmware: ${{ fromJSON(needs.firmwares.outputs.list) }}
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false

- name: Cache ESP-IDF tooling
uses: actions/cache@v4
Expand Down Expand Up @@ -115,9 +142,9 @@ jobs:
# The IDF target follows the firmware-key prefix: esp32s3* → esp32s3,
# esp32p4* → esp32p4 (the only target that pulls the ip101 PHY +
# esp_hosted, both manifest-gated on target == esp32p4), everything
# else → esp32. (esp32p4-eth-wifi is intentionally NOT in the matrix
# yet — its C6-slave Kconfig defaults don't survive a plain CI build;
# tracked in backlog § ESP32-P4 round 3.)
# else → esp32. (The matrix is the `ships` subset of firmwares.json;
# esp32p4-eth-wifi has ships=False in build_esp32.py so it stays out —
# its C6-slave Kconfig defaults don't survive a plain CI build.)
target: ${{ startsWith(matrix.firmware, 'esp32s3') && 'esp32s3' || startsWith(matrix.firmware, 'esp32p4') && 'esp32p4' || 'esp32' }}
path: 'esp32'
# We run our own builder (not the action's default `idf.py build`)
Expand All @@ -136,10 +163,18 @@ jobs:
# CI uses one per matrix job, dev machines use as many as built.
B=build/esp32-${{ matrix.firmware }}
PREFIX="firmware-${{ matrix.firmware }}-v$V"
cp "$B/projectMM.bin" "dist/${PREFIX}.bin"
cp "$B/bootloader/bootloader.bin" "dist/${PREFIX}-bootloader.bin"
cp "$B/partition_table/partition-table.bin" "dist/${PREFIX}-partition-table.bin"
cp "$B/ota_data_initial.bin" "dist/${PREFIX}-ota-data.bin"
# App + bootloader are unique per firmware — staged with the prefix.
cp "$B/projectMM.bin" "dist/${PREFIX}.bin"
cp "$B/bootloader/bootloader.bin" "dist/${PREFIX}-bootloader.bin"
# ota-data is byte-identical for ALL firmwares, and partition-table is
# identical within a flash-size group — staged under SHARED names so the
# release uploads each once instead of one-per-firmware. Every matrix job
# writes the same shared filename(s); the flatten step keeps one copy
# (byte-identical, so last-writer-wins is correct). The generated
# manifests point at these same names (generate_manifest.py).
SIZE=$(jq -r .flash_settings.flash_size "$B/flasher_args.json" | tr 'A-Z' 'a-z')
cp "$B/partition_table/partition-table.bin" "dist/partition-table-$SIZE.bin"
cp "$B/ota_data_initial.bin" "dist/shared-ota-data.bin"
# Per-firmware flasher_args.json — the release job feeds it to
# generate_manifest.py so offsets come from the real build.
cp "$B/flasher_args.json" "dist/flasher-${{ matrix.firmware }}.json"
Expand All @@ -154,6 +189,8 @@ jobs:
runs-on: macos-14
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
# CMakeLists.txt calls find_program(UV_EXECUTABLE … REQUIRED) so the
# build-host Python (gzip / build_info.h) is reached through uv. The
# runners don't ship uv by default — install it before package_desktop.
Expand All @@ -170,6 +207,8 @@ jobs:
runs-on: windows-latest
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
# Same uv prerequisite as build-macos — see the comment there.
- uses: astral-sh/setup-uv@v3
- name: Build + package Windows x64
Expand All @@ -194,8 +233,13 @@ jobs:
# tagged releases with zero binaries. Pages deploy lives in its own job
# below (gated to main, the only ref the environment trusts).
steps:
# NB: this checkout KEEPS persisted credentials (unlike the other jobs) —
# the "Re-create latest" step below force-pushes the `latest` tag with git,
# which needs the token in .git/config.
- uses: actions/checkout@v4

- uses: astral-sh/setup-uv@v3

- uses: actions/download-artifact@v4
with:
path: artifacts
Expand Down Expand Up @@ -235,11 +279,10 @@ jobs:
# — no CORS). The Pages-relative manifests are generated in the
# deploy-pages job, where the web installer (CORS-bound) consumes them.
BASE="https://github.com/${REPO}/releases/download/$TAG"
# Keep in sync with the build-esp32 matrix above (the source of truth
# for which firmware variants exist). A variant missing here builds but
# gets no manifest, so the installer can't flash it.
for F in esp32 esp32-16mb esp32-eth esp32-eth-wifi esp32s3-n16r8 esp32s3-n8r8 esp32p4-eth; do
python scripts/build/generate_manifest.py \
# The shipping firmware list — the same docs/install/firmwares.json the
# build matrix reads, so manifests and builds can't drift.
for F in $(jq -r '.firmwares[] | select(.ships) | .name' docs/install/firmwares.json); do
uv run python scripts/build/generate_manifest.py \
--firmware "$F" \
--version "$V" \
--release-url "$BASE" \
Expand Down Expand Up @@ -303,6 +346,8 @@ jobs:
# is a glob pattern, the action does NOT strip `#` as comment syntax.
files: |
dist/firmware-*.bin
dist/shared-ota-data.bin
dist/partition-table-*.bin
dist/manifest-*.json
dist/projectMM-*.tar.gz
dist/projectMM-*.zip
Expand All @@ -326,6 +371,10 @@ jobs:
url: ${{ steps.deploy-pages.outputs.page_url }}
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false

- uses: astral-sh/setup-uv@v3

- uses: actions/download-artifact@v4
with:
Expand All @@ -346,10 +395,9 @@ jobs:
# because the web installer is same-origin-bound by CORS. The
# absolute release-asset manifests live in the `release` job.
mkdir -p pages-manifests
# Keep in sync with the build-esp32 matrix above (the source of truth
# for which firmware variants exist). A variant missing here builds but
# gets no manifest, so the installer can't flash it.
for F in esp32 esp32-16mb esp32-eth esp32-eth-wifi esp32s3-n16r8 esp32s3-n8r8 esp32p4-eth; do
# The shipping firmware list — the same docs/install/firmwares.json the
# build matrix reads, so manifests and builds can't drift.
for F in $(jq -r '.firmwares[] | select(.ships) | .name' docs/install/firmwares.json); do
python scripts/build/generate_manifest.py \
--firmware "$F" \
--version "$V" \
Expand Down Expand Up @@ -391,6 +439,8 @@ jobs:
gh release download "$T" \
--dir "pages/install/releases/$T" \
--pattern 'firmware-*.bin' \
--pattern 'shared-ota-data.bin' \
--pattern 'partition-table-*.bin' \
--pattern 'projectMM-*.tar.gz' \
--pattern 'projectMM-*.zip' \
|| echo " (skip $T — no matching assets)"
Expand Down Expand Up @@ -423,6 +473,19 @@ jobs:
cp src/ui/install-picker.js pages/install/
# library.json — install page reads the project version from it.
cp library.json pages/install/
# Board picker images live in docs/assets/boards/ (the project's asset
# home, also a library for boards not yet in the catalog). Stage ONLY the
# images a boards.json entry actually references, under install/assets/,
# so an "image": "assets/boards/<slug>.jpg" resolves same-origin from
# /install/ without shipping the unused library to Pages.
mkdir -p pages/install/assets/boards
# rel is "assets/boards/<slug>.<ext>" (the path served from /install/);
# the source file lives in docs/<rel> (i.e. docs/assets/boards/...).
jq -r '.[].image // empty' docs/install/boards.json | while read -r rel; do
src="docs/$rel"
[ -f "$src" ] && cp "$src" "pages/install/$rel" \
|| echo "WARNING: boards.json image not found: $src"
done
# Root landing page (moonmodules.org/projectMM/) → Flash button + repo
# links. Without it the bare root 404s; only /install/ would exist.
cp docs/landing/index.html pages/index.html
Expand Down
4 changes: 3 additions & 1 deletion CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,8 +110,10 @@ The narrow safety net: "this snapshot is internally consistent."
5. Platform boundary, `check_platform_boundary.py`, if any file under `src/` (excluding `src/platform/`) changed.
6. ESP32 build, `build_esp32.py`, if any file under `src/` (excluding `src/platform/desktop/`), `esp32/`, `CMakeLists.txt`, or `library.json` changed.
7. KPI collection, `collect_kpi.py --commit`, if any file under `src/` changed. **The one-liner MUST include `tick:Xus(FPS:Y)` for every supported target** (PC + ESP32 today; Teensy/RPi when added). If a target's tick/FPS is missing (e.g. ESP32 wasn't monitored recently and `esp32/monitor.log` is stale), re-run a short live capture before committing, or note explicitly in the commit body why the value is absent.
8. Board catalog, `check_boards.py`, fast (<1s), if `docs/install/boards.json` or `scripts/check/check_boards.py` changed. Validates the installer catalog: required fields, `default_firmware ∈ firmwares`, every `image` resolves on disk, each `Board.board` control equals its entry `name`, module `type`s are factory-registered (or boot-wired singletons), `pins` controls live only on `*LedDriver` modules, and `supported` capabilities stay within the known vocabulary.
9. Firmware list, `check_firmwares.py`, fast (<1s), if `scripts/build/build_esp32.py`, `docs/install/firmwares.json`, or `scripts/check/check_firmwares.py` changed. Regenerates the firmware projection from the `FIRMWARES` dict and fails on drift from the committed `docs/install/firmwares.json` (so a `FIRMWARES` edit without regenerating is caught). Trigger includes `build_esp32.py` because that dict is the upstream source.

A commit that touches *only* `.github/`, `docs/`, `scripts/` (non-test), `README.md`, `CLAUDE.md`, or `.claude/` therefore runs only the spec check; the rest are no-ops because their triggers don't fire. This is the intended pre-commit cost for CI-only or doc-only changes.
A commit that touches *only* `.github/`, `docs/`, `scripts/` (non-test), `README.md`, `CLAUDE.md`, or `.claude/` therefore runs only the spec check (plus the board-catalog and firmware checks when their specific files changed); the build/test/ESP32/KPI gates are no-ops because their triggers don't fire. This is the intended pre-commit cost for CI-only or doc-only changes.

**Recommended (manual, not blocking):**

Expand Down
10 changes: 5 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ If you like projectMM, give it a ⭐️, fork it, or open an issue or pull reque

💡 **DMX *and* addressable LEDs in one setup**: RGB strips, RGBW pixels, par lights, moving heads, all through the same pipeline.

🔌 **Parallel WS2812 output**: drive many strands at once over three ESP32 peripherals — RMT (every chip), the S3's LCD_CAM i80 bus (8 lanes), and the P4's Parlio engine (up to 20 simultaneous strands on the P4) — each with an on-device loopback self-test that bit-verifies the wire signal.
🔌 **Parallel WS2812 output**: drive many strands at once over three ESP32 peripherals — RMT (every chip), the S3's LCD_CAM i80 bus (8 lanes), and the P4's Parlio engine (up to 8 lanes) — each with an on-device loopback self-test that bit-verifies the wire signal.

🌐 **Industry protocols, both directions**: send *and* receive [Art-Net](https://art-net.org.uk/), [E1.31/sACN](https://tsp.esta.org/tsp/documents/docs/ANSI_E1-31-2018.pdf), and [DDP](http://www.3waylabs.com/ddp/) over the network — interoperable with Falcon, Advatek, xLights, Resolume, LedFx and other industry gear.

Expand Down Expand Up @@ -58,20 +58,20 @@ The **Desktop** column is host-CPU-bound, not OS-bound: the numbers track the ma

### Frames per second

| Grid | Lights | Desktop | Olimex `esp32-eth-wifi` | Olimex `esp32-eth` | LOLIN S3 N16R8 `esp32s3-n16r8` |
| Grid | Lights | Desktop | Olimex `esp32` | Olimex `esp32-eth` | LOLIN S3 N16R8 `esp32s3-n16r8` |
|---|---:|---:|---:|---:|---:|
| 16×16 | 256 | *(below host clock resolution)* | 1,543 | 1,628 | 1,672 |
| 32×32 | 1,024 | 166,667 | 447 | 432 | 287 |
| 64×64 | 4,096 | 40,000 | 81 | 71 | 25 |
| 128×128 | 16,384 | 9,708 | 11 | 10 | 6 |

The LOLIN S3 N16R8 is WiFi-only and runs with `Network.txPowerSetting` capped to 8 dBm (the brown-out fix, see below). At 128×128 it's bound by ArtNet over WiFi at reduced TX power (~93 ms of the ~164 ms tick), which is why it trails the Ethernet boards despite a faster core. The board's niche is PSRAM headroom (8 MB) for large pixel buffers, not raw ArtNet FPS; use an Ethernet board when frame rate matters.
The Olimex `esp32` figures were measured on the WiFi+Ethernet build (the pre-collapse `esp32-eth-wifi`, now the default `esp32`). The LOLIN S3 N16R8 was measured over WiFi with `Network.txPowerSetting` capped to 8 dBm (the brown-out fix, see below); at 128×128 it's bound by ArtNet over WiFi at reduced TX power (~93 ms of the ~164 ms tick), which is why it trails the Ethernet boards despite a faster core. (The S3 now also supports W5500 SPI Ethernet, which sidesteps that WiFi bottleneck on boards wired for it.) The board's niche is PSRAM headroom (8 MB) for large pixel buffers; use an Ethernet board when frame rate matters.

### Free heap

Each cell is **free internal RAM / largest contiguous internal-RAM block**. Internal RAM is the scarce, comparable resource across all boards, so for PSRAM boards (the S3) this is internal-only, NOT the PSRAM-merged total (we assume the 8 MB PSRAM pool is large enough that it isn't the constraint). The block size is the memory-pressure signal that matters: free RAM can be ample while fragmentation leaves no single block big enough for the next allocation.

| Grid | Desktop | Olimex `esp32-eth-wifi` | Olimex `esp32-eth` | LOLIN S3 N16R8 `esp32s3-n16r8` |
| Grid | Desktop | Olimex `esp32` | Olimex `esp32-eth` | LOLIN S3 N16R8 `esp32s3-n16r8` |
|---|---:|---:|---:|---:|
| 16×16 | unlimited | 139 KB / 52 KB | 178 KB / 100 KB | 238 KB / 160 KB |
| 32×32 | unlimited | 132 KB / 50 KB | 172 KB / 92 KB | 240 KB / 152 KB |
Expand All @@ -80,7 +80,7 @@ Each cell is **free internal RAM / largest contiguous internal-RAM block**. Inte

The S3's internal-free stays flat across grid sizes because its Layer buffer + LUT live in PSRAM: growing the grid consumes PSRAM, not internal RAM. The Olimex boards hold those buffers in internal RAM, so their free heap drops as the grid grows.

Build variants differ structurally: `esp32-eth-wifi` includes the WiFi stack (~270 KB flash, ~28 KB heap). `esp32-eth` drops WiFi for more free heap, at the cost of slightly slower tick on large grids (lwIP buffer-pool sizing is tuned for the eth-wifi sdkconfig). The right variant depends on whether the deployment needs WiFi, Ethernet, or large buffers.
Build variants differ structurally: the default `esp32` includes the WiFi stack (~270 KB flash, ~28 KB heap) alongside Ethernet. `esp32-eth` drops WiFi for more free heap, at the cost of slightly slower tick on large grids (lwIP buffer-pool sizing is tuned for the WiFi+Ethernet sdkconfig). The right variant depends on whether the deployment needs WiFi at all, or only Ethernet plus the extra buffers.

The numbers above are observations. The **contracts** projectMM commits to, what the device must hit on every CI run, live in [`test/scenarios/*.json`](test/scenarios/) as per-step `contract.<target>` blocks; see [docs/testing.md § Performance contracts](docs/testing.md#performance-contracts-contracttarget) for how they're set and renegotiated. The [docs/performance.md](docs/performance.md) page covers the *why* (WiFi vs Ethernet physics, sizeof tables, build-variant deltas).

Expand Down
Loading