From 10f688c5bfe406bb8b95ef79c1b391b5da23ad37 Mon Sep 17 00:00:00 2001 From: Will Miles Date: Sun, 31 May 2026 23:17:01 +0000 Subject: [PATCH 01/11] Update usermod documentation Remove the in-tree example in favor of the out-of-tree example instead. Encourage people to start their project out-of-tree instead of here. --- AGENTS.md | 27 +- usermods/EXAMPLE/library.json | 5 - usermods/EXAMPLE/readme.md | 14 +- usermods/EXAMPLE/usermod_v2_example.cpp | 407 ------------------------ usermods/readme.md | 38 ++- 5 files changed, 50 insertions(+), 441 deletions(-) delete mode 100644 usermods/EXAMPLE/library.json delete mode 100644 usermods/EXAMPLE/usermod_v2_example.cpp diff --git a/AGENTS.md b/AGENTS.md index c1ce6a510f..f5d1fd7d35 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -175,7 +175,9 @@ Background Info: ## Usermod Pattern -Usermods live in `usermods//` with a `.cpp`, optional `.h`, `library.json`, and `readme.md`. +The preferred approach for new usermods is **out-of-tree**: click **Use this template** on [github.com/wled/wled-usermod-example](https://github.com/wled/wled-usermod-example) to create your own copy, add your code, and reference it from a WLED build — no changes to the WLED source tree needed. The fully annotated reference implementation lives there. Full guide at [kno.wled.ge/advanced/custom-features](https://kno.wled.ge/advanced/custom-features/). + +In-tree usermods in `usermods//` (with a `.cpp`, optional `.h`, `library.json`, and `readme.md`) are for commonly-needed features the core team maintains; the bar for inclusion is high (see `usermods/readme.md`). ```cpp class MyUsermod : public Usermod { @@ -183,23 +185,26 @@ class MyUsermod : public Usermod { bool enabled = false; static const char _name[]; public: - void setup() override { /* ... */ } // runs once at start-up - void loop() override { /* ... */ } // runs once per main loop iteration - void addToConfig(JsonObject& root) override { /* ... */ } // create/add persistent settings (usermod settings) - bool readFromConfig(JsonObject& root) override { /* ... */ } // read from persistent settings (usermod settings UI) - uint16_t getId() override { return USERMOD_ID_MYMOD; } - void addToJsonInfo(JsonObject& root) override { /* ... */ } // Add custom items to the "info" page and to /json/info - void appendConfigData() override { /* ... */ } // Customize the settings page: dropdowns, checkboxes, extra text, etc. Buffer size is limited! + void setup() override { /* ... */ } // runs once at start-up + void loop() override { /* ... */ } // runs once per main loop iteration + void addToConfig(JsonObject& root) override { /* ... */ } // persist settings to cfg.json + bool readFromConfig(JsonObject& root) override { /* ... */ } // restore settings; return false if keys missing + void addToJsonInfo(JsonObject& root) override { /* ... */ } // add entries to /json/info + void appendConfigData(Print& settingsScript) override { /* ... */ } // customize the Usermod Settings page + // uint16_t getId() override { return USERMOD_ID_MYMOD; } // only needed — see "Usermod IDs" below }; const char MyUsermod::_name[] PROGMEM = "MyUsermod"; static MyUsermod myUsermod; REGISTER_USERMOD(myUsermod); ``` -refer to detailed examples in `usermods/EXAMPLE/`, `usermods/user_fx/` and [in the user documentation for custom features](https://kno.wled.ge/advanced/custom-features/). +For additional lifecycle hooks (`connected`, `handleOverlayDraw`, `handleButton`, MQTT callbacks, etc.) see the annotated example at [github.com/wled/wled-usermod-example](https://github.com/wled/wled-usermod-example) and `usermods/user_fx/`. -- Activate via `custom_usermods = ` in platformio build config. The `usermod_v2_` prefix or `_v2` suffix can be omitted. -- Base new usermods on `usermods/EXAMPLE/` (never edit the example directly) +- **Out-of-tree** (preferred): reference via `custom_usermods` in `platformio_override.ini`: + - Local clone: `symlink:///absolute/path/to/your-usermod` + - Published: `https://github.com/you/your-usermod.git#main` +- **In-tree**: activate via `custom_usermods = ` in the platformio build config. The `usermod_v2_` prefix or `_v2` suffix can be omitted. +- Base new usermods on the out-of-tree template above; if contributing in-tree, use `usermods/user_fx/` as a reference (never edit in-tree examples directly) - Store repeated strings as `static const char[] PROGMEM` - Add usermod IDs to `wled00/const.h` **only when a unique ID is required** (see below) diff --git a/usermods/EXAMPLE/library.json b/usermods/EXAMPLE/library.json deleted file mode 100644 index d0dc2f88e6..0000000000 --- a/usermods/EXAMPLE/library.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "name": "EXAMPLE", - "build": { "libArchive": false }, - "dependencies": {} -} diff --git a/usermods/EXAMPLE/readme.md b/usermods/EXAMPLE/readme.md index ee8a2282a0..58dcef7bd3 100644 --- a/usermods/EXAMPLE/readme.md +++ b/usermods/EXAMPLE/readme.md @@ -1,9 +1,13 @@ -# Usermods API v2 example usermod +# Looking for the example usermod? -In this usermod file you can find the documentation on how to take advantage of the new version 2 usermods! +The annotated example has moved to its own repository: -## Installation +**[github.com/wled/wled-usermod-example](https://github.com/wled/wled-usermod-example)** -Add `EXAMPLE` to `custom_usermods` in your PlatformIO environment and compile! -_(You shouldn't need to actually install this, it does nothing useful)_ +Click **Use this template** on GitHub to create your own copy and start your own usermod. It contains a fully annotated implementation and a `library.json` template covering everything you need. +For the full guide — enabling usermods, the `custom_usermods` build system, persistent settings, effects, and more — see: + +**[kno.wled.ge/advanced/custom-features](https://kno.wled.ge/advanced/custom-features/)** + +Please use the example repo above as the starting point for your own work. diff --git a/usermods/EXAMPLE/usermod_v2_example.cpp b/usermods/EXAMPLE/usermod_v2_example.cpp deleted file mode 100644 index 65f3eda457..0000000000 --- a/usermods/EXAMPLE/usermod_v2_example.cpp +++ /dev/null @@ -1,407 +0,0 @@ -#include "wled.h" - -/* - * Usermods allow you to add own functionality to WLED more easily - * See: https://github.com/wled-dev/WLED/wiki/Add-own-functionality - * - * This is an example for a v2 usermod. - * v2 usermods are class inheritance based and can (but don't have to) implement more functions, each of them is shown in this example. - * Multiple v2 usermods can be added to one compilation easily. - * - * Creating a usermod: - * This file serves as an example. If you want to create a usermod, it is recommended to use usermod_v2_empty.h from the usermods folder as a template. - * Please remember to rename the class and file to a descriptive name. - * You may also use multiple .h and .cpp files. - * - * Using a usermod: - * 1. Copy the usermod into the sketch folder (same folder as wled00.ino) - * 2. Register the usermod by adding #include "usermod_filename.h" in the top and registerUsermod(new MyUsermodClass()) in the bottom of usermods_list.cpp - */ - -//class name. Use something descriptive and leave the ": public Usermod" part :) -class MyExampleUsermod : public Usermod { - - private: - - // Private class members. You can declare variables and functions only accessible to your usermod here - bool enabled = false; - bool initDone = false; - unsigned long lastTime = 0; - - // set your config variables to their boot default value (this can also be done in readFromConfig() or a constructor if you prefer) - bool testBool = false; - unsigned long testULong = 42424242; - float testFloat = 42.42; - String testString = "Forty-Two"; - - // These config variables have defaults set inside readFromConfig() - int testInt; - long testLong; - int8_t testPins[2]; - - // string that are used multiple time (this will save some flash memory) - static const char _name[]; - static const char _enabled[]; - - - // any private methods should go here (non-inline method should be defined out of class) - void publishMqtt(const char* state, bool retain = false); // example for publishing MQTT message - - - public: - - // non WLED related methods, may be used for data exchange between usermods (non-inline methods should be defined out of class) - - /** - * Enable/Disable the usermod - */ - inline void enable(bool enable) { enabled = enable; } - - /** - * Get usermod enabled/disabled state - */ - inline bool isEnabled() { return enabled; } - - // in such case add the following to another usermod: - // in private vars: - // #ifdef USERMOD_EXAMPLE - // MyExampleUsermod* UM; - // #endif - // in setup() - // #ifdef USERMOD_EXAMPLE - // UM = (MyExampleUsermod*) UsermodManager::lookup(USERMOD_ID_EXAMPLE); - // #endif - // somewhere in loop() or other member method - // #ifdef USERMOD_EXAMPLE - // if (UM != nullptr) isExampleEnabled = UM->isEnabled(); - // if (!isExampleEnabled) UM->enable(true); - // #endif - - - // methods called by WLED (can be inlined as they are called only once but if you call them explicitly define them out of class) - - /* - * setup() is called once at boot. WiFi is not yet connected at this point. - * readFromConfig() is called prior to setup() - * You can use it to initialize variables, sensors or similar. - */ - void setup() override { - // do your set-up here - //Serial.println("Hello from my usermod!"); - initDone = true; - } - - - /* - * connected() is called every time the WiFi is (re)connected - * Use it to initialize network interfaces - */ - void connected() override { - //Serial.println("Connected to WiFi!"); - } - - - /* - * loop() is called continuously. Here you can check for events, read sensors, etc. - * - * Tips: - * 1. You can use "if (WLED_CONNECTED)" to check for a successful network connection. - * Additionally, "if (WLED_MQTT_CONNECTED)" is available to check for a connection to an MQTT broker. - * - * 2. Try to avoid using the delay() function. NEVER use delays longer than 10 milliseconds. - * Instead, use a timer check as shown here. - */ - void loop() override { - // if usermod is disabled or called during strip updating just exit - // NOTE: on very long strips strip.isUpdating() may always return true so update accordingly - if (!enabled || (strip.isUpdating() && (millis() - lastTime < 200))) return; // adjust "200" (in millisecond) to your needs - prevents starvation with very long strips - - // do your magic here - if (millis() - lastTime > 1000) { - //Serial.println("I'm alive!"); - lastTime = millis(); - } - } - - - /* - * addToJsonInfo() can be used to add custom entries to the /json/info part of the JSON API. - * Creating an "u" object allows you to add custom key/value pairs to the Info section of the WLED web UI. - * Below it is shown how this could be used for e.g. a light sensor - */ - void addToJsonInfo(JsonObject& root) override - { - // if "u" object does not exist yet wee need to create it - JsonObject user = root["u"]; - if (user.isNull()) user = root.createNestedObject("u"); - - //this code adds "u":{"ExampleUsermod":[20," lux"]} to the info object - //int reading = 20; - //JsonArray lightArr = user.createNestedArray(FPSTR(_name))); //name - //lightArr.add(reading); //value - //lightArr.add(F(" lux")); //unit - - // if you are implementing a sensor usermod, you may publish sensor data - //JsonObject sensor = root[F("sensor")]; - //if (sensor.isNull()) sensor = root.createNestedObject(F("sensor")); - //temp = sensor.createNestedArray(F("light")); - //temp.add(reading); - //temp.add(F("lux")); - } - - - /* - * addToJsonState() can be used to add custom entries to the /json/state part of the JSON API (state object). - * Values in the state object may be modified by connected clients - */ - void addToJsonState(JsonObject& root) override - { - if (!initDone || !enabled) return; // prevent crash on boot applyPreset() - - JsonObject usermod = root[FPSTR(_name)]; - if (usermod.isNull()) usermod = root.createNestedObject(FPSTR(_name)); - - //usermod["user0"] = userVar0; - } - - - /* - * readFromJsonState() can be used to receive data clients send to the /json/state part of the JSON API (state object). - * Values in the state object may be modified by connected clients - */ - void readFromJsonState(JsonObject& root) override - { - if (!initDone) return; // prevent crash on boot applyPreset() - - JsonObject usermod = root[FPSTR(_name)]; - if (!usermod.isNull()) { - // expect JSON usermod data in usermod name object: {"ExampleUsermod:{"user0":10}"} - userVar0 = usermod["user0"] | userVar0; //if "user0" key exists in JSON, update, else keep old value (userVar0 is defined in wled.h) - } - // you can as well check WLED state JSON keys - //if (root["bri"] == 255) Serial.println(F("Don't burn down your garage!")); - } - - - /* - * addToConfig() can be used to add custom persistent settings to the cfg.json file in the "um" (usermod) object. - * It will be called by WLED when settings are actually saved (for example, LED settings are saved) - * If you want to force saving the current state, use serializeConfig() in your loop(). - * - * CAUTION: serializeConfig() will initiate a filesystem write operation. - * It might cause the LEDs to stutter and will cause flash wear if called too often. - * Use it sparingly and always in the loop, never in network callbacks! - * - * addToConfig() will make your settings editable through the Usermod Settings page automatically. - * - * Usermod Settings Overview: - * - Numeric values are treated as floats in the browser. - * - If the numeric value entered into the browser contains a decimal point, it will be parsed as a C float - * before being returned to the Usermod. The float data type has only 6-7 decimal digits of precision, and - * doubles are not supported, numbers will be rounded to the nearest float value when being parsed. - * The range accepted by the input field is +/- 1.175494351e-38 to +/- 3.402823466e+38. - * - If the numeric value entered into the browser doesn't contain a decimal point, it will be parsed as a - * C int32_t (range: -2147483648 to 2147483647) before being returned to the usermod. - * Overflows or underflows are truncated to the max/min value for an int32_t, and again truncated to the type - * used in the Usermod when reading the value from ArduinoJson. - * - Pin values can be treated differently from an integer value by using the key name "pin" - * - "pin" can contain a single or array of integer values - * - On the Usermod Settings page there is simple checking for pin conflicts and warnings for special pins - * - Red color indicates a conflict. Yellow color indicates a pin with a warning (e.g. an input-only pin) - * - Tip: use int8_t to store the pin value in the Usermod, so a -1 value (pin not set) can be used - * - * See usermod_v2_auto_save.h for an example that saves Flash space by reusing ArduinoJson key name strings - * - * If you need a dedicated settings page with custom layout for your Usermod, that takes a lot more work. - * You will have to add the setting to the HTML, xml.cpp and set.cpp manually. - * See the WLED Soundreactive fork (code and wiki) for reference. https://github.com/atuline/WLED - * - * I highly recommend checking out the basics of ArduinoJson serialization and deserialization in order to use custom settings! - */ - void addToConfig(JsonObject& root) override - { - JsonObject top = root.createNestedObject(FPSTR(_name)); - top[FPSTR(_enabled)] = enabled; - //save these vars persistently whenever settings are saved - top["great"] = userVar0; - top["testBool"] = testBool; - top["testInt"] = testInt; - top["testLong"] = testLong; - top["testULong"] = testULong; - top["testFloat"] = testFloat; - top["testString"] = testString; - JsonArray pinArray = top.createNestedArray("pin"); - pinArray.add(testPins[0]); - pinArray.add(testPins[1]); - } - - - /* - * readFromConfig() can be used to read back the custom settings you added with addToConfig(). - * This is called by WLED when settings are loaded (currently this only happens immediately after boot, or after saving on the Usermod Settings page) - * - * readFromConfig() is called BEFORE setup(). This means you can use your persistent values in setup() (e.g. pin assignments, buffer sizes), - * but also that if you want to write persistent values to a dynamic buffer, you'd need to allocate it here instead of in setup. - * If you don't know what that is, don't fret. It most likely doesn't affect your use case :) - * - * Return true in case the config values returned from Usermod Settings were complete, or false if you'd like WLED to save your defaults to disk (so any missing values are editable in Usermod Settings) - * - * getJsonValue() returns false if the value is missing, or copies the value into the variable provided and returns true if the value is present - * The configComplete variable is true only if the "exampleUsermod" object and all values are present. If any values are missing, WLED will know to call addToConfig() to save them - * - * This function is guaranteed to be called on boot, but could also be called every time settings are updated - */ - bool readFromConfig(JsonObject& root) override - { - // default settings values could be set here (or below using the 3-argument getJsonValue()) instead of in the class definition or constructor - // setting them inside readFromConfig() is slightly more robust, handling the rare but plausible use case of single value being missing after boot (e.g. if the cfg.json was manually edited and a value was removed) - - JsonObject top = root[FPSTR(_name)]; - - bool configComplete = !top.isNull(); - - configComplete &= getJsonValue(top["great"], userVar0); - configComplete &= getJsonValue(top["testBool"], testBool); - configComplete &= getJsonValue(top["testULong"], testULong); - configComplete &= getJsonValue(top["testFloat"], testFloat); - configComplete &= getJsonValue(top["testString"], testString); - - // A 3-argument getJsonValue() assigns the 3rd argument as a default value if the Json value is missing - configComplete &= getJsonValue(top["testInt"], testInt, 42); - configComplete &= getJsonValue(top["testLong"], testLong, -42424242); - - // "pin" fields have special handling in settings page (or some_pin as well) - configComplete &= getJsonValue(top["pin"][0], testPins[0], -1); - configComplete &= getJsonValue(top["pin"][1], testPins[1], -1); - - return configComplete; - } - - - /* - * appendConfigData() is called when user enters usermod settings page - * it may add additional metadata for certain entry fields (adding drop down is possible) - * be careful not to add too much as oappend() buffer is limited to 3k - */ - void appendConfigData() override - { - oappend(F("addInfo('")); oappend(String(FPSTR(_name)).c_str()); oappend(F(":great")); oappend(F("',1,'(this is a great config value)');")); - oappend(F("addInfo('")); oappend(String(FPSTR(_name)).c_str()); oappend(F(":testString")); oappend(F("',1,'enter any string you want');")); - oappend(F("dd=addDropdown('")); oappend(String(FPSTR(_name)).c_str()); oappend(F("','testInt');")); - oappend(F("addOption(dd,'Nothing',0);")); - oappend(F("addOption(dd,'Everything',42);")); - } - - - /* - * handleOverlayDraw() is called just before every show() (LED strip update frame) after effects have set the colors. - * Use this to blank out some LEDs or set them to a different color regardless of the set effect mode. - * Commonly used for custom clocks (Cronixie, 7 segment) - */ - void handleOverlayDraw() override - { - //strip.setPixelColor(0, RGBW32(0,0,0,0)) // set the first pixel to black - } - - - /** - * handleButton() can be used to override default button behaviour. Returning true - * will prevent button working in a default way. - * Replicating button.cpp - */ - bool handleButton(uint8_t b) override { - yield(); - // ignore certain button types as they may have other consequences - if (!enabled - || buttons[b].type == BTN_TYPE_NONE - || buttons[b].type == BTN_TYPE_RESERVED - || buttons[b].type == BTN_TYPE_PIR_SENSOR - || buttons[b].type == BTN_TYPE_ANALOG - || buttons[b].type == BTN_TYPE_ANALOG_INVERTED) { - return false; - } - - bool handled = false; - // do your button handling here - return handled; - } - - -#ifndef WLED_DISABLE_MQTT - /** - * handling of MQTT message - * topic only contains stripped topic (part after /wled/MAC) - */ - bool onMqttMessage(char* topic, char* payload) override { - // check if we received a command - //if (strlen(topic) == 8 && strncmp_P(topic, PSTR("/command"), 8) == 0) { - // String action = payload; - // if (action == "on") { - // enabled = true; - // return true; - // } else if (action == "off") { - // enabled = false; - // return true; - // } else if (action == "toggle") { - // enabled = !enabled; - // return true; - // } - //} - return false; - } - - /** - * onMqttConnect() is called when MQTT connection is established - */ - void onMqttConnect(bool sessionPresent) override { - // do any MQTT related initialisation here - //publishMqtt("I am alive!"); - } -#endif - - - /** - * onStateChanged() is used to detect WLED state change - * @mode parameter is CALL_MODE_... parameter used for notifications - */ - void onStateChange(uint8_t mode) override { - // do something if WLED state changed (color, brightness, effect, preset, etc) - } - - - /* - * getId() allows you to optionally give your V2 usermod an unique ID (please define it in const.h!). - * This could be used in the future for the system to determine whether your usermod is installed. - */ - uint16_t getId() override - { - return USERMOD_ID_EXAMPLE; - } - - //More methods can be added in the future, this example will then be extended. - //Your usermod will remain compatible as it does not need to implement all methods from the Usermod base class! -}; - - -// add more strings here to reduce flash memory usage -const char MyExampleUsermod::_name[] PROGMEM = "ExampleUsermod"; -const char MyExampleUsermod::_enabled[] PROGMEM = "enabled"; - - -// implementation of non-inline member methods - -void MyExampleUsermod::publishMqtt(const char* state, bool retain) -{ -#ifndef WLED_DISABLE_MQTT - //Check if MQTT Connected, otherwise it will crash the 8266 - if (WLED_MQTT_CONNECTED) { - char subuf[64]; - strcpy(subuf, mqttDeviceTopic); - strcat_P(subuf, PSTR("/example")); - mqtt->publish(subuf, 0, retain, state); - } -#endif -} - -static MyExampleUsermod example_usermod; -REGISTER_USERMOD(example_usermod); diff --git a/usermods/readme.md b/usermods/readme.md index eefb64dbd0..8258877517 100644 --- a/usermods/readme.md +++ b/usermods/readme.md @@ -1,21 +1,33 @@ # Usermods -This folder serves as a repository for usermods (custom `usermod.cpp` files)! +This folder contains usermods, optional WLED components maintained by the core WLED team and some legacy modules with separate maintainers. Usermods are self-contained modules that add functionality without modifying core source files. -If you have created a usermod you believe is useful (for example to support a particular sensor, display, feature...), feel free to contribute by opening a pull request! +## Writing your own usermod -In order for other people to be able to have fun with your usermod, please keep these points in mind: +Start from the official example repository — click **Use this template** on GitHub to create your own copy: -* Create a folder in this folder with a descriptive name (for example `usermod_ds18b20_temp_sensor_mqtt`) -* Include your custom files -* If your usermod requires changes to other WLED files, please write a `readme.md` outlining the steps one needs to take -* Create a pull request! -* If your feature is useful for the majority of WLED users, I will consider adding it to the base code! +**[github.com/wled/wled-usermod-example](https://github.com/wled/wled-usermod-example)** -While I do my best to not break too much, keep in mind that as WLED is updated, usermods might break. -I am not actively maintaining any usermod in this directory, that is your responsibility as the creator of the usermod. +It contains a fully annotated implementation and a `library.json` template. Keep your usermod in its own repository and reference it from your WLED build via `custom_usermods` — no changes to the WLED source tree needed. -For new usermods, I would recommend trying out the new v2 usermod API, which allows installing multiple usermods at once and new functions! -You can take a look at `EXAMPLE_v2` for some documentation and at `Temperature` for a completed v2 usermod! +For the complete guide see **[kno.wled.ge/advanced/custom-features](https://kno.wled.ge/advanced/custom-features/)**, covering: -Thank you for your help :) +- Enabling usermods via `custom_usermods` in `platformio_override.ini` +- Local development with `symlink://` references +- Sharing via git URL +- `library.json` structure and the required `"libArchive": false` setting +- All lifecycle methods (`setup`, `loop`, `addToConfig`, `readFromConfig`, etc.) +- Adding custom LED effects via usermod + +Once your usermod is ready, add it to the [Community Usermods index](https://kno.wled.ge/advanced/community-usermods/) and tag your repository with the [`wled-usermod`](https://github.com/topics/wled-usermod) GitHub topic so others can find it. + +## Contributing a usermod to this folder + +The preferred approach is an independent repository (see above). If you strongly believe that your module adds a commonly missing feature that would be useful in most WLED installations, and maintenance can be managed by the core WLED team, you can suggest it for inclusion here: + +- Create a subfolder with a descriptive name +- Include a `library.json` and your source files +- Add a `README.md` describing what the mod does and any wiring or configuration required +- Open a pull request on the WLED repo + +The bar for inclusion in the main WLED repository is quite high. We encourage you to consider maintaining the module in your own repository for a period of time first. From c5baf7e3cd6271544c96f677c9aaa6b94e343100 Mon Sep 17 00:00:00 2001 From: Will Miles Date: Sun, 21 Dec 2025 18:05:09 -0500 Subject: [PATCH 02/11] build_ui: Build only when needed Use SCons dependencies to only run npm if we need it. --- pio-scripts/build_ui.py | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/pio-scripts/build_ui.py b/pio-scripts/build_ui.py index eb7a01b366..3dda3f90b9 100644 --- a/pio-scripts/build_ui.py +++ b/pio-scripts/build_ui.py @@ -1,4 +1,5 @@ Import("env") +from pathlib import Path # For OS-agnostic path manipulation import shutil node_ex = shutil.which("node") @@ -7,7 +8,9 @@ print('\x1b[0;31;43m' + 'Node.js is not installed or missing from PATH html css js will not be processed check https://kno.wled.ge/advanced/compiling-wled/' + '\x1b[0m') exitCode = env.Execute("null") exit(exitCode) -else: + + +def build_ui(target, source, env): # Install the necessary node packages for the pre-build asset bundling script print('\x1b[6;33;42m' + 'Installing node packages' + '\x1b[0m') env.Execute("npm ci") @@ -19,3 +22,31 @@ if (exitCode): print('\x1b[0;31;43m' + 'npm run build fails check https://kno.wled.ge/advanced/compiling-wled/' + '\x1b[0m') exit(exitCode) + +ui_srcdir = Path(env["PROJECT_DIR"]).resolve() / "wled00" / "data" + +build_ui_cmd = env.Command( + target=env.File('$PROJECT_DIR/wled00/html_ui.h'), # A representative file + source=[ + env.File('$PROJECT_DIR/package-lock.json'), + env.File('$PROJECT_DIR/package.json'), + env.File('$PROJECT_DIR/tools/cdata.js'), + *[env.File(str(p)) for p in ui_srcdir.rglob("*") if p.is_file()] + ], + action=build_ui) + +# Ensure UI gets built before any cpp files by hooking the Object constructor +def _wrap_object_method(name): + if not hasattr(env, name): return + orig = getattr(env, name) + def wrapped(*args, **kwargs): + nodes = orig(*args, **kwargs) + env.Requires(nodes, build_ui_cmd) + return nodes + setattr(env, name, wrapped) + +for m in ("Object", "StaticObject", "SharedObject"): + _wrap_object_method(m) + +# Also make an an explicit dependency of the final build target, so it will always be built the first time +env.Depends("$BUILD_DIR/$PROGNAME$PROGSUFFIX", build_ui_cmd) From ffad7004f8d705814be8ae57908bb1cfcf172002 Mon Sep 17 00:00:00 2001 From: Will Miles Date: Mon, 25 May 2026 14:26:47 +0000 Subject: [PATCH 03/11] feat: usermod static HTML support Usermods can now provide a cdata.json manifest alongside their source, declaring HTML/JS/CSS assets to be compiled into a gzipped C header at build time. The file schema is described in `tools/cdata.schema.json` These manifests are processed concordantly with SCons build rules, so the build system can naturally track dependencies and rebuild only when a manifest or its source files change. Includes numerous error handling fixes inside `cdata.js`. Co-Authored-By: Claude Sonnet 4.6 Co-Authored-By: Claude Opus 4.8 --- .gitignore | 1 + pio-scripts/build_ui.py | 259 ++++++++++-- pio-scripts/load_usermods.py | 13 + tools/cdata-test.js | 211 +++++++--- tools/cdata.js | 776 ++++++++++++++++++++++------------- tools/cdata.schema.json | 83 ++++ 6 files changed, 943 insertions(+), 400 deletions(-) create mode 100644 tools/cdata.schema.json diff --git a/.gitignore b/.gitignore index a2f10883a6..d5209fe0f2 100644 --- a/.gitignore +++ b/.gitignore @@ -31,3 +31,4 @@ wled-update.sh /wled00/wled00.ino.cpp /wled00/html_*.h /wled00/js_*.h +/usermods/*/html_*.h diff --git a/pio-scripts/build_ui.py b/pio-scripts/build_ui.py index 3dda3f90b9..c1e05bbc64 100644 --- a/pio-scripts/build_ui.py +++ b/pio-scripts/build_ui.py @@ -1,6 +1,26 @@ Import("env") +import os +import re +import subprocess from pathlib import Path # For OS-agnostic path manipulation import shutil +from SCons.Script import Exit + +# The web UI build banner lives here (Python), not in cdata.js, so it prints +# once per build during script evaluation -- before any compilation output -- +# instead of appearing partway through the build (or not at all) whenever the +# node process happened to run. +WLED_BANNER = ( + "\n" + "\t\x1b[34m ## ## ## ###### ######\n" + "\t\x1b[34m## ## ## ## ## ## ##\n" + "\t\x1b[34m## ## ## ## ###### ## ##\n" + "\t\x1b[34m## ## ## ## ## ## ##\n" + "\t\x1b[34m ## ## ###### ###### ######\n" + "\t\t\x1b[36m build script for web UI\n" + "\x1b[0m" +) +print(WLED_BANNER) node_ex = shutil.which("node") # Check if Node.js is installed and present in PATH if it failed, abort the build @@ -9,44 +29,203 @@ exitCode = env.Execute("null") exit(exitCode) +PROJECT_DIR = Path(env["PROJECT_DIR"]).resolve() +CDATA_JS = PROJECT_DIR / "tools" / "cdata.js" + + +# --- Dependency graph ----------------------------------------------------------- +# +# cdata.js is the single source of truth for which web UI headers exist and what +# they depend on; it emits that graph as a Makefile-style ".d" depfile. We read +# that cached depfile to declare a single SCons Command -- covering the main UI +# and every usermod -- with its complete list of target headers and input files. +# The actual build (one node invocation) then runs as a normal build node, at +# build time, ordered ahead of anything that #includes the generated headers. + +def _graph_inputs(manifests): + """Files whose change may alter the *shape* of the graph (which outputs exist + / what depends on what), so the cached depfile can no longer be trusted.""" + inputs = [CDATA_JS, PROJECT_DIR / "package.json"] + for name in ("platformio.ini", "platformio_override.ini"): + p = PROJECT_DIR / name + if p.exists(): + inputs.append(p) + inputs.extend(Path(m) for m in manifests) + return inputs + + +def _depfile_structure_current(depfile, manifests): + """True if the depfile exists and is newer than every structural input.""" + if not depfile.exists(): + return False + depfile_mtime = depfile.stat().st_mtime + for src in _graph_inputs(manifests): + try: + if src.stat().st_mtime > depfile_mtime: + return False + except OSError: + pass + return True + + +def _resolve_dep(token): + """Un-escape a depfile token ("\\ " -> " ") and resolve it against the project.""" + token = token.strip().replace('\\ ', ' ') + return os.path.normpath(os.path.join(str(PROJECT_DIR), token)) + + +def _parse_depfile(depfile): + """Read a Makefile-style depfile into (targets, sources) lists of absolute + paths. Paths are project-relative with forward slashes (no Windows + drive-letter colons); spaces within a path are escaped as "\\ ", so the + dependency list is split only on *unescaped* whitespace.""" + targets, sources, seen = [], [], set() + for line in depfile.read_text().splitlines(): + line = line.strip() + if not line or line.startswith('#') or ':' not in line: + continue + target, deps = line.split(':', 1) + targets.append(_resolve_dep(target)) + for dep in re.split(r'(? node_modules.stat().st_mtime + ) + if stale: + print('\x1b[6;33;42m' + 'Installing node packages' + '\x1b[0m') + rc = env.Execute("npm ci") + if rc: + print('\x1b[0;31;43m' + 'npm ci failed check https://kno.wled.ge/advanced/compiling-wled/' + '\x1b[0m') + Exit(rc) + + +def _wire_object_methods(target_env, cmd): + """Make every object compiled by target_env require the UI build command, so + the generated headers exist before anything #including them compiles -- even + under a parallel (-j) build.""" + for name in ("Object", "StaticObject", "SharedObject"): + if not hasattr(target_env, name): + continue + orig = getattr(target_env, name) + def wrapped(*args, _orig=orig, _env=target_env, **kwargs): + nodes = _orig(*args, **kwargs) + _env.Requires(nodes, cmd) + return nodes + setattr(target_env, name, wrapped) + + +# Guard so the UI build is registered at most once per environment. +_registered = {"done": False} + + +def _register_ui_build(xenv, result): + if _registered["done"]: + return + _registered["done"] = True + + # Usermod HTML manifests discovered by load_usermods.py (may be empty). + manifests = list(xenv.get("WLED_UI_MANIFESTS", [])) + depfile = Path(xenv.subst("$BUILD_DIR")).resolve() / "ui_deps.d" + + # Make sure the cached graph is present and structurally current so the + # Command is declared with the right targets/sources. This launches node + # only to (re)describe the graph, and only when a manifest / cdata.js / + # platformio.ini changed -- never to build the UI. The build itself is the + # deferred Command below. + if not _depfile_structure_current(depfile, manifests): + _emit_depfile(depfile, manifests) + + targets, sources = _parse_depfile(depfile) + + # The single node invocation that builds the whole web UI (main + usermods), + # run as a normal build node when SCons finds a target stale. + node_cmd = " ".join( + ['node', '"%s"' % CDATA_JS, '--depfile', '"%s"' % depfile] + + ['--manifest "%s"' % m for m in manifests] + ) + + def _build_ui(target, source, env, _cmd=node_cmd): + _ensure_node_modules(env) + rc = env.Execute(_cmd) + if rc: + print('\x1b[0;31;43m' + 'Web UI build failed check https://kno.wled.ge/advanced/compiling-wled/' + '\x1b[0m') + return rc + + ui_cmd = env.Command( + target=[env.File(t) for t in targets], + source=[env.File(s) for s in sources], + action=env.VerboseAction(_build_ui, "Building web UI"), + ) + + # By default SCons removes a builder's targets before running its action. + # Because every header is a target of this one Command, editing any single + # UI source would delete ALL of them, cdata.js would then see them missing + # and rebuild every one (each with a fresh WEB_BUILD_TIME), and everything + # that #includes a generated header would needlessly recompile. Mark the + # targets Precious so SCons leaves them in place and cdata.js's own + # per-job staleness check does the incremental rebuild. (Precious only + # affects pre-build removal, not `pio run -t clean`.) + env.Precious(ui_cmd) + + # Order the build ahead of compilation. PlatformIO compiles the main WLED + # sources with result.env (the project-as-library builder's env; see + # ProcessProjectDeps -> plb.env.BuildSources), and each usermod's sources + # with that module's own env -- so we wire both. + _wire_object_methods(result.env, ui_cmd) + # Match on realpath both sides: manifests come from load_usermods.py already + # resolved, and a symlink:// usermod's src_dir may reach through a symlink, + # so a plain normpath comparison could miss and leave its objects unwired. + manifest_dirs = {os.path.realpath(os.path.dirname(m)) for m in manifests} + for dep in result.depbuilders: + if os.path.realpath(str(dep.src_dir)) in manifest_dirs: + _wire_object_methods(dep.env, ui_cmd) + + # Belt-and-suspenders: also an explicit prerequisite of the final firmware. + xenv.Depends("$BUILD_DIR/$PROGNAME$PROGSUFFIX", ui_cmd) + + +# Hook ConfigureProjectLibBuilder *after* load_usermods.py's wrapper, so the +# usermod manifest list is already populated and the returned builder (whose env +# compiles the main sources) is available. This script is listed after +# load_usermods.py in platformio.ini, so our wrapper is outermost and runs last. +# We only register nodes here; the build runs later, at build time. +_old_ConfigureProjectLibBuilder = env.ConfigureProjectLibBuilder + +def _wrapped_ConfigureProjectLibBuilder(xenv): + result = _old_ConfigureProjectLibBuilder.clone(xenv)() + _register_ui_build(xenv, result) + return result -def build_ui(target, source, env): - # Install the necessary node packages for the pre-build asset bundling script - print('\x1b[6;33;42m' + 'Installing node packages' + '\x1b[0m') - env.Execute("npm ci") - - # Call the bundling script - exitCode = env.Execute("npm run build") - - # If it failed, abort the build - if (exitCode): - print('\x1b[0;31;43m' + 'npm run build fails check https://kno.wled.ge/advanced/compiling-wled/' + '\x1b[0m') - exit(exitCode) - -ui_srcdir = Path(env["PROJECT_DIR"]).resolve() / "wled00" / "data" - -build_ui_cmd = env.Command( - target=env.File('$PROJECT_DIR/wled00/html_ui.h'), # A representative file - source=[ - env.File('$PROJECT_DIR/package-lock.json'), - env.File('$PROJECT_DIR/package.json'), - env.File('$PROJECT_DIR/tools/cdata.js'), - *[env.File(str(p)) for p in ui_srcdir.rglob("*") if p.is_file()] - ], - action=build_ui) - -# Ensure UI gets built before any cpp files by hooking the Object constructor -def _wrap_object_method(name): - if not hasattr(env, name): return - orig = getattr(env, name) - def wrapped(*args, **kwargs): - nodes = orig(*args, **kwargs) - env.Requires(nodes, build_ui_cmd) - return nodes - setattr(env, name, wrapped) - -for m in ("Object", "StaticObject", "SharedObject"): - _wrap_object_method(m) - -# Also make an an explicit dependency of the final build target, so it will always be built the first time -env.Depends("$BUILD_DIR/$PROGNAME$PROGSUFFIX", build_ui_cmd) +env.AddMethod(_wrapped_ConfigureProjectLibBuilder, "ConfigureProjectLibBuilder") diff --git a/pio-scripts/load_usermods.py b/pio-scripts/load_usermods.py index 18852ff30b..22aa1255dc 100644 --- a/pio-scripts/load_usermods.py +++ b/pio-scripts/load_usermods.py @@ -201,6 +201,19 @@ def wrapped_ConfigureProjectLibBuilder(xenv): fg="red", err=True) Exit(1) + # Collect the HTML-asset manifests of the selected usermods and publish them + # for build_ui.py, which builds all UI assets (main WLED + usermods) in a + # single node invocation after this discovery step. The generated header + # lives in each usermod's own source directory, so SCons's C/C++ scanner + # detects the #include and orders compilation correctly. + ui_manifests = [] + for dep in wled_deps: + manifest_path = Path(dep.src_dir) / 'cdata.json' + if manifest_path.exists(): + ui_manifests.append(str(manifest_path.resolve())) + + xenv['WLED_UI_MANIFESTS'] = ui_manifests + # Save the depbuilders list for later validation xenv.Replace(WLED_MODULES=wled_deps) diff --git a/tools/cdata-test.js b/tools/cdata-test.js index 169756475e..c25137c668 100644 --- a/tools/cdata-test.js +++ b/tools/cdata-test.js @@ -3,76 +3,21 @@ const assert = require('node:assert'); const { describe, it, before, after } = require('node:test'); const fs = require('fs'); +const os = require('os'); const path = require('path'); const child_process = require('child_process'); const util = require('util'); const execPromise = util.promisify(child_process.exec); -process.env.NODE_ENV = 'test'; // Set the environment to testing -const cdata = require('./cdata.js'); +// Importing the build script must be side-effect free: with NODE_ENV=test it +// defines its helpers but must not run a build. +process.env.NODE_ENV = 'test'; +require('./cdata.js'); -describe('Function', () => { - const testFolderPath = path.join(__dirname, 'testFolder'); - const oldFilePath = path.join(testFolderPath, 'oldFile.txt'); - const newFilePath = path.join(testFolderPath, 'newFile.txt'); - - // Create a temporary file before the test - before(() => { - // Create test folder - if (!fs.existsSync(testFolderPath)) { - fs.mkdirSync(testFolderPath); - } - - // Create an old file - fs.writeFileSync(oldFilePath, 'This is an old file.'); - // Modify the 'mtime' to simulate an old file - const oldTime = new Date(); - oldTime.setFullYear(oldTime.getFullYear() - 1); - fs.utimesSync(oldFilePath, oldTime, oldTime); - - // Create a new file - fs.writeFileSync(newFilePath, 'This is a new file.'); - }); - - // delete the temporary files after the test - after(() => { - fs.rmSync(testFolderPath, { recursive: true }); - }); - - describe('isFileNewerThan', async () => { - it('should return true if the file is newer than the provided time', async () => { - const pastTime = Date.now() - 10000; // 10 seconds ago - assert.strictEqual(cdata.isFileNewerThan(newFilePath, pastTime), true); - }); - - it('should return false if the file is older than the provided time', async () => { - assert.strictEqual(cdata.isFileNewerThan(oldFilePath, Date.now()), false); - }); - - it('should throw an exception if the file does not exist', async () => { - assert.throws(() => { - cdata.isFileNewerThan('nonexistent.txt', Date.now()); - }); - }); - }); - - describe('isAnyFileInFolderNewerThan', async () => { - it('should return true if a file in the folder is newer than the given time', async () => { - const time = fs.statSync(path.join(testFolderPath, 'oldFile.txt')).mtime; - assert.strictEqual(cdata.isAnyFileInFolderNewerThan(testFolderPath, time), true); - }); - - it('should return false if no files in the folder are newer than the given time', async () => { - assert.strictEqual(cdata.isAnyFileInFolderNewerThan(testFolderPath, new Date()), false); - }); - - it('should throw an exception if the folder does not exist', async () => { - assert.throws(() => { - cdata.isAnyFileInFolderNewerThan('nonexistent', new Date()); - }); - }); - }); -}); +// The CLI decides whether to build from NODE_ENV, so child processes must not +// inherit the test harness's NODE_ENV=test (which suppresses the build). +const buildEnv = { ...process.env, NODE_ENV: 'production' }; +const runCdata = (args = '') => execPromise('node tools/cdata.js ' + args, { env: buildEnv }); describe('Script', () => { const folderPath = 'wled00'; @@ -216,3 +161,141 @@ describe('Script', () => { }); }); }); + +describe('Dependency graph (--emit-deps / --depfile)', () => { + const depfile = path.join(os.tmpdir(), `cdata-deps-${process.pid}.d`); + + after(() => { + try { fs.rmSync(depfile); } catch { /* ignore */ } + }); + + it('emits a depfile listing every generated header, without building', async () => { + const { stdout } = await runCdata(`--emit-deps --depfile "${depfile}"`); + assert.match(stdout, /Wrote dependency graph/); + assert.doesNotMatch(stdout, /Minified/); // nothing was actually built + + const dep = fs.readFileSync(depfile, 'utf8'); + for (const header of ['wled00/html_ui.h', 'wled00/html_settings.h', 'wled00/js_iro.h']) { + assert.match(dep, new RegExp('^' + header.replace(/\./g, '\\.') + ':', 'm'), + `depfile is missing a rule for ${header}`); + } + }); + + it('records cdata.js and package.json as inputs of every header', async () => { + await runCdata(`--emit-deps --depfile "${depfile}"`); + const dep = fs.readFileSync(depfile, 'utf8'); + for (const line of dep.split('\n')) { + if (!line || line.startsWith('#') || !line.includes(':')) continue; + assert.match(line, /tools\/cdata\.js/, 'every rule should depend on cdata.js'); + assert.match(line, /package\.json/, 'every rule should depend on package.json'); + } + }); + + it('escapes spaces in dependency paths so each stays one token', async () => { + // wled00/data/icons-ui/Read Me.txt has a space in its name + await runCdata(`--emit-deps --depfile "${depfile}"`); + const dep = fs.readFileSync(depfile, 'utf8'); + assert.match(dep, /icons-ui\/Read\\ Me\.txt/); // escaped: one token + assert.doesNotMatch(dep, /icons-ui\/Read Me\.txt/); // never an unescaped space + }); +}); + +describe('Usermod manifests', () => { + let modDir; + const manifest = () => path.join(modDir, 'cdata.json'); + + before(() => { + modDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cdata-mod-')); + fs.mkdirSync(path.join(modDir, 'data')); + fs.writeFileSync(path.join(modDir, 'data', 'page.htm'), + '

Hi ##VERSION##

'); + fs.writeFileSync(manifest(), JSON.stringify({ + header: [{ + output: 'html_mod.h', + srcDir: 'data', + specs: [{ file: 'page.htm', name: 'PAGE_mod', method: 'gzip', filter: 'html-minify' }], + }], + })); + }); + + after(() => { + fs.rmSync(modDir, { recursive: true, force: true }); + }); + + it('legacy single-manifest mode builds only that header', async () => { + const out = path.join(modDir, 'html_mod.h'); + const { stdout } = await runCdata(`"${manifest()}"`); + assert(fs.existsSync(out), 'usermod header was not built'); + assert.match(fs.readFileSync(out, 'utf8'), /PAGE_mod/); + assert.doesNotMatch(stdout, /index\.htm|html_ui\.h/); // main UI is not part of this run + }); + + it('--manifest adds the usermod header to the graph alongside the main UI', async () => { + const depfile = path.join(modDir, 'deps.d'); + await runCdata(`--emit-deps --depfile "${depfile}" --manifest "${manifest()}"`); + const dep = fs.readFileSync(depfile, 'utf8'); + assert.match(dep, /html_mod\.h:/); // the usermod output + assert.match(dep, /wled00\/html_ui\.h:/); // together with the main UI + }); + + it('rejects a manifest with an unknown top-level key', async () => { + const bad = path.join(modDir, 'bad.json'); + fs.writeFileSync(bad, JSON.stringify({ + inject: {}, // reserved for the future, not accepted by current tooling + header: [{ + output: 'html_bad.h', + specs: [{ file: 'page.htm', name: 'PAGE_bad', method: 'gzip', filter: 'html-minify' }], + }], + })); + await assert.rejects(runCdata(`"${bad}"`), /unknown top-level key/); + fs.rmSync(bad); + }); + + it('builds one header per header op in a multi-header manifest', async () => { + const multi = path.join(modDir, 'multi.json'); + fs.writeFileSync(multi, JSON.stringify({ + header: [ + { output: 'html_a.h', srcDir: 'data', + specs: [{ file: 'page.htm', name: 'PAGE_a', method: 'gzip', filter: 'html-minify' }] }, + { output: 'html_b.h', srcDir: 'data', + specs: [{ file: 'page.htm', name: 'PAGE_b', method: 'gzip', filter: 'html-minify' }] }, + ], + })); + await runCdata(`"${multi}"`); + assert.match(fs.readFileSync(path.join(modDir, 'html_a.h'), 'utf8'), /PAGE_a/); + assert.match(fs.readFileSync(path.join(modDir, 'html_b.h'), 'utf8'), /PAGE_b/); + fs.rmSync(multi); + }); + + // The build hard-fails on any invalid manifest (cdata.js is the single + // validator; the Python layer only forwards paths). Each case asserts the + // CLI exits non-zero with a message identifying the specific problem. + const spec = { file: 'page.htm', name: 'PAGE_x', method: 'gzip', filter: 'html-minify' }; + const invalidManifests = { + 'malformed JSON': { raw: '{ "header": [ }', error: /Could not read manifest/ }, + 'a header op missing output': { + json: { header: [{ specs: [spec] }] }, error: /missing a string 'output'/ }, + 'an empty header array': { + json: { header: [] }, error: /missing or empty 'header' array/ }, + 'no header key at all': { + json: { schemaVersion: 1 }, error: /missing or empty 'header' array/ }, + 'an unsupported schemaVersion': { + json: { schemaVersion: 2, header: [{ output: 'x.h', specs: [spec] }] }, + error: /unsupported schemaVersion 2/ }, + 'two header ops writing the same output': { + json: { header: [ + { output: 'dup.h', srcDir: 'data', specs: [{ ...spec, name: 'PAGE_a' }] }, + { output: 'dup.h', srcDir: 'data', specs: [{ ...spec, name: 'PAGE_b' }] }, + ] }, + error: /collides with an earlier header op/ }, + }; + + for (const [desc, { raw, json, error }] of Object.entries(invalidManifests)) { + it(`rejects ${desc}`, async () => { + const bad = path.join(modDir, 'invalid.json'); + fs.writeFileSync(bad, raw !== undefined ? raw : JSON.stringify(json)); + await assert.rejects(runCdata(`"${bad}"`), error); + fs.rmSync(bad); + }); + } +}); diff --git a/tools/cdata.js b/tools/cdata.js index 5ae7088b3e..a258f94ffd 100644 --- a/tools/cdata.js +++ b/tools/cdata.js @@ -12,31 +12,26 @@ * * How it works? * - * It uses NodeJS packages to inline, minify and GZIP files. See writeHtmlGzipped and writeChunks invocations at the bottom of the page. + * It uses NodeJS packages to inline, minify and GZIP files. See the mainJobs table and writeChunks/writeHtmlGzipped below. + * + * Command line: + * node cdata.js build the main web UI (skips up-to-date outputs) + * node cdata.js -f | --force rebuild everything unconditionally + * node cdata.js --manifest ... additionally build these usermod manifests in the same run + * node cdata.js --depfile write a Makefile-style dependency graph for the build system + * node cdata.js --emit-deps --depfile only write the dependency graph, build nothing + * node cdata.js legacy single-manifest mode (build just that manifest) */ const fs = require("node:fs"); const path = require("path"); -const inline = require("web-resource-inliner"); const zlib = require("node:zlib"); -const CleanCSS = require("clean-css"); -const minifyHtml = require("html-minifier-terser").minify; const packageJson = require("../package.json"); -// Export functions for testing -module.exports = { isFileNewerThan, isAnyFileInFolderNewerThan }; - -const output = ["wled00/html_ui.h", "wled00/html_pixart.h", "wled00/html_cpal.h", "wled00/html_edit.h", "wled00/html_pxmagic.h", "wled00/html_pixelforge.h", "wled00/html_settings.h", "wled00/html_other.h", "wled00/js_iro.h", "wled00/js_omggif.h"] - -// \x1b[34m is blue, \x1b[36m is cyan, \x1b[0m is reset -const wledBanner = ` -\t\x1b[34m ## ## ## ###### ###### -\t\x1b[34m## ## ## ## ## ## ## -\t\x1b[34m## ## ## ## ###### ## ## -\t\x1b[34m## ## ## ## ## ## ## -\t\x1b[34m ## ## ###### ###### ###### -\t\t\x1b[36m build script for web UI -\x1b[0m`; +// The heavy third-party packages (web-resource-inliner, clean-css, +// html-minifier-terser) are require()d lazily inside the functions that use +// them. This keeps the --emit-deps dependency-graph pass working before +// `npm ci` has populated node_modules, and keeps node startup cheap. // Generate build timestamp as UNIX timestamp (seconds since epoch) function generateBuildTime() { @@ -46,14 +41,14 @@ function generateBuildTime() { const singleHeader = `/* * Binary array for the Web UI. * gzip is used for smaller size and improved speeds. - * + * * Please see https://kno.wled.ge/advanced/custom-features/#changing-web-ui * to find out how to easily modify the web UI source! */ // Automatically generated build time for cache busting (UNIX timestamp) #define WEB_BUILD_TIME ${generateBuildTime()} - + `; const multiHeader = `/* @@ -123,40 +118,51 @@ async function minify(str, type = "plain") { if (type == "plain") { return str; } else if (type == "css-minify") { + const CleanCSS = require("clean-css"); return new CleanCSS({}).minify(str).styles; } else if (type == "js-minify") { + const minifyHtml = require("html-minifier-terser").minify; let js = await minifyHtml('', options); return js.replace(/<[\/]*script>/g, ''); } else if (type == "html-minify") { + const minifyHtml = require("html-minifier-terser").minify; return await minifyHtml(str, options); } throw new Error("Unknown filter: " + type); } +// Promisified wrapper around web-resource-inliner's callback API. Resolves with +// the inlined HTML (css/js pulled in) or rejects on error. Without this, the +// enclosing async function used to resolve *before* the callback ran, so callers +// could not await the real work and a thrown error vanished into an unhandled +// rejection instead of propagating. +function inlineHtml(sourceFile, inlineCss) { + const inline = require("web-resource-inliner"); + return new Promise((resolve, reject) => { + inline.html({ + fileContent: fs.readFileSync(sourceFile, "utf8"), + relativeTo: path.dirname(sourceFile), + strict: inlineCss, // when not inlining css, ignore errors (enables linking style.css from subfolder htm files) + stylesheets: inlineCss // when true (default), css is inlined + }, (error, html) => error ? reject(error) : resolve(html)); + }); +} + async function writeHtmlGzipped(sourceFile, resultFile, page, inlineCss = true) { console.info("Reading " + sourceFile); - inline.html({ - fileContent: fs.readFileSync(sourceFile, "utf8"), - relativeTo: path.dirname(sourceFile), - strict: inlineCss, // when not inlining css, ignore errors (enables linking style.css from subfolder htm files) - stylesheets: inlineCss // when true (default), css is inlined - }, - async function (error, html) { - if (error) throw error; - - html = adoptVersionAndRepo(html); - const originalLength = html.length; - html = await minify(html, "html-minify"); - const result = zlib.gzipSync(html, { level: zlib.constants.Z_BEST_COMPRESSION }); - console.info("Minified and compressed " + sourceFile + " from " + originalLength + " to " + result.length + " bytes"); - const array = hexdump(result); - let src = singleHeader; - src += `const uint16_t PAGE_${page}_length = ${result.length};\n`; - src += `const uint8_t PAGE_${page}[] PROGMEM = {\n${array}\n};\n\n`; - console.info("Writing " + resultFile); - fs.writeFileSync(resultFile, src); - }); + let html = await inlineHtml(sourceFile, inlineCss); + html = adoptVersionAndRepo(html); + const originalLength = html.length; + html = await minify(html, "html-minify"); + const result = zlib.gzipSync(html, { level: zlib.constants.Z_BEST_COMPRESSION }); + console.info("Minified and compressed " + sourceFile + " from " + originalLength + " to " + result.length + " bytes"); + const array = hexdump(result); + let src = singleHeader; + src += `const uint16_t PAGE_${page}_length = ${result.length};\n`; + src += `const uint8_t PAGE_${page}[] PROGMEM = {\n${array}\n};\n\n`; + console.info("Writing " + resultFile); + fs.writeFileSync(resultFile, src); } async function specToChunk(srcDir, s) { @@ -201,280 +207,458 @@ async function writeChunks(srcDir, specs, resultFile) { fs.writeFileSync(resultFile, src); } -// Check if a file is newer than a given time -function isFileNewerThan(filePath, time) { - const stats = fs.statSync(filePath); - return stats.mtimeMs > time; +// List every file under a folder, recursively (absolute or srcDir-relative paths). +function listFilesRecursive(folderPath) { + const out = []; + for (const entry of fs.readdirSync(folderPath, { withFileTypes: true })) { + const p = path.join(folderPath, entry.name); + if (entry.isDirectory()) out.push(...listFilesRecursive(p)); + else out.push(p); + } + return out; } -// Check if any file in a folder (or its subfolders) is newer than a given time -function isAnyFileInFolderNewerThan(folderPath, time) { - const files = fs.readdirSync(folderPath, { withFileTypes: true }); - for (const file of files) { - const filePath = path.join(folderPath, file.name); - if (isFileNewerThan(filePath, time)) { - return true; +// Every generated header, described as data so a single code path can both build +// it and report its input dependencies. +// kind:'html' — a single .htm inlined + gzipped into a PAGE_ array +// kind:'chunks' — one or more specs concatenated into a header +const mainJobs = [ + { kind: 'html', src: "wled00/data/index.htm", out: "wled00/html_ui.h", page: 'index' }, + { kind: 'html', src: "wled00/data/pixart/pixart.htm", out: "wled00/html_pixart.h", page: 'pixart' }, + { kind: 'html', src: "wled00/data/pxmagic/pxmagic.htm", out: "wled00/html_pxmagic.h", page: 'pxmagic' }, + { kind: 'html', src: "wled00/data/pixelforge/pixelforge.htm", out: "wled00/html_pixelforge.h", page: 'pixelforge', inlineCss: false }, // do not inline css + //{ kind: 'html', src: "wled00/data/edit.htm", out: "wled00/html_edit.h", page: 'edit' }, + + { + kind: 'chunks', + srcDir: "wled00/data/", + out: "wled00/js_iro.h", + specs: [ + { + file: "iro.js", + name: "JS_iro", + method: "gzip", + filter: "plain", // no minification, it is already minified + mangle: (s) => s.replace(/^\/\*![\s\S]*?\*\//, '') // remove license comment at the top + } + ], + }, + + { + kind: 'chunks', + srcDir: "wled00/data/pixelforge", + out: "wled00/js_omggif.h", + specs: [ + { + file: "omggif.js", + name: "JS_omggif", + method: "gzip", + filter: "js-minify", + mangle: (s) => s.replace(/^\/\*![\s\S]*?\*\//, '') // remove license comment at the top + } + ], + }, + + { + kind: 'chunks', + srcDir: "wled00/data", + out: "wled00/html_edit.h", + specs: [ + { + file: "edit.htm", + name: "PAGE_edit", + method: "gzip", + filter: "html-minify" + } + ], + }, + + { + kind: 'chunks', + srcDir: "wled00/data/cpal", + out: "wled00/html_cpal.h", + specs: [ + { + file: "cpal.htm", + name: "PAGE_cpal", + method: "gzip", + filter: "html-minify" + } + ], + }, + + { + kind: 'chunks', + srcDir: "wled00/data", + out: "wled00/html_settings.h", + specs: [ + { + file: "style.css", + name: "PAGE_settingsCss", + method: "gzip", + filter: "css-minify", + mangle: (str) => + str + .replace("%%", "%") + }, + { + file: "common.js", + name: "JS_common", + method: "gzip", + filter: "js-minify", + }, + { + file: "settings.htm", + name: "PAGE_settings", + method: "gzip", + filter: "html-minify", + }, + { + file: "settings_wifi.htm", + name: "PAGE_settings_wifi", + method: "gzip", + filter: "html-minify", + }, + { + file: "settings_leds.htm", + name: "PAGE_settings_leds", + method: "gzip", + filter: "html-minify", + }, + { + file: "settings_dmx.htm", + name: "PAGE_settings_dmx", + method: "gzip", + filter: "html-minify", + }, + { + file: "settings_ui.htm", + name: "PAGE_settings_ui", + method: "gzip", + filter: "html-minify", + }, + { + file: "settings_sync.htm", + name: "PAGE_settings_sync", + method: "gzip", + filter: "html-minify", + }, + { + file: "settings_time.htm", + name: "PAGE_settings_time", + method: "gzip", + filter: "html-minify", + }, + { + file: "settings_sec.htm", + name: "PAGE_settings_sec", + method: "gzip", + filter: "html-minify", + }, + { + file: "settings_um.htm", + name: "PAGE_settings_um", + method: "gzip", + filter: "html-minify", + }, + { + file: "settings_2D.htm", + name: "PAGE_settings_2D", + method: "gzip", + filter: "html-minify", + }, + { + file: "settings_pin.htm", + name: "PAGE_settings_pin", + method: "gzip", + filter: "html-minify" + }, + { + file: "settings_pininfo.htm", + name: "PAGE_settings_pininfo", + method: "gzip", + filter: "html-minify" + } + ], + }, + + { + kind: 'chunks', + srcDir: "wled00/data", + out: "wled00/html_other.h", + specs: [ + { + file: "usermod.htm", + name: "PAGE_usermod", + method: "gzip", + filter: "html-minify", + mangle: (str) => + str.replace(/fetch\("http\:\/\/.*\/win/gms, 'fetch("/win'), + }, + { + file: "msg.htm", + name: "PAGE_msg", + prepend: "=====(", + append: ")=====", + method: "plaintext", + filter: "html-minify", + mangle: (str) => str.replace(/\.*\<\/body\>/gms, "

%MSG%"), + }, + { + file: "dmxmap.htm", + name: "PAGE_dmxmap", + prepend: "=====(", + append: ")=====", + method: "plaintext", + filter: "html-minify", + mangle: (str) => ` +#ifdef WLED_ENABLE_DMX +${str.replace(/function FM\(\)[ ]?\{/gms, "function FM() {%DMXVARS%\n")} +#else +const char PAGE_dmxmap[] PROGMEM = R"=====()====="; +#endif +`, + }, + { + file: "update.htm", + name: "PAGE_update", + method: "gzip", + filter: "html-minify", + }, + { + file: "welcome.htm", + name: "PAGE_welcome", + method: "gzip", + filter: "html-minify", + }, + { + file: "liveview.htm", + name: "PAGE_liveview", + method: "gzip", + filter: "html-minify", + }, + { + file: "liveviewws2D.htm", + name: "PAGE_liveviewws2D", + method: "gzip", + filter: "html-minify", + }, + { + file: "404.htm", + name: "PAGE_404", + method: "gzip", + filter: "html-minify", + }, + { + file: "favicon.ico", + name: "favicon", + method: "binary", + } + ], + }, +]; + +// Top-level keys a usermod manifest (cdata.json) is allowed to carry. Any other +// key is a validation error: the schema is closed so typos and not-yet-supported +// features (e.g. 'inject', 'defaults') fail loudly instead of being ignored. +const MANIFEST_ALLOWED_KEYS = new Set(['header', 'schemaVersion', '$schema']); + +// Turn a usermod manifest (cdata.json) into one chunks job per header op. +// The manifest is an externally-tagged object: `header` is an array of +// header-operations, each of which produces one generated header file. +function manifestToJobs(manifestArg) { + const manifestPath = path.resolve(manifestArg); + const modDir = path.dirname(manifestPath); + const fail = (msg) => { + console.error("Invalid manifest " + manifestPath + ": " + msg); + process.exit(1); + }; + + let manifest; + try { + manifest = JSON.parse(fs.readFileSync(manifestPath, 'utf8')); + } catch (e) { + console.error("Could not read manifest " + manifestPath + ": " + e.message); + process.exit(1); + } + + if (typeof manifest !== 'object' || manifest === null || Array.isArray(manifest)) { + fail("top-level value must be an object"); + } + for (const key of Object.keys(manifest)) { + if (!MANIFEST_ALLOWED_KEYS.has(key)) fail("unknown top-level key '" + key + "'"); + } + // Absent schemaVersion means the baseline contract (implicit v1); only absent + // or an explicit 1 is understood by this tooling. + if ('schemaVersion' in manifest && manifest.schemaVersion !== 1) { + fail("unsupported schemaVersion " + JSON.stringify(manifest.schemaVersion) + " (expected 1 or absent)"); + } + if (!Array.isArray(manifest.header) || manifest.header.length === 0) { + fail("missing or empty 'header' array"); + } + + const seenOut = new Set(); + return manifest.header.map((op, i) => { + if (typeof op !== 'object' || op === null || Array.isArray(op)) { + fail("header[" + i + "] must be an object"); } - if (file.isDirectory() && isAnyFileInFolderNewerThan(filePath, time)) { - return true; + if (typeof op.output !== 'string' || op.output.length === 0) { + fail("header[" + i + "] is missing a string 'output'"); } - } - return false; + if (!Array.isArray(op.specs) || op.specs.length === 0) { + fail("header[" + i + "] is missing a non-empty 'specs' array"); + } + // Two ops writing the same file would declare two SCons builders for one + // target (an error); catch it here with a clear message instead. + const out = path.join(modDir, op.output); + if (seenOut.has(out)) { + fail("header[" + i + "] output '" + op.output + "' collides with an earlier header op; each must write a distinct file"); + } + seenOut.add(out); + return { + kind: 'chunks', + srcDir: path.join(modDir, op.srcDir || 'data'), + out: out, + specs: op.specs, + manifestPath: manifestPath, // the manifest itself is an input + }; + }); } -// Check if the web UI is already built -function isAlreadyBuilt(webUIPath, packageJsonPath = "package.json") { - let lastBuildTime = Infinity; +// The set of input files a job's output depends on. An html job inlines +// arbitrary resources from its source folder, so we conservatively depend on the +// whole folder; a chunks job depends on exactly its listed specs. Every job also +// depends on this script and package.json (they affect the generated output). +function jobInputs(job) { + let files; + if (job.kind === 'html') { + files = listFilesRecursive(path.dirname(job.src)); + } else { + files = job.specs.map(s => path.join(job.srcDir, s.file)); + } + files = files.concat(__filename, 'package.json'); + if (job.manifestPath) files.push(job.manifestPath); + return files; +} - for (const file of output) { +// A job is stale if its output is missing or older than any of its inputs. +function isJobStale(job) { + let outMs; + try { + outMs = fs.statSync(job.out).mtimeMs; + } catch (e) { + if (e.code === 'ENOENT') return true; + throw e; + } + for (const input of jobInputs(job)) { try { - lastBuildTime = Math.min(lastBuildTime, fs.statSync(file).mtimeMs); + if (fs.statSync(input).mtimeMs > outMs) return true; } catch (e) { - if (e.code !== 'ENOENT') throw e; - console.info("File " + file + " does not exist. Rebuilding..."); - return false; + if (e.code !== 'ENOENT') throw e; // a missing input can't make us stale } } + return false; +} - return !isAnyFileInFolderNewerThan(webUIPath, lastBuildTime) && !isFileNewerThan(packageJsonPath, lastBuildTime) && !isFileNewerThan(__filename, lastBuildTime); +async function buildJob(job) { + if (job.kind === 'html') { + await writeHtmlGzipped(job.src, job.out, job.page, job.inlineCss !== false); + } else { + await writeChunks(job.srcDir, job.specs, job.out); + } } -// Don't run this script if we're in a test environment -if (process.env.NODE_ENV === 'test') { - return; +// Emit a Makefile-style depfile: one `output: input1 input2 ...` rule per job. +// Paths are written relative to the current directory (the project root) with +// forward slashes, so the file is portable and free of Windows drive-letter +// colons that would confuse a depfile parser. Spaces in file names are escaped +// as "\ " (Makefile style) so a name like "Read Me.txt" stays a single token. +function toDepPath(p) { + return path.relative(process.cwd(), path.resolve(p)).split(path.sep).join('/'); } -console.info(wledBanner); +function escapeDepPath(p) { + return toDepPath(p).replace(/ /g, '\\ '); +} -if (isAlreadyBuilt("wled00/data") && process.argv[2] !== '--force' && process.argv[2] !== '-f') { - console.info("Web UI is already built"); - return; +function emitDepfile(depfilePath, jobs) { + let out = "# Auto-generated by tools/cdata.js -- web UI dependency graph. Do not edit.\n"; + for (const job of jobs) { + const target = escapeDepPath(job.out); + const deps = jobInputs(job).map(escapeDepPath); + out += `${target}: ${deps.join(' ')}\n`; + } + fs.mkdirSync(path.dirname(path.resolve(depfilePath)), { recursive: true }); + fs.writeFileSync(depfilePath, out); + console.info("Wrote dependency graph (" + jobs.length + " targets) to " + depfilePath); } -writeHtmlGzipped("wled00/data/index.htm", "wled00/html_ui.h", 'index'); -writeHtmlGzipped("wled00/data/pixart/pixart.htm", "wled00/html_pixart.h", 'pixart'); -writeHtmlGzipped("wled00/data/pxmagic/pxmagic.htm", "wled00/html_pxmagic.h", 'pxmagic'); -writeHtmlGzipped("wled00/data/pixelforge/pixelforge.htm", "wled00/html_pixelforge.h", 'pixelforge', false); // do not inline css -//writeHtmlGzipped("wled00/data/edit.htm", "wled00/html_edit.h", 'edit'); - -writeChunks( - "wled00/data/", - [ - { - file: "iro.js", - name: "JS_iro", - method: "gzip", - filter: "plain", // no minification, it is already minified - mangle: (s) => s.replace(/^\/\*![\s\S]*?\*\//, '') // remove license comment at the top - } - ], - "wled00/js_iro.h" -); - -writeChunks( - "wled00/data/pixelforge", - [ - { - file: "omggif.js", - name: "JS_omggif", - method: "gzip", - filter: "js-minify", - mangle: (s) => s.replace(/^\/\*![\s\S]*?\*\//, '') // remove license comment at the top - } - ], - "wled00/js_omggif.h" -); - -writeChunks( - "wled00/data", - [ - { - file: "edit.htm", - name: "PAGE_edit", - method: "gzip", - filter: "html-minify" - } - ], - "wled00/html_edit.h" -); - -writeChunks( - "wled00/data/cpal", - [ - { - file: "cpal.htm", - name: "PAGE_cpal", - method: "gzip", - filter: "html-minify" +// Await every task; if any fail, report them all and throw once. Using +// allSettled rather than Promise.all means one failing conversion cannot abort +// its siblings mid-write and leave a truncated header behind. +async function runAll(tasks) { + const results = await Promise.allSettled(tasks); + const errors = results.filter(r => r.status === 'rejected').map(r => r.reason); + if (errors.length > 0) { + for (const err of errors) console.error(err); + throw new Error(errors.length + " web UI build task(s) failed"); + } +} + +function parseArgs(args) { + const opts = { force: false, depfile: null, emitDepsOnly: false, manifests: [], legacyManifest: null }; + for (let i = 0; i < args.length; i++) { + const a = args[i]; + if (a === '-f' || a === '--force') opts.force = true; + else if (a === '--emit-deps') opts.emitDepsOnly = true; + else if (a === '--depfile') { + opts.depfile = args[++i]; + if (opts.depfile === undefined) { console.error("--depfile requires a path"); process.exit(1); } } - ], - "wled00/html_cpal.h" -); - -writeChunks( - "wled00/data", - [ - { - file: "style.css", - name: "PAGE_settingsCss", - method: "gzip", - filter: "css-minify", - mangle: (str) => - str - .replace("%%", "%") - }, - { - file: "common.js", - name: "JS_common", - method: "gzip", - filter: "js-minify", - }, - { - file: "settings.htm", - name: "PAGE_settings", - method: "gzip", - filter: "html-minify", - }, - { - file: "settings_wifi.htm", - name: "PAGE_settings_wifi", - method: "gzip", - filter: "html-minify", - }, - { - file: "settings_leds.htm", - name: "PAGE_settings_leds", - method: "gzip", - filter: "html-minify", - }, - { - file: "settings_dmx.htm", - name: "PAGE_settings_dmx", - method: "gzip", - filter: "html-minify", - }, - { - file: "settings_ui.htm", - name: "PAGE_settings_ui", - method: "gzip", - filter: "html-minify", - }, - { - file: "settings_sync.htm", - name: "PAGE_settings_sync", - method: "gzip", - filter: "html-minify", - }, - { - file: "settings_time.htm", - name: "PAGE_settings_time", - method: "gzip", - filter: "html-minify", - }, - { - file: "settings_sec.htm", - name: "PAGE_settings_sec", - method: "gzip", - filter: "html-minify", - }, - { - file: "settings_um.htm", - name: "PAGE_settings_um", - method: "gzip", - filter: "html-minify", - }, - { - file: "settings_2D.htm", - name: "PAGE_settings_2D", - method: "gzip", - filter: "html-minify", - }, - { - file: "settings_pin.htm", - name: "PAGE_settings_pin", - method: "gzip", - filter: "html-minify" - }, - { - file: "settings_pininfo.htm", - name: "PAGE_settings_pininfo", - method: "gzip", - filter: "html-minify" + else if (a === '--manifest') { + const m = args[++i]; + if (m === undefined) { console.error("--manifest requires a path"); process.exit(1); } + opts.manifests.push(m); } - ], - "wled00/html_settings.h" -); - -writeChunks( - "wled00/data", - [ - { - file: "usermod.htm", - name: "PAGE_usermod", - method: "gzip", - filter: "html-minify", - mangle: (str) => - str.replace(/fetch\("http\:\/\/.*\/win/gms, 'fetch("/win'), - }, - { - file: "msg.htm", - name: "PAGE_msg", - prepend: "=====(", - append: ")=====", - method: "plaintext", - filter: "html-minify", - mangle: (str) => str.replace(/\.*\<\/body\>/gms, "

%MSG%"), - }, - { - file: "dmxmap.htm", - name: "PAGE_dmxmap", - prepend: "=====(", - append: ")=====", - method: "plaintext", - filter: "html-minify", - mangle: (str) => ` -#ifdef WLED_ENABLE_DMX -${str.replace(/function FM\(\)[ ]?\{/gms, "function FM() {%DMXVARS%\n")} -#else -const char PAGE_dmxmap[] PROGMEM = R"=====()====="; -#endif -`, - }, - { - file: "update.htm", - name: "PAGE_update", - method: "gzip", - filter: "html-minify", - }, - { - file: "welcome.htm", - name: "PAGE_welcome", - method: "gzip", - filter: "html-minify", - }, - { - file: "liveview.htm", - name: "PAGE_liveview", - method: "gzip", - filter: "html-minify", - }, - { - file: "liveviewws2D.htm", - name: "PAGE_liveviewws2D", - method: "gzip", - filter: "html-minify", - }, - { - file: "404.htm", - name: "PAGE_404", - method: "gzip", - filter: "html-minify", - }, - { - file: "favicon.ico", - name: "favicon", - method: "binary", + else if (!a.startsWith('-')) opts.legacyManifest = a; + else { console.error("Unknown option: " + a); process.exit(1); } + } + return opts; +} + +async function main() { + const opts = parseArgs(process.argv.slice(2)); + + // Assemble the job list. Legacy single-manifest mode builds only that + // manifest; otherwise we build the main UI plus any --manifest usermods. + const jobs = opts.legacyManifest + ? manifestToJobs(opts.legacyManifest) + : mainJobs.concat(opts.manifests.flatMap(manifestToJobs)); + + if (opts.emitDepsOnly) { + if (!opts.depfile) { + console.error("--emit-deps requires --depfile "); + process.exit(1); } - ], - "wled00/html_other.h" -); + emitDepfile(opts.depfile, jobs); + return; + } + + const stale = opts.force ? jobs : jobs.filter(isJobStale); + if (stale.length === 0) { + console.info("Web UI is already built"); + } else { + await runAll(stale.map(buildJob)); + } + + // Refresh the dependency graph for the build system, if requested. + if (opts.depfile) emitDepfile(opts.depfile, jobs); +} + +// Don't run the build when imported by the test harness. +if (process.env.NODE_ENV !== 'test') { + main().catch((err) => { + console.error(err); + process.exitCode = 1; + }); +} diff --git a/tools/cdata.schema.json b/tools/cdata.schema.json new file mode 100644 index 0000000000..c15df6f93c --- /dev/null +++ b/tools/cdata.schema.json @@ -0,0 +1,83 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "$id": "https://raw.githubusercontent.com/wled-dev/WLED/main/tools/cdata.schema.json", + "title": "WLED usermod cdata.json manifest", + "description": "Describes how a usermod's web UI source files are compiled into generated C headers at build time. Consumed by tools/cdata.js. Keep in sync with the validation in tools/cdata.js.", + "type": "object", + "additionalProperties": false, + "required": ["header"], + "properties": { + "$schema": { + "type": "string", + "description": "Optional JSON Schema reference for editor validation. Ignored by the build." + }, + "schemaVersion": { + "type": "integer", + "enum": [1], + "description": "Manifest schema version. Omit to use the baseline schema (implicit v1); only 1 is currently supported. A future breaking change would introduce version 2." + }, + "header": { + "type": "array", + "minItems": 1, + "description": "Header-generation operations. Each produces one generated .h file in the usermod's own directory.", + "items": { "$ref": "#/definitions/headerOp" } + } + }, + "definitions": { + "headerOp": { + "type": "object", + "additionalProperties": false, + "required": ["output", "specs"], + "properties": { + "srcDir": { + "type": "string", + "default": "data", + "description": "Folder, relative to the manifest, holding the source files. Defaults to \"data\"." + }, + "output": { + "type": "string", + "minLength": 1, + "description": "Filename of the generated header, written into the usermod's own directory (e.g. \"example_ui.h\")." + }, + "specs": { + "type": "array", + "minItems": 1, + "description": "One entry per source file / C symbol to embed in this header.", + "items": { "$ref": "#/definitions/spec" } + } + } + }, + "spec": { + "type": "object", + "additionalProperties": false, + "required": ["file", "name", "method"], + "properties": { + "file": { + "type": "string", + "description": "Source filename under srcDir." + }, + "name": { + "type": "string", + "description": "C symbol to generate; e.g. \"PAGE_example\" yields a PAGE_example[] PROGMEM array plus PAGE_example_length." + }, + "method": { + "enum": ["gzip", "plaintext", "binary"], + "description": "How the embedded data is encoded: gzip-compressed, verbatim text, or a raw byte array." + }, + "filter": { + "enum": ["html-minify", "css-minify", "js-minify", "plain"], + "default": "plain", + "description": "Pre-processing applied to the source before embedding. Defaults to plain (no minification)." + }, + "prepend": { + "type": "string", + "description": "Raw-string delimiter prefix; only meaningful with the plaintext method." + }, + "append": { + "type": "string", + "description": "Raw-string delimiter suffix; only meaningful with the plaintext method." + } + } + } + } +} From cd32e2d8df4db99fc79d514debdf53c44d2ee7d1 Mon Sep 17 00:00:00 2001 From: Will Miles Date: Mon, 25 May 2026 19:45:12 +0000 Subject: [PATCH 04/11] feat: expose handleStaticContent() for usermod use MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removes the static linkage from handleStaticContent() in wled_server.cpp and adds a forward declaration in fcn_declare.h, making it available to usermods that want to serve their own gzip-compressed static assets via the same ETag/cache/gzip mechanism the core pages use. This is intentionally minimal — the full server API refactor (wled::serve namespace, header decomposition) remains deferred. Co-Authored-By: Claude Sonnet 4.6 --- wled00/fcn_declare.h | 1 + wled00/wled_server.cpp | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/wled00/fcn_declare.h b/wled00/fcn_declare.h index 6201a19192..d5471be63c 100644 --- a/wled00/fcn_declare.h +++ b/wled00/fcn_declare.h @@ -595,6 +595,7 @@ void serveMessage(AsyncWebServerRequest* request, uint16_t code, const String& h void serveJsonError(AsyncWebServerRequest* request, uint16_t code, uint16_t error); void serveSettings(AsyncWebServerRequest* request, bool post = false); void serveSettingsJS(AsyncWebServerRequest* request); +void handleStaticContent(AsyncWebServerRequest *request, const String &path, int code, const String &contentType, const uint8_t *content, size_t len, bool gzip = true, uint16_t eTagSuffix = 0); //ws.cpp void handleWs(); diff --git a/wled00/wled_server.cpp b/wled00/wled_server.cpp index 0b4d0fb546..12515fb9f1 100644 --- a/wled00/wled_server.cpp +++ b/wled00/wled_server.cpp @@ -127,7 +127,7 @@ static bool handleIfNoneMatchCacheHeader(AsyncWebServerRequest *request, int cod * @param gzip Optional. Defaults to true. If false, the gzip header will not be added. * @param eTagSuffix Optional. Defaults to 0. A suffix that will be added to the ETag header. This can be used to invalidate the cache for a specific page. */ -static void handleStaticContent(AsyncWebServerRequest *request, const String &path, int code, const String &contentType, const uint8_t *content, size_t len, bool gzip = true, uint16_t eTagSuffix = 0) { +void handleStaticContent(AsyncWebServerRequest *request, const String &path, int code, const String &contentType, const uint8_t *content, size_t len, bool gzip, uint16_t eTagSuffix) { if (path != "" && handleFileRead(request, path)) return; if (handleIfNoneMatchCacheHeader(request, code, eTagSuffix)) return; AsyncWebServerResponse *response = request->beginResponse_P(code, contentType, content, len); From 185e4ef8356c1569239b937e49a213ad795a0b43 Mon Sep 17 00:00:00 2001 From: Will Miles Date: Mon, 25 May 2026 19:46:08 +0000 Subject: [PATCH 05/11] feat: add _name to Usermod base class Adds a _name member (const char*, PROGMEM ok) to the Usermod protected section, with getName(). Field names are intentionally compatible with MoonModules' fork to allow MM usermods to work with minimal changes. For mods that don't pass a name to the constructor, probe addToConfig() with a small stack-local JsonDocument and take the first top-level key as the name. (H/T MoonModules for this idea.) Key differences from MM's approach: - No base class _enabled: there's no practical solution for backwards compatibility with usermods that don't provide their own handling. - Explicit getter rather than direct field access Also adds UsermodManager::lookup(const char*) to find a usermod by name. Co-Authored-By: Claude Sonnet 4.6 --- wled00/fcn_declare.h | 9 +++++++-- wled00/um_manager.cpp | 43 ++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 47 insertions(+), 5 deletions(-) diff --git a/wled00/fcn_declare.h b/wled00/fcn_declare.h index d5471be63c..e2dbd3c760 100644 --- a/wled00/fcn_declare.h +++ b/wled00/fcn_declare.h @@ -346,8 +346,9 @@ const unsigned int um_data_size = sizeof(um_data_t); // 12 bytes class Usermod { protected: um_data_t *um_data; // um_data should be allocated using new in (derived) Usermod's setup() or constructor + const char *_name = nullptr; // PROGMEM pointer (set by constructor) or heap string (set by probeNameFromConfig()) public: - Usermod() : um_data(nullptr) {}; + Usermod(const char* name = nullptr) : um_data(nullptr), _name(name) {} virtual ~Usermod() { if (um_data) delete um_data; } virtual void setup() = 0; // pure virtual, has to be overriden virtual void loop() = 0; // pure virtual, has to be overriden @@ -360,7 +361,7 @@ class Usermod { virtual void addToJsonInfo(JsonObject& obj) {} // add JSON objects for UI Info page virtual void readFromJsonState(JsonObject& obj) {} // process JSON messages received from web server virtual void addToConfig(JsonObject& obj) {} // add JSON entries that go to cfg.json - virtual bool readFromConfig(JsonObject& obj) { return true; } // Note as of 2021-06 readFromConfig() now needs to return a bool, see usermod_v2_example.h + virtual bool readFromConfig(JsonObject& obj) { return true; } // apply config from JSON; return true if no changes were made virtual void onMqttConnect(bool sessionPresent) {} // fired when MQTT connection is established (so usermod can subscribe) virtual bool onMqttMessage(char* topic, char* payload) { return false; } // fired upon MQTT message received (wled topic) virtual bool onEspNowMessage(uint8_t* sender, uint8_t* payload, uint8_t len) { return false; } // fired upon ESP-NOW message received @@ -369,6 +370,9 @@ class Usermod { virtual void onStateChange(uint8_t mode) {} // fired upon WLED state change virtual uint16_t getId() {return USERMOD_ID_UNSPECIFIED;} + inline const char* getName() const { return _name; } + void probeNameFromConfig(); // set _name from first addToConfig() key if not already set; defined in um_manager.cpp + // API shims private: static Print* oappend_shim; @@ -408,6 +412,7 @@ namespace UsermodManager { void onUpdateBegin(bool); void onStateChange(uint8_t); Usermod* lookup(uint16_t mod_id); + Usermod* lookup(const char* mod_name); // find usermod by name (case-sensitive), safe for PROGMEM strings size_t getModCount(); }; diff --git a/wled00/um_manager.cpp b/wled00/um_manager.cpp index 504b5ba97c..222d7741b1 100644 --- a/wled00/um_manager.cpp +++ b/wled00/um_manager.cpp @@ -18,7 +18,12 @@ static size_t getCount() { //Usermod Manager internals -void UsermodManager::setup() { for (auto mod = DYNARRAY_BEGIN(usermods); mod < DYNARRAY_END(usermods); ++mod) (*mod)->setup(); } +void UsermodManager::setup() { + for (auto mod = DYNARRAY_BEGIN(usermods); mod < DYNARRAY_END(usermods); ++mod) { + if ((*mod)->getName() == nullptr) (*mod)->probeNameFromConfig(); // set _name from addToConfig() key if not provided by constructor + (*mod)->setup(); + } +} void UsermodManager::connected() { for (auto mod = DYNARRAY_BEGIN(usermods); mod < DYNARRAY_END(usermods); ++mod) (*mod)->connected(); } void UsermodManager::loop() { for (auto mod = DYNARRAY_BEGIN(usermods); mod < DYNARRAY_END(usermods); ++mod) (*mod)->loop(); } void UsermodManager::handleOverlayDraw() { for (auto mod = DYNARRAY_BEGIN(usermods); mod < DYNARRAY_END(usermods); ++mod) (*mod)->handleOverlayDraw(); } @@ -46,9 +51,13 @@ void UsermodManager::addToJsonInfo(JsonObject& obj) { } } void UsermodManager::readFromJsonState(JsonObject& obj) { for (auto mod = DYNARRAY_BEGIN(usermods); mod < DYNARRAY_END(usermods); ++mod) (*mod)->readFromJsonState(obj); } -void UsermodManager::addToConfig(JsonObject& obj) { for (auto mod = DYNARRAY_BEGIN(usermods); mod < DYNARRAY_END(usermods); ++mod) (*mod)->addToConfig(obj); } +void UsermodManager::addToConfig(JsonObject& obj) { + for (auto mod = DYNARRAY_BEGIN(usermods); mod < DYNARRAY_END(usermods); ++mod) { + (*mod)->addToConfig(obj); + } +} bool UsermodManager::readFromConfig(JsonObject& obj) { - bool allComplete = true; + bool allComplete = true; for (auto mod = DYNARRAY_BEGIN(usermods); mod < DYNARRAY_END(usermods); ++mod) { if (!(*mod)->readFromConfig(obj)) allComplete = false; } @@ -86,8 +95,36 @@ Usermod* UsermodManager::lookup(uint16_t mod_id) { return nullptr; } +// mod_name is a RAM string; _name may be PROGMEM (from constructor) or heap RAM (from probe). +// strcmp_P is safe for both on ESP32; on ESP8266 probe-set names are already RAM so strcmp_P works too. +Usermod* UsermodManager::lookup(const char* mod_name) { + if (!mod_name) return nullptr; + for (auto mod = DYNARRAY_BEGIN(usermods); mod < DYNARRAY_END(usermods); ++mod) { + const char* name = (*mod)->getName(); + if (name && strcmp_P(mod_name, name) == 0) return *mod; + } + return nullptr; +} + size_t UsermodManager::getModCount() { return getCount(); }; +/* Usermod base class name probe */ +void Usermod::probeNameFromConfig() { + if (_name) return; // already named by constructor + pDoc->clear(); // NOTE: this is safe only because it's called from `setup()` alone + JsonObject root = pDoc->to(); + addToConfig(root); + auto it = root.begin(); + if (it == root.end()) return; // mod wrote nothing + const char* key = it->key().c_str(); + if (!key || !key[0]) return; + char* buf = new char[strlen(key) + 1]; + if (buf) { + strcpy(buf, key); + _name = buf; + } // otherwise we're in big trouble anyways this early in the boot... +} + /* Usermod v2 interface shim for oappend */ Print* Usermod::oappend_shim = nullptr; From 2f7b32361d88fccff24e70b5b882f270fd80f56e Mon Sep 17 00:00:00 2001 From: Will Miles Date: Mon, 25 May 2026 19:47:09 +0000 Subject: [PATCH 06/11] feat: add per-usermod settings UI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Registers GET and POST handlers for /settings/um/*, placed before the /settings catch-all so the more-specific prefix wins. `settings_um.htm` is updated to support a per-usermod endpoint, so only the one usermod's fields appear. The global I2C/SPI bus section is moved to `/settings/hw/if`. The original `/settings/um` page is also preserved for backwards compatibility with old UI integrations. The global settings panel is expanded to generate links to the per- usermod page. GET /settings/mod/: - Looks up the mod by name via UsermodManager::lookup(const char*) - Returns 404 if the mod isn't found - Otherwise serves settings_um.htm (JS uses window.location.pathname to filter the display to the named mod — implemented in next commit) - Respects the PIN gate by delegating to serveSettings() when locked Usermods that register server.on("/settings/mod/mymod") during setup() shadow this catch-all for their specific path via first-registered-wins ordering. --- wled00/data/settings.htm | 10 +++- wled00/data/settings_um.htm | 113 +++++++++++++++++++++++++++++++++--- wled00/fcn_declare.h | 1 + wled00/um_manager.cpp | 8 ++- wled00/wled_server.cpp | 34 +++++++++++ wled00/xml.cpp | 14 +++++ 6 files changed, 169 insertions(+), 11 deletions(-) diff --git a/wled00/data/settings.htm b/wled00/data/settings.htm index ef20671c87..de5984d7e1 100644 --- a/wled00/data/settings.htm +++ b/wled00/data/settings.htm @@ -17,6 +17,13 @@ l.onerror = () => setTimeout(loadFiles, 100); document.head.appendChild(l); })(); + function addUMBtn(n) { + var b = document.createElement('button'); + b.type = 'button'; + b.onclick = function() { window.location = getURL('/settings/um/' + encodeURIComponent(n)); }; + b.textContent = n; + gId('umlist').appendChild(b); + }