diff --git a/Documentation/applications/index.rst b/Documentation/applications/index.rst index bed0932dc556f..a037d65ff6409 100644 --- a/Documentation/applications/index.rst +++ b/Documentation/applications/index.rst @@ -26,7 +26,7 @@ These can be found in the `apps `_ reposit logging/index.rst lte/index.rst math/index.rst - mlearing/index.rst + mlearning/index.rst netutils/index.rst nsh/index.rst sdr/index.rst diff --git a/Documentation/applications/mlearing/cmsis/index.rst b/Documentation/applications/mlearing/cmsis/index.rst deleted file mode 100644 index a1a066b9eba49..0000000000000 --- a/Documentation/applications/mlearing/cmsis/index.rst +++ /dev/null @@ -1,3 +0,0 @@ -========================= -``cmsis`` CMSIS Libraries -========================= diff --git a/Documentation/applications/mlearing/darknet/index.rst b/Documentation/applications/mlearing/darknet/index.rst deleted file mode 100644 index d366659234740..0000000000000 --- a/Documentation/applications/mlearing/darknet/index.rst +++ /dev/null @@ -1,3 +0,0 @@ -============================================ -``darknet`` YOLO: Real-Time Object Detection -============================================ diff --git a/Documentation/applications/mlearing/index.rst b/Documentation/applications/mlearing/index.rst deleted file mode 100644 index 5186f123f98d2..0000000000000 --- a/Documentation/applications/mlearing/index.rst +++ /dev/null @@ -1,11 +0,0 @@ -======================== -Machine Learning Support -======================== - -.. toctree:: - :glob: - :maxdepth: 1 - :titlesonly: - :caption: Contents - - */index* diff --git a/Documentation/applications/mlearing/libnnablart/index.rst b/Documentation/applications/mlearing/libnnablart/index.rst deleted file mode 100644 index 4fc9249f4ee61..0000000000000 --- a/Documentation/applications/mlearing/libnnablart/index.rst +++ /dev/null @@ -1,3 +0,0 @@ -======================================== -``libnnablart`` NNABLA Runtime Libraries -======================================== diff --git a/Documentation/applications/mlearning/cmsis-nn/index.rst b/Documentation/applications/mlearning/cmsis-nn/index.rst new file mode 100644 index 0000000000000..c56cde351bec6 --- /dev/null +++ b/Documentation/applications/mlearning/cmsis-nn/index.rst @@ -0,0 +1,36 @@ +============================= +``cmsis-nn`` CMSIS-NN Library +============================= + +``apps/mlearning/cmsis-nn`` builds the standalone `CMSIS-NN +`_ kernels. This is the package +TensorFlow Lite Micro uses when ``CONFIG_MLEARNING_CMSIS_NN`` is enabled. + +It is distinct from :doc:`../cmsis/index`, which vendors the older CMSIS 5 +tree (DSP plus NN) for NNABLA. + +Enable with ``CONFIG_MLEARNING_CMSIS_NN``. The Makefile downloads a pinned +CMSIS-NN commit and compiles: + +- Activation functions +- Basic math +- Concatenation +- Convolution +- Fully connected +- LSTM +- NN support functions +- Pooling +- Reshape +- Softmax +- SVD + +When ``CONFIG_ARM_NEON`` is also set, a few kernels are omitted here +because TFLM supplies NEON replacements under +``apps/mlearning/tflite-micro/operators/neon``. + +Headers are added to the include path from +``apps/mlearning/cmsis-nn/cmsis-nn/Include``. + +For TFLM, enabling this option defines ``CMSIS_NN`` and swaps TFLM +reference kernels for the CMSIS-NN variants. See +:doc:`../tflite-micro/index`. diff --git a/Documentation/applications/mlearning/cmsis/index.rst b/Documentation/applications/mlearning/cmsis/index.rst new file mode 100644 index 0000000000000..4bd872ceb7687 --- /dev/null +++ b/Documentation/applications/mlearning/cmsis/index.rst @@ -0,0 +1,33 @@ +========================= +``cmsis`` CMSIS Libraries +========================= + +``apps/mlearning/cmsis`` fetches `CMSIS 5 +`_ and builds CMSIS-DSP and +CMSIS-NN from that tree. Default version is ``5.8.0`` +(``CONFIG_CMSIS_VER``). + +This package is aimed at NNABLA and generic DSP use. For TensorFlow Lite +Micro kernel acceleration, use :doc:`../cmsis-nn/index` instead. + +Enable ``CONFIG_CMSIS`` from :menuselection:`Application Configuration --> +Machine Learning Support --> CMSIS Libraries`. + +``CONFIG_CMSIS_DSP`` + CMSIS-DSP. Optional ``CONFIG_CMSIS_DSP_ARM_MATH_MATRIX_CHECK`` and + ``CONFIG_CMSIS_DSP_ARM_MATH_ROUNDING`` map to ``ARM_MATH_MATRIX_CHECK`` + and ``ARM_MATH_ROUNDING``. ``CONFIG_ARCH_FPU`` defines + ``__FPU_PRESENT``. + +``CONFIG_CMSIS_NN`` + CMSIS-NN from the CMSIS 5 tree. Depends on ``CONFIG_CMSIS_DSP``. A + NuttX patch (``cmsis-nn-support_nnabla.patch``) adds CHW convolution + helpers used by NNABLA. + +Public headers are added from: + +- ``CMSIS_5/CMSIS/Core/Include`` +- ``CMSIS_5/CMSIS/DSP/Include`` (when DSP is enabled) +- ``CMSIS_5/CMSIS/NN/Include`` (when NN is enabled) + +See https://arm-software.github.io/CMSIS_5/General/html/index.html. diff --git a/Documentation/applications/mlearning/darknet/index.rst b/Documentation/applications/mlearning/darknet/index.rst new file mode 100644 index 0000000000000..f1f129e4f2ea3 --- /dev/null +++ b/Documentation/applications/mlearning/darknet/index.rst @@ -0,0 +1,18 @@ +============================================ +``darknet`` YOLO: Real-Time Object Detection +============================================ + +``apps/mlearning/darknet`` vendors `Darknet +`_, the framework behind YOLO (You +Only Look Once) object detection. + +Enable ``CONFIG_DARKNET_YOLO``. ``CONFIG_DARKNET_YOLO_VER`` is the Git +branch fetched at build time (default ``master``). The Makefile compiles +the core Darknet sources (network, parser, convolutional and connected +layers, YOLO/region/detection layers, RNN/GRU/LSTM, and image helpers). + +Include path: ``apps/mlearning/darknet/darknet/include``. + +Darknet is a larger, floating-point oriented stack compared with +TensorFlow Lite Micro. Prefer :doc:`../tflite-micro/index` for +microcontroller-class inference. diff --git a/Documentation/applications/mlearning/index.rst b/Documentation/applications/mlearning/index.rst new file mode 100644 index 0000000000000..54fddfd30bd63 --- /dev/null +++ b/Documentation/applications/mlearning/index.rst @@ -0,0 +1,33 @@ +======================== +Machine Learning Support +======================== + +NuttX machine learning support lives in the `apps +`_ repository under +``apps/mlearning``. These packages wrap upstream TinyML runtimes and +optimized kernels. The build system fetches the selected upstream sources +when the corresponding Kconfig option is enabled. + +Enable the libraries from:: + + Application Configuration ---> + Machine Learning Support ---> + +The following packages are available: + +- :doc:`cmsis/index` — ARM CMSIS 5 DSP and NN libraries +- :doc:`cmsis-nn/index` — standalone ARM CMSIS-NN kernels for TensorFlow Lite Micro +- :doc:`darknet/index` — Darknet YOLO object detection +- :doc:`libnnablart/index` — Sony NNABLA C inference runtime +- :doc:`tflite-micro/index` — TensorFlow Lite for Microcontrollers + +A ready-to-build simulator configuration is ``sim:tflm``. See +:doc:`/platforms/sim/sim/boards/sim/index`. + +.. toctree:: + :glob: + :maxdepth: 1 + :titlesonly: + :caption: Contents + + */index* diff --git a/Documentation/applications/mlearning/libnnablart/index.rst b/Documentation/applications/mlearning/libnnablart/index.rst new file mode 100644 index 0000000000000..489818a08f6de --- /dev/null +++ b/Documentation/applications/mlearning/libnnablart/index.rst @@ -0,0 +1,23 @@ +======================================== +``libnnablart`` NNABLA Runtime Libraries +======================================== + +``apps/mlearning/libnnablart`` builds the Sony `NNABLA C Runtime +`_ so NuttX applications can run +networks produced by Neural Network Libraries. + +Enable ``CONFIG_NNABLA_RT``. ``CONFIG_NNABLA_RT_VER`` selects the upstream +tag (default ``1.24.0``). The Makefile downloads that release and compiles +the runtime plus a subset of operators, including: + +- Activations (ReLU, sigmoid, tanh, softmax, ELU, leaky ReLU, Swish, ...) +- Arithmetic and reduction +- Affine (float and fixed-point) +- Convolution, depthwise convolution, deconvolution, and pooling +- Batch normalization, dropout, reshape, concatenate, and related array ops +- Binary-weight / binary-connect quantization helpers + +Include path: ``apps/mlearning/libnnablart/nnabla-c-runtime/include``. + +CMSIS 5 NN kernels with the NNABLA CHW helpers can be enabled together +via :doc:`../cmsis/index`. diff --git a/Documentation/applications/mlearning/tflite-micro/index.rst b/Documentation/applications/mlearning/tflite-micro/index.rst new file mode 100644 index 0000000000000..23eb04f16271f --- /dev/null +++ b/Documentation/applications/mlearning/tflite-micro/index.rst @@ -0,0 +1,180 @@ +====================================== +``tflite-micro`` TensorFlow Lite Micro +====================================== + +TensorFlow Lite for Microcontrollers (TFLM) is a C++ interpreter for +running ``.tflite`` models on memory-constrained targets. NuttX integrates +it from ``apps/mlearning/tflite-micro``. + +The build downloads a pinned TFLM snapshot from +https://github.com/tensorflow/tflite-micro and applies NuttX patches that +add INT8-only operator registrations and an ``extern "C"`` entry point for +the hello-world example. + +Dependencies +============ + +``CONFIG_TFLITEMICRO`` depends on all of: + +- ``CONFIG_SYSTEM_FLATBUFFERS`` — FlatBuffers headers (``apps/system/flatbuffers``) +- ``CONFIG_MATH_GEMMLOWP`` — gemmlowp (``apps/math/gemmlowp``) +- ``CONFIG_MATH_KISSFFT`` — KissFFT (``apps/math/kissfft``) +- ``CONFIG_MATH_RUY`` — Ruy (``apps/math/ruy``) + +TFLM is C++, so the configuration also needs C++ support (``CONFIG_HAVE_CXX`` +and a C++ standard library such as ``CONFIG_LIBCXX``). + +Optional acceleration: + +- ``CONFIG_MLEARNING_CMSIS_NN`` — replace reference kernels with CMSIS-NN + implementations. See :doc:`../cmsis-nn/index`. +- ``CONFIG_ARM_NEON`` — when CMSIS-NN is enabled, a few kernels are taken + from ``apps/mlearning/tflite-micro/operators/neon`` instead of CMSIS-NN. + +Configuration +============= + +From :menuselection:`Application Configuration --> Machine Learning Support`: + +``CONFIG_TFLITEMICRO`` + Enable the TFLM library. + +``CONFIG_TFLITEMICRO_DEBUG`` + Keep TFLM error strings and print memory-use information + (``TF_LITE_SHOW_MEMORY_USE``). When disabled, ``TF_LITE_STRIP_ERROR_STRINGS`` + is set to reduce code size. + +``CONFIG_TFLITEMICRO_SYSLOG`` + Route TFLM logging through NuttX syslog via ``tflm_syslog.cc``. + ``CONFIG_TFLITEMICRO_SYSLOG_LEVEL`` selects the syslog priority + (default ``6``, ``LOG_INFO``). See ``include/syslog.h``. + +``CONFIG_TFLITEMICRO_TOOL`` + Build the ``tflm`` NSH command from ``tflm_tool.cc``. + ``CONFIG_TFLITEMICRO_TOOL_PRIORITY`` and + ``CONFIG_TFLITEMICRO_TOOL_STACKSIZE`` set the task attributes + (defaults: priority 100, stack 4096). + +``CONFIG_TFLITEMICRO_HELLOWORLD`` + Build the ``tflm_hello`` example (CMake builds only). The example runs + the upstream TFLM hello-world float and INT8 sine models. + ``CONFIG_TFLITEMICRO_HELLOWORLD_PRIORITY`` and + ``CONFIG_TFLITEMICRO_HELLOWORLD_STACKSIZE`` set the task attributes + (defaults: priority 100, stack 4096). + +Building +======== + +The ``sim:tflm`` defconfig enables TFLM, the ``tflm`` tool, the hello-world +example, libc++, FlatBuffers, gemmlowp, KissFFT, and Ruy. + +Makefile build (produces the ``tflm`` tool; hello-world is CMake-only): + +.. code-block:: console + + $ cd nuttx + $ make distclean + $ ./tools/configure.sh sim:tflm + $ make -j$(nproc) + $ ./nuttx + +CMake build (produces both ``tflm`` and ``tflm_hello``): + +.. code-block:: console + + $ cd nuttx + $ cmake -B build -DBOARD_CONFIG=sim:tflm -GNinja + $ cmake --build build + $ ./build/nuttx + +The first build downloads the TFLM sources. Subsequent builds reuse the +unpacked tree under ``apps/mlearning/tflite-micro/tflite-micro``. +``make distclean`` in the apps tree removes that snapshot. + +Using the ``tflm`` tool +======================= + +``tflm`` loads a ``.tflite`` file from the filesystem, constructs a +``tflite::MicroInterpreter``, and can invoke the model once for profiling +or emit compiled C++ (when TFLM was built with ``TFLITE_MODEL_COMPILER``). + +.. code-block:: console + + nsh> tflm -h + + Utility to use tflite micro on nuttx. + [ -C ] Compile tflite model into c++ codes. + [ -E ] Do once evaluation (for profiling). + [ -i ] Readable model file path. + [ -o ] Writable c++ file path. + [ -p ] Prefix of compiled code. + [ -a ] Arena size (mempool). + [ -h ] Print this message. + +Both ``-i`` and ``-o`` are required. Defaults are prefix ``NXAI`` and +arena size 8192 bytes. + +The built-in operator resolver registers eight INT8-oriented ops: + +- ``CONV_2D`` (INT8) +- ``MAX_POOL_2D`` (INT8) +- ``QUANTIZE`` (float32 to INT8) +- ``DEQUANTIZE`` (INT8) +- ``MEAN`` (INT8) +- ``RESHAPE`` +- ``FULLY_CONNECTED`` (INT8) +- ``SOFTMAX`` (INT8) + +Models that need other operators must change the resolver in +``apps/mlearning/tflite-micro/tflm_tool.cc``. + +Hello-world example +=================== + +With a CMake ``sim:tflm`` image: + +.. code-block:: console + + nsh> tflm_hello + +This runs the upstream hello-world test: it profiles memory and latency, +then loads the float and INT8 sine models that are converted to C arrays +at build time with ``xxd``. + +Embedding a model in an application +=================================== + +TFLM is designed for targets without a filesystem and without dynamic +allocation for the model itself. Typical NuttX applications compile the +``.tflite`` file into a C array and pass it to ``tflite::GetModel()``. + +The CMake helper ``tflite_generate_data()`` in +``apps/mlearning/tflite-micro/CMakeLists.txt`` wraps ``xxd -i`` for that +purpose. The same conversion can be done manually:: + + xxd -i model.tflite model_data.h + +Provide a tensor arena (the ``-a`` size in ``tflm``, or a static buffer +in application code) large enough for the model's scratch tensors. +Insufficient arena size makes ``MicroInterpreter::AllocateTensors()`` +fail. + +Patches applied by NuttX +======================== + +- ``tflite-micro.patch`` — guard the ``ASSERT`` macro in the TFLM signal + circular buffer. +- ``0001-dequantize-int8.patch`` — ``Register_DEQUANTIZE_INT8()``. +- ``0002-quantize-int8.patch`` — ``Register_QUANTIZE_FLOAT32_INT8()``. +- ``0003-mean-int8.patch`` — ``Register_MEAN_INT8()``. +- ``0004-tflite-add-extern-C-to-main-function-to-avoid-c-mang.patch`` — + ``extern "C"`` on the hello-world ``main`` so NuttX can call it + (applied by the CMake fetch). + +See also +======== + +- Upstream TFLM: https://github.com/tensorflow/tflite-micro +- Hello-world example: + https://github.com/tensorflow/tflite-micro/blob/main/tensorflow/lite/micro/examples/hello_world/README.md +- :doc:`../cmsis-nn/index` for ARM kernel acceleration diff --git a/Documentation/platforms/sim/sim/boards/sim/index.rst b/Documentation/platforms/sim/sim/boards/sim/index.rst index f7f7293193d5b..9e53a777565a7 100644 --- a/Documentation/platforms/sim/sim/boards/sim/index.rst +++ b/Documentation/platforms/sim/sim/boards/sim/index.rst @@ -1446,6 +1446,31 @@ To use IPv4, modify these settings in the defconfig file: -CONFIG_NET_IPv6=y -CONFIG_NET_IPv6_NCONF_ENTRIES=4 +tflm +---- + +Builds TensorFlow Lite for Microcontrollers from +``apps/mlearning/tflite-micro``, including the ``tflm`` command-line tool. +A CMake build of this configuration also includes the ``tflm_hello`` +example. + +.. code-block:: console + + $ ./tools/configure.sh sim:tflm + $ make -j$(nproc) + $ ./nuttx + +With CMake:: + + $ cmake -B build -DBOARD_CONFIG=sim:tflm -GNinja + $ cmake --build build + $ ./build/nuttx + +From NSH, ``tflm -h`` prints the tool usage. ``tflm_hello`` (CMake image) +runs the upstream sine-model test. + +See :doc:`/applications/mlearning/tflite-micro/index`. + touchscreen -----------