Machine learning possible on microcontrollers

ARM’s Zach Shelby introduced the use of microcontrollers for machine learning and artificial intelligence at the ECF19 event in Helsinki on last Friday. The talk showed that that artificial intelligence and machine learning can be applied to small embedded devices in addition to the cloud-based model. In particular, artificial intelligence is well suited to the devices of the Internet of Things. The use of machine learning in IoT is also sensible from an energy efficiency point of view if unnecessary power-consuming communication can be avoided (for example local keyword detection before sending voice data to cloud more more detailed analysis).

According to Shelby , we are now moving to a third wave of IoT that comes with comprehensive equipment security and voice control. In this model, machine learning techniques are one new application that can be added to previous work done on IoT.

In order to successfully use machine learning in small embedded devices, the problem to be solved is that it has reasonably little incoming information and a very limited number of possible outcomes. ARM Cortex M4 processor equipped with a DSP unit is powerful enough for simple hand writing decoding or detecting few spoken words with machine learning model. In examples the used machine learning models needed less than 100 kilobytes of memory.

zackdscf6473

The presentation can be now viewed on YouTube:

Important tools and projects mentioned on the presentation:

TinyML

TensorFlow Lite

uTensor (ARM MicroTensor)

TensorFlow Lite Micro

Articles on presentation:

https://www.uusiteknologia.fi/2019/05/20/ecf19-koneoppiminen-mahtuu-mikro-ohjaimeen/

http://www.etn.fi/index.php/72-ecf/9495-koneoppiminen-mullistaa-sulautetun-tekniikan

 

33 Comments

  1. Tomi Engdahl says:

    Merging TensorFlow Lite and μTensor
    A new inference engine for micro-controllers?
    https://blog.hackster.io/merging-tensorflow-lite-and-%CE%BCtensor-c7abfa38208f

    In a joint announcement today by the TensorFlow Lite team at Google and the microTensor team at Arm, came the news that the two major inference engine platforms for micro-controllers will be joining forces.

    Reply
  2. Tomi Engdahl says:

    TensorFlow Lite Ported to Arduino
    Adafruit ports TensorFlow for Micro-controllers to Arduino IDE!
    https://blog.hackster.io/tensorflow-lite-ported-to-arduino-5e851c094ddc

    Reply
  3. Tomi Engdahl says:

    TensorFlow Lite Ported to Arduino @HacksterIO by @aallan @TensorFlow @arduino #MachineLearning #EdgeComputing #Arduino #IoT
    https://blog.adafruit.com/2019/07/03/tensorflow-lite-ported-to-arduino-hacksterio-by-aallan-tensorflow-arduino-machinelearning-edgecomputing-arduino-iot/

    Reply
  4. Tomi Engdahl says:

    Playing rock-paper-scissors slightly better than random, using a recurrent neural network running on an 8-bit MCU.

    Neural Network Device ‘Solves’ Rock-Paper-Scissors
    https://blog.hackster.io/neural-network-device-solves-rock-paper-scissors-e37cabafd69

    To accomplish this AI feat, Klinger uses a three-layer recurrent neural network (RNN) trained by looking at over 80,000 games played on roshambo.me. This runs on a Microchip ATtiny1614 microcontroller, with a 3D-printed case, CR2032 battery, and other electronics. A button starts the round, and inputs are available for rock, paper, and scissors, depending on the human selection. Three LEDs correspond to the computer’s move

    https://github.com/PaulKlinger/rps-rnn

    Reply
  5. Tomi Engdahl says:

    Want to get started with machine learning on MCUs? This example demonstrates a full end-to-end workflow of training a model, converting it for use with TensorFlow Lite, and deploying it to an Arduino.

    https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/experimental/micro/examples/hello_world#deploy-to-arduino

    Reply
  6. Tomi Engdahl says:

    MACHINE LEARNING MONDAY! TinyML comes to Circuit Playground Bluefruit
    https://blog.adafruit.com/2019/08/19/machine-learning-monday-tinyml-comes-to-circuit-playground-bluefruit-tensorflow-tensorflow-tensorflowlite-tinyml-nordictweets-nrf52840-nordicsemi-arduino/

    Circuit Playground Bluefruit features the nRF52840 Cortex M4 process, and has all those sensors you love from the previous Circuit Playgrounds: light, temperature, touch, and…sound!

    Reply
  7. Tomi Engdahl says:

    TensorFlow on the Nordic nRF52840
    Running TensorFlow Lite for Micro-controllers on the nRF52840
    https://blog.hackster.io/tensorflow-on-the-nordic-nrf52840-ed9f03326a23

    Reply
  8. Tomi Engdahl says:

    Building Brains on the Edge
    Running TensorFlow Lite models on micro-controllers
    https://blog.hackster.io/building-brains-on-the-edge-245ce1153eb1

    Reply
  9. Tomi Engdahl says:

    Embedded ML for All Developers
    https://blog.hackster.io/embedded-ml-for-all-developers-1f000ccdaddd

    Over the next decade, embedded is going to experiencing the kind of innovation we haven’t seen since the late 2000s when open wireless, protocols and cryptography (and as a result, 32-bit MCUs) were introduced. Today most people think about Machine Learning as highly complex, large, and extremely memory and compute hungry — with clusters of GPUs/TPUs heating whole towns…

    Reply
  10. Tomi Engdahl says:

    Speech recognition on Arduino is just one TensorFlow Lite Micro example in the Library Manager! Install a neural network on your Nano 33 BLE Sense to recognize simple voice commands: arduino.cc/ai

    Reply
  11. Tomi Engdahl says:

    Here’s a quick end-to-end demo of machine learning running on a Nano 33 BLE Sense! The same framework can be used to sample different sensors and train more complex models.

    https://blog.arduino.cc/2019/11/07/fruit-identification-using-arduino-and-tensorflow/

    Reply
  12. Tomi Engdahl says:

    The resources around TinyML are still emerging but there’s a great opportunity to get a head start and meet experts coming up December 2nd-3rd in Mountain View, California at the Arm AIoT Dev Summit. This includes workshops from Sandeep Mistry, Arduino technical lead for on-device ML, along with Google’s Pete Warden and Daniel Situnayake, who literally wrote the book on TinyML: http://bit.ly/2NNGGVO

    Fruit identification using Arduino and TensorFlow
    https://blog.arduino.cc/2019/11/07/fruit-identification-using-arduino-and-tensorflow/

    Reply
  13. Tomi Engdahl says:

    “However perhaps the most interesting thing about the software on the Bangle.js is that there is TensorFlow Lite for Micro-controllers built into the firmware, so you can run machine learning models on your wrist.”

    The Bangle.js, an Open Sourced JavaScript-Powered Smart Watch
    https://www.hackster.io/news/the-bangle-js-an-open-sourced-javascript-powered-smart-watch-6359962646c2

    It may be built using off-the shelf hardware, but the firmware of this JavaScript-powered smart watch is entirely open source and hackable.

    Reply
  14. Tomi Engdahl says:

    Antmicro has published demonstrations of TensorFlow Lite running on a RISC-V soft-core processor via the Zephyr Project RTOS — a port it is calling TF Lite micro — though warns that additional work needs to be done before its efforts will be reflected upstream.

    Antmicro Releases TensorFlow Lite on RISC-V Demos Using Zephyr, LiteX, VexRiscV: TF Lite Micro

    https://www.hackster.io/news/antmicro-releases-tensorflow-lite-on-risc-v-demos-using-zephyr-litex-vexriscv-tf-lite-micro-cde3123d5700

    Two demos developed ahead of the RISC-V Summit taking place this week.

    Reply
  15. Tomi Engdahl says:

    https://eloquentarduino.github.io/2019/12/how-to-do-gesture-identification-on-arduino/

    In this Arduno Machine learning project we’re going to use an accelerometer sensor to identify the gestures you play.
    This is a remake of the project found on the Tensorflow blog. We’re going to use a lot less powerful chip in this tutorial, tough: an Arduino Nano (old generation), equipped with 32 kb of flash and only 2 kb of RAM.

    On my machine, the sketch targeted at the Arduino Nano (old generation) requires 25310 bytes (82%) of program space and 1228 bytes (59%) of RAM. This means you could actually run machine learning in even less space than what the Arduino Nano provides.

    How-to Get Started with Machine Learning on Arduino
    https://blog.tensorflow.org/2019/11/how-to-get-started-with-machine.html?m=1

    Reply
  16. Tomi Engdahl says:

    Tiny Machine Learning On The Attiny85
    https://hackaday.com/2020/01/07/tiny-machine-learning-on-the-attiny85/

    We tend to think that the lowest point of entry for machine learning (ML) is on a Raspberry Pi, which it definitely is not. [EloquentArduino] has been pushing the limits to the low end of the scale, and managed to get a basic classification model running on the ATtiny85.

    Using his experience of running ML models on an old Arduino Nano, he had created a generator that can export C code from a scikit-learn. He tried using this generator to compile a support-vector colour classifier for the ATtiny85, but ran into a problem with the Arduino ATtiny85 compiler not supporting a variadic function used by the generator.

    Embedded Machine learning on Attiny85
    https://eloquentarduino.github.io/2019/12/machine-learning-on-attiny85/

    You won’t believe it, but you can run Machine learning on embedded systems like an Attiny85 (and many others Attiny)!

    Reply
  17. Tomi Engdahl says:

    Train a neural network using TensorFlow and run it on Avnet’s Azure Sphere Starter Kit and an ESP32.

    How to Build a Neural Network in Microcontrollers
    https://www.hackster.io/Robertodcrsj/how-to-build-a-neural-network-in-microcontrollers-1236a7

    In this project we can see how to train a neural network using TensorFlow and implement it in Avnet Azure Sphere MT3620 and ESP32.

    Reply
  18. Tomi Engdahl says:

    How to Do Gesture Identification Through Machine Learning on Arduino
    https://www.hackster.io/news/how-to-do-gesture-identification-through-machine-learning-on-arduino-39c9f5f55092

    Classify gestures from accelerometer data directly on your Arduino with the power of machine learning.

    In this Arduino machine learning project, we’re going to use an accelerometer sensor to identify the gestures you play. This is a remake of the project found on the TensorFlow blog. We’re going to use a lot less powerful chip in this tutorial, though: an older Arduino Nano, equipped with 32KB of flash and only 2KB of RAM.

    How-to Get Started with Machine Learning on Arduino
    https://blog.tensorflow.org/2019/11/how-to-get-started-with-machine.html?m=1

    Reply
  19. Tomi Engdahl says:

    Perform WiFi indoor position with any Espressif Systems ESP8266/ESP32 board and machine learning in the Arduino environment.

    Indoor Positioning Using Arduino and Machine Learning in 4 Easy Steps
    https://www.hackster.io/news/indoor-positioning-using-arduino-and-machine-learning-in-4-easy-steps-295d39e5e7c9

    Learn how to do WiFi indoor position with any ESP8266/ESP32 board and machine learning in the Arduino environment.

    In this Arduino machine learning project, we’re going to use the nearby WiFi access points to locate where we are. For this project to work, you will need a WiFi-equipped board, such as ESP8266, ESP32, or MKR WiFI 1010.

    In this project, we’re going to use WiFi indoor positioning to detect in which room of our house/office we are. This is the most basic task we can accomplish and will get us a feeling of level of accuracy that we can achieve with such a simple setup

    So, how exactly does WiFi indoor positioning works in conjunction with machine learning?

    As you can see, there are two markers on the map — each of these markers will “see” different networks, with different signal strengths (a.k.a RSSI).

    The features for this project are going to be the RSSIs (received signal strength indication) of the known WiFi networks. If a network is out of range, it will have an RSSI equal to 0.

    First of all, we need to enumerate all the networks we will encounter during the inference process.

    Now that we have a bunch of SSIDs, we need to assign each SSID to a fixed index, from 0 to MAX_NETWORKS.

    The second step is to convert the scan results into a features vector. Each feature will be the RSSI of the given SSID, in the exact order we populated the knownNetworks array.

    Grab some recordings just staying in a location for a few seconds and save the serial output to a file, then move to the next location and repeat — 10-15 samples for each location will suffice.

    If you do a good job, you should end with distinguible features

    if you’re getting poor results you may filter out those low values.

    Train and Export the SVM Classifier

    you have to copy the printed code and import it in your Arduino project, in a file called model.h

    Run the Inference

    Move around your house/office/whatever and see your location printed on the serial monitor!

    Reply
  20. Tomi Engdahl says:

    Edge Impulse launches TinyML as a service to enable machine learning for all embedded developers with open source device SDKs.

    TinyML for All Developers with Edge Impulse
    https://www.hackster.io/news/tinyml-for-all-developers-with-edge-impulse-2cfbbcc14b90

    Edge Impulse launches TinyML as a service to enable machine learning for all embedded developers with open source device SDKs.

    Edge Impulse enables the easy collection of real sensor data, live signal processing from raw data to neural networks, testing and deployment to any target device. Sign up for a free developer account

    https://www.edgeimpulse.com/

    Reply
  21. Tomi Engdahl says:

    Easy TinyML on ESP32 and Arduino
    https://www.hackster.io/news/easy-tinyml-on-esp32-and-arduino-a9dbc509f26c

    The easiest way to deploy TensorFlow Lite models onto your ESP32 with just two lines of code.

    As a first step, I downloaded the free chapters from the TinyML book website and rapidly skimmed through them.

    Once done reading the six chapters, I wanted to try the described tutorial on my ESP32. Sadly, it is not mentioned in the supported boards on the book, so I had to solve it by myself.

    In this post I’m going to make a sort of recap of my learnings about the steps you need to follow to implement TF models to a microcontroller and introduce you to a tiny library I wrote for the purpose of facilitating the deployment in the Arduino IDE: EloquentTinyML.

    https://github.com/eloquentarduino/EloquentTinyML

    This Arduino library is here to simplify the deployment of Tensorflow Lite for Microcontrollers models to Arduino boards using the Arduino IDE.

    Including all the required files for you, the library exposes an eloquent interface to load a model and run inferences.

    Reply
  22. Tomi Engdahl says:

    Apple or Orange? Image Recognition with ESP32 and Arduino
    How to apply an SVM classifier to an image classification task.
    https://www.hackster.io/news/apple-or-orange-image-recognition-with-esp32-and-arduino-ff4f85459a9a

    Do you have an ESP32 camera?

    Want to do image recognition directly on your ESP32, without a PC?

    In this post, we’ll look into a very basic image recognition task: distinguish apples from oranges with machine learning.

    Image recognition is a very hot topic these days in the AI/ML landscape. Convolutional neural networks really shines in this task and can achieve almost perfect accuracy on many scenarios.

    Sadly, you can’t run CNN on your ESP32, as they’re just too large for a microcontroller.

    The objective of this post, instead, is to investigate if we can use the MicroML framework to do simple image recognition on the images from an ESP32 camera.

    In this context, image recognition means deciding which class (from the trained ones) the current image belongs to. This algorithm can’t locate interesting objects in the image, neither detect if an object is present in the frame. It will classify the current image based on the samples recorded during training.

    Even at the lowest resolution of 160×120 pixels, a raw RGB image from the camera would generate 160x120x3 = 57600 features: way too much.

    We need to reduce this number to the bare minimum.

    How many pixels do you think are necessary to get reasonable results in this task of classifying apples from oranges?

    You would be surprised to know that I got 90% accuracy with an RGB image of 8×6!

    Obviously, this is a tradeoff: you can’t expect to achieve 99% accuracy while mantaining the model size small enough to fit on a microcontroller. 90% is an acceptable accuracy for me in this context.

    To train the classifier, save the features for each object in a file, one features vector per line. Then follow the steps on how to train a ML classifier for Arduino to get the exported model.

    You can experiment with different classifier configurations.

    My features were well distinguishable, so I had great results (100% accuracy) with any kernel (even linear).

    This is not full-fledged object recognition: it can’t label objects while you walk as TensorFlow can do, for example.

    You have to carefully craft your setup and be as consistent as possible between training and inferencing.

    Still, I think this is a fun proof-of-concept that can have useful applications in simple scenarios where you can live with a fixed camera and don’t want to use a full Raspberry Pi.

    In the upcoming weeks, I settled to finally try TensorFlow Lite for Microcontrollers on my ESP32

    Reply
  23. Tomi Engdahl says:

    Easily deploy your TensorFlow Lite model to an ESP32 using the Arduino IDE without any compilation stuff.

    Easy TinyML on ESP32 and Arduino
    https://www.hackster.io/news/easy-tinyml-on-esp32-and-arduino-a9dbc509f26c

    The easiest way to deploy TensorFlow Lite models onto your ESP32 with just two lines of code.

    Reply
  24. Tomi Engdahl says:

    Classify words using a cheap microphone and an Arduino Nano, equipped with 32KB of flash and only 2 kb of RAM: https://bit.ly/2Hct17G

    eloquentarduino.github.io

    Reply
  25. Tomi Engdahl says:

    Indoor Positioning Using Arduino and Machine Learning in 4 Easy Steps
    https://www.hackster.io/news/indoor-positioning-using-arduino-and-machine-learning-in-4-easy-steps-295d39e5e7c9

    Learn how to do WiFi indoor position with an ESP8266, ESP32 or MKR WiFi 1010 board and machine learning in the Arduino environment.

    Reply

Leave a Comment

Your email address will not be published. Required fields are marked *

*

*