ARM’s Zach Shelby introduced the use of microcontrollers for machine learning and artificial intelligence at the ECF19 event in Helsinki on last Friday. The talk showed that that artificial intelligence and machine learning can be applied to small embedded devices in addition to the cloud-based model. In particular, artificial intelligence is well suited to the devices of the Internet of Things. The use of machine learning in IoT is also sensible from an energy efficiency point of view if unnecessary power-consuming communication can be avoided (for example local keyword detection before sending voice data to cloud more more detailed analysis).
According to Shelby , we are now moving to a third wave of IoT that comes with comprehensive equipment security and voice control. In this model, machine learning techniques are one new application that can be added to previous work done on IoT.
In order to successfully use machine learning in small embedded devices, the problem to be solved is that it has reasonably little incoming information and a very limited number of possible outcomes. ARM Cortex M4 processor equipped with a DSP unit is powerful enough for simple hand writing decoding or detecting few spoken words with machine learning model. In examples the used machine learning models needed less than 100 kilobytes of memory.
The presentation can be now viewed on YouTube:
Important tools and projects mentioned on the presentation:
uTensor (ARM MicroTensor)
Articles on presentation:
https://www.uusiteknologia.fi/2019/05/20/ecf19-koneoppiminen-mahtuu-mikro-ohjaimeen/
http://www.etn.fi/index.php/72-ecf/9495-koneoppiminen-mullistaa-sulautetun-tekniikan
406 Comments
Tomi Engdahl says:
Merging TensorFlow Lite and μTensor
A new inference engine for micro-controllers?
https://blog.hackster.io/merging-tensorflow-lite-and-%CE%BCtensor-c7abfa38208f
In a joint announcement today by the TensorFlow Lite team at Google and the microTensor team at Arm, came the news that the two major inference engine platforms for micro-controllers will be joining forces.
Tomi Engdahl says:
Embedded Machine Learning on 8-Bit Microcontrollers, Including Arduino
https://blog.hackster.io/embedded-machine-learning-on-8-bit-microcontrollers-including-arduino-783155e7a135
Tomi Engdahl says:
https://phys.org/news/2019-06-machine-sensors.html
Tomi Engdahl says:
TensorFlow Lite Ported to Arduino
Adafruit ports TensorFlow for Micro-controllers to Arduino IDE!
https://blog.hackster.io/tensorflow-lite-ported-to-arduino-5e851c094ddc
Tomi Engdahl says:
TensorFlow Lite Ported to Arduino @HacksterIO by @aallan @TensorFlow @arduino #MachineLearning #EdgeComputing #Arduino #IoT
https://blog.adafruit.com/2019/07/03/tensorflow-lite-ported-to-arduino-hacksterio-by-aallan-tensorflow-arduino-machinelearning-edgecomputing-arduino-iot/
Tomi Engdahl says:
Playing rock-paper-scissors slightly better than random, using a recurrent neural network running on an 8-bit MCU.
Neural Network Device ‘Solves’ Rock-Paper-Scissors
https://blog.hackster.io/neural-network-device-solves-rock-paper-scissors-e37cabafd69
To accomplish this AI feat, Klinger uses a three-layer recurrent neural network (RNN) trained by looking at over 80,000 games played on roshambo.me. This runs on a Microchip ATtiny1614 microcontroller, with a 3D-printed case, CR2032 battery, and other electronics. A button starts the round, and inputs are available for rock, paper, and scissors, depending on the human selection. Three LEDs correspond to the computer’s move
https://github.com/PaulKlinger/rps-rnn
Tomi Engdahl says:
Want to get started with machine learning on MCUs? This example demonstrates a full end-to-end workflow of training a model, converting it for use with TensorFlow Lite, and deploying it to an Arduino.
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/experimental/micro/examples/hello_world#deploy-to-arduino
Tomi Engdahl says:
https://spectrum.ieee.org/geek-life/hands-on/machine-learning-thats-light-enough-for-an-arduino
Tomi Engdahl says:
MACHINE LEARNING MONDAY! TinyML comes to Circuit Playground Bluefruit
https://blog.adafruit.com/2019/08/19/machine-learning-monday-tinyml-comes-to-circuit-playground-bluefruit-tensorflow-tensorflow-tensorflowlite-tinyml-nordictweets-nrf52840-nordicsemi-arduino/
Circuit Playground Bluefruit features the nRF52840 Cortex M4 process, and has all those sensors you love from the previous Circuit Playgrounds: light, temperature, touch, and…sound!
Tomi Engdahl says:
TensorFlow on the Nordic nRF52840
Running TensorFlow Lite for Micro-controllers on the nRF52840
https://blog.hackster.io/tensorflow-on-the-nordic-nrf52840-ed9f03326a23
Tomi Engdahl says:
Building Brains on the Edge
Running TensorFlow Lite models on micro-controllers
https://blog.hackster.io/building-brains-on-the-edge-245ce1153eb1
Tomi Engdahl says:
Embedded ML for All Developers
https://blog.hackster.io/embedded-ml-for-all-developers-1f000ccdaddd
Over the next decade, embedded is going to experiencing the kind of innovation we haven’t seen since the late 2000s when open wireless, protocols and cryptography (and as a result, 32-bit MCUs) were introduced. Today most people think about Machine Learning as highly complex, large, and extremely memory and compute hungry — with clusters of GPUs/TPUs heating whole towns…
Tomi Engdahl says:
https://blog.hackster.io/deep-learning-at-the-edge-on-an-arm-cortex-powered-camera-board-3ca16eb60ef7
Tomi Engdahl says:
Blisteringly Fast Machine Learning On An Arduino Uno
https://hackaday.com/2019/06/30/blisteringly-fast-machine-learning-on-an-arduino-uno/
Tomi Engdahl says:
Speech recognition on Arduino is just one TensorFlow Lite Micro example in the Library Manager! Install a neural network on your Nano 33 BLE Sense to recognize simple voice commands: arduino.cc/ai
Tomi Engdahl says:
How-to Get Started with Machine Learning on Arduino
https://medium.com/tensorflow/how-to-get-started-with-machine-learning-on-arduino-7daf95b4157
Tomi Engdahl says:
https://blog.arduino.cc/2019/10/15/get-started-with-machine-learning-on-arduino/
Tomi Engdahl says:
Here’s a quick end-to-end demo of machine learning running on a Nano 33 BLE Sense! The same framework can be used to sample different sensors and train more complex models.
https://blog.arduino.cc/2019/11/07/fruit-identification-using-arduino-and-tensorflow/
Tomi Engdahl says:
The resources around TinyML are still emerging but there’s a great opportunity to get a head start and meet experts coming up December 2nd-3rd in Mountain View, California at the Arm AIoT Dev Summit. This includes workshops from Sandeep Mistry, Arduino technical lead for on-device ML, along with Google’s Pete Warden and Daniel Situnayake, who literally wrote the book on TinyML: http://bit.ly/2NNGGVO
Fruit identification using Arduino and TensorFlow
https://blog.arduino.cc/2019/11/07/fruit-identification-using-arduino-and-tensorflow/
Tomi Engdahl says:
“However perhaps the most interesting thing about the software on the Bangle.js is that there is TensorFlow Lite for Micro-controllers built into the firmware, so you can run machine learning models on your wrist.”
The Bangle.js, an Open Sourced JavaScript-Powered Smart Watch
https://www.hackster.io/news/the-bangle-js-an-open-sourced-javascript-powered-smart-watch-6359962646c2
It may be built using off-the shelf hardware, but the firmware of this JavaScript-powered smart watch is entirely open source and hackable.
Tomi Engdahl says:
Antmicro has published demonstrations of TensorFlow Lite running on a RISC-V soft-core processor via the Zephyr Project RTOS — a port it is calling TF Lite micro — though warns that additional work needs to be done before its efforts will be reflected upstream.
Antmicro Releases TensorFlow Lite on RISC-V Demos Using Zephyr, LiteX, VexRiscV: TF Lite Micro
https://www.hackster.io/news/antmicro-releases-tensorflow-lite-on-risc-v-demos-using-zephyr-litex-vexriscv-tf-lite-micro-cde3123d5700
Two demos developed ahead of the RISC-V Summit taking place this week.
Tomi Engdahl says:
https://eloquentarduino.github.io/2019/12/how-to-do-gesture-identification-on-arduino/
In this Arduno Machine learning project we’re going to use an accelerometer sensor to identify the gestures you play.
This is a remake of the project found on the Tensorflow blog. We’re going to use a lot less powerful chip in this tutorial, tough: an Arduino Nano (old generation), equipped with 32 kb of flash and only 2 kb of RAM.
On my machine, the sketch targeted at the Arduino Nano (old generation) requires 25310 bytes (82%) of program space and 1228 bytes (59%) of RAM. This means you could actually run machine learning in even less space than what the Arduino Nano provides.
How-to Get Started with Machine Learning on Arduino
https://blog.tensorflow.org/2019/11/how-to-get-started-with-machine.html?m=1
Tomi Engdahl says:
Tiny Machine Learning On The Attiny85
https://hackaday.com/2020/01/07/tiny-machine-learning-on-the-attiny85/
We tend to think that the lowest point of entry for machine learning (ML) is on a Raspberry Pi, which it definitely is not. [EloquentArduino] has been pushing the limits to the low end of the scale, and managed to get a basic classification model running on the ATtiny85.
Using his experience of running ML models on an old Arduino Nano, he had created a generator that can export C code from a scikit-learn. He tried using this generator to compile a support-vector colour classifier for the ATtiny85, but ran into a problem with the Arduino ATtiny85 compiler not supporting a variadic function used by the generator.
Embedded Machine learning on Attiny85
https://eloquentarduino.github.io/2019/12/machine-learning-on-attiny85/
You won’t believe it, but you can run Machine learning on embedded systems like an Attiny85 (and many others Attiny)!
Tomi Engdahl says:
Train a neural network using TensorFlow and run it on Avnet’s Azure Sphere Starter Kit and an ESP32.
How to Build a Neural Network in Microcontrollers
https://www.hackster.io/Robertodcrsj/how-to-build-a-neural-network-in-microcontrollers-1236a7
In this project we can see how to train a neural network using TensorFlow and implement it in Avnet Azure Sphere MT3620 and ESP32.
Tomi Engdahl says:
How to Do Gesture Identification Through Machine Learning on Arduino
https://www.hackster.io/news/how-to-do-gesture-identification-through-machine-learning-on-arduino-39c9f5f55092
Classify gestures from accelerometer data directly on your Arduino with the power of machine learning.
In this Arduino machine learning project, we’re going to use an accelerometer sensor to identify the gestures you play. This is a remake of the project found on the TensorFlow blog. We’re going to use a lot less powerful chip in this tutorial, though: an older Arduino Nano, equipped with 32KB of flash and only 2KB of RAM.
How-to Get Started with Machine Learning on Arduino
https://blog.tensorflow.org/2019/11/how-to-get-started-with-machine.html?m=1
Tomi Engdahl says:
Perform WiFi indoor position with any Espressif Systems ESP8266/ESP32 board and machine learning in the Arduino environment.
Indoor Positioning Using Arduino and Machine Learning in 4 Easy Steps
https://www.hackster.io/news/indoor-positioning-using-arduino-and-machine-learning-in-4-easy-steps-295d39e5e7c9
Learn how to do WiFi indoor position with any ESP8266/ESP32 board and machine learning in the Arduino environment.
In this Arduino machine learning project, we’re going to use the nearby WiFi access points to locate where we are. For this project to work, you will need a WiFi-equipped board, such as ESP8266, ESP32, or MKR WiFI 1010.
In this project, we’re going to use WiFi indoor positioning to detect in which room of our house/office we are. This is the most basic task we can accomplish and will get us a feeling of level of accuracy that we can achieve with such a simple setup
So, how exactly does WiFi indoor positioning works in conjunction with machine learning?
As you can see, there are two markers on the map — each of these markers will “see” different networks, with different signal strengths (a.k.a RSSI).
The features for this project are going to be the RSSIs (received signal strength indication) of the known WiFi networks. If a network is out of range, it will have an RSSI equal to 0.
First of all, we need to enumerate all the networks we will encounter during the inference process.
Now that we have a bunch of SSIDs, we need to assign each SSID to a fixed index, from 0 to MAX_NETWORKS.
The second step is to convert the scan results into a features vector. Each feature will be the RSSI of the given SSID, in the exact order we populated the knownNetworks array.
Grab some recordings just staying in a location for a few seconds and save the serial output to a file, then move to the next location and repeat — 10-15 samples for each location will suffice.
If you do a good job, you should end with distinguible features
if you’re getting poor results you may filter out those low values.
Train and Export the SVM Classifier
you have to copy the printed code and import it in your Arduino project, in a file called model.h
Run the Inference
Move around your house/office/whatever and see your location printed on the serial monitor!
Tomi Engdahl says:
Edge Impulse launches TinyML as a service to enable machine learning for all embedded developers with open source device SDKs.
TinyML for All Developers with Edge Impulse
https://www.hackster.io/news/tinyml-for-all-developers-with-edge-impulse-2cfbbcc14b90
Edge Impulse launches TinyML as a service to enable machine learning for all embedded developers with open source device SDKs.
Edge Impulse enables the easy collection of real sensor data, live signal processing from raw data to neural networks, testing and deployment to any target device. Sign up for a free developer account
https://www.edgeimpulse.com/
Tomi Engdahl says:
https://hackaday.com/2019/11/29/simplified-ai-on-microcontrollers/
Tomi Engdahl says:
Easy TinyML on ESP32 and Arduino
https://www.hackster.io/news/easy-tinyml-on-esp32-and-arduino-a9dbc509f26c
The easiest way to deploy TensorFlow Lite models onto your ESP32 with just two lines of code.
As a first step, I downloaded the free chapters from the TinyML book website and rapidly skimmed through them.
Once done reading the six chapters, I wanted to try the described tutorial on my ESP32. Sadly, it is not mentioned in the supported boards on the book, so I had to solve it by myself.
In this post I’m going to make a sort of recap of my learnings about the steps you need to follow to implement TF models to a microcontroller and introduce you to a tiny library I wrote for the purpose of facilitating the deployment in the Arduino IDE: EloquentTinyML.
https://github.com/eloquentarduino/EloquentTinyML
This Arduino library is here to simplify the deployment of Tensorflow Lite for Microcontrollers models to Arduino boards using the Arduino IDE.
Including all the required files for you, the library exposes an eloquent interface to load a model and run inferences.
Tomi Engdahl says:
Apple or Orange? Image Recognition with ESP32 and Arduino
How to apply an SVM classifier to an image classification task.
https://www.hackster.io/news/apple-or-orange-image-recognition-with-esp32-and-arduino-ff4f85459a9a
Do you have an ESP32 camera?
Want to do image recognition directly on your ESP32, without a PC?
In this post, we’ll look into a very basic image recognition task: distinguish apples from oranges with machine learning.
Image recognition is a very hot topic these days in the AI/ML landscape. Convolutional neural networks really shines in this task and can achieve almost perfect accuracy on many scenarios.
Sadly, you can’t run CNN on your ESP32, as they’re just too large for a microcontroller.
The objective of this post, instead, is to investigate if we can use the MicroML framework to do simple image recognition on the images from an ESP32 camera.
In this context, image recognition means deciding which class (from the trained ones) the current image belongs to. This algorithm can’t locate interesting objects in the image, neither detect if an object is present in the frame. It will classify the current image based on the samples recorded during training.
Even at the lowest resolution of 160×120 pixels, a raw RGB image from the camera would generate 160x120x3 = 57600 features: way too much.
We need to reduce this number to the bare minimum.
How many pixels do you think are necessary to get reasonable results in this task of classifying apples from oranges?
You would be surprised to know that I got 90% accuracy with an RGB image of 8×6!
Obviously, this is a tradeoff: you can’t expect to achieve 99% accuracy while mantaining the model size small enough to fit on a microcontroller. 90% is an acceptable accuracy for me in this context.
To train the classifier, save the features for each object in a file, one features vector per line. Then follow the steps on how to train a ML classifier for Arduino to get the exported model.
You can experiment with different classifier configurations.
My features were well distinguishable, so I had great results (100% accuracy) with any kernel (even linear).
This is not full-fledged object recognition: it can’t label objects while you walk as TensorFlow can do, for example.
You have to carefully craft your setup and be as consistent as possible between training and inferencing.
Still, I think this is a fun proof-of-concept that can have useful applications in simple scenarios where you can live with a fixed camera and don’t want to use a full Raspberry Pi.
In the upcoming weeks, I settled to finally try TensorFlow Lite for Microcontrollers on my ESP32
Tomi Engdahl says:
Easily deploy your TensorFlow Lite model to an ESP32 using the Arduino IDE without any compilation stuff.
Easy TinyML on ESP32 and Arduino
https://www.hackster.io/news/easy-tinyml-on-esp32-and-arduino-a9dbc509f26c
The easiest way to deploy TensorFlow Lite models onto your ESP32 with just two lines of code.
Tomi Engdahl says:
Classify words using a cheap microphone and an Arduino Nano, equipped with 32KB of flash and only 2 kb of RAM: https://bit.ly/2Hct17G
eloquentarduino.github.io
Tomi Engdahl says:
Indoor Positioning Using Arduino and Machine Learning in 4 Easy Steps
https://www.hackster.io/news/indoor-positioning-using-arduino-and-machine-learning-in-4-easy-steps-295d39e5e7c9
Learn how to do WiFi indoor position with an ESP8266, ESP32 or MKR WiFi 1010 board and machine learning in the Arduino environment.
Tomi Engdahl says:
Handwritten Digit Classification with Arduino and MicroML
How to recognize handwritten digits in real-time directly on an ESP32 camera.
https://www.hackster.io/news/handwritten-digit-classification-with-arduino-and-microml-4d7d4ffec6b7
Tomi Engdahl says:
Cartesiam helps developers bring AI to microcontrollers
https://tcrn.ch/2T2f1nS
Cartesiam, a startup that aims to bring machine learning to edge devices powered by microcontrollers, has launched a new tool for developers who want an easier way to build services for these devices. The new NanoEdge AI Studio is the first IDE specifically designed for enabling machine learning and inferencing on Arm Cortex-M microcontrollers, which power billions of devices already.
Tomi Engdahl says:
https://spectrum.ieee.org/tech-talk/semiconductors/processors/cartesiam-startup-news-embedded-ai-arm-microcontrollers
Tomi Engdahl says:
AI Sound Recognition on a Cortex-M0: Data is King
https://www.eetimes.com/ai-sound-recognition-on-a-cortex-m0-data-is-king/
Tomi Engdahl says:
Arduino Day 2020 Bandung online event:
Magic Wand using Machine Learning running on an ESP32 board. ML inference runs right on the ESP32 MCU.
Here’s the source code: https://github.com/andriyadi/MagicWand-TFLite-ESP32
It’s based on the example from Tensorflow Lite for Microcontroller. The challenge is how to make it work with ESP32 Arduino framework and PlatformIO. Need to make adjustments on tfmicro library.
For ESPectro32 development board.
Tomi Engdahl says:
The presentation slides: https://www.slideshare.net/andri_yadi/introduction-to-aiot-tinyml-with-arduino
Tomi Engdahl says:
Demo 50: Bring Tensorflow Lite to ESP32 Arduino – person detection application using deep learning with ESP32 CAM
http://www.iotsharing.com/2020/03/bring-tensorflow-lite-to-esp32-person-detection-deep-learning.html?m=1
Tomi Engdahl says:
In this Digi-Key Electronics tutorial, Shawn Hymel loads a TensorFlow Lite model onto a Nano 33 BLE Sense and uses it to run inference to generate an approximation of a sinewave.
Intro to TinyML Part 1: Training a Model for Arduino in TensorFlow
https://www.digikey.com/en/maker/projects/intro-to-tinyml-part-1-training-a-model-for-arduino-in-tensorflow/8f1fc8c0b83d417ab521c48864d2a8ec
Intro to TinyML Part 2: Deploying a TensorFlow Lite Model to Arduino
https://www.digikey.com/en/maker/projects/intro-to-tinyml-part-2-deploying-a-tensorflow-lite-model-to-arduino/59bf2d67256f4b40900a3fa670c14330
Tomi Engdahl says:
Intro to TinyML Part 2: Deploying a TensorFlow Lite Model to Arduino
https://www.digikey.com/en/maker/projects/intro-to-tinyml-part-2-deploying-a-tensorflow-lite-model-to-arduino/59bf2d67256f4b40900a3fa670c14330
Tomi Engdahl says:
Gesture Control of a Self-Balancing Robot Using TensorFlow © GPL3+
https://create.arduino.cc/projecthub/RolfK/gesture-control-of-a-self-balancing-robot-using-tensorflow-52037e
Gesture control of a self-balancing robot using deep learning with TensorFlow Lite.
a self-balancing robot.
The robot could be controlled by a joystick. In this project the robot is controlled by gestures. The gestures are recognized by a neural network. The neural network is based on TensorFlow Lite.
TensorFlow Lite is TensorFlow’s lightweight solution for mobile and embedded devices. It lets you run machine-learned models on mobile devices like Arduino.
My sketch was implemented on a Arduino Nano BLE Sense. The board has a built-in nine-axis IMU. This sensor module contains a 3-axis gyroscope, a 3-axis accelerometer, and a 3-axis magnetic sensor.
Tomi Engdahl says:
Edge Impulse launches full support for the Nano 33 BLE Sense and other 32-bit Arduino boards, making machine learning accessible to millions of Arduino developers.
Edge Impulse Brings TinyML to Millions of Arduino Developers
https://www.hackster.io/news/edge-impulse-brings-tinyml-to-millions-of-arduino-developers-91cec576dc99
Edge Impulse has launched full support for Arduino, making machine learning accessible to millions of Arduino developers
Tomi Engdahl says:
https://www.edgeimpulse.com/
Using Edge Impulse you can now quickly collect real-world sensor data, train ML models on this data in the cloud, and then deploy the model back to your Arduino device. From there you can integrate the model into your Arduino sketches with a single function call. Your sensors are then a whole lot smarter, being able to make sense of complex events in the real world. The built-in examples allow you to collect data from the accelerometer and the microphone, but it’s easy to integrate other sensors with a few lines of code.
Excited? This is how you build your first deep learning model with the Arduino Nano 33 BLE Sense (there’s also a video tutorial here):
https://docs.edgeimpulse.com/docs/arduino-nano-33-ble-sense
Tomi Engdahl says:
Use Edge Impulse machine learning on a Nano 33 BLE Sense to detect the presence of coughing in real-time audio.
Cough Detection with TinyML on Arduino © CC BY
https://create.arduino.cc/projecthub/edge-impulse/cough-detection-with-tinyml-on-arduino-417f37
In this tutorial we show how to build a cough detection system for the Arduino Nano BLE Sense using TinyML and Edge Impulse.
Tomi Engdahl says:
Sipeed MAix and Arduino IDE/Micropython | Image Recognition
https://www.youtube.com/watch?v=NA4JgWccu8s
Tomi Engdahl says:
TinyML is giving hardware new life
The latest embedded software technology moves hardware into an almost magical realm
https://techcrunch.com/2020/05/29/tinyml-is-giving-hardware-new-life/?tpcc=ECFB2020
Tomi Engdahl says:
No Hardware? No Problem: Simulating TensorFlow Lite Targets with Renode
https://www.hackster.io/news/no-hardware-no-problem-simulating-tensorflow-lite-targets-with-renode-8532d0cb4635
Renode 1.9 accelerates machine learning development at the edge by simulating physical hardware systems.
Continuous integration and delivery tools have experienced widespread adoption by traditional software developers thanks to tools like Travis CI, CircleCI, and Concourse. While solutions are being developed to bring this productivity-enhancing workflow to the hardware world, the ability to create a complete pipeline can be complicated by the need to incorporate actual hardware targets to execute and test code on. Renode, an open-source project that spun out of and is commercially supported by Antmicro, allows simulation of Arm and RISC-V hardware, bringing the rapid development and iteration that most software developers now take for granted to the world of IoT. When the TensorFlow Lite team themselves needed to accelerate development by automating testing of Arm and RISC-V platforms, Antmicro and Renode made this possible, without the need for physical hardware.
https://blog.tensorflow.org/2020/06/running-and-testing-tf-lite-on-microcontrollers.html?m=1
Tomi Engdahl says:
Perform gesture recognition using TensorFlow Lite on your M5Stack.
Gesture Recognition with M5Stack + TensorFlow Lite
https://www.hackster.io/boochowp/gesture-recognition-with-m5stack-tensorflow-lite-da212b