New approaches for embedded development

The idea for this posting started when I read New approaches to dominate in embedded development article. Then I found some ther related articles and here is the result: long article.

Embedded devices, or embedded systems, are specialized computer systems that constitute components of larger electromechanical systems with which they interface. The advent of low-cost wireless connectivity is altering many things in embedded development: With a connection to the Internet, an embedded device can gain access to essentially unlimited processing power and memory in cloud service – and at the same time you need to worry about communication issues like breaks connections, latency and security issues.

Those issues are espcecially in the center of the development of popular Internet of Things device and adding connectivity to existing embedded systems. All this means that the whole nature of the embedded development effort is going to change. A new generation of programmers are already making more and more embedded systems. Rather than living and breathing C/C++, the new generation prefers more high-level, abstract languages (like Java, Python, JavaScript etc.). Instead of trying to craft each design to optimize for cost, code size, and performance, the new generation wants to create application code that is separate from an underlying platform that handles all the routine details. Memory is cheap, so code size is only a minor issue in many applications.

Historically, a typical embedded system has been designed as a control-dominated system using only a state-oriented model, such as FSMs. However, the trend in embedded systems design in recent years has been towards highly distributed architectures with support for concurrency, data and control flow, and scalable distributed computations. For example computer networks, modern industrial control systems, electronics in modern car,Internet of Things system fall to this category. This implies that a different approach is necessary.

Companies are also marketing to embedded developers in new ways. Ultra-low cost development boards to woo makers, hobbyists, students, and entrepreneurs on a shoestring budget to a processor architecture for prototyping and experimentation have already become common.If you look under the hood of any connected embedded consumer or mobile device, in addition to the OS you will find a variety of middleware applications. As hardware becomes powerful and cheap enough that the inefficiencies of platform-based products become moot. Leaders with Embedded systems development lifecycle management solutions speak out on new approaches available today in developing advanced products and systems.

Traditional approaches

C/C++

Tradionally embedded developers have been living and breathing C/C++. For a variety of reasons, the vast majority of embedded toolchains are designed to support C as the primary language. If you want to write embedded software for more than just a few hobbyist platforms, your going to need to learn C. Very many embedded systems operating systems, including Linux Kernel, are written using C language. C can be translated very easily and literally to assembly, which allows programmers to do low level things without the restrictions of assembly. When you need to optimize for cost, code size, and performance the typical choice of language is C. Still C is today used for maximum efficiency instead of C++.

C++ is very much alike C, with more features, and lots of good stuff, while not having many drawbacks, except fror it complexity. The had been for years suspicion C++ is somehow unsuitable for use in small embedded systems. At some time many 8- and 16-bit processors were lacking a C++ compiler, that may be a concern, but there are now 32-bit microcontrollers available for under a dollar supported by mature C++ compilers.Today C++ is used a lot more in embedded systems. There are many factors that may contribute to this, including more powerful processors, more challenging applications, and more familiarity with object-oriented languages.

And if you use suitable C++ subset for coding, you can make applications that work even on quite tiny processors, let the Arduino system be an example of that: You’re writing in C/C++, using a library of functions with a fairly consistent API. There is no “Arduino language” and your “.ino” files are three lines away from being standard C++.

Today C++ has not displaced C. Both of the languages are widely used, sometimes even within one system – for example in embedded Linux system that runs C++ application. When you write a C or C++ programs for modern Embedded Linux you typically use GCC compiler toolchain to do compilation and make file to manage compilation process.

Most organization put considerable focus on software quality, but software security is different. When the security is very much talked about topic todays embedded systems, the security of the programs written using C/C++ becomes sometimes a debated subject. Embedded development presents the challenge of coding in a language that’s inherently insecure; and quality assurance does little to ensure security. The truth is that majority of today’s Internet connected systems have their networking fuctionality written using C even of the actual application layer is written using some other methods.

Java

Java is a general-purpose computer programming language that is concurrent, class-based and object-oriented.The language derives much of its syntax from C and C++, but it has fewer low-level facilities than either of them. Java is intended to let application developers “write once, run anywhere” (WORA), meaning that compiled Java code can run on all platforms that support Java without the need for recompilation.Java applications are typically compiled to bytecode that can run on any Java virtual machine (JVM) regardless of computer architecture. Java is one of the most popular programming languages in use, particularly for client-server web applications. In addition to those it is widely used in mobile phones (Java apps in feature phones,) and some embedded applications. Some common examples include SIM cards, VOIP phones, Blu-ray Disc players, televisions, utility meters, healthcare gateways, industrial controls, and countless other devices.

Some experts point out that Java is still a viable option for IoT programming. Think of the industrial Internet as the merger of embedded software development and the enterprise. In that area, Java has a number of key advantages: first is skills – there are lots of Java developers out there, and that is an important factor when selecting technology. Second is maturity and stability – when you have devices which are going to be remotely managed and provisioned for a decade, Java’s stability and care about backwards compatibility become very important. Third is the scale of the Java ecosystem – thousands of companies already base their business on Java, ranging from Gemalto using JavaCard on their SIM cards to the largest of the enterprise software vendors.

Although in the past some differences existed between embedded Java and traditional PC based Java solutions, the only difference now is that embedded Java code in these embedded systems is mainly contained in constrained memory, such as flash memory. A complete convergence has taken place since 2010, and now Java software components running on large systems can run directly with no recompilation at all on design-to-cost mass-production devices (consumers, industrial, white goods, healthcare, metering, smart markets in general,…) Java for embedded devices (Java Embedded) is generally integrated by the device manufacturers. It is NOT available for download or installation by consumers. Originally Java was tightly controlled by Sun (now Oracle), but in 2007 Sun relicensed most of its Java technologies under the GNU General Public License. Others have also developed alternative implementations of these Sun technologies, such as the GNU Compiler for Java (bytecode compiler), GNU Classpath (standard libraries), and IcedTea-Web (browser plugin for applets).

My feelings with Java is that if your embedded systems platform supports Java and you know hot to code Java, then it could be a good tool. If your platform does not have ready Java support, adding it could be quite a bit of work.

 

Increasing trends

Databases

Embedded databases are coming more and more to the embedded devices. If you look under the hood of any connected embedded consumer or mobile device, in addition to the OS you will find a variety of middleware applications. One of the most important and most ubiquitous of these is the embedded database. An embedded database system is a database management system (DBMS) which is tightly integrated with an application software that requires access to stored data, such that the database system is “hidden” from the application’s end-user and requires little or no ongoing maintenance.

There are many possible databases. First choice is what kind of database you need. The main choices are SQL databases and simpler key-storage databases (also called NoSQL).

SQLite is the Database chosen by virtually all mobile operating systems. For example Android and iOS ship with SQLite. It is also built into for example Firefox web browser. It is also often used with PHP. So SQLite is probably a pretty safe bet if you need relational database for an embedded system that needs to support SQL commands and does not need to store huge amounts of data (no need to modify database with millions of lines of data).

If you do not need relational database and you need very high performance, you need probably to look somewhere else.Berkeley DB (BDB) is a software library intended to provide a high-performance embedded database for key/value data. Berkeley DB is written in Cwith API bindings for many languages. BDB stores arbitrary key/data pairs as byte arrays. There also many other key/value database systems.

RTA (Run Time Access) gives easy runtime access to your program’s internal structures, arrays, and linked-lists as tables in a database. When using RTA, your UI programs think they are talking to a PostgreSQL database (PostgreSQL bindings for C and PHP work, as does command line tool psql), but instead of normal database file you are actually accessing internals of your software.

Software quality

Building quality into embedded software doesn’t happen by accident. Quality must be built-in from the beginning. Software startup checklist gives quality a head start article is a checklist for embedded software developers to make sure they kick-off their embedded software implementation phase the right way, with quality in mind

Safety

Traditional methods for achieving safety properties mostly originate from hardware-dominated systems. Nowdays more and more functionality is built using software – including safety critical functions. Software-intensive embedded systems require new approaches for safety. Embedded Software Can Kill But Are We Designing Safely?

IEC, FDA, FAA, NHTSA, SAE, IEEE, MISRA, and other professional agencies and societies work to create safety standards for engineering design. But are we following them? A survey of embedded design practices leads to some disturbing inferences about safety.Barr Group’s recent annual Embedded Systems Safety & Security Survey indicate that we all need to be concerned: Only 67 percent are designing to relevant safety standards, while 22 percent stated that they are not—and 11 percent did not even know if they were designing to a standard or not.

If you were the user of a safety-critical embedded device and learned that the designers had not followed best practices and safety standards in the design of the device, how worried would you be? I know I would be anxious, and quite frankly. This is quite disturbing.

Security

The advent of low-cost wireless connectivity is altering many things in embedded development – it has added to your list of worries need to worry about communication issues like breaks connections, latency and security issues. Understanding security is one thing; applying that understanding in a complete and consistent fashion to meet security goals is quite another. Embedded development presents the challenge of coding in a language that’s inherently insecure; and quality assurance does little to ensure security.

Developing Secure Embedded Software white paper  explains why some commonly used approaches to security typically fail:

MISCONCEPTION 1: SECURITY BY OBSCURITY IS A VALID STRATEGY
MISCONCEPTION 2: SECURITY FEATURES EQUAL SECURE SOFTWARE
MISCONCEPTION 3: RELIABILITY AND SAFETY EQUAL SECURITY
MISCONCEPTION 4: DEFENSIVE PROGRAMMING GUARANTEES SECURITY

Many organizations are only now becoming aware of the need to incorporate security into their software development lifecycle.

Some techniques for building security to embedded systems:

Use secure communications protocols and use VPN to secure communications
The use of Public Key Infrastructure (PKI) for boot-time and code authentication
Establishing a “chain of trust”
Process separation to partition critical code and memory spaces
Leveraging safety-certified code
Hardware enforced system partitioning with a trusted execution environment
Plan the system so that it can be easily and safely upgraded when needed

Flood of new languages

Rather than living and breathing C/C++, the new generation prefers more high-level, abstract languages (like Java, Python, JavaScript etc.). So there is a huge push to use interpreted and scripting also in embedded systems. Increased hardware performance on embedded devices combined with embedded Linux has made the use of many scripting languages good tools for implementing different parts of embedded applications (for example web user interface). Nowadays it is common to find embedded hardware devices, based on Raspberry Pi for instance, that are accessible via a network, run Linux and come with Apache and PHP installed on the device.  There are also many other relevant languages

One workable solution, especially for embedded Linux systems is that part of the activities organized by totetuettu is a C program instead of scripting languages ​​(Scripting). This enables editing operation simply script files by editing without the need to turn the whole system software again.  Scripting languages ​​are also tools that can be implemented, for example, a Web user interface more easily than with C / C ++ language. An empirical study found scripting languages (such as Python) more productive than conventional languages (such as C and Java) for a programming problem involving string manipulation and search in a dictionary.

Scripting languages ​​have been around for a couple of decades Linux and Unix server world standard tools. the proliferation of embedded Linux and resources to merge systems (memory, processor power) growth has made them a very viable tool for many embedded systems – for example, industrial systems, telecommunications equipment, IoT gateway, etc . Some of the command language is suitable for up well even in quite small embedded environments.
I have used with embedded systems successfully mm. Bash, AWK, PHP, Python and Lua scripting languages. It works really well and is really easy to make custom code quickly .It doesn’t require a complicated IDE; all you really need is a terminal – but if you want there are many IDEs that can be used.
High-level, dynamically typed languages, such as Python, Ruby and JavaScript. They’re easy—and even fun—to use. They lend themselves to code that easily can be reused and maintained.

There are some thing that needs to be considered when using scripting languages. Sometimes lack of static checking vs a regular compiler can cause problems to be thrown at run-time. But it is better off practicing “strong testing” than relying on strong typing. Other ownsides of these languages is that they tend to execute more slowly than static languages like C/C++, but for very many aplications they are more than adequate. Once you know your way around dynamic languages, as well the frameworks built in them, you get a sense of what runs quickly and what doesn’t.

Bash and other shell scipting

Shell commands are the native language of any Linux system. With the thousands of commands available for the command line user, how can you remember them all? The answer is, you don’t. The real power of the computer is its ability to do the work for you – the power of the shell script is the way to easily to automate things by writing scripts. Shell scripts are collections of Linux command line commands that are stored in a file. The shell can read this file and act on the commands as if they were typed at the keyboard.In addition to that shell also provides a variety of useful programming features that you are familar on other programming langauge (if, for, regex, etc..). Your scripts can be truly powerful. Creating a script extremely straight forward: It can be created by opening a separate editor such or you can do it through a terminal editor such as VI (or preferably some else more user friendly terminal editor). Many things on modern Linux systems rely on using scripts (for example starting and stopping different Linux services at right way).

One of the most useful tools when developing from within a Linux environment is the use of shell scripting. Scripting can help aid in setting up environment variables, performing repetitive and complex tasks and ensuring that errors are kept to a minimum. Since scripts are ran from within the terminal, any command or function that can be performed manually from a terminal can also be automated!

The most common type of shell script is a bash script. Bash is a commonly used scripting language for shell scripts. In BASH scripts (shell scripts written in BASH) users can use more than just BASH to write the script. There are commands that allow users to embed other scripting languages into a BASH script.

There are also other shells. For example many small embedded systems use BusyBox. BusyBox providesis software that provides several stripped-down Unix tools in a single executable file (more than 300 common command). It runs in a variety of POSIX environments such as Linux, Android and FreeeBSD. BusyBox become the de facto standard core user space toolset for embedded Linux devices and Linux distribution installers.

Shell scripting is a very powerful tool that I used a lot in Linux systems, both embedded systems and servers.

Lua

Lua is a lightweight  cross-platform multi-paradigm programming language designed primarily for embedded systems and clients. Lua was originally designed in 1993 as a language for extending software applications to meet the increasing demand for customization at the time. It provided the basic facilities of most procedural programming languages. Lua is intended to be embedded into other applications, and provides a C API for this purpose.

Lua has found many uses in many fields. For example in video game development, Lua is widely used as a scripting language by game programmers. Wireshark network packet analyzer allows protocol dissectors and post-dissector taps to be written in Lua – this is a good way to analyze your custom protocols.

There are also many embedded applications. LuCI, the default web interface for OpenWrt, is written primarily in Lua. NodeMCU is an open source hardware platform, which can run Lua directly on the ESP8266 Wi-Fi SoC. I have tested NodeMcu and found it very nice system.

PHP

PHP is a server-side HTML embedded scripting language. It provides web developers with a full suite of tools for building dynamic websites but can also be used as a general-purpose programming language. Nowadays it is common to find embedded hardware devices, based on Raspberry Pi for instance, that are accessible via a network, run Linux and come with Apache and PHP installed on the device. So on such enviroment is a good idea to take advantage of those built-in features for the applications they are good – for building web user interface. PHP is often embedded into HTML code, or it can be used in combination with various web template systems, web content management system and web frameworks. PHP code is usually processed by a PHP interpreter implemented as a module in the web server or as a Common Gateway Interface (CGI) executable.

Python

Python is a widely used high-level, general-purpose, interpreted, dynamic programming language. Its design philosophy emphasizes code readability. Python interpreters are available for installation on many operating systems, allowing Python code execution on a wide variety of systems. Many operating systems include Python as a standard component; the language ships for example with most Linux distributions.

Python is a multi-paradigm programming language: object-oriented programming and structured programming are fully supported, and there are a number of language features which support functional programming and aspect-oriented programming,  Many other paradigms are supported using extensions, including design by contract and logic programming.

Python is a remarkably powerful dynamic programming language that is used in a wide variety of application domains. Since 2003, Python has consistently ranked in the top ten most popular programming languages as measured by the TIOBE Programming Community Index. Large organizations that make use of Python include Google, Yahoo!, CERN, NASA. Python is used successfully in thousands of real world business applications around globally, including many large and mission-critical systems such as YouTube.com and Google.com.

Python was designed to be highly extensible. Libraries like NumPy, SciPy and Matplotlib allow the effective use of Python in scientific computing. Python is intended to be a highly readable language. Python can also be embedded in existing applications and hasbeen successfully embedded in a number of software products as a scripting language. Python can serve as a scripting language for web applications, e.g., via mod_wsgi for the Apache web server.

Python can be used in embedded, small or minimal hardware devices. Some modern embedded devices have enough memory and a fast enough CPU to run a typical Linux-based environment, for example, and running CPython on such devices is mostly a matter of compilation (or cross-compilation) and tuning. Various efforts have been made to make CPython more usable for embedded applications.

For more limited embedded devices, a re-engineered or adapted version of CPython, might be appropriateExamples of such implementations include the following: PyMite, Tiny Python, Viper. Sometimes the embedded environment is just too restrictive to support a Python virtual machine. In such cases, various Python tools can be employed for prototyping, with the eventual application or system code being generated and deployed on the device. Also MicroPython and tinypy have been ported Python to various small microcontrollers and architectures. Real world applications include Telit GSM/GPRS modules that allow writing the controlling application directly in a high-level open-sourced language: Python.

Python on embedded platforms? It is quick to develop apps, quick to debug – really easy to make custom code quickly. Sometimes lack of static checking vs a regular compiler can cause problems to be thrown at run-time. To avoid those try to have 100% test coverage. pychecker is a very useful too also which will catch quite a lot of common errors. The only downsides for embedded work is that sometimes python can be slow and sometimes it uses a lot of memory (relatively speaking). An empirical study found scripting languages (such as Python) more productive than conventional languages (such as C and Java) for a programming problem involving string manipulation and search in a dictionary. Memory consumption was often “better than Java and not much worse than C or C++”.

JavaScript and node.js

JavaScript is a very popular high-level language. Love it or hate it, JavaScript is a popular programming language for many, mainly because it’s so incredibly easy to learn. JavaScript’s reputation for providing users with beautiful, interactive websites isn’t where its usefulness ends. Nowadays, it’s also used to create mobile applications, cross-platform desktop software, and thanks to Node.js, it’s even capable of creating and running servers and databases!  There is huge community of developers. JavaScript is a high-level language.

Its event-driven architecture fits perfectly with how the world operates – we live in an event-driven world. This event-driven modality is also efficient when it comes to sensors.

Regardless of the obvious benefits, there is still, understandably, some debate as to whether JavaScript is really up to the task to replace traditional C/C++ software in Internet connected embedded systems.

It doesn’t require a complicated IDE; all you really need is a terminal.

JavaScript is a high-level language. While this usually means that it’s more human-readable and therefore more user-friendly, the downside is that this can also make it somewhat slower. Being slower definitely means that it may not be suitable for situations where timing and speed are critical.

JavaScript is already in embedded boards. You can run JavaScipt on Raspberry Pi and BeagleBone. There are also severa other popular JavaScript-enabled development boards to help get you started: The Espruino is a small microcontroller that runs JavaScript. The Tessel 2 is a development board that comes with integrated wi-fi, an ethernet port, two USB ports, and companion source library downloadable via the Node Package Manager. The Kinoma Create, dubbed the “JavaScript powered Internet of Things construction kit.”The best part is that, depending on the needs of your device, you can even compile your JavaScript code into C!

JavaScript for embedded systems is still in its infancy, but we suspect that some major advancements are on the horizon.We for example see a surprising amount of projects using Node.js.Node.js is an open-source, cross-platform runtime environment for developing server-side Web applications. Node.js has an event-driven architecture capable of asynchronous I/O that allows highly scalable servers without using threading, by using a simplified model of event-driven programming that uses callbacks to signal the completion of a task. The runtime environment interprets JavaScript using Google‘s V8 JavaScript engine.Node.js allows the creation of Web servers and networking tools using JavaScript and a collection of “modules” that handle various core functionality. Node.js’ package ecosystem, npm, is the largest ecosystem of open source libraries in the world. Modern desktop IDEs provide editing and debugging features specifically for Node.js applications

JXcore is a fork of Node.js targeting mobile devices and IoTs. JXcore is a framework for developing applications for mobile and embedded devices using JavaScript and leveraging the Node ecosystem (110,000 modules and counting)!

Why is it worth exploring node.js development in an embedded environment? JavaScript is a widely known language that was designed to deal with user interaction in a browser.The reasons to use Node.js for hardware are simple: it’s standardized, event driven, and has very high productivity: it’s dynamically typed, which makes it faster to write — perfectly suited for getting a hardware prototype out the door. For building a complete end-to-end IoT system, JavaScript is very portable programming system. Typically an IoT projects require “things” to communicate with other “things” or applications. The huge number of modules available in Node.js makes it easier to generate interfaces – For example, the HTTP module allows you to create easily an HTTP server that can easily map the GET method specific URLs to your software function calls. If your embedded platform has ready made Node.js support available, you should definately consider using it.

Future trends

According to New approaches to dominate in embedded development article there will be several camps of embedded development in the future:

One camp will be the traditional embedded developer, working as always to craft designs for specific applications that require the fine tuning. These are most likely to be high-performance, low-volume systems or else fixed-function, high-volume systems where cost is everything.

Another camp might be the embedded developer who is creating a platform on which other developers will build applications. These platforms might be general-purpose designs like the Arduino, or specialty designs such as a virtual PLC system.

This third camp is likely to become huge: Traditional embedded development cannot produce new designs in the quantities and at the rate needed to deliver the 50 billion IoT devices predicted by 2020.

Transition will take time. The enviroment is different than computer and mobile world. There are too many application areas with too widely varying requirements for a one-size-fits-all platform to arise.

But the shift will happen as hardware becomes powerful and cheap enough that the inefficiencies of platform-based products become moot.

 

Sources

Most important information sources:

New approaches to dominate in embedded development

A New Approach for Distributed Computing in Embedded Systems

New Approaches to Systems Engineering and Embedded Software Development

Lua (programming language)

Embracing Java for the Internet of Things

Node.js

Wikipedia Node.js

Writing Shell Scripts

Embedded Linux – Shell Scripting 101

Embedded Linux – Shell Scripting 102

Embedding Other Languages in BASH Scripts

PHP Integration with Embedded Hardware Device Sensors – PHP Classes blog

PHP

Python (programming language)

JavaScript: The Perfect Language for the Internet of Things (IoT)

Node.js for Embedded Systems

Embedded Python

MicroPython – Embedded Pytho

Anyone using Python for embedded projects?

Telit Programming Python

JavaScript: The Perfect Language for the Internet of Things (IoT)

MICROCONTROLLERS AND NODE.JS, NATURALLY

Node.js for Embedded Systems

Why node.js?

Node.JS Appliances on Embedded Linux Devices

The smartest way to program smart things: Node.js

Embedded Software Can Kill But Are We Designing Safely?

DEVELOPING SECURE EMBEDDED SOFTWARE

 

 

 

1,432 Comments

  1. Tomi Engdahl says:

    Scripting Language Rapidly Develops A Clock
    https://hackaday.com/2019/06/27/scripting-language-rapidly-develops-a-clock/

    In the past, you might very well have started programming in Basic. It wasn’t very powerful language and it was difficult to build big projects with, but it was simple to learn, easy to use, and the interpreter made it easy to try things out without a big investment of time. Today you are more likely to get started using something like an Arduino, but it is easy to miss the accessible language and immediate feedback when you are doing simple projects. Annex WiFi RDS (Rapid Development Suite) is a scripting language for the ESP8266 that isn’t quite Basic, but it shares a lot of the same attributes. One example project from [cicciocb] is a scrolling dot matrix LED clock.

    https://sites.google.com/site/annexwifi/projects/3d-printed-clock

    Reply
  2. Tomi Engdahl says:

    MicroPython Robot Kit
    Rapid Prototyping Open Source Robot Kit
    https://hackaday.io/project/164744-micropython-robot-kit

    MicroPython Robot kit is a constructor, it changable, you can using your own types of sensors, that gives more possibilities for learning and practice.

    Main core, the “heart” of the robot, is ESP32. It giving us more compact and flexible system with BLE and Wi-Fi onboard.

    You can use MicroPython for rapid prototyping and shortener development time, but hardware of the robot give possibilities to use C/C++ on FreeRTOS for more complex and fast applications.

    Published under GNU GPL v3.0 license + CERN Open Hardware License

    Reply
  3. Tomi Engdahl says:

    Hands-On: GreatFET Is An Embedded Tool That Does It All
    https://hackaday.com/2019/07/02/hands-on-greatfet-is-an-embedded-tool-that-does-it-all/

    There’s a new embedded hacking tool on the scene that gives you an interactive Python interface for a speedy chip on a board with oodles of GPIO, the ability to masquerade as different USB devices, and a legacy of tricks up its sleeve. This is the GreatFET, the successor to the much loved GoodFET.

    I first heard this board was close to launch almost a year ago and asked for an early look. When shipping began at the end of April, they sent me one. Let’s dig in for a hands-on review of the GreatFET from Great Scott Gadgets.

    Lots of Fast I/O with Direct Access From Your Computer

    https://greatscottgadgets.com/greatfet/

    Reply
  4. Tomi Engdahl says:

    Lisp Runs on a $1 ATtiny3216
    https://www.hackster.io/news/lisp-runs-on-a-1-attiny3216-21a7f02fc7de

    David Johnson-Davies has come up with a way to run the high-level language uLisp —his version of Lisp written for microcontrollers — on an ATtiny3216. This relatively new microcontroller costs just over a dollar, but packs in some impressive specs, including a 32kB of flash memory and 2kB of RAM. This just so happens to match up with the ATmega328P that was previously the smallest MCU that could run this interpreter

    http://www.technoblogy.com/show?2R5C

    Reply
  5. Tomi Engdahl says:

    Arm shows roadmap for collaborating on embedded operating systems
    https://venturebeat.com/2019/10/08/arm-shows-roadmap-for-collaborating-on-embedded-operating-systems/

    Arm revealed its roadmap for Mbed OS, the embedded software operating system that serves as a foundation for the internet of things (IoT), or making everyday objects smart and connected.

    Reply
  6. Tomi Engdahl says:

    Worn Out EMMC Chips Are Crippling Older Teslas
    https://hackaday.com/2019/10/17/worn-out-emmc-chips-are-crippling-older-teslas/

    Unfortunately, it seems the rise of high-tech EVs is also ushering in a new error of unexpected failures and maintenance woes. Case in point, some owners of older model Teslas are finding they’re at risk of being stranded on the side of the road by a failure most of us would more likely associate with losing some documents or photos: a disk read error.

    Linux Loudly Logging

    Much like the rockets and spacecraft of sister company SpaceX, Tesla’s vehicles are powered by Linux running on what’s essentially off-the-shelf computing hardware. Until 2018 the Model S and X were running the open source operating system on a NVIDIA Tegra 3, at which point they switched the Media Control Unit (MCU) over to an Intel Atom solution. In either event, the Linux system is stored on an embedded Multi-Media Controller (eMMC) flash chip instead of a removable storage device as you might expect.

    Now under normal circumstances, this wouldn’t be an issue. There are literally billions of devices running Linux from an eMMC chip. But any competent embedded Linux developer would take the steps necessary to make sure the operating system’s various log files are not being written to a non-replaceable storage device soldered onto the board

    Unfortunately, for reasons that still remain somewhat unclear, the build of Linux running on the MCU is doing exactly that. What’s worse, Tesla’s graphical interface appears to be generating its own additional log messages.

    Reply
  7. Tomi Engdahl says:

    Numpy Comes To Micro Python
    https://hackaday.com/2019/10/29/numpy-comes-to-micro-python/

    [Zoltán] sends in his very interesting implementation of a NumPy-like library for micropython called ulab.

    He had a project in MicroPython that needed a very fast FFT on a micro controller, and was looking at all of the options when it occurred to him that a more structured approach like the one we all know and love in CPython would be possible on a micro controller too. He thus ended up with a python library that could do the FFT 50 times faster than the the pure Python implementation while providing all the readability and ease of use benefits that NumPy and Python together provide.

    https://github.com/v923z/micropython-ulab

    Reply
  8. Tomi Engdahl says:

    The Arduino IDE Finally Grows Up
    https://hackaday.com/2019/10/21/the-arduino-ide-finally-grows-up/

    While the Arduino has a very vocal fan club, there are always a few people less than thrilled with the ubiquitous ecosystem. While fans may just dismiss it as sour grapes, there are a few legitimate complaints you can fairly level at the stock setup. To address at least some of those concerns, Arduino is rolling out the Arduino Pro IDE and while it doesn’t completely address every shortcoming, it is worth a look and may grow to quiet down some of the other criticisms, given time.

    https://blog.arduino.cc/2019/10/18/arduino-pro-ide-alpha-preview-with-advanced-features/

    Reply
  9. Tomi Engdahl says:

    A Web API For Your Pi
    https://hackaday.com/2019/10/01/a-web-api-for-your-pi/

    There are many ways to attach a project to the Internet, and a plethora of Internet-based services that can handle talking to hardware. But probably the most ubiquitous of Internet protocols for the average Joe or Jane is the web browser, and one of the most accessible of programming environments lies within it. If only somebody with a bit of HTML and Javascript could reach a GPIO pin on their Raspberry Pi!

    If that’s your wish, then help could be at hand in the form of [Victor Ribeiro]’s RPiAPI. As its name suggests, it’s an API for your Raspberry Pi, and in particular it provides a simple web-accessible endpoint wrapper for the Pi’s GPIO library from which its expansion port pins can be accessed.

    RPiAPI
    https://hackaday.io/project/167837-rpiapi

    RPiAPI is a lightweight WSGI API built on top of the RPi GPIO library. It provides endpoints so you can interact with your Raspberry Pi

    Reply
  10. harshnithasri says:

    Computer Support Specialists or Help Desk Technicians, these technicians assist employees in resolving computer-related issues. This position requires both excellent customer service and problem-solving skills.
    Technical Support Technicians work with organizations, vendors or companies that provide third-party contract technical support. Whether in-house or outsourced, the technician provides primary point-of-contact assistance when a problem arises. Apart from solving complicated issues, technicians create manuals and other documentation intended to guide users in the use of new hardware and software. Regardless of client and customer needs, support technicians should be willing and able to stay abreast of new computing technologies.
    https://www.fieldengineer.com/skills/technical-support-technician

    Reply
  11. Tomi Engdahl says:

    Why We Need to Retire the Blinking LED as the Entry Into Embedded Programming
    https://www.hackster.io/news/why-we-need-to-retire-the-blinking-led-as-the-entry-into-embedded-programming-662c0eb5e751

    Something that was true since 1985 changed in Q4 of 2018 and we are yet to feel all the repercussions.

    Every piece of electronics which we geeked out about, used, abused, eventually bricked then forgot… Each and every one of them hails from the same embryo.

    The blinking LED

    It used to be the classic “Hello World” of the embedded world.

    But for a while now it has been due for an update. An update that would reflect the vanishing boundaries between different domains of software development, as brought on by IoT. Consider Gartner’s prediction: by 2020, 20.4 billion devices will be connected to the Internet. It’s time to replace the LED blink with a “Hello, Cloud.”

    We could have spent a few more years debating about it but a major turning point happened in the last quarter of last year, giving us a perfect excuse.

    What happened in Q4 of 2018 is that Python became the most popular programming language, according to a five-minute video that recently went viral. To become number one, Python toppled Java. Java sat on the top spot since 2000 after dethroning C, which in turn knocked down Pascal in 1985.

    How did a scripting language replace a comparatively low-level language as the most popular in the world? Will Python seep into all aspects of programming from supercomputers to embedded systems the way C did?

    It’s impossible to tell, of course. But whatever skeptics and detractors may say, Python is more diverse than given credit for. It has only started to show its potential for programming embedded systems. What it lacks in efficiency and speed of execution, it makes up for in speed of development and deployment. And in the world of 32-bit microcontrollers, and short, Agile-based development cycles, this is often a welcome trade-off.

    In just 12 lines of readable Python code, you can connect your device to the cloud and send a message.

    Reply
  12. Tomi Engdahl says:

    One of the biggest challenges for system designers is building a system that can prevent, detect and react to system faults and failures.

    Based on our deep experience with analog technologies, we see a world that continually needs efficient power, precision measurement, rugged connectivity, and reliable protection.

    https://www.electronicdesign.com/webcasts/3-ways-make-your-system-more-robust?partnerref=ED3&utm_rid=CPG05000002750211&utm_campaign=30180&utm_medium=email&elq2=2eff8fc4f0b14e0685d9d07e46df8e58&oly_enc_id=0452E0081834E9U

    Reply
  13. Tomi Engdahl says:

    Industrial virtualization heads to the plant floor
    End users are comfortable with virtualization in their information technology (IT) server rooms and
    desktops, and many similar benefits now can be realized with industrial programmable logic controllers (PLCs) and programmable automation controllers (PACs).
    https://www.controleng.com/articles/industrial-virtualization-heads-to-the-plant-floor/

    A basic definition of virtualization is providing the ability to run more than one VM software operating
    system (OS) on a single hardware platform, allowing one physical computer to be more utilized as many

    virtual computers. Each VM must operate independently.
    A common class of industrial controller for typical automation applications is the PLC, which has used a

    dedicated processor and specific real-time OS (RTOS) to provide high-speed deterministic control. The challenge to virtualizing PLC functions is maintaining high-speed determinism.

    For edge-located controllers, the possibility of running multiple OSs is a dramatic shift (Figure 2). The primary concern is maintaining a robust RTOS for control functionality, as with PLCs, so the automation functionality is not compromised. A secondary OS, operating as supplemental to the RTOS, provides additional computing options. The two OSs must be independent, but maintain the ability to interact.

    The concept of running two OSs in an edge-located industrial controller can be further explained in terms of an “inner” and an “outer” loop, which may be familiar to users of cascaded process control loops (Figure 3). In this case, the inner loop is the RTOS VM for control, while the outer loop is a standard OS VM for added functions. For instance, the inner loop could monitor a process flow input, perform the proportional-integral-derivative (PID) calculations, and command the control valve output. The outer loop can advise the inner loop regarding the optimal flow rate, but it will not impact inner loop operation otherwise.

    Control remains the inner loop

    PLCs and PACs use many types of programming languages; ladder logic is the most popular. A basic measurement of PLC performance is how fast the controller can scan through the ladder logic, typically measured in milliseconds. All other overhead tasks must be handled so a deterministic scan time is preserved. The OS from a PC is not a good candidate for millisecond control because it must handle many overhead tasks such as graphics and user interface.

    Superseding a PLC or PAC with a controller-based VM while maintaining the equivalent functionality requires the use of a RTOS. This RTOS provides the features needed for inner-loop PLC functionality without any performance-sapping features.

    Extended capabilities live in the outer loop

    Since control functionality remains essentially the same, the real advantage to controller virtualization is the addition of an outer-loop Linux VM carefully integrated in a cooperative manner. This VM can do anything a dedicated PC could, but at a lower cost, and packaged in a more compact form factor, with no need to integrate a PC to the controller.

    It is not necessary for industrial users to take advantage of the extra OS right away, as they could use a virtualized controller for the basic PLC functionality. However, many users are finding a general-purpose Linux OS at the edge can enhance their applications by:

    Running machine learning elements
    Performing analytics
    Communicating to the cloud
    Using messaging queuing telemetry transport (MQTT) or some other publish-subscribe model to exchange information
    Executing optimization calculations to inform the controller VM
    Driving a local display
    Serving web pages.

    These functions have required upstream computing resources. Users benefit from the efficiency of implementing these functions out at an edge-located OS, because they are taking advantage of locally available processing power and acting on the data as close to the source as possible. This removes layers of computing and streamlines networking usage. The ability to drive a local display or directly serve web pages out of the controller are examples of this.

    Some of these functions are possible in standard PLCs today, but adding these capabilities into a traditional PLC must be carefully performed to prevent overloading. Also, when advanced features are added in a traditional PLC, it means users must work in the active control environment to make changes, which increases the risk of disrupting automation.

    When implemented properly, an outer loop VM lets users safely perform processing out at the edge closer to the source of data, which unloads upstream networking and processing requirements.

    Reply
  14. Tomi Engdahl says:

    Ethernet for process and discrete devices
    https://www.controleng.com/articles/ethernet-for-process-and-discrete-devices/

    Think Again: Separate efforts aim to extend Ethernet to process and discrete devices. Standards can promote interoperability for devices and software if standards are specific enough.

    Advanced Physical Layer for industrial Ethernet

    The Advanced Physical Layer for industrial Ethernet consists of IEEE P802.3cg 10Mbits single twisted pair long reach Ethernet (from IEEE Task Force; approved Nov. 7) plus Power on Single Twisted Pair plus an Intrinsically safe frontend (both from APL Industry Partners R&D project). The goal is to update industrial Ethernet specifications in 2020 (FieldComm Group, ODVA and PI). First infrastructure and field devices are expected in 2021. Companies involved so far include Endress+Hauser, Pepperl+Fuchs, Rockwell Automation, and Samson, to bring interoperability, Ethernet communications, and hazardous area protection to field instruments.

    Extending the discrete Ethernet side, Festo’s CPX-AP-I Remote I/O, launched in October, extends performance of mixed valve terminals and I/O systems and is compatible with most communication protocols, including industrial Ethernets, such as ODVA’s EtherNet/IP, PI’s Profinet, and Ethernet Technology Group’s EtherCAT.

    Latency is virtually nonexistent, Festo said: “Scan cycles for a mix of both valve terminals and I/O are below one millisecond and are expected to approach microseconds.”

    Reply
  15. Tomi Engdahl says:

    4 Tips for Launching Effective Continuous Testing
    https://www.ranorex.com/blog/effective-continuous-testing/

    It’s important that tests be automatic and thorough. Continuous testing is an approach for automated testing that gives fast feedback by testing early and often. But “continuous” doesn’t mean all tests need to run at all times. Here are four tips for organizing your continuous testing system.

    1. Know yourself
    Continuous testing (CT) is valuable to essentially all developers. But different developers are different enough that the value will vary a great deal. Run CT for the specific benefits it brings in your situation, not just because CT is a hot topic right now.

    2. Everybody runs unit tests
    Unit tests need to be automatic, cheap, and easy. If they take 40 minutes to run, they’re not unit tests. If they can only be run on certain hardware, they’re probably not unit tests. Tests that demand human oversight—such as “Is that click in the right place on the screen?”—are not unit tests. If a supervisor has to enter a secret passphrase at a front-line contributor’s desk, the context is definitely not for unit tests.

    That doesn’t mean those other tests aren’t valuable; they might even become unit tests some day.

    3. Select which tests work for continuous testing
    Next up, identify tests that are feasible for your developers but not your CT system, or vice versa.

    4. Plan for the future you want
    Does your testing plan still have gaps? Are there error-handling segments of source that are hard to test, or tests that require human intervention and judgment, or errors that show up only for end-users?

    Like anything in life worth doing, your CT’s unlikely to be perfect at the beginning. Document your gaps, and figure out which ones will reward correction the quickest and best. Go slow on tests that promise little return, and learn to recognize tests that cost more than they’re worth.

    Reply
  16. Tomi Engdahl says:

    With the rapid pace of innovation, it has become increasingly challenging for system architects to keep embedded systems development and deployment efficient, reliable, and cost-effective. Virtualization is the bridge that enables system architects to connect today’s legacy software with tomorrow’s innovation.

    https://www.techonline.com/electrical-engineers/education-training/courses/4462235/Future-Proof-your-critical-infrastructure?acctid=6786&utm_source=newsletter&utm_campaign=ad&utm_medium=EDNNPN-20191224

    Reply
  17. Tomi Engdahl says:

    5 Tips for Versioning Embedded Systems
    https://www.designnews.com/electronics-test/5-tips-versioning-embedded-systems/93647737161546?ADTRK=InformaMarkets&elq_mid=10357&elq_cid=876648

    Versioning an embedded system doesn’t have to be complicated. The simpler you make it, the smaller the chances there will be confusion from using the wrong version of hardware or software.

    One topic that can be tricky for embedded systems engineers is how to version their embedded system. Every embedded system has two primary systems that need to be versioned, the hardware and the software. While versioning the system sounds simple, it can be confusing on the best way to version the system. In this post, we will examine five tips for versioning your embedded system.

    Tips #1 – Use GPIO to Hard Code PCB Versions

    Tip #2 – Use the Major.Minor.Patch Semantic

    Tip #3 – Create a version.h module

    Tip #4 – Avoid module and function versioning if possible

    Tip #5 – Integrate VCS with your IDE

    Conclusions

    Versioning an embedded system doesn’t have to be complicated. In fact, the simpler you make versioning your system, the smaller the chances will be that there will be confusion or issues from using the wrong version of hardware or software. As we have seen, there are several tricks developers can leverage to simplify versioning such as using an integrated version log and hardware-based versioning. No matter what system is adopted, the only way it will be successful is if the developers involved adopt a disciplined approach to versioning and don’t cut corners when they commit their software.

    Reply
  18. Tomi Engdahl says:

    Don’t get locked up into avoiding lock-in
    https://martinfowler.com/articles/oss-lockin.html

    A significant share of architectural energy is spent on reducing or avoiding lock-in. That’s a rather noble objective: architecture is meant to give us options and lock-in does the opposite. However, lock-in isn’t a simple true-or-false matter: avoiding being locked into one aspect often locks you into another. Also, popular notions, such as open source automagically eliminating lock-in, turn out to be not entirely true. Time to have a closer look at lock-in, so you don’t get locked up into avoiding it!

    Reply
  19. Tomi Engdahl says:

    Virtualizing a PLC?
    http://scadamag.infracritical.com/index.php/2019/07/15/virtualizing-a-plc/

    A very similar effort was tried about 25 years ago and it failed in multiple implementations among various vendors. It was the PLC on the PC concept. The problem is that many other things were also going on in that PC. Even though the PLC process was given priority even higher than the OS, and could be given processor affinity and full time processor availability, it still didn’t do well. Why?

    1. The PC is not in the field. It’s usually in the administration building. The PLC process has to coexist with a lot of other applications. The network connections to get the I/O traffic to that office or server were not particularly reliable.

    2. If the OS crashes, yes, the PLC can continue to work most of the time (I actually saw this failure mode). But that’s not good enough. Even if the PLC continues to run, someone is going to want the rest of that PC to work. This results in a full boot-up sequence, during which the PLC becomes unavailable.

    3. PC hardware is not hardened environmentally. So if the building HVAC failed over the weekend, the PLC software might not stay online because the building itself was too hot for the PC to operate.

    Now let’s review why we put the PLC in the field to begin with.

    Networks are not reliable enough. Yes, the components themselves can be hardened. Yes, they can be pretty capable. But if you add a switch here, a firewall there, a fiber optic trunk, another router, and then a firewall to get to the VM host, your potential for failure is still higher than if you just stuffed a PLC in to that field cabinet.

    Let’s suppose you have on-premise cloud hardware. A reliable VM host (and UPS) is probably more expensive and power hungry than most of those PLCs, but let’s set that aside for a minute. Your latency will be significantly more. Keep in mind that the network extent to the I/O is greater. From the admin building to the site, you may have many failures: Flooded conduits, Electric faults, Fires, or even a misidentified cable where someone disconnects the wrong thing. Oh, but there is redundant networking, right?

    Here’s where I put on my hat of experience. I’ve seen redundant networks. Do you know where they get installed? They usually traverse the same conduits or conduit banks as the other network cable.

    Let me reiterate one thing about using cloud resources: Do not use Internet based cloud services for ICS! Internet cloud services are designed for serving traffic at high bandwidth to the rest of the Internet, but not necessarily you. It is optimized for resiliency of a web site.

    Reply
  20. Tomi Engdahl says:

    Improving Algorithms With High-Level Synthesis
    New directions for a commonly used technology.
    https://semiengineering.com/improving-algorithms-with-high-level-synthesis/

    Most computer algorithms today are developed in high-level languages on general-purpose computers. But someday they may be deployed in embedded systems where the development, verification, and validation of algorithms is done in languages like python, Java, C++, or even numerical frameworks like MatLab.

    This is the goal of high-level synthesis (HLS), and it aims to solve a fundamental problem in system design today. The basic idea is allowing hardware designers to build and verify hardware, with better control over optimization of the design architecture, describing the design at a higher level of abstraction while the tool implements the RTL. It also may be possible to use HLS to improve today’s algorithms that run on that hardware.

    Reply
  21. Tomi Engdahl says:

    Implementing IoT sensor based designs that crosses multiple domains is challenging.

    across the design, simulation, layout, and verification of MEMS, Photonics, and AMS designs

    Reply
  22. Tomi Engdahl says:

    5 Tips For Modernizing Legacy Code
    Here are several strategies teams can follow to start modernizing their legacy code projects.
    https://www.designnews.com/electronics-test/5-tips-modernizing-legacy-code/6734291362137?ADTRK=InformaMarkets&elq_mid=11716&elq_cid=876648

    Teams often struggle with legacy code bases for reasons such as:

    The sheer number of features and code size (100,000’s of lines of code if not millions)
    Tight coupling between application code and hardware
    Antiquated tools and development processes
    Inability to effectively deploy new developers to work on the code base

    Tip #1 – Audit your application code

    Tip #2 – Generate a function list and dependency graph

    Tip #3 – Modularize the application

    Tip #4 – Let go of how things “are” or “were”

    Tip #5 – Modernize through incremental improvements

    Conclusions

    In today’s post, we have just scratched the surface on how you can start to modernize a legacy code base. It’s important to recognize that modernization won’t be done fast or overnight if it is going to be done right. A successful modernization effort will be incremental, but the benefits can be nearly instantaneous if teams focus on the low hanging, high value areas that are identified in a code audit

    Reply
  23. Tomi Engdahl says:

    5 Tips for Improving Embedded Development in 2020
    https://www.designnews.com/electronics-test/5-tips-improving-embedded-development-2020/142340837962172?ADTRK=InformaMarkets&elq_mid=11753&elq_cid=876648

    Modernizing your embedded techniques can accelerate development, improve quality, and allow a team to get more done in the same amount of time.

    Tip #1 – Start Measuring Key Metrics and Drivers

    Tip #2 – Start having Code Reviews

    Tip #3 – Master Debugging Skills

    Tip #4 – Start Automating Testing

    Tip #5 – Separate Software Concerns

    Conclusions

    There is never a best time to start improving the way that you design and build embedded systems. Waiting for the perfect time will simply result in the same old day to day fires, long nights and working weekends. Don’t wait for the perfect time to measure and improve your development processes, that time will never come. The five tips we’ve discussed in todays post are low hanging fruit that can be easily implemented and contribute to a feedback loop that will allow development to become more efficient, cost effective and more consistent.

    Reply
  24. Tomi Engdahl says:

    Designing Peer-To-Peer Command and Control
    https://posts.specterops.io/designing-peer-to-peer-command-and-control-ad2c61740456

    In this post we will discuss the design and implementation of peer-to-peer command and control protocols in general, as well as the concrete example of the peer-to-peer design implemented in Covenant, an open-source command and control framework

    Reply
  25. Tomi Engdahl says:

    Open Source Hardware Risks

    Open-source hardware is gaining attention on a variety of fronts, from chiplets and the underlying infrastructure to the ecosystems required to support open-source and hybrid open-source and proprietary designs. Open-source development is hardly a new topic. It has proven to be a successful strategy in the Linux world, but far less so on the hardware side. That is beginning to change, fueled …
    Semiconductor Engineering

    Open Source Hardware Risks
    https://semiengineering.com/open-source-hardware-risks/

    There’s still much work to be done to enable an open source hardware ecosystem.

    Reply
  26. Tomi Engdahl says:

    Using Static Analysis For Functional Safety
    Why this technology is suddenly so important.
    https://semiengineering.com/using-static-analysis-for-functional-safety/

    Fadi Maamari, group director for R&D at Synopsys, explains why static analysis is suddenly in demand in auto chip design, how it can help to choose the best implementation of functional safety approaches, and where it fits into the design flow.

    Reply
  27. Tomi Engdahl says:

    Determining What Really Needs To Be Secured In A Chip
    Experts at the Table: Why previous approaches at security have only limited success.
    https://semiengineering.com/determining-what-really-needs-to-be-secured-in-a-chip/

    Reply
  28. Tomi Engdahl says:

    How Chips Age
    https://semiengineering.com/how-chips-age/

    Are current methodologies sufficient for ensuring that chips will function as expected throughout their expected lifetimes?

    Reply
  29. Tomi Engdahl says:

    The future of smart devices: contextual awareness
    https://www.eetimes.com/the-future-of-smart-devices-contextual-awareness/

    Radar technology will enhance the usability of smart devices, making them “context-aware.” They will be able to understand their environment and react much more purposefully. Infineon is a leader in this space, having successfully implemented the complex radar system into a mobile device for the first time.

    Reply
  30. Tomi Engdahl says:

    https://www.sqlite.org/howtocorrupt.html

    An SQLite database is highly resistant to corruption. If an application crash, or an operating-system crash, or even a power failure occurs in the middle of a transaction, the partially written transaction should be automatically rolled back the next time the database file is accessed. The recovery process is fully automatic and does not require any action on the part of the user or the application.

    Reply
  31. Tomi Engdahl says:

    On aika siirtyä 32-bittisiin
    https://etn.fi/index.php?option=com_content&view=article&id=1371&via=n&datum=2020-01-30_15:20:29&mottagare=30929

    Sulautetuissa suunnitteluissa on tullut aika siirtyä 8- ja 16-bittisistä ohjaimista 32-bittisiin. Sen myötä saadaan lisää suorituskykyä, parempi energiatehokkuus ja tiheämpi koodi. 32-bittisyyteen liittyvät vanhat virhekäsitykset on syytä korjata.

    Jokaisen uuden projektin yhteydessä kysymykset lisääntyvät: Mitä menetän, jos pysyttelen 8-bittisissä? Mitä 32-bittinen arkkitehtuuri minulle oikeasti tuo lisää? Ja mitkä 32-bittiset ohjaimet hallitsevat nyt sulautettujen ohjaimien myyntitilastoja, onko tullut aika siirtyä ennen kuin jään kehityksestä jälkeen?

    Reply

Leave a Comment

Your email address will not be published. Required fields are marked *

*

*