Best practice - Threading, Events, Interrupts and Memory

[nathanh]

Hi,

I am relatively new to MicroPython (background of C#/Java) but have been loving getting my hands dirty with the SiPy for a project I am working on. However, I have been having some issues when it is coming to running multiple threads and memory management. Specifically I keep having issues at times where I cannot allocate enough memory to create a new thread (as they seem quite expensive to run).

So I wanted to make this post to try and get some advice on what the best programming practice would be to achieve what I want.

The background to the project is that it is a SiPy running 3 sensors (GPS, Temp and Light). I want all sensors to be logging at certain intervals (e.g. Temp reading every 5 mins, GPS every 10 mins, Light every 60 minutes). On top of that I want to set an alert that if any of the sensors hit a certain value and use that alert to send a message over SigFox. So in effect, an event driven functionality.

My issue is that I could not find any real built event driven methods/methodology in Python (might of overlooked something). The closest I found was using a hardware interrupt or the Timer.Alarm() functionality which both feel like I am hammering a square peg into a round hole a bit.

My current solution involves running each sensor logging function (GPS, Temp and Light) on a separate thread that loops, logs and sleeps. If a value of interest is detected, then each individual thread would create another thread that would handle the value (typically by sending a SigFox message). Now, all individual parts of my code work fine but I find that at time my memory is low and cannot create the threads to send the message, especially when two or more are trying to do it at the same time.

So again, with my limited exposure to Python/MicroPython I was wondering what would be the best practice here to create this style of event/interrupt driven functionality. I seem to feel the only other options, as mentioned, would be to use Timer.Alarm() or a hardware pin interrupt and set it via software.

Hopefully that makes some sense. Appreciate any help or feedback.

[nathanh]

@robert-hh Thanks for that (and your assistance on my other question).

I would be interested in your thoughts on how the "main" thread would check these flags as that goes back into using some kind of event/interrupt to know the timer/sensor readings have read something in. My first thoughts would be a while loop with a sleep count and checking periodically but feels like that is not the correct way to go about it.

At this stage I am currently working with two or three period alarms executed from the "main" thread and a single secondary thread that is in a while loop and constantly sleeping looking for values at certain intervals which it then does the "heavy" processing and sending.

@crumble - Yes I must admit I need to get out of some of my older habits to focus more on efficiency than simplicity/readability. However, I must admit I have been struggling to find material on how Micropython, specifically, works with allocation of memory for objects/variables/functions and what is the most efficient practice. Most examples go back to pure Python requiring multiple libraries to do memory statistics, even then not actually demonstrating what is the best way.

Simple questions as to how the GC works, if I need to set variables to 'None' at the end of a function, or the use of a list over three distinct variables. The only way I am really finding it is by doing it both ways and testing it with gc.mem_free() which is time consuming and feels more like guess work. Would be interested on your point 4 about modules and compiling at boot time.

[crumble]

Do not focus in a nice object orientated way. Design your software so, that it 1) that it need low energy -> send your device into deepsleep when possible.
2) that it uses as a low memory as you still have a nice readable source. Use fix buffers for string manipulation for example. Parsing the NMEA codes with normal string methods will create too many short living objects. So you have to call gc.collect() very often. Avoiding these objects will use less CPU time & energy, creates less fragmentation and you have plenty of free memory for other stuff.
3) read the micropython documentation. They have a chapter about performance. Sadly many perfomance related stuff is based on semantic and not on syntax.
4) you can try to call all your uploaded modules during boot time, so that they are compiled. before you use them and you still have huge free memory fragments. As soon as you deal with a lot of short living objects like the string manipulation of the GPS sentences, you may have enough free memory, but it may be too fragmented for compiling seldom used modules.

Sadly all those modern languages are not designed to run on small devices. You have to quite often against their features instead of using them. Sadly there are often more user friendly libraries for python than good old C. A modern C or Basic environment would be much better on such devices like the *Pys. Sadly someone decided that python is the new beginners language. 99% of the beginner friendly features of python belong to the huge amount of libraries and examples. Not because the core language itself is user friendly.

[robert-hh]

@nathanh Using Timer alarms, which read the sensor in various periods and a main thread which checks for the upcoming results is a very efficient and elegant method. Each timer callback would get a result and set a flag, telling, that a new value is present. A "main" thread would check these flags and maybe create a new thread for sending these values. You could do that even w/o theading, but then you might need a value queue for sensor results if a new value comes up before sending has finished.