Main thread blocks auxiliary threads (_thread module)

[d.alvrzx]

Dear all:

With the new Pycom firmware (v1.20.2.rc6), the main thread appears to block auxiliary threads.

Issue description:

To repeat this issue, run the following code with a LoPy4 with Pycom's v1.20.2.rc6 firmware:

import pycom
import time
import _thread

pycom.heartbeat(False)

def blinkLoop():
    while True:
        pycom.rgbled(0xff0000)
        time.sleep(0.5)
        pycom.rgbled(0x00ff00)
        time.sleep(0.5)

blink_process = _thread.start_new_thread(blinkLoop, ())

x = 0

while True:
    #time.sleep(0.1)
    x = x + 1

When time.sleep(0.1) is commented out, the blinkLoop thread does not execute (no blinking is observed in the LED). If you add the 0.1 second delay (uncomment that line), the blinkLoop thread executes without any (discernible) problems.

When did this problem come up?

In v1.20.0.rc13 of Pycom's firmware, the 0.1 second delay is not necessary in order for the blinkLoop thread to execute correctly. When updating to v1.20.2.rc6, the delay is necessary. This is an undesirable behaviour for my application.

These are the 2 firmwares mentioned above:

(sysname='LoPy4', nodename='LoPy4', release='1.20.0.rc13', version='v1.9.4-94bb382 on 2019-08-22', machine='LoPy4 with ESP32', lorawan='1.0.2', sigfox='1.0.1')

and

(sysname='LoPy4', nodename='LoPy4', release='1.20.2.rc6', version='v1.11-01f49f7 on 2020-02-28', machine='LoPy4 with ESP32', lorawan='1.0.2', sigfox='1.0.1', pybytes='1.3.1')

Why does this happen? Has something been changed in the _thread module during the v1.20.2.rc6 firmware release? Or maybe in the change to micropython v1.11? Any ideas or help is appreciated.

Thank you in advance.

Best regards,

Dan

[peterp]

Hey Dan,

I've noticed as well. It is a change in behaviour indeed. Unfortunately, determining the root cause is not trivial though. I suspect it is some change inside IDF, maybe FreeRTOS inside the IDF.

Can you elaborate why a small sleep in the main thread is undesirable for your application?

[robert-hh]

@d-alvrzx The observation that busy loops block other activities is present since long, even before v1.20.0. The cure is typically to add machine.idle() or a short sleep into the loop. In your example, machine.idle() does not work, and the shortest possible sleep time seems to be 1 ms or 1001µs.

[d.alvrzx]

@peterp Hello Peter. Thank you for your prompt response. Regarding your question, it is not "undesirable" as such, but I did not like the idea of having to add a delay (even if it was 1 ms), when in the past my code used to work without it. But I understand it seems to be necessary, and so I can add it without problem (although it does make me feel a bit uncomfortable, haha).

One last question I would have would be: Do you know of any other issues that might arise regarding threads in this new firmware version? Or is it realistic to say that I can just add this 1 ms delay and then forget about it? i.e. Would I encounter no other issues in the future? I understand this is a very, very broad question, but I'm just looking for any info/experience you might have on this.

Thanks!

[peterp]

@d-alvrzx said in Main thread blocks auxiliary threads (_thread module):

@peterp Hello Peter. Thank you for your prompt response. Regarding your question, it is not "undesirable" as such, but I did not like the idea of having to add a delay (even if it was 1 ms), when in the past my code used to work without it. But I understand it seems to be necessary, and so I can add it without problem (although it does make me feel a bit uncomfortable, haha).

Great! (me too ;) )

One last question I would have would be: Do you know of any other issues that might arise regarding threads in this new firmware version? Or is it realistic to say that I can just add this 1 ms delay and then forget about it? i.e. Would I encounter no other issues in the future? I understand this is a very, very broad question, but I'm just looking for any info/experience you might have on this.

Thanks!

A very broad question indeed :) To the best of my knowledge there are no other changes that you need to be aware of. I expect threads to "just work", as long as the main thread "yields" by sleeping a little at some point.

[d.alvrzx]

@robert-hh Thank you very much for your detailed response. I have in fact tried the 1 ms delay within my actual code (the example I posted was just a short piece of code to replicate the problem) and it in fact works. So thanks for that. Maybe you could give me you insight into the question I posed to @peterp, which was:

@d-alvrzx said in Main thread blocks auxiliary threads (_thread module):

One last question I would have would be: Do you know of any other issues that might arise regarding threads in this new firmware version? Or is it realistic to say that I can just add this 1 ms delay and then forget about it? i.e. Would I encounter no other issues in the future? I understand this is a very, very broad question, but I'm just looking for any info/experience you might have on this.

Best wishes,

Dan

[alvaroav]

Dear all,

Yesterday I updated the firmware of some units, in order to follow the supported firmware versions (previously I used the v1.20.0r3 legacy and I went directly to the latest v1.20.2r4 pybytes).
Then I debugged the code I have for the wipys, and everthing works except the _thread running in paralell to the main, in order to listen UDP sockets for start/stop data transmission. I was looking for info related to the changes and I found this post, which shows exactly the problem I had.
I have also applied to proposed solution, and this makes my code operational again, BUT I lost 10% of speed performance, since I have to place a time.sleep_ms(1) at the end of the while(True) of the main loop.
Does someone look for a better solution, or have clear idea of what was changed at _thread implementation?
Thanks to all!

[Pablo Woolvett]

Hi all,

We did not like the idea of sleeping, so we implemented an alternative workaround using two locks, which we believe it to be more robust.

import time
import _thread

class Thread:

    def init(self, target):
        self.result = None
        self.target = target
        self.lock = _thread.allocate_lock()
        self.aux_lock = _thread.allocate_lock()

    def real_call(self):
        with self.lock:
            self.aux_lock.release()
            self.result = self.target()

    def call(self):
        self.aux_lock.acquire()
        _thread.start_new_thread(self.real_call, ())
        with self.aux_lock:
            pass

    def join(self):
        with self.lock:
            pass

def demo():
    print("Starting")
    time.sleep(10)
    print("Ensing")
    return 42


def main():

    t = Thread(target = demo)
    t()
    t.join()
    print("Result: " + str(t.result))

if name == "main":
    main()