Guru Meditation Error - Pycom WiPy

[timh]

@martinnn I wonder if your guru meditation could be due to AWS MQTT lib (threads).

I found I had to make some fixes to that library. (I use the MQTTClient and MQTTMessageHandler classes to connect to a none AWS service that uses TLS.)

I found that there where problems,

for instance I found self.connect() is called in _verify_connection_state() but connect method isn't defined.

That in itself should cause a guru but the next problem could lead to it thread issues.

I found that if a disconnection occurs then underlying thread isn't being killed for the message handler.

In fact I think there is no mechanism in that code to explicitly disconnect reliably and re-connect as threads are left running over time. This causes a fault eventually.

I added connect and modified disconnect

def disconnect(self):
        if self._sock:
            self._sock.close()
            self._sock = None
        self._conn_state_mutex.acquire()
        self._exitRequest = True
        self._conn_state_mutex.release()

    def connect(self):
        self.disconnect()
        self._conn_state_mutex.acquire()
        self._exitRequest = False
        self._conn_state_mutex.release()
        self.createSocketConnection()

Added a flag in the handler to mark a request to close the thread and added a guard in the io_thread loop

if self._exitRequest:
     _thread.exit()
     raise mqttConst.MQTTDisconnected

This seems to have made the library more robust, at least for me.

There does still seem to be one other run away thread issue, but my WDT catches that one.

[Martinnn]

@timh I also suspect AWSIoT or threads. I'll look into your suggestions - thanks!

[Milan]

@robert-hh Unfortunately I can't share code with you, sorry. Firmware version is v1.18.1.r7. Also, where can I find .elf file from this version, I couldn't find it on GitHub?

@Martinnn @timh I'm not using AWSIoT or threads but same thing happens to me anyway. I hope we will get some help on this, device is so unreliable this way.

[robert-hh]

@milan You have to build you own firmware to get the .elf file. It will be in esp32/build/WIPY/release

[Milan]

@robert-hh Ok, but I can't find this release on https://github.com/pycom/pycom-micropython-sigfox/releases
Where could I found source for this release?

[robert-hh]

@milan The master branch at github https://github.com/pycom/pycom-micropython-sigfox always almost always reflects the latest build, which you can install with pyupgrade. The version number is in the file https://github.com/pycom/pycom-micropython-sigfox/blob/master/esp32/pycom_version.h.
I am not sure which commit to check out to get 1.18.1.r7. The log is not detailed in respect to that file.

[timh]

@timh been going over the AWS MQTT msgHandler further. I see a number of more bugs.

For instance if a the handler see's a disconnection, and (tries to reconnect - of course connect() is missing)
The poll registration on the original socket still exists. No poll registration is created when a new socket is created.

When I finish reviewing I will post separately a updated message handler.

Some of my earlier changes (above) don't make things worse but are incorrect/incomplete.

T

[Martinnn]

... and one more! Had five Wipys running over the weekend, one died after roughly 37000 sent messages to AWS:

Guru Meditation Error: Core 1 panic'ed (Unknown reason)

And that was it. I'd really appreciate a fix here.

[oligauc]

@timh When the handler sees a disconnection, _connect_helper() is called, which in turn calls connect in the mqtt client class.

The poll registration is done each time a new socket is created. Look at createSocketConnection

The firmware has an embedded garbage collector

[oligauc]

@timh self.connect is defined in the MQTTClient class and is passed to the MsgHandler class using self._msgHandler=msgHandler.MsgHandler(self._recv_callback, self.connect)

Your connect function creates a socket, but it does not send the connect packet to the aws server

[oligauc]

@milan Please can you post the output of:

import os
os.uname()

[Milan]

@oligauc Output is:

>>> os.uname()
(sysname='LoPy4', nodename='LoPy4', release='1.18.1.r7', version='v1.8.6-849-d1c5ea9 on 2018-12-17', machine='LoPy4 with ESP32', lorawan='1.0.2', sigfox='1.0.1')

[timh]

@oligauc Hmm

May be I am missing something but _verify_connection_state in MQTTMSgHandler has the code below .

self.connect() is in the scope of the MQTTMSgHandler instance - I can't see how self.connect() can be referring to the instance in MQTTClient class.

   def _verify_connection_state(self):
        elapsed = time.time() - self._start_time
        if not self._waiting_ping_resp and elapsed > self._ping_interval:
            if self._connection_state == mqttConst.STATE_CONNECTED:
                self._pingSent=False
                self._send_pingreq()
                self._waiting_ping_resp=True
            elif self._connection_state == mqttConst.STATE_DISCONNECTED:
                self.connect()

            self._start_time = time.time()
        elif self._waiting_ping_resp and (self._connection_state == mqttConst.STATE_CONNECTED or elapsed > self._mqttOperationTimeout):
            if not self._pingSent:
                if self._ping_failures <= self._ping_cutoff:
                    self._ping_failures+=1
                else:
                    self.connect()
            else:
                self._ping_failures=0

            self._start_time = time.time()
            self._waiting_ping_resp=False

[timh]

@oligauc said in Guru Meditation Error - Pycom WiPy:

_connect_helper()

Yes that is correct.

However you have a disconnect() method defined in MQTTMsgHandler class which has the following code.
if this is called then _sock is set to None, and will never be deregistered in the createSocketConnection method

def disconnect(self):
       if self._sock:
           self._sock.close()
           self._sock = None

[timh]

@timh May be these methods don't belong in the class, but given they don't have preceding _ one tends to assume they are public and are there to be used.

the disconnect() method in the MSGHandler class is called by disconnect() in the MQTTClient, so any explicit disconnect will end up with sock poll registrations not being cleaned up as per previous post.

[Martinnn]

two more dead over night.

Guru Meditation Error: Core  1 panic'ed (Unknown reason)

I wonder if I should dump the _thread implementation and make a callback for MQTT transmission. Currently in MQTTMsgHandler:

class MsgHandler:

    def __init__(self, receive_callback, connect_helper):
        ...
        _thread.stack_size(8192)
        _thread.start_new_thread(self._io_thread_func, ())

If I would call _io_thread_func (modified of course without while(1)) from my main loop, I might get nearly the same functionality without the guruous _thread.

[combaindeft]

Hi all,

I've run into the same "Guru Meditation Error" issue with our "PySense + FiPy" and "PyTrack + FiPy's" ...

Though the weirdest thing is, it wasn't running any code ... freshly firmware'd ... both boards to version 8 ... and the FiPy's to 1.18.r7 ...

We where doing some field tests ... and noticed we where close to a BIG Power Line ... so we choose to just drive a bit further away ... and wham ... it stopped!

Drove back towards the same spot ... and wham ... the board just ran in an endless loop "Guru Meditation Error" ...

Drove away again ... and wham ... it stopped!

So my question is, how sensitive to magnetic interference (EMI/RFI) are all of PyCom's products?? @robert-hh @Paul-Thornton

[oligauc]

@timh Please see here:

https://github.com/pycom/pycom-micropython-sigfox/blob/master/esp32/frozen/Common/MQTTMsgHandler.py

[Paul Thornton]

@combaindeft It certainly not something I see reported. Ill ask internally to see if anything more is known.

[combaindeft]

@paul-thornton

Thanks!

I have btw started a different topic for EMI/RFI Guru-Core-Dump-loop error

https://forum.pycom.io/topic/4342/guru-meditation-error-magnetic-interference-emi-rfi