Guru Meditation Error - Pycom WiPy

[Martinnn]

... and one more! Had five Wipys running over the weekend, one died after roughly 37000 sent messages to AWS:

Guru Meditation Error: Core 1 panic'ed (Unknown reason)

And that was it. I'd really appreciate a fix here.

[oligauc]

@timh When the handler sees a disconnection, _connect_helper() is called, which in turn calls connect in the mqtt client class.

The poll registration is done each time a new socket is created. Look at createSocketConnection

The firmware has an embedded garbage collector

[oligauc]

@timh self.connect is defined in the MQTTClient class and is passed to the MsgHandler class using self._msgHandler=msgHandler.MsgHandler(self._recv_callback, self.connect)

Your connect function creates a socket, but it does not send the connect packet to the aws server

[oligauc]

@milan Please can you post the output of:

import os
os.uname()

[Milan]

@oligauc Output is:

>>> os.uname()
(sysname='LoPy4', nodename='LoPy4', release='1.18.1.r7', version='v1.8.6-849-d1c5ea9 on 2018-12-17', machine='LoPy4 with ESP32', lorawan='1.0.2', sigfox='1.0.1')

[timh]

@oligauc Hmm

May be I am missing something but _verify_connection_state in MQTTMSgHandler has the code below .

self.connect() is in the scope of the MQTTMSgHandler instance - I can't see how self.connect() can be referring to the instance in MQTTClient class.

   def _verify_connection_state(self):
        elapsed = time.time() - self._start_time
        if not self._waiting_ping_resp and elapsed > self._ping_interval:
            if self._connection_state == mqttConst.STATE_CONNECTED:
                self._pingSent=False
                self._send_pingreq()
                self._waiting_ping_resp=True
            elif self._connection_state == mqttConst.STATE_DISCONNECTED:
                self.connect()

            self._start_time = time.time()
        elif self._waiting_ping_resp and (self._connection_state == mqttConst.STATE_CONNECTED or elapsed > self._mqttOperationTimeout):
            if not self._pingSent:
                if self._ping_failures <= self._ping_cutoff:
                    self._ping_failures+=1
                else:
                    self.connect()
            else:
                self._ping_failures=0

            self._start_time = time.time()
            self._waiting_ping_resp=False

[timh]

@oligauc said in Guru Meditation Error - Pycom WiPy:

_connect_helper()

Yes that is correct.

However you have a disconnect() method defined in MQTTMsgHandler class which has the following code.
if this is called then _sock is set to None, and will never be deregistered in the createSocketConnection method

def disconnect(self):
       if self._sock:
           self._sock.close()
           self._sock = None

[timh]

@timh May be these methods don't belong in the class, but given they don't have preceding _ one tends to assume they are public and are there to be used.

the disconnect() method in the MSGHandler class is called by disconnect() in the MQTTClient, so any explicit disconnect will end up with sock poll registrations not being cleaned up as per previous post.

[Martinnn]

two more dead over night.

Guru Meditation Error: Core  1 panic'ed (Unknown reason)

I wonder if I should dump the _thread implementation and make a callback for MQTT transmission. Currently in MQTTMsgHandler:

class MsgHandler:

    def __init__(self, receive_callback, connect_helper):
        ...
        _thread.stack_size(8192)
        _thread.start_new_thread(self._io_thread_func, ())

If I would call _io_thread_func (modified of course without while(1)) from my main loop, I might get nearly the same functionality without the guruous _thread.

[combaindeft]

Hi all,

I've run into the same "Guru Meditation Error" issue with our "PySense + FiPy" and "PyTrack + FiPy's" ...

Though the weirdest thing is, it wasn't running any code ... freshly firmware'd ... both boards to version 8 ... and the FiPy's to 1.18.r7 ...

We where doing some field tests ... and noticed we where close to a BIG Power Line ... so we choose to just drive a bit further away ... and wham ... it stopped!

Drove back towards the same spot ... and wham ... the board just ran in an endless loop "Guru Meditation Error" ...

Drove away again ... and wham ... it stopped!

So my question is, how sensitive to magnetic interference (EMI/RFI) are all of PyCom's products?? @robert-hh @Paul-Thornton

[oligauc]

@timh Please see here:

https://github.com/pycom/pycom-micropython-sigfox/blob/master/esp32/frozen/Common/MQTTMsgHandler.py

[Paul Thornton]

@combaindeft It certainly not something I see reported. Ill ask internally to see if anything more is known.

[combaindeft]

@paul-thornton

Thanks!

I have btw started a different topic for EMI/RFI Guru-Core-Dump-loop error

https://forum.pycom.io/topic/4342/guru-meditation-error-magnetic-interference-emi-rfi

[oligauc]

@milan Please can you send your code to: support@pycom.io

Are you using PWM ?

Did you modify the low level c firmware ?

[Milan]

@oligauc I can't send your my code, sorry. Important thing is that this error occur when pycom is sending data to Azure IoTHub using HTTPS or MQTTS connections. Pycom is connected to WiFi. I'm suspecting that something is wrong with socket connections.

When I'm sending data every minute to server, error appears few times a day. It is tested with 2 devices and 2 different codes. Common thing is socket connection, and that error always appear after that.

I'm not using PWM and I haven't modified low level c firmware.

[Martinnn]

@milan I found that my devices (pysense+wipy) work fine when just powered by the battery connector and fail quickly when connected via USB. I also suspected MQTT, HTTPS, SSL first, but in my case it looks more like an issue related to the USB connection.

[Milan]

@martinnn My pycom devices is also connected via USB. I will try some scenario using battery connector. Thank you.

The reason I think that is not problem in my case is that error hapends on same line (part) of the code. In case of USB connection error I expect more randomness in error scenario.

[Paul Thornton]

@martinnn What Amperage is the usb power supply your using rated for? I have seen a few situations where at peak power consumption it'll consume out more than a normal old style USB-A socket can put out (this was on an older laptop, however). causing these issues. Would explain why using a LiPo battery with presumably higher max amperage could fix the issues.

[Martinnn]

@paul-thornton I have five devices connected to a powered hub with external 5 V 2 A supply. I also have one that is directly connected to a laptop socket. So I'd say all of those should have enough power. All of those crash sooner or later.
Also I have five devices connected to a lab supply at 4.2 V, drawing 0.55 A, so the 2 A of the USB hub should be plenty.
I also unplugged the USB hub from my laptop so that there was no communication from the computer to the devices (just power from the external supply), still fails (today 2 out of 5 dead).
What does the controller (PIC) on the pysense do? Any chance it generates some glitches on supply/reset when connected to USB?

[Martinnn]

@milan said in Guru Meditation Error - Pycom WiPy:

@martinnn My pycom devices is also connected via USB. I will try some scenario using battery connector. Thank you.

The reason I think that is not problem in my case is that error hapends on same line (part) of the code. In case of USB connection error I expect more randomness in error scenario.

I agree. But maybe it's just that your code spends most of the time doing SSL encryption (or wherever your crashes happen) so it's statistical. Or maybe the crypto module is as high quality as the ADC?
A true debugger with "catch unhandled exception" like the Segger J-Link would be nice. I wonder if something like this exists or everyone just does printf debugging? At least it seems possible https://www.esp32.com/viewtopic.php?t=7097