Guru Meditation Error - Pycom WiPy

[Milan]

Hi,

I've got Guru Meditation Error when running my Pycom code.
Below is what I've received on log:

Guru Meditation Error: Core  1 panic'ed (LoadProhibited)
. Exception was unhandled.
Register dump:
PC      : 0x4008ccdf  PS      : 0x00060033  A0      : 0x8008b7a5  A1      : 0x3ffc1370
A2      : 0x3ffd455c  A3      : 0x00060023  A4      : 0x3ffc52a4  A5      : 0x00000001
A6      : 0x000000fe  A7      : 0x00000008  A8      : 0x00000000  A9      : 0x3ffd455c
A10     : 0x3ffd455c  A11     : 0x00000000  A12     : 0x800fc98d  A13     : 0x3ffd9490
A14     : 0x3ffc5fe4  A15     : 0x3f409f14  SAR     : 0x00000003  EXCCAUSE: 0x0000001c
EXCVADDR: 0x00000004  LBEG    : 0x400e675c  LEND    : 0x400e6798  LCOUNT  : 0x00000000

Backtrace: 0x4008ccdf:0x3ffc1370 0x4008b7a2:0x3ffc1390 0x4008a3e9:0x3ffc13b0 0x40086319:0x3ffc13d0 0x40083519:0x3ffc1410 0x4000bfed:0x00000000

================= CORE DUMP START =================
.
.
.
================= CORE DUMP END =================
E (24354) esp_core_dump: Skipped 1 tasks with bad TCB!
E (24359) esp_core_dump: Crashed task has been skipped!
Rebooting...

Can someone tell me more information about this error?

[andreas]

Dear @Milan and @Martinnn,

if you feel lucky, you might want to try our custom build just released on [1]. More background about this is available through [2].

Please be aware that you will have to erase your device completely before flashing in order to keep things straight. You will find respective references to this on the forum. Hint: Use pycom-fwtool-cli --port /dev/ttyUSB0 erase_all, see also [3].

With kind regards,
Andreas.

[1] https://packages.hiveeyes.org/hiveeyes/foss/pycom/vanilla/WiPy-1.20.1.r1-0.6.0-vanilla-dragonfly.tar.gz
[2] https://community.hiveeyes.org/t/investigating-random-core-panics-on-pycom-esp32-devices/2480
[3] https://community.hiveeyes.org/t/installing-the-recent-pycom-firmware-1-20-1-r1-requires-erasing-the-flash-memory-completely/2688

[Martinnn]

Last few days: 2 of 6 dead with power only connection, also 2 of 6 dead with USB. So my previous assumption that the crashes are related to the USB cable doesn't hold.
I have to say if this wouldn't be a simple pretty stateless (serial in, MQTT out) sensor application (where I don't mind the sledgehammer approach of an external watchdog reboot), I'd be getting nervous by now and having second thoughts whether these boards were the right choice.

[Martinnn]

@paul-thornton The last two crashes were around 14 and 19 UTC. Doesn't look like it...

[Milan]

@martinnn I've to correct myself. I don't use USB, my pycom is directly powered to Vin Pin.

[Paul Thornton]

@Martinnn Dont suppose your logging lets you know if the boards all fail at the same time during the evening. or are the failures spread across the night?

[Martinnn]

@milan said in Guru Meditation Error - Pycom WiPy:

@martinnn My pycom devices is also connected via USB. I will try some scenario using battery connector. Thank you.

The reason I think that is not problem in my case is that error hapends on same line (part) of the code. In case of USB connection error I expect more randomness in error scenario.

I agree. But maybe it's just that your code spends most of the time doing SSL encryption (or wherever your crashes happen) so it's statistical. Or maybe the crypto module is as high quality as the ADC?
A true debugger with "catch unhandled exception" like the Segger J-Link would be nice. I wonder if something like this exists or everyone just does printf debugging? At least it seems possible https://www.esp32.com/viewtopic.php?t=7097

[Martinnn]

@paul-thornton I have five devices connected to a powered hub with external 5 V 2 A supply. I also have one that is directly connected to a laptop socket. So I'd say all of those should have enough power. All of those crash sooner or later.
Also I have five devices connected to a lab supply at 4.2 V, drawing 0.55 A, so the 2 A of the USB hub should be plenty.
I also unplugged the USB hub from my laptop so that there was no communication from the computer to the devices (just power from the external supply), still fails (today 2 out of 5 dead).
What does the controller (PIC) on the pysense do? Any chance it generates some glitches on supply/reset when connected to USB?

[Paul Thornton]

@martinnn What Amperage is the usb power supply your using rated for? I have seen a few situations where at peak power consumption it'll consume out more than a normal old style USB-A socket can put out (this was on an older laptop, however). causing these issues. Would explain why using a LiPo battery with presumably higher max amperage could fix the issues.

[Milan]

@martinnn My pycom devices is also connected via USB. I will try some scenario using battery connector. Thank you.

The reason I think that is not problem in my case is that error hapends on same line (part) of the code. In case of USB connection error I expect more randomness in error scenario.

[Martinnn]

@milan I found that my devices (pysense+wipy) work fine when just powered by the battery connector and fail quickly when connected via USB. I also suspected MQTT, HTTPS, SSL first, but in my case it looks more like an issue related to the USB connection.

[Milan]

@oligauc I can't send your my code, sorry. Important thing is that this error occur when pycom is sending data to Azure IoTHub using HTTPS or MQTTS connections. Pycom is connected to WiFi. I'm suspecting that something is wrong with socket connections.

When I'm sending data every minute to server, error appears few times a day. It is tested with 2 devices and 2 different codes. Common thing is socket connection, and that error always appear after that.

I'm not using PWM and I haven't modified low level c firmware.

[oligauc]

@milan Please can you send your code to: support@pycom.io

Are you using PWM ?

Did you modify the low level c firmware ?

[combaindeft]

@paul-thornton

Thanks!

I have btw started a different topic for EMI/RFI Guru-Core-Dump-loop error

https://forum.pycom.io/topic/4342/guru-meditation-error-magnetic-interference-emi-rfi

[Paul Thornton]

@combaindeft It certainly not something I see reported. Ill ask internally to see if anything more is known.

[oligauc]

@timh Please see here:

https://github.com/pycom/pycom-micropython-sigfox/blob/master/esp32/frozen/Common/MQTTMsgHandler.py

[combaindeft]

Hi all,

I've run into the same "Guru Meditation Error" issue with our "PySense + FiPy" and "PyTrack + FiPy's" ...

Though the weirdest thing is, it wasn't running any code ... freshly firmware'd ... both boards to version 8 ... and the FiPy's to 1.18.r7 ...

We where doing some field tests ... and noticed we where close to a BIG Power Line ... so we choose to just drive a bit further away ... and wham ... it stopped!

Drove back towards the same spot ... and wham ... the board just ran in an endless loop "Guru Meditation Error" ...

Drove away again ... and wham ... it stopped!

So my question is, how sensitive to magnetic interference (EMI/RFI) are all of PyCom's products?? @robert-hh @Paul-Thornton

[Martinnn]

two more dead over night.

Guru Meditation Error: Core  1 panic'ed (Unknown reason)

I wonder if I should dump the _thread implementation and make a callback for MQTT transmission. Currently in MQTTMsgHandler:

class MsgHandler:

    def __init__(self, receive_callback, connect_helper):
        ...
        _thread.stack_size(8192)
        _thread.start_new_thread(self._io_thread_func, ())

If I would call _io_thread_func (modified of course without while(1)) from my main loop, I might get nearly the same functionality without the guruous _thread.

[timh]

@timh May be these methods don't belong in the class, but given they don't have preceding _ one tends to assume they are public and are there to be used.

the disconnect() method in the MSGHandler class is called by disconnect() in the MQTTClient, so any explicit disconnect will end up with sock poll registrations not being cleaned up as per previous post.

[timh]

@oligauc said in Guru Meditation Error - Pycom WiPy:

_connect_helper()

Yes that is correct.

However you have a disconnect() method defined in MQTTMsgHandler class which has the following code.
if this is called then _sock is set to None, and will never be deregistered in the createSocketConnection method

def disconnect(self):
       if self._sock:
           self._sock.close()
           self._sock = None