Guru Meditation Error - Pycom WiPy

[combaindeft]

Hi all,

I've run into the same "Guru Meditation Error" issue with our "PySense + FiPy" and "PyTrack + FiPy's" ...

Though the weirdest thing is, it wasn't running any code ... freshly firmware'd ... both boards to version 8 ... and the FiPy's to 1.18.r7 ...

We where doing some field tests ... and noticed we where close to a BIG Power Line ... so we choose to just drive a bit further away ... and wham ... it stopped!

Drove back towards the same spot ... and wham ... the board just ran in an endless loop "Guru Meditation Error" ...

Drove away again ... and wham ... it stopped!

So my question is, how sensitive to magnetic interference (EMI/RFI) are all of PyCom's products?? @robert-hh @Paul-Thornton

[oligauc]

@timh Please see here:

https://github.com/pycom/pycom-micropython-sigfox/blob/master/esp32/frozen/Common/MQTTMsgHandler.py

[Paul Thornton]

@combaindeft It certainly not something I see reported. Ill ask internally to see if anything more is known.

[combaindeft]

@paul-thornton

Thanks!

I have btw started a different topic for EMI/RFI Guru-Core-Dump-loop error

https://forum.pycom.io/topic/4342/guru-meditation-error-magnetic-interference-emi-rfi

[oligauc]

@milan Please can you send your code to: support@pycom.io

Are you using PWM ?

Did you modify the low level c firmware ?

[Milan]

@oligauc I can't send your my code, sorry. Important thing is that this error occur when pycom is sending data to Azure IoTHub using HTTPS or MQTTS connections. Pycom is connected to WiFi. I'm suspecting that something is wrong with socket connections.

When I'm sending data every minute to server, error appears few times a day. It is tested with 2 devices and 2 different codes. Common thing is socket connection, and that error always appear after that.

I'm not using PWM and I haven't modified low level c firmware.

[Martinnn]

@milan I found that my devices (pysense+wipy) work fine when just powered by the battery connector and fail quickly when connected via USB. I also suspected MQTT, HTTPS, SSL first, but in my case it looks more like an issue related to the USB connection.

[Milan]

@martinnn My pycom devices is also connected via USB. I will try some scenario using battery connector. Thank you.

The reason I think that is not problem in my case is that error hapends on same line (part) of the code. In case of USB connection error I expect more randomness in error scenario.

[Paul Thornton]

@martinnn What Amperage is the usb power supply your using rated for? I have seen a few situations where at peak power consumption it'll consume out more than a normal old style USB-A socket can put out (this was on an older laptop, however). causing these issues. Would explain why using a LiPo battery with presumably higher max amperage could fix the issues.

[Martinnn]

@paul-thornton I have five devices connected to a powered hub with external 5 V 2 A supply. I also have one that is directly connected to a laptop socket. So I'd say all of those should have enough power. All of those crash sooner or later.
Also I have five devices connected to a lab supply at 4.2 V, drawing 0.55 A, so the 2 A of the USB hub should be plenty.
I also unplugged the USB hub from my laptop so that there was no communication from the computer to the devices (just power from the external supply), still fails (today 2 out of 5 dead).
What does the controller (PIC) on the pysense do? Any chance it generates some glitches on supply/reset when connected to USB?

[Martinnn]

@milan said in Guru Meditation Error - Pycom WiPy:

@martinnn My pycom devices is also connected via USB. I will try some scenario using battery connector. Thank you.

The reason I think that is not problem in my case is that error hapends on same line (part) of the code. In case of USB connection error I expect more randomness in error scenario.

I agree. But maybe it's just that your code spends most of the time doing SSL encryption (or wherever your crashes happen) so it's statistical. Or maybe the crypto module is as high quality as the ADC?
A true debugger with "catch unhandled exception" like the Segger J-Link would be nice. I wonder if something like this exists or everyone just does printf debugging? At least it seems possible https://www.esp32.com/viewtopic.php?t=7097

[Paul Thornton]

@Martinnn Dont suppose your logging lets you know if the boards all fail at the same time during the evening. or are the failures spread across the night?

[Milan]

@martinnn I've to correct myself. I don't use USB, my pycom is directly powered to Vin Pin.

[Martinnn]

@paul-thornton The last two crashes were around 14 and 19 UTC. Doesn't look like it...

[Martinnn]

Last few days: 2 of 6 dead with power only connection, also 2 of 6 dead with USB. So my previous assumption that the crashes are related to the USB cable doesn't hold.
I have to say if this wouldn't be a simple pretty stateless (serial in, MQTT out) sensor application (where I don't mind the sledgehammer approach of an external watchdog reboot), I'd be getting nervous by now and having second thoughts whether these boards were the right choice.

[andreas]

Dear @Milan and @Martinnn,

if you feel lucky, you might want to try our custom build just released on [1]. More background about this is available through [2].

Please be aware that you will have to erase your device completely before flashing in order to keep things straight. You will find respective references to this on the forum. Hint: Use pycom-fwtool-cli --port /dev/ttyUSB0 erase_all, see also [3].

With kind regards,
Andreas.

[1] https://packages.hiveeyes.org/hiveeyes/foss/pycom/vanilla/WiPy-1.20.1.r1-0.6.0-vanilla-dragonfly.tar.gz
[2] https://community.hiveeyes.org/t/investigating-random-core-panics-on-pycom-esp32-devices/2480
[3] https://community.hiveeyes.org/t/installing-the-recent-pycom-firmware-1-20-1-r1-requires-erasing-the-flash-memory-completely/2688