Random Guru meditation Wipy3-1.20.2.r4

[serafimsaudade]

Hi,
I'm having random Guru meditation.
I'm using a WiPy3 with custom hardware and the release v1.20.2.r4 from git with custom frozen files.
I find this thread on esp32 forum.

I was able to log one core dump, and decoding It.

Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC      : 0x40092ffb  PS      : 0x00060e33  A0      : 0x80091523  A1      : 0x3ffdebe0  
A2      : 0x3ffdfd1c  A3      : 0x00060e20  A4      : 0x3ffdcbc8  A5      : 0x00000001  
A6      : 0x000000fe  A7      : 0x00000001  A8      : 0x00000000  A9      : 0x3ffc3fa0  
A10     : 0x3ffc3fa0  A11     : 0x0000abab  A12     : 0x3ffdcbc8  A13     : 0x00000001  
A14     : 0x000000fe  A15     : 0x3ffdfd14  SAR     : 0x00000000  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00000004  LBEG    : 0x40094354  LEND    : 0x40094382  LCOUNT  : 0xffffffff  

ELF file SHA256: 0000000000000000000000000000000000000000000000000000000000000000

Backtrace: 0x40092ffb:0x3ffdebe0 0x40091520:0x3ffdec00 0x400f372c:0x3ffdec20 0x400e045d:0x3ffdecb0

================= CORE DUMP START =================

Decode:

xtensa-esp32-elf-addr2line -fe build/WIPY/release/application.elf 0x40092ffb:0x3ffdebe0 0x40091520:0x3ffdec00 0x400f372c:0x3ffdec20 0x400e045d:0x3ffdecb0
uxListRemove
/home/luis/Desktop/Lumitio/software/pycom-esp-idf/components/freertos/list.c:218
vTaskDelete
/home/luis/Desktop/Lumitio/software/pycom-esp-idf/components/freertos/tasks.c:5063
TASK_Interrupts
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/util/mpirq.c:72
freertos_entry
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/mpthreadport.c:140

Update:
Using ExpansionBoard v2.1A with Gpy, fw release v1.20.2.r4 from git with custom frozen files. I also have the same Guru meditation.

Best regards,

[Gijs]

Yes, as @robert-hh suggested, I'm very much looking for a reproduction, or a series of things I can do that make this coredump happen (like running this micropython code in combination with pressing button x). The functions calls listed in this coredump decode happen quite a lot all over the firmware, making me think it could be related to running out of memory, or a memory leak somewhere, but again, it depends on the location and order of your function calls.

Also, quick side note, you mention in your previous post you're using a Gpy, but have the firmware.elf path going through WIPY, is that intentional? It is possible to run Wipy firmware on a Gpy, (disabling the usage of the LTE modem). If its not intentional. it might show us the wrong function calls here.

Let me know

[robert-hh]

@serafimsaudade What @Gijs probably asked is, if you can tell what the code is doing at the moment of the core dump. The decoding of the core dump does not look strange. The code seems to deal with function arguments down in a series of function calls.

[serafimsaudade]

@Gijs I'm being developing for the last year a device can read multiple sensors. And also local webserver for configuration. The sofware is custom and save into frozen folder to be compile.
So far I have install +- 100 devices. Normally they are install and forget. But now some clients want to interact with the device locally.
And we start receiving some complains, about random reboots.

After some testing we also see some random coredumps. We test with different fw releases.
The felling we get is that the device do a coredump when is in heavy load, like accessing to websever over wifi_AP.

Update:
New coredump

Guru Meditation Error: Core  1 panic'ed (StoreProhibited). Exception was unhandled.
Core 1 register dump:
PC      : 0x4010238c  PS      : 0x00060530  A0      : 0x8010273c  A1      : 0x3ffd8c40  
A2      : 0x00000004  A3      : 0x3ffd8c60  A4      : 0x3f9c8000  A5      : 0x3f4736d0  
A6      : 0x3f450d0d  A7      : 0x00000f02  A8      : 0x00000000  A9      : 0x3ffd8c20  
A10     : 0x3f9c7ff0  A11     : 0x00000005  A12     : 0x00000000  A13     : 0x3f40c13c  
A14     : 0x3f40c13c  A15     : 0x3f40ad00  SAR     : 0x00000004  EXCCAUSE: 0x0000001d  
EXCVADDR: 0x00000004  LBEG    : 0x400f66fc  LEND    : 0x400f6727  LCOUNT  : 0x0001f80d  

ELF file SHA256: 0000000000000000000000000000000000000000000000000000000000000000

Backtrace: 0x4010238c:0x3ffd8c40 0x40102739:0x3ffd8c60 0x40104a72:0x3ffd8ca0 0x400fce69:0x3ffd8cc0 0x4010833d:0x3ffd8ce0 0x40100710:0x3ffd8d80 0x400fce69:0x3ffd8df0 0x400fceed:0x3ffd8e10 0x401083c3:0x3ffd8e30 0x40100710:0x3ffd8ed0 0x400fce69:0x3ffd8f00 0x400fceed:0x3ffd8f20 0x401083c3:0x3ffd8f40 0x40100710:0x3ffd8fe0 0x400fce69:0x3ffd9010 0x400fceed:0x3ffd9030 0x401083c3:0x3ffd9050 0x40100710:0x3ffd90f0 0x400fce69:0x3ffd9120 0x400fceed:0x3ffd9140 0x401083c3:0x3ffd9160 0x40100710:0x3ffd9200 0x400fce69:0x3ffd9230 0x400fce96:0x3ffd9250 0x400e0f1a:0x3ffd9270 0x400e11c1:0x3ffd9310 0x400dfe98:0x3ffd9340

Decode:

xtensa-esp32-elf-addr2line -fe build/WIPY/release/application.elf 0x4010238c:0x3ffd8c40 0x40102739:0x3ffd8c60 0x40104a72:0x3ffd8ca0 0x400fce69:0x3ffd8cc0 0x4010833d:0x3ffd8ce0 0x40100710:0x3ffd8d80 0x400fce69:0x3ffd8df0 0x400fceed:0x3ffd8e10 0x401083c3:0x3ffd8e30 0x40100710:0x3ffd8ed0 0x400fce69:0x3ffd8f00 0x400fceed:0x3ffd8f20 0x401083c3:0x3ffd8f40 0x40100710:0x3ffd8fe0 0x400fce69:0x3ffd9010 0x400fceed:0x3ffd9030 0x401083c3:0x3ffd9050 0x40100710:0x3ffd90f0 0x400fce69:0x3ffd9120 0x400fceed:0x3ffd9140 0x401083c3:0x3ffd9160 0x40100710:0x3ffd9200 0x400fce69:0x3ffd9230 0x400fce96:0x3ffd9250 0x400e0f1a:0x3ffd9270 0x400e11c1:0x3ffd9310 0x400dfe98:0x3ffd9340
mp_obj_new_str_from_vstr
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/objstr.c:2082
mp_obj_str_make_new
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/objstr.c:2082
type_call
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/objtype.c:997
mp_call_function_n_kw
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/runtime.c:624
mp_execute_bytecode
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/vm.c:919
fun_bc_call
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/objfun.c:287
mp_call_function_n_kw
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/runtime.c:624
mp_call_method_n_kw
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/runtime.c:640
mp_execute_bytecode
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/vm.c:1002
fun_bc_call
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/objfun.c:287
mp_call_function_n_kw
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/runtime.c:624
mp_call_method_n_kw
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/runtime.c:640
mp_execute_bytecode
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/vm.c:1002
fun_bc_call
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/objfun.c:287
mp_call_function_n_kw
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/runtime.c:624
mp_call_method_n_kw
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/runtime.c:640
mp_execute_bytecode
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/vm.c:1002
fun_bc_call
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/objfun.c:287
mp_call_function_n_kw
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/runtime.c:624
mp_call_method_n_kw
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/runtime.c:640
mp_execute_bytecode
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/vm.c:1002
fun_bc_call
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/objfun.c:287
mp_call_function_n_kw
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/runtime.c:624
mp_call_function_0
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/runtime.c:598
parse_compile_execute
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../lib/utils/pyexec.c:601
pyexec_frozen_module
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../lib/utils/pyexec.c:588
TASK_Micropython
/home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/mptask.c:328

Best Regards,

[Gijs]

Nice work decoding the coredump! Can you also describe what you are trying to achieve, and where it breaks? That would really help us to find the cause of the coredump!
Gijs