Random Core Panic with Lora Socket send/recv, device doesnt restart - try with CONFIG_ESP32_PANIC_SILENT_REBOOT



  • Dear Pycom Forum

    We are running Pycom Firmware LoPy4-1.20.1.r1 with frozen modules.
    Randomly our Device crashes with attached core panic when multiple threats are using the lora socket send/recv.
    Apart from the crash, it would be no problem, if the device would restart. But it doesn't.
    All threats and actions are locked.
    Only manually taking of the power helps.

    I read that core panics should always restart. per default following option is true.
    CONFIG_ESP32_PANIC_PRINT_REBOOT

    I was thinking that because the core panic print breaks somehow in the middle, the esp32 can't recover, and in the core panic handler is a problem, which cause that reboot part is not triggered.

    I had the idea to disable the print, that the reboot is directly triggered by disabling CONFIG_ESP32_PANIC_PRINT_REBOOT
    but enabling

    CONFIG_ESP32_PANIC_SILENT_REBOOT
    

    After searching around I changed the values with no effect in
    pycom-micropython-sigfox/esp32/sdkconfig.h

    after I found out that the
    pycom-micropython-sigfox is using the pycom-esp-idf/wifi/scan reference.
    So I changed there the sdkconfig and runned "make all"

    When building the firmware with, also no change. Core panic is still printed.

    make BOARD=LOPY4 FS=LFS release
    

    I saw that the .a reference files are not updated in
    pycom-micropython-sigfox/esp32/

    when trying to run the copy over script get_idf_libs.py , i get a path missmatch.

    ascals-MacBook-Pro:esp32 pascalschaefer$ python get_idf_libs.py --idflibs ~/development/pycom-esp-idf/examples/wifi/scan/build
    Couldn't Copy IDF libs defaulting to Local Lib Folders!
    Traceback (most recent call last):
      File "get_idf_libs.py", line 26, in main
        shutil.copy(src + '/bootloader/bootloader_support/libbootloader_support.a', dsttmpbl)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py", line 245, in copy
        copyfile(src, dst, follow_symlinks=follow_symlinks)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py", line 120, in copyfile
        with open(src, 'rb') as fsrc:
    FileNotFoundError: [Errno 2] No such file or directory: '/Users/pascalschaefer/development/pycom-esp-idf/examples/wifi/scan/build/bootloader/bootloader_support/libbootloader_support.a'
    

    The File is actually in

    /wifi/scan/build/bootloader_support/libbootloader_support.a
    

    I am a bit confused.
    Does somebody know how it is intended to copy over the full modified esp32 build into the pycom folder. that the following esp32 configuration value changes, and maybe the device restarts ? Does the sdkconfig.h also needs to be changed?

    CONFIG_ESP32_PANIC_SILENT_REBOOT
    

    Many thanks in advance

    TRACE [LORA] > sending 7 bytes: b'040200188afb5d'
    Guru Meditation Error: Core  1 panic'ed (Cache disabled but cached memory region accessed)
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    ....
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    .
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core  1 panic'ed (IllegalInstruction). Exception was unhandled.
    Memory dump at 0x40200744: bad00bad bad00bad bad00bad
    Guru Meditation Error: Core�```


  • Dear community,

    updating to the recent 1.20.2.rc3 release seems to help here, see also [1].

    @ps47 said:

    I tried the new build LoPy4-1.20.2.rc3-0.8.0-vanilla-squirrel.tar.gz and not any “bad00bad bad00bad bad00bad” Core Panic happened till now.

    Thanks @husigeza and the whole Pycom team for mitigating this issue!

    With kind regards,
    Andreas.

    [1] https://community.hiveeyes.org/t/investigating-core-panics-on-the-lopy4/2878/29



  • Hi there,

    we just wanted to drop a note here that we also started investigating these core panics on [1]. It might help others coming along here to follow the conversation.

    With kind regards,
    Andreas.

    [1] https://community.hiveeyes.org/t/investigating-core-panics-on-the-lopy4/2878



  • I have found out that following make addition would do it.

    COPY_IDF_LIB=1
    

    but still, when I have build the "pycom-esp-idf/example/wifi/scan" following files i don't have in the build directory of pycom-esp-idf/example/wifi/scan/build and they exists but without the '/bootloader' praefix.

        shutil.copy(src + '/bootloader/bootloader_support/libbootloader_support.a', dst)
        shutil.copy(src + '/bootloader/log/liblog.a', dst)
        shutil.copy(src + '/bootloader/micro-ecc/libmicro-ecc.a', dst)
        shutil.copy(src + '/bootloader/soc/libsoc.a', dst)
        shutil.copy(src + '/bootloader/spi_flash/libspi_flash.a', dst)
    

Log in to reply
 

Pycom on Twitter