Guru meditation revisted



  • I've been relatively free of guru meditation these errors these last 12 months. On the odd occasion where a guru meditation error occurs I've found a

    pycom-fwtool-cli.exe -p COM3 erase_all
    

    via the windows cmd line usually fixes the problem.

    Recently I erased & reflashed a brand new gpy, tested it several times to check boot.py was free of core panics & then wandered off to do other things, leaving the gpy in the familiar 'no user code' state where it flashes blue on the rgb led every 4s. After a day I went to back to the gpy. To my great surprise it had had 2 core panics (one 'illegalInstruction' & one 'LoadProhibited') in the 24 hours it had been just sitting there flashing the led.

    I'm not sure what to make of this. I guess I figured the frozen byte code that flashes the led was too simple to core panic. If I do a pycom.heartbeat(False) should that make core panics impossible or could there still be some other code running?



  • @peterp Swapped to a different gpy & problem gone so I'm assuming the other gpy had dodgy flash out-of-the box. As soon as dev board is free I'll plug in the dodgy & supply the requested. Maybe I'll get another rare 'double exception' error. I have to say I'm sceptical any of this will lead anywhere given the random nature of the errors it throws but always happy to proven wrong.



  • @kjm please share more details when reporting issues. Just saying there is a "guru meditation error" only tells us that there is a problem, but there is practically zero chance to help with it, because of the lack of details. It's like saying "my windows machine had a blue screen". Well, it's a problem alright, but not nearly enough detail to do anything about it.

    Also saying 'LoadProhibited', or 'illegalInstruction' is not enough. At least copy paste the core dump, as serafimsaudade did above. Also specify which firmware version.

    I tried to follow your reproduction as far as I understand it:

    • GPy
    • erase_all
    • flash v1.20.2.r4
    • let it sit and blink and capture the log with putty to see if it core dumps.

    -> so far it is 46 hours, no core dump

    Is it still happening for you? Similar rate, like 2 in 24h? Can you confirm that my reproduction above is what you do? You also say something about boot.py and frozen code - note that there is nothing about this in the repro I have written above, so please clarify if there are any steps missing. Is it one particular GPy or can you reproduce on multiple different modules? Also please paste the output of

            print('heartbeat', pycom.heartbeat_on_boot())
            print('lte', pycom.lte_modem_en_on_boot())
            print('pybytes', pycom.pybytes_on_boot())
            print('smart_config', pycom.smart_config_on_boot())
            print('wdt', pycom.wdt_on_boot())
            print('wifi', pycom.wifi_on_boot())
    

    Update: 94h



  • @Gijs By function call you mean stuff like 'store prohibited', 'illegalinstruction', 'LoadProhibited', etc? They seem to be random.



  • Hi,

    @kjm You mention just leaving an empty wipy with heartbeat enabled (&disabled?) running idle for a prolonged period of time. I will try to reproduce the case here, as the random coredumps you experience should not happen. Do you get similar function calls every time or no?

    @serafimsaudade This seems to be a different issue. I think your issue will be solved by adding time.sleep(1) in the while loop? It does not seem like a normal use case to me, and it could be related to there being no downtime for the microcontroller to do its garbage collecting etc. Let me know if that helps



  • I wish I knew how core dumps are supposed to help with the problem. This gpy continues to crash just sitting here not running any user code, I even disabled the pycom heartbeat.



  • Hi,

    I'm also having random coredumps.

    I was able to log one, and decoding.

    xtensa-esp32-elf-addr2line -fe build/WIPY/release/application.elf 0x40092ffb:0x3ffdec60 0x40091520:0x3ffdec80 0x400f372c:0x3ffdeca0 0x400e045d:0x3ffded30 
    
    uxListRemove 
    
    /home/luis/Desktop/Lumitio/software/pycom-esp-idf/components/freertos/list.c:218 
    
    vTaskDelete 
    
    /home/luis/Desktop/Lumitio/software/pycom-esp-idf/components/freertos/tasks.c:5063 
    
    TASK_Interrupts 
    
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/util/mpirq.c:72 
    
    freertos_entry 
    
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/mpthreadport.c:140 
    

    I will continue saving to file the pycom log.

    I'm using a Wipy3 and the release v1.20.2.r4 from git.

    Update:
    I create a simple program, using ExpansionBoard v2.1A with Wipy 3, fw release v1.20.2.r4 from git..

    print("Start While Loop")
    while True:
        wdt.feed()
        wlan.deinit()
        wlan = WLAN(mode=WLAN.STA_AP,ssid="lumitio_ap", auth=(WLAN.WPA2, 'www.lumit.io'))
        print("mem_free: ",gc.mem_free(), "mem_alloc",gc.mem_alloc())
    print("done")
    

    Wipy log output:

    mem_free:  2503744 mem_alloc 57600
    mem_free:  2503728 mem_alloc 57616
    mem_free:  2503712 mem_alloc 57632
    mem_free:  2503696 mem_alloc 57648
    mem_free:  2503680 mem_alloc 57664
    Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.
    Core 1 register dump:
    PC      : 0x400fe296  PS      : 0x00060b30  A0      : 0x800e6353  A1      : 0x3ffd8e20  
    A2      : 0x00000000  A3      : 0x00000004  A4      : 0x3ffc43d4  A5      : 0x00000001  
    A6      : 0x000000fe  A7      : 0x00000000  A8      : 0x3f40f128  A9      : 0x00000013  
    A10     : 0x00000000  A11     : 0x3ffc419c  A12     : 0x8008ff87  A13     : 0x3ffd8d30  
    A14     : 0x00000003  A15     : 0x00060023  SAR     : 0x00000010  EXCCAUSE: 0x0000001c  
    EXCVADDR: 0x00000000  LBEG    : 0x40093d60  LEND    : 0x40093d8e  LCOUNT  : 0xffffffff  
    
    ELF file SHA256: 0000000000000000000000000000000000000000000000000000000000000000
    
    Backtrace: 0x400fe296:0x3ffd8e20 0x400e6350:0x3ffd8e40 0x400e6509:0x3ffd8e60 0x400e81d1:0x3ffd8e80 0x400e8766:0x3ffd8f20 0x400e8c0d:0x3ffd8ff0 0x400e8c89:0x3ffd9080 0x40104a7a:0x3ffd90e0 0x400fce71:0x3ffd9100 0x40108345:0x3ffd9120 0x40100718:0x3ffd91c0 0x400fce71:0x3ffd91f0 0x400fce9e:0x3ffd9210 0x400e0f1a:0x3ffd9230 0x400e105d:0x3ffd92d0 0x400dfe80:0x3ffd9300
    

    Core dump decode:

    xtensa-esp32-elf-addr2line -fe build/WIPY/release/application.elf 0x400fe296:0x3ffd8e20 0x400e6350:0x3ffd8e40 0x400e6509:0x3ffd8e60 0x400e81d1:0x3ffd8e80 0x400e8766:0x3ffd8f20 0x400e8c0d:0x3ffd8ff0 0x400e8c89:0x3ffd9080 0x40104a7a:0x3ffd90e0 0x400fce71:0x3ffd9100 0x40108345:0x3ffd9120 0x40100718:0x3ffd91c0 0x400fce71:0x3ffd91f0 0x400fce9e:0x3ffd9210 0x400e0f1a:0x3ffd9230 0x400e105d:0x3ffd92d0 0x400dfe80:0x3ffd9300
    mp_obj_get_type
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/obj.c:43
    network_select_nic
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/mods/modnetwork.c:194
    mod_network_register_nic
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/mods/modnetwork.c:96
    wlan_setup_ap
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/mods/modwlan.c:542
    wlan_setup
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/mods/modwlan.c:312
    wlan_init_helper
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/mods/modwlan.c:1212
    wlan_make_new
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/mods/modwlan.c:1251
    type_call
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/objtype.c:997
    mp_call_function_n_kw
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/runtime.c:624
    mp_execute_bytecode
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/vm.c:919
    fun_bc_call
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/objfun.c:287
    mp_call_function_n_kw
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/runtime.c:624
    mp_call_function_0
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../py/runtime.c:598
    parse_compile_execute
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../lib/utils/pyexec.c:601
    pyexec_raw_repl
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/../lib/utils/pyexec.c:405
    TASK_Micropython
    /home/luis/Desktop/Lumitio/software/pycom-micropython-sigfox/esp32/mptask.c:356
    

    Best Regards,


Log in to reply
 

Pycom on Twitter