Running test on 100 wipy over a month

[serafimsaudade]

I produced 100 pieces of equipment with wipy. Everyone is running fw v1.20.1.r2.
50 units have python scripts in the flash. The rest were copied to frozen and the fw was compiled.
We have the equipment in tests + - 1 month.
So far about 30 are frozen. 20 reboot with guru meditation randomly. The rest are working correctly.
All equipment is in the same condition. Everyone has a dedicated power supply with enough power.

With this scenario I will be forced to patch the hardware and add an external wdt to try to mitigate the freezing problem.

Note: I am using the internal wdt, but as the tests indicate, it is not reliable.

[peterp]

Could you retry with 1.20.2.rc6? The last couple fw version had some stability improvements

[serafimsaudade]

@peterp Ok. I will try

[serafimsaudade]

Hi, again.

I have update 10 wipy and 2 gpy with the latest fw (1.20.2.rc6).
With this fw I'm getting Guru Meditation Error. Some times the pycom became freeze on the Guru Meditation Error.

Guru Meditation Error: Core  1 panic'ed (InstrFetchProhibited). Exception was unhandled.
Core 1 register dump:
PC      : 0x3ffb624c  PS      : 0x00060630  A0      : 0x80095188  A1      : 0x3ffdcc00  
A2      : 0x3ffb61dc  A3      : 0x00060620  A4      : 0x3ffdabe8  A5      : 0x00000001  
A6      : 0x000000fe  A7      : 0x00000001  A8      : 0x800947bc  A9      : 0x3ffdcbf0  
A10     : 0x00000000  A11     : 0x3ffc9509  A12     : 0x00060620  A13     : 0x00000001  
A14     : 0x000000fe  A15     : 0x3ffb61dc  SAR     : 0x00000000  EXCCAUSE: 0x00000014  
EXCVADDR: 0x3ffb624c  LBEG    : 0x40091ad4  LEND    : 0x40091b02  LCOUNT  : 0xffffffff  

ELF file SHA256: ets Jun  8 2016 00:22:57

But most of the times the guru meditation error do a core dump, and the cpu reset.

Guru Meditation Error: Core  1 panic'ed (StoreProhibited). Exception was unhandled.
Core 1 register dump:
PC      : 0x40096c29  PS      : 0x00060533  A0      : 0x80095107  A1      : 0x3ffdcc00  
A2      : 0x3ffb61ec  A3      : 0x00060520  A4      : 0x3ffdabe8  A5      : 0x00000001  
A6      : 0x000000fe  A7      : 0x00000001  A8      : 0x3ffb6210  A9      : 0x3ffb61fc  
A10     : 0x00000001  A11     : 0x0000abab  A12     : 0x3ffdabe8  A13     : 0x00000001  
A14     : 0x000000fe  A15     : 0x3ffb61e4  SAR     : 0x00000000  EXCCAUSE: 0x0000001d  
EXCVADDR: 0x00000009  LBEG    : 0x40091ad4  LEND    : 0x40091b02  LCOUNT  : 0xffffffff  

ELF file SHA256: 0000000000000000000000000000000000000000000000000000000000000000

Backtrace: 0x40096c29:0x3ffdcc00 0x40095104:0x3ffdcc20 0x400f31b8:0x3ffdcc40 0x400e0715:0x3ffdccd0

================= CORE DUMP START =================

This guru meditations are random. Can happen on the first 5 min or after 5 days.