Guru Meditation Error

[andreas]

Thanks @sita. This yields...

==================== CURRENT THREAD STACK =====================
#0  0x4009774e in vTaskDelay (xTicksToDelay=<optimized out>) at /home/gezahusi/Work/Repositories/pycom-esp-idf/components/freertos/tasks.c:1484
#1  0x4011338c in TASK_LoRa (pvParameters=<optimized out>) at mods/modlora.c:804

[andreas]

Dear @sita,

@ps47 also experienced similar errors like you have been observing [1]. He is also using pure LoRa. Using the most recent 1.20.2.rc3 release seems to help here [2]. Thanks @husigeza and the whole Pycom team for mitigating this issue.

I tried the new build LoPy4-1.20.2.rc3-0.8.0-vanilla-squirrel.tar.gz and not any “bad00bad bad00bad bad00bad” Core Panic happened till now.

With kind regards,
Andreas.

[1] https://forum.pycom.io/topic/5514/random-core-panic-with-lora-socket-send-recv-device-doesnt-restart-try-with-config_esp32_panic_silent_reboot
[2] https://community.hiveeyes.org/t/investigating-core-panics-on-the-lopy4/2878/29

[andreas]

Dear @sita,

any news on your side about this issue?

With kind regards,
Andreas.

[sita]

Hello @andreas,

I am tried all the ways (like removing SD card, changing the time ..etc) still same error is continuing :( :(
Now i will test the device by sending data via TTN to check whether this error is coming in ad-hoc lora mode or not and let you know.

Thanks for your support.

Have a nice day

[peterp]

Which firmware are you using now? Or let me ask it differently, can you reproduce the error with v1.20.2.rc3?

[sita]

Hello @peterp,

I am using latest firmware ( v1.20.2.rc3 ). I have tested the device different codes.
If i send basic message then i didn't find any error, but if send GPS the error is still coming.

[andreas]

@sita said in Guru Meditation Error:

If i send basic message then i didn't find any error, but if send GPS the error is still coming.

Thanks for clarifying this. While I believe there has to be some protection for that case, might it be possible that the message is just getting too long?

coord = l76.coordinates()
gps_datetime = l76.get_datetime()
lat,lon = coord

timestamp =('20' + gps_datetime[4][4] + gps_datetime[4][5]+gps_datetime[4][2] + gps_datetime[4][3]+gps_datetime[4][0] + gps_datetime[4][1]+gps_datetime[2][0] + gps_datetime[2][1]+gps_datetime[2][2] + gps_datetime[2][3]+gps_datetime[2][4] + gps_datetime[2][5] )

message = ''+str(lat)+','+str(lon)+','+str(timestamp)+','+str(value)

s.send(message)

a) Can you tell us about the final result of len(message) here?
b) Can you make up another arbitrary message of the same length (e.g. by using message = 'X' * length) without reading the L76 in order to check if both things produce the error independently from each other?

[sita]

Hello @andreas,

a) The length of message is 35
b) With same length (35) message without L76 i tested the device, there is no error present.

[andreas]

@sita said in Guru Meditation Error:

without L76 i tested the device, there is no error present

So, do you believe we can deduce from that that doing both things (reading the L76 and submitting data using LoRa) influence each other unfavourably?

That might finally be a trace we could follow up here.

[sita]

Hello @andreas,

So as per my opinion, these things (reading the L76 and submitting data using LoRa) effecting the device.

[sita]

Hello @andreas,

After transmitting the data 2 hours (without L76 lib and message length 35), just now i have observed Guru meditation error.

[g0hww]

My Lopy4 has been sending position info from the PyTrack GPS over LoRa every 10 seconds since startup, and has 420254.5 seconds of uptime right now. That's pretty much a record up-time for me, and I have done a lot of refactoring and simplification of my code-base recently to get this kind of stability. It also publishes info on the GPS status and node health over MQTT at the same rate. This is with FW 1.20.2.rc3. I did have quite a few issues with LoRa on previous FW versions.

[peterp]

@sita so let me recap and see whether I understand your reproduction.

• lopy4
• pytrack
• v1.20.2.rc3
• this script https://forum.pycom.io/topic/5560/guru-meditation-error/16
• except, do not import and use L76GNSS
• crash after 2h

That about right?
How quickly, reliable can you reproduce it WITH L76GNSS?

Could you try without sdcard, and/or without writing to logfile?

[peterp]

@g0hww anything that comes to your mind what could be different with your setup?

[sita]

Hello @peterp,

For a clarification, i had tried all the type of experiment like

1. With L76 library
   • Transmitted only numbers (length of 1 -4 max)
   • Transmitted data (length of 3)
   • Transmitted data (length of 35)
   • Transmitted GPS (length of 25)
   • Transmitted only time (length od 10)
   • Transmitted both time & GPS (length of 35)
2. Without L76 Library
   • Transmitted only numbers (length of 1 -4 max)
   • Transmitted data (length of 3)
   • Transmitted data (length of 35)

With above all the experiments I got error. I have tried with & without SD card also. One thing is previously the frequency of error is very often. But after updating the firmware it is reduced.

[peterp]

@sita so, lets focus on the simplest reproduction, ie

• without L76
• without SD card

Can you try without writing the logfile? (fyi, with some of the previous panics, there were indications that it has to do with writing to flash)

[g0hww]

@peterp My implementation is quite different to that shown above and uses a home-baked Navigator class, which provides a work() method that is called in a loop from main().

It directly invokes L76GNSS._read() to consume data. Every 10 secs, when a repeating timer callback sets a flag to process the GPS data, Navigator.work() repeatedly reads a block of 256 bytes and feeds it to a temporary instance of MicropyGPS (which is always in a pristine state due to it being a single use temporary instance). Blocks of 256 bytes are fed into that MicropyGPS instance until a specific set of NMEA messages have been ingested and MicropyGPS.satellite_data_updated() returns true. It then extracts nav solution info from the MicropyGPS instance and transmits/publishes it, before clearing the flag.

Subsequent calls to Navigator.work() cause it to invoke L76GNSS._read() and discard the data, until the flag is set again by the timer callback.

I found this approach to be necessary as a backlog of data from the GPS would sometimes build up if I attempted to ingest every byte from the GPS into MicropyGPS, with MQTT publishables indicating a latency of up to 10 seconds. With this approach, I can get the current nav solution, prepare an AES-128 encrypted payload, add HMAC-MD5, send it over LoRa, receive it on another LoPy4 acting as a LoRa/MQTT bridge and publish it on MQTT within 2 seconds of the nav solution timestamp, which is pretty good IMHO.

The LoPy4 with Pytrack running this stuff is now at an uptime of 485234 secs (5 days,15 hrs) and seems to be still going strong.. The software load was new at the start of this boot cycle, so I have no prior experience with this specific configuration to indicate how long I might expect it to actually run for, but as of this time it's beaten my previous best uptime for any software load by about 2 days. I'm now preparing for the jinx effect to make it break.

UPDATE: Uptime experiment terminated due to lack of patience (and no apparent jinx effect) after 645684 (about 7.5 days) as I needed to transplant the LoPy4/Pytack into its new IP67 case.

[Don-iot]

I just want to share my experience. I think this can happen if your power supply voltage is not sufficient. For example the device we are working on will handle most commands but when it tries to write to memory and voltage is below 3 V it will stall and output this kind of message. I have had this issue on two GPys and could replicate that the problem was solved by merely upping the voltage and rebooting the device.