Guru Meditation Error & Magnetic Interference (EMI/RFI)

[combaindeft]

@martinnn said in Guru Meditation Error & Magnetic Interference (EMI/RFI):

@combaindeft
What does your physical setup look like? Cables, connections...?

• 3 pcs of PyTrack+FiPys .... 2pcs ran Freelance (on battery power) ... 1pcs ran on battery+usb-cable ...

• 1pcs of PySense+FiPys ... we running on battery power ... and we attached & detached the usb-cable from it .. on matter what we did to it ... it just ran ... ergo dubed The Joker

• USB-Cable(s)? .... we choose to use the best ones ... rated for Power & Data connection ... as some only do Power and never Data (or say, do Power, and extremely poorly Data over it ... guessing the ones that do both have better shielding inside of them)

I honestly doubt that any module alone could be affected by any kind of power line - power lines are 50 Hz mostly, field strength might be something but yours does not look like a 380 kV one. I would suspect that your setup contains some long wires as antenna or worse: loops that pick up some signal.

• And yeah, we're running a MacBook Pro ... Aluminium Case ... and one thought we have is that it might be the Aluminum Casing of the Mac that Act's as an Antenna or sorts ... and creates EMI in the USB-Cable that is attached to one of the devices ... But this doesn't quite answer the entire picture ... since we noticed when we drove away from the Power Line(s) that all 3pcs (two Freelance & one Hitched) ... all booted up at the same time ... *hummMmm*

• According to Wikipedia, SLF (Super low frequency),
  30 Hz to 300 Hz have been used by:

Radio amateurs have received signals in this range using simple receivers built around personal computers, with coil or loop antennas connected to the PCs sound card. Signals are analysed by a software fast Fourier transform algorithm and converted into audible sound.[3]

• Using the EMF Detector on my MacBook, it is at the speakers that I get the highest reading of ~2800-3000 uT

You write that you use the Pysense. I am also using a pysense/wipy combination

• Yes we are.(though with FiPy's...)

and I suspect these are extremely sensitive with a USB cable connected. In my office, I had six of those running on 4.2V (battery connector powered from lab power supply), which are still running (MQTT send every 2s via wifi). Five units are connected via USB (one to my laptop, four to a powered hub), of which three died in the past two days. Same pattern in the days before. I have standalone units running for two weeks now.

• Once in an office we had so much Electrostatic charged air air ... you couldn't touch any electronic boards without having a ground wire attached to yourself ... or you'd get a nice shock ...

• And yes, the usb-cable does make sense to look at .... but it doesn't explain why all PyTrack+FiPy devices (Freelance & Hitched) came back online at the exakt moment ... and began running their code ... if it was the usb-cable then the device hooked to it shouldn't have come back online (and the logs should have kept loging the Panic Core dump ... until x-reboots ... before it ran the code once well...)

So it seems you don't need a power line to make those things fail.

• Just some EMI to interfere with something on the board ... According to our Hardware guy ... it should be a software issue and not a Hardware-component issue.

• In our case we did get an explicit pointer that it must be some software/firmware issue ... as it states "Guru Meditation Error: Core 0 panic'ed (InstrFetchProhibited). Exception was unhandled." ... and that our code was running after x-reboots ... and then failed ... and ran again .... if it where a HW issue ... we shouldn't have been able to run any code ... also ... it does look that since we We have improved our code for: pycoproc.py the PySence+FiPy is working without any issue(s) ...

Guru Meditation Error: Core  0 panic'ed (InstrFetchProhibited)
. Exception was unhandled.
Register dump:
PC      : 0x00000000  PS      : 0x00060e30  A0      : 0x8011a4c9  A1      : 0x3ffbd650  
A2      : 0x00000000  A3      : 0x00000037  A4      : 0xffffffa4  A5      : 0x3fbc1e24  
A6      : 0x00000008  A7      : 0x00000000  A8      : 0x801257a4  A9      : 0x3ffbd640  
A10     : 0x3fbc1e40  A11     : 0x00000037  A12     : 0xffffffa4  A13     : 0x3ffbd6e4  
A14     : 0x7777715b  A15     : 0x7777715b  SAR     : 0x00000010  EXCCAUSE: 0x00000014  
EXCVADDR: 0x00000000  LBEG    : 0x4009b2dd  LEND    : 0x4009b311  LCOUNT  : 0xffffffff  

Backtrace: 0x00000000:0x3ffbd650 0x4011a4c6:0x3ffbd670 0x4011b4d3:0x3ffbd740 0x40131a71:0x3ffbd760 0x401335d3:0x3ffbd7b0

Can you try running the pysense boards without connections? Just add a battery and set them to communicate wirelessly?
@pycom: it seems you might have some work to do regarding the pysense

• From our test yesterday, The Joker (PySense+FiPy) devices worked without any issues ... and it was only with the PyTrack+FiPy that we all the issues with.

• Although, do note: We have improved our code for: pycoproc.py
  Since we noticed that we had a lot of issues with I2C Bus Error's

• @Martinnn - Maybe try it out yourself? and see if it solve your issue with the PySense+WiPy?

• Loosely & Planing of running more tests ... and in one of them I'll use the standard lib from PyCom GitHub ... vs our own improved one ... and seee.

• Also thinking of writing a specific test script ... where I'll cycle through all colours (Rolling-Rainbow...) ... and systematically one by one run one pice of our code-line ... and see how far it'll run through the code ... before it stats panicing.

[Paul Thornton]

@combaindeft Thanks for this. (and your email). I've shared the info with the team to see what they can come up with.

will update this thread with whatever they come back with asap.

[combaindeft]

@paul-thornton Hi again, any News?

[combaindeft]

@Paul-Thornton @crumble @Martinnn

So, we did some more tests .... wrote the following test code (see below)

Loaded it up on our FiPy+PyTrack ... and drove around the town (Veberöd, in the South of Sweden...) ...

We can Conclude

• It's not the High Power Lines at all! ... as we followed it and drove very close to it at multiple locations ... and none other place had the issue.

• We drove around more, and found more places that the Guru Meditation Error-loop happens. *YAY-US...* ... we have found more locations.

• After driving around, a lot more, we started to see a pattern ... and that it has to be something underground that generates some sort of EMF/EMI/RFI ...
  Note: In Sweden almost 99,9% of all cables & stuff, are put underground

• If, I put my hand around the case (the plastic-box the FiPy ships in) .... in other words ... Shield it ... in an affected area ... it code runs as it was supposed to do. Take my hand of, again, and it Errors ...

• Further more, I saw on the bus home, that in Dalby (15km away) it happened to at a certain place ... so it's Not-One-City-Only... issue ... Mabye? somebody else in the community can upload the debug-code (see below)? ... and test at their location? ... (You will NOTICE: As the colors will stop changing ... or changing different time patterns ... or that it suddenly runs through one Loop ... and then Stop ... and a long while later, it Runs again ... so forth and so on ...)

Will Test Soon

• To run a "FiPy+PyTrack" vs "FiPy+PySense" combo next to each other ... and validate that it's only the "FiPy+PyTrack" that has the issue(s).

Code: boot_debug_Lights.py

print('DEBUG :: @Import')

import machine
import time
import pycom

print('DEBUG :: @Defining Colors')
# Colors
off = 0x000000
gray = 0x121212
white = 0xffffff
red = 0xff0000
green = 0x00ff00
blue = 0x0000ff
purple = 0x800080
orange = 0xFFA500
# DEft Colors
dTorq = 0x00CED1
dPink = 0xFF1493
dRed = 0xFF4500
dMag = 0xFF00FF
dViol = 0x9400D3
dLime = 0x32CD32
dOlive = 0x808000
dTeal = 0x008080
dCyan = 0x00FFFF
dBrown = 0xA52A2A
dChoco = 0xD2691E
dAzure = 0xF0FFFF
dSgray = 0x708090

print('DEBUG :: @Setting up RainBow')
RainBow = [gray, white, red, green, blue, purple, orange, dTorq, dPink, dRed, dMag, dViol, dLime, dOlive, dTeal, dCyan, dBrown, dChoco, dAzure, dSgray]

print('DEBUG :: @HeartBeat OFF')
pycom.heartbeat(False)

print('DEBUG :: @Vars')
RUN = True
i = 0
j = 0 

print('DEBUG :: @Running RainDow Lights Show')
while RUN:
        pycom.rgbled(RainBow[i])
        time.sleep_ms(150)
        i = i + 1
        if i == 20:
                i = 0
                j = j + 1                
                print('DEBUG :: @Loop: ' + str(j))

[robert-hh]

@combaindeft YES! That could be the proof for earth rays, long sought by esotarian people!

[combaindeft]

@robert-hh said in Guru Meditation Error & Magnetic Interference (EMI/RFI):

@combaindeft YES! That could be the proof for earth rays, long sought by esotarian people!

HaHa 😇

But NOT! ... more like they didn't put good shielding around the things they put underground .... and yes ... it only follows the road(s) that are paved / have put something along underground ...

and, @robert-hh ... having looked more into the Error code in the Core-Dump ... it's looks like there are places in the Firmware code one needs to add Try-Catch(es) ...

Since, say after 12-20 reboots & core dumps ... it does actually run the code once to the end (last line) and then reboots 12-20 times ... and runs once again ... so forth and so on ...

[Martinnn]

@combaindeft Very interesting. In my case, the proposed correlation between USB cable and gurus did not hold. It more seems that some devices crash more regularly than others.
Maybe you should make yourself a pair of divining rods. This can be a very enlightening experience, particularly for a science-believing engineer (not joking)!
Do you have a spectrum analyzer? I'd suggest using it together with some near field probes (E and H field) when driving around.

[crumble]

@robert-hh said in Guru Meditation Error & Magnetic Interference (EMI/RFI):

@combaindeft YES! That could be the proof for earth rays, long sought by esotarian people!

Or a good place to dig for gold. @combaindeft you can easily test if it an expensive metal. Switch the GPS antennawit a plug and an active antenna. Shield the device and tell us the result. It will be only copper when it works, so you share the result ;)

[Paul Thornton]

@combaindeft I should be at my local hackspace this weekend at somepoint. Ill give your test script a run then. That place has enough RF/EMF noise that all laptop trackpads die when entering the building and refuse to work untill we leave (You try debugging that one in the room full of random hardware...)

That said. I've run a ton of Pycom boards there in the past without major issue.

[combaindeft]

@paul-thornton

Any results? :-)

@crumble, @Martinnn, @Paul-Thornton:

So we ran both PySense+FiPY & PyTrack+FiPy devices side by side (in the car, on the dashboard), no usb, both running same "boot_debug_Lights.py" script.

Our findings:

• Both had issues :-(
• The PySense seamed to bit a bit less vunerable and ran a little better ... but it it failed.

[Paul Thornton]

@combaindeft said in Guru Meditation Error & Magnetic Interference (EMI/RFI):

@paul-thornton

Any results? :-)

@crumble, @Martinnn, @Paul-Thornton:

So we ran both PySense+FiPY & PyTrack+FiPy devices side by side (in the car, on the dashboard), no usb, both running same "boot_debug_Lights.py" script.

Our findings:

• Both had issues :-(
• The PySense seamed to bit a bit less vunerable and ran a little better ... but it it failed.

Sadly no results (they worked fine). Will mention this issue again in tomorrows standup.

[combaindeft]

@paul-thornton

Here is a video you can show them:
Direkt download Link
InDirekt Link

Would it be possible for you to send us one of your devices that worked in your "test area"? ... so we can field test them in our "Area"? (and we'll send it back ASAP...)

So we can exclude if it's something between different HW Devices or not.

[johand]

@combaindeft
Have you found a soultion for this issue? I have the same problem (also in Sweden) with a W01 on a custom board. Have found a couple of places where this happens everytime passing with a car. This is on the country side, there could be smaller power lines under ground (power lines feeding a few houses, nothing big).

[johand]

Seems like this is related to WIFI in some way. When turning off WIFI (pycom.wifi_on_boot(False)) problem goes away. I am not using WIFI in any way, but maybe powerlines or WIFI accesspoint makes my W01 panic?

This is the application I have been running during the test:

from machine import Pin
import utime

LED.append(Pin('P3', mode=Pin.OUT))

while True:
    LED.value(1)
    utime.sleep_ms(100)
    LED.value(0)
    utime.sleep_ms(100)

I havent been able to decode the core dumps, can I send them to someone for investigation? This forum didn´t allow me to post it here.