Pymesh crashes Lopy

Jordan Reyes

@catalin This is what comes back when pyMesh crashes with lopy and mesh that is running, it doesn't happen reliably, but it eventually crashes the entire system:

Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited). Exception was unhandled.
Core 1 register dump:
PC : 0x00013ffe PS : 0x00050033 A0 : 0x00013ffe A1 : 0x3ffe12f0
A2 : 0x00230000 A3 : 0x00000006 A4 : 0x08700000 A5 : 0x00003ffb
A6 : 0xd2550000 A7 : 0xd26b4009 A8 : 0xfffb4009 A9 : 0x39a4ffff
A10 : 0x00014008 A11 : 0x69ac0000 A12 : 0x08704009 A13 : 0x00003ffb
A14 : 0x00000000 A15 : 0x00000000 SAR : 0x00000000 EXCCAUSE: 0x00000014
EXCVADDR: 0x00013ffc LBEG : 0x00000000 LEND : 0x00000000 LCOUNT : 0x00000000

Backtrace: 0x00013ffe:0x3ffe12f0 0x00013ffb:0x000000ff

Guru Meditation Error: Core 1 panic'ed (IntegerDivideByZero). Exception was unhandled.
Core 1 register dump:
PC : 0x40098f0b PS : 0x00060033 A0 : 0x80098822 A1 : 0x3ffe11b0
A2 : 0x3ffe1230 A3 : 0x00000000 A4 : 0x00002800 A5 : 0xc0051ff0
A6 : 0x00051fe0 A7 : 0x00000032 A8 : 0x00000000 A9 : 0x3ffe11a0
A10 : 0x2c42f5e1 A11 : 0x00000000 A12 : 0x00000018 A13 : 0x3ffe11c8
A14 : 0x3ffc3b00 A15 : 0x00000001 SAR : 0x00000017 EXCCAUSE: 0x00000006
EXCVADDR: 0x00013ffc LBEG : 0x4009cfd8 LEND : 0x4009cfe3 LCOUNT : 0x00000000

Backtrace: 0x40098f0b:0x3ffe11b0 0x4009881f:0x3ffe11f0 0x40098a5a:0x3ffe1210 0x4008378a:0x3ffe1230 0x40013ffb:0x3ffe12f0

Re-entered core dump! Exception happened during core dump!
Rebooting...
ets Jun 8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff8028,len:8
load:0x3fff8030,len:2064
load:0x4009fa00,len:0
load:0x4009fa00,len:19208
entry 0x400a05f8
Initializing filesystem as FatFS!
3
2
1

catalin

hi @Jordan-Reyes, I've forked your issue here, to discuss it out of Pymesh announcements topic.
So, are you using Lopy_v1, release 1.20.0.RC8 and scripts from here: https://docs.pycom.io/tutorials/lora/lora-mesh.html?

Danne

Just want to confirm, that I have seen this on three Lopy4 units running the Loramesh library. Firmware 1.20.0.rc11

Seems to crash in init on the line:

self.mesh = self.lora.Mesh()

Same code runs on several other Lopy4 units without problem and this unit did run the same code before without problems then suddenly started crashing and I thought the unit to be completely broken since I could not get them to run the mesh code again even when I re-flashed the firmware.

@catalin: Since I have some (3) of these nodes crashing It would be nice to be able to build the firmware myself to debug. I built the latest version successfully (without mesh) but I am unable to build with openthread support, I guess I need to add the openthread component to pycom-esp-idf somehow?

catalin

hi @Danne,

Since 1.20.0.rc9, the Pymesh component was excluded from development firmware releases (I hope to be right, but from July '19). The idea is to add a separate firmware release for Pymesh, due to licensing issues. This is not yet done, but we are working on it.

I will announce the release on this channel: https://forum.pycom.io/topic/4449/pymesh-updates

Danne

Thanks @catalin!

(it seems to have mesh up until rc11, at least I'm using that one )

While we wait I use the old legacy firmware to play with Loramesh.

However, this issue reported with instrFetchProhibited seems to happen on some boards and not others (regardless of firmware version) so it would still be nice to be able to build my own version of the firmware do investigate it further while I wait for your release... is there a version of the FW-sourcecode with openmesh included?

For future reference, if the issue remains...
The following code does cause the problem on some Lopy4 boards

from loramesh import Loramesh
from network import LoRa
import pycom
pycom.wifi_on_boot(False)
pycom.heartbeat(False)
lora = LoRa(mode=LoRa.LORA, region=LoRa.EU868, bandwidth=LoRa.BW_125KHZ, sf=7)
print("pre loramesh")
mesh = Loramesh(lora)
print("Lora mesh complete")

andreas

Dear @Jordan-Reyes and @Danne,

we also have been annoyed by different random core panics people from our community have been observing.

@lars_r also experienced core panics on his LoPy4 and we have been able to help him out of being stuck with it just today [1]. However, the fixes improve general stability which is not specific to any mesh networking features. Saying this, it might still improve things for you.

Please read [1] carefully regarding the erase_all step and how to eventually restore your LoRa MAC. Thanks again to @robert-hh for his knowledge on how to do that.

With kind regards,
Andreas.

[1] https://github.com/pycom/pycom-micropython-sigfox/issues/361

catalin

Hi @andreas, let's keep things real and not indirect people to false solutions (fyi the solution you proposed is for a Pybytes release, which is not the case here).

I agree with erase_all risk.

The issues from this topic are related to miss-match between Lopy4 (or Fipy) binary and the Pymesh micropython library. All the Pymesh releases(until the licensed v1.20.1.r2 one) were for experimenting and trying Pymesh. Instability came from incompatible micropy scripts.

robert-hh

@catalin No python script should cause a crash dump. If it does, there is a problem with the firmware, which requires analysis, as you most likely did with the early Pymesh firmware.
The Pybytes firmware is more a pain for the users, because different to the Pymesh firmware it claims to be 'standard' firmware, but breaks existing behaviour and for some people it causes the firmware to crash, which in turn points at bugs in the firmware.

andreas

Hi there,

@robert-hh said in Pymesh crashes Lopy:

No python script should cause a crash dump.

I believe the Python snippets outlined within this topic by @Jordan-Reyes and @Danne will perfectly work for many people.

Unfortunately, I haven't been able to create and publish a short and concise Python program which deterministically reproduces the indeterministic core panics some people are observing in various conditions. In fact, many things are involved here beyond the sole scope of the MicroPython interpreter.

While I can't say this matches the observations of @Jordan-Reyes and @Danne in this particular case, the issues are going down to the silicon level. To be clear on that detail: Let's not blame Pycom here, I believe they are doing a great job of keeping things running but the nitty gritty details within "turtles all the way down" are more than overwhelming.

@robert-hh said in Pymesh crashes Lopy:

For some people it causes the firmware to crash, which in turn points at bugs in the firmware.

That is probably the case here. We have investigated potential issues thoroughly during the past two months and tried to report about them as good as possible on [1].

However, wrapping things up requires more dedication and we are running out of everything on these details. So, I am currently trying to get in touch with @daniel and @Xykon in order to discuss further details. Maybe @catalin is also the right person to talk to as I just found out he is also member of Pycom [2].

With kind regards,
Andreas.

[1] https://community.hiveeyes.org/t/investigating-random-core-panics-on-pycom-esp32-devices/2480
[2] https://github.com/orgs/pycom/people

andreas

Hi there,

@catalin said in Pymesh crashes Lopy:

Since 1.20.0.rc9, the Pymesh component was excluded from development firmware releases.

I have not been aware of that detail. Thanks for telling us, @catalin. That said, I am afraid you will not have any fun with the custom builds coming from our pen unless the updates might have been mainlined.

So, sorry for the noise everybody!

With kind regards,
Andreas.

ricardolima

@catalin
Hello all,
i'am trying to implement a mesh using your new release and I had tried to send a message from one node to the border router. But I think i'am getting some problem in configuring the border router.
To configure the border router I uncommented "pymesh.br_set(PymeshConfig.BR_PRIORITY_NORM, new_br_message_cb)", "new_br_message_cb" function, connected the device to the wifi and running the codes it returns

State: True BR:  []
Force add BR
BR:  [(net='2001:cafe:cafe:cafe::/64', preference=0)]
done Pymesh init, forever loop, exit/stop with Ctrl+C multiple times

Which seems like it is working. But now I have two problem:
1 - It is getting blocked (the node br)

Neighbors: Router MAC 0x70B3D5499F8MASTER, rloc16 0x7800, coord (51.45, 5.45313), neigh_num 1, ts 126
MAC 0x70B3D5499SENDER, rloc16 0x3000, role 3, rssi -16, age 98

And this does not pass from where.. it is really blocked, even if I press enter it does not change anything.

2 - Then in the other node, the one how is spamming messages I tried

while True:
    #print("sent to br")
    #ip = "1:2:3::4"
    #port = 5555
    #pymesh.send_mess_external(ip, port, "Hello World number 2)
    pymesh.send_mess(0x70B3D5499F8DBDD0, "Hello World number 2")
    time.sleep(3)

the option with pymesh.send_mess was working, but then it stopped and I don't know why .. using the function pymesh.send_mess_external it gives the error "no BR (mesh-external IPv6) found"

Any idea ? sorry for the long post

Thank you

catalin

@ricardolima After declaring the BR, do you see in the network the nodes starting to have a new IPv6 unicast address with prefix `2001:cafe:cafe:cafe:'?

The error no BR (mesh-external IPv6) found comes from not being able to find this ipv6 prefix.

If this is the case, probably I should start investigating this issue.

ricardolima

@catalin
the node sending to the br says

Role 4, Single False, IPv6: ['fdde:ad00:beef:0:0:ff:fe00:fc00', 'fdde:ad00:beef:0:0:ff:fe00:1400', 'fdde:ad00:beef:0:0:0:0:3bda', 'fdde:ad00:beef:0:cb3d:2abf:bcb3:839', 'fe80:0:0:0:72b3:d549:9b77:3bda']
Neighbors: Router MAC 0x70B3D5499B773BDA, rloc16 0x1400, coord (51.45, 5.45313), neigh_num 1, ts 485
MAC 0x70B3D5499777124D, rloc16 0x5800, role 3, rssi -26, age 11

And the br says

Role 3, Single False, IPv6: ['fdde:ad00:beef:0:0:ff:fe00:5800', 'fdde:ad00:beef:0:0:0:0:124d', 'fdde:ad00:beef:0:6b76:e43d:813b:f49d', 'fe80:0:0:0:72b3:d549:9777:124d']

What do you think? it is a issue?

Thank you

ricardolima

@catalin
No news about this ?
Happy new year for all

catalin

@ricardolima
Sorry for the late answer. I am investigating this, seems from the last openthread update, some BR functionality fails.

ricardolima

@catalin Ok thank you.
Waiting for news :D

catalin

@ricardolima
Yes, the BR functionality was re-factorized in a new module, and it was fixed.
LoRaWAN coexistence with Pymesh, is also being exposed in the micropython API.
Refreshing the base software (probably esp32 idf v.3.3.1) will also be integrated.
I am targeting this week(Th) to re-release Pymesh.

ricardolima

Ok thank you ! You will resend the file to all users that have asked permission to use Pymesh ?

catalin

@ricardolima Yes, I'll make an email list with everyone that already signed the licence.

jclo13

hello @catalin, I I have followed the procedures to download the latest firmware through Pybytes and I am using pymesh microphyton API, the problem I have is that with the following code the system chashes or not connect whit the other nodes, I do not know if it is a firmware problem or what it is due to, I will appreciate your help.

from network import LoRa
from mesh import LoraMesh
import ubinascii
import socket
import utime
from  LoraWan import lorawan
import cayenneLPP
import config
import machine
from machine import SD

pycom.rgbled(0x0000ff)                  #blue
time.sleep(0.2)

pycom.rgbled(0x000000)

sd = SD()
os.mount(sd, '/sd')
f= open('/sd/reenvio.csv', 'a+')
r= open('/sd/reenvio.csv', 'r')
os.listdir('/sd')



while True:

    print('##################### Conectando  MESH ##################')
    lora = LoRa(mode=LoRa.LORA, region=LoRa.US915, frequency = 904300000, bandwidth=LoRa.BW_125KHZ, sf=7)
    masterkey = ubinascii.unhexlify("12345678")
    pymesh = lora.Mesh(key=masterkey)
    pymesh.border_router("2001:dead:beef:caff::/64", 0)
    time.sleep(5)
    print("IPv6 unicast addresses: %s"%pymesh.ipaddr())
    print("4")
    pycom.rgbled(0xff00)                 #GREEN
    time.sleep(0.3)
    pycom.rgbled(0x000000)
    #IP=mesh.ipaddr()
    #ipBR=str(IP[0])

    #creando sockets:

    s_EID = socket.socket(socket.AF_LORA, socket.SOCK_RAW)
    s_EID.bind(config.puertoEID)

    s_BR = socket.socket(socket.AF_LORA, socket.SOCK_RAW)
    s_BR.bind(config.puertoBR)

    s_mesh=[]
    s_mesh.append(s_EID)
    s_mesh.append(s_BR)

    data= "hola"
    id=0
    ip=0
    cont=0

    #escuchando mesh
    while cont<config.acumulador:
        time.sleep(4)
        paquete= Loramesh.receive_pack(s_mesh)
        if paquete!=None:
            data=paquete[0]
            ip=paquete[1]
            port=paquete[2]
            print("-----------PAQUETE:%d-------"%cont)
            print ('datos recividos:  %s'% (data))
            pycom.rgbled(0xff00)                 #verde
            time.sleep(0.2)
            pycom.rgbled(0xff00)                 #verde
            time.sleep(0.2)
            pycom.rgbled(0x000000)
            time.sleep(2)
            f.write('{},{},{}\n'.format(data,port,utime.localtime()))
            cont+=1

Explore Pybytes | Official Documentation | Report a Firmware Bug/Issue | GitHub

Pymesh crashes Lopy

Pycom on Twitter