How long have you had a WiPy run for?

[smssms]

I've got two WiPy (latest firmware) and they are reading an I2C sensor and writing the results to the Extension Board SD card in a loop. They are powered via a mains adapter on a UPS.

**Problem is they randomly restart or crash (after 5 hours to 3 days so far).

Have others managed to get their WiPy to run continuously for long periods?**

[Shaun]

I have experienced similar long term stability issues.

One very strange one was that after some hours of operation my code would go into a "hiccup" execution state where in my code would be executed for 1 second followed by 13.7 seconds of no code execution. Eventually my code would run normally again for a while and then the cycle would repeat.

I traced this problem down to the Telnet session I had open to monitor my code. I set the Telnet timeout at 100 000s.

My application so far has run for some days now, with no Telnet session active.

[RobTuDelft]

On the Wipy you can use the watchdog timer to restart on a crash/hiccup.

[livius]

@smssms
look here:
https://forum.pycom.io/post/7234

If you are talking about wipi2
Then my wipy work continously more then 6 days with firmware 1.6.12.
With previous software i have longer continous times.
Do you have all soldered or on breadboard with air wires?
And do you use gc.collect()?

What more do you use rtc, onwire, spi, uart, adc, threads, callbacks....?
I have:

• i2c:
  ** BH1750
  **BMP180
  **HMC5883L
  **PCF8574
  **2x16 blue LCD
• onewire:
  ** 4xds18b20
• SD card for storing values
• wifi:
  ** ap_sta mode connected to the router
  ** socket send data to server every minute - if avaiable - if not mark data on sd card to send in another time
• IR
• few buttons
• 4 servo

and all work ok - but all is soldered

[smssms]

@Shaun I have noticed exactly the same thing! A simple loop will run normally for a few hours and then it just stops for upto 300 seconds for no reason. After an hour or so it just goes back to normal or the board resets (there are also just some random board resets too). There is no heavy maths or processing and I'm running a manual garbage collect every loop. The graph below shows the time between running the loop over about 24 hours:

[Shaun]

@smssms
Sorry to hear you are experience the same annoying problems.
Do you have a Telnet session active when this happens?

I too was running a manual GC every run through my main loop.
I ran GC just before starting a self terminating thread each 1s cycle.

At least a WDT is now available, for applications that can tolerate a restart and the associated 6s restart period.
The WDT was not available untill a week or two ago. My application could not tolerate missing 6s of data while the system reset and restarted every so often in any case, probably another 2s of WD time out delay as well making for a total 8s gap in data.

I chose to develop with the WiPy as I had hoped it would save me time.
I purchased close on 1k pcs, up front, to secure personal development support, and Daniel was very supportive, but in the end I abandoned the WiPy and have almost completed a new design without the WiPy.

The ESP32 silicon is what I would refer to as beta silicon samples as the brown out reset circuit does not work and the ADC is next to useless. (Our production lines do not have the ability to calibrate each and every product coming off the line due to each and every ESP32 ADC having different characteristics.)
Input only pins have strange low going output glitches when reading the pins.
The above problems can be worked around using additional external BOR and ADC silicon, but in the end it was the stability issues that caused me to abandon the product for any volume production.

I'll consider the product for volume production again, once Espressif release corrected silicon and once the Python VM and libraries has matured on the ESP32.

[jpalbinet]

@Shaun I'm experiencing the same type of issues with a LoPy. The lack of stability is a real concern for me and I'm now also considering to abandon the product for volume production too unless we can get some extensive support from the Pycom team to tackle these stability issues some of us have found.

[RichardTXD]

@Shaun
Hi. We are developing an application to listen to BLE advertizing packets, process them, then write to I2C memory. Later, use WiFi to upload to a server.

I'm really sorry to hear your experiences as they may become mine over the coming weeks. The BOD and WDT issues and the ADC are all significant problems for us also. I'm thinking the external silicon to do these functions is now mandatory. We need a NiMh charger and offloading that from WiPy seems a better idea now.

I found the WiPy crashed after a number of hours reading BLE packets. GC delayed the ineviatable and the longest it ran was about 36 hours.

I then decided to try turning Bluetooth off, then on again, and that lets it run for at least 2 days. I've not tried 3 days yet. I'm wondering if the same (silly) solution would exist with I2C and SD to help?

I've not got I2C running to some 24LC512 eeproms yet and probably won't be using the SD card.

Your comment about ESP32 silicon is well taken.

[RichardTXD]

@jpalbinet
Hi, same here. Lack of stability is a show-stopper for us. We're using WiPy 2's

I'm wondering what PyCom can actually achieve if the ESP32 support in firmware and silicon is 'limited' or worse. It has been quite a few months since the ESP32 came out.

Just as a matter of interest, what are the alternatives? Developing for ESP32 'native' seems like using the same thing in a different way and hence exposed to the same issues.

I'm sitting on the fence, hoping like hell the solutions to these issues will be found and our R&D effort won't be lost. I spent time looking for alternatives and didn't come to any conclusions that were attractive.

[jmarcelino]

@Shaun
Problems like the Brown Out Reset are known and documented in the official ECO from Espressif:

https://espressif.com/sites/default/files/documentation/eco_and_workarounds_for_bugs_in_esp32_en.pdf

According to it that has been fixed in revision 1 of the silicon, which is what's now on the Pycom W01 and L01 OEM modules ( the development kits are still revision 0)

Yes things aren't stable yet and features are not where we all would like - I'd also love to move all my project to the LoPy/ESP32 ASAP, failures can be frustrating at times - but this is a very well featured chip and unparalleled in cost/features with more amazing things on the way like getting a full 4 Megabytes of RAM for 'free' (already on the W01/L01 modules)

Looking at the big picture the progress isn't going too badly - 6 months ago there wasn't even a BLE API for MicroPython, it came first on the Pycom modules! On the core SDK side Espressif is getting ready to launch a new stable release - ESP-IDF 2.1 again with many improvements.

So I'm still confident we'll get to a good place and I think the R&D involved will pay off. It feels to me very like when I started to use Linux around version 0.9 - man that was painful, once I had to code my own scanner drivers - but look where things got to.

[RichardTXD]

@jmarcelino
Thank you very much.
It is similar to what I was thinking. PyCom have a huge investment in it and I don't doubt are working hard. It must be frustrating for them. I feel their pain - we develop for Apple.
The problem is as you said... it is frustrating.

[jmarcelino]

Worth reminding that if you have crash reports (especially serial captures showing the guru meditation error/crash dumps) please submit them on Github as requested: https://github.com/pycom/pycom-micropython-sigfox/issues

Any hint will probably help a lot in improving things. Both the Pycom team and other interested parties (e.g. moi ;-) can have a look and try to trace the problem.

Right now there's only a handful of issues reported there (and many aren't even issues at all)

[RichardTXD]

@jmarcelino
Fair point. Poblem is that the crashes I've had do it silently. No messages. It just appears to lockup. Reset is required.
Yes - I am sure they need all the help they can get in resolving issues.

[TravisT]

Sorry to revive an old thread, but it seems relevant enough to include with the other information provided.

I have a couple proof of concepts working in the field using the Wipy2 development module. Not to say it is conclusive but I appear to be having random crashing that vary from hours to months. I admit there are some potential environmental influence, but it would appear to likely be the module.

The code that is running is only about 60 lines and it basically is in a endless loop that if it sees a message it rebroadcasts it in a slightly different format. This happens about 5 times a second, so I am reasonably confident in the code stability. Even with that I have added try/excepts, garbage collection on each loop, and other things to help minimize more normal exceptions.

Since there is some known issue with the original development Wipy2 modules I understand to avoid those now. The issue sounds like it was likely fixed in the OEM modules, what about the newer Wipy3 modules? Are these using the latest silicon?

Like others I want to use this in products, but need more confidence or risk the need to change to a more mature product.

[crumble]

@travist the versions with 4MB RAM use the latest hardware revision. But more important, pycom fixed a lot of bugs inbetween.

The 4MB has enough memory to run python scripts. If you stay with the old hardware versions you have to keep always the small amount of memory in mind. Python is not written to run with such low amount of RAM. If you only read some data into buffers which you allocate for the whole runtime, everthing seems to work fine. If you do a lot of huge string operations, there is a good chance for an out of memory exception. I miss a remote debuging interface to see the recources during runtime.

Many methods of the external communication devices seems to need some time until you can do the next step. Sadly you cannot wait for a certain status/event

something like

wifi.deinit()
lora.deinit()
machine.wait(READY_FOR_DEEPSLEEP)

would be awesome. Now we have to browse the forum for the right values which may change on the next firmware update.

I had some problems with softreboot. Sometimes the LoPy had not collected enough carbage which ended up in memory exceptions from the GPS library. It would be great if there will be a debug mode writing all exceptions into a log file. Would be much easier than guessing and writing from random places into the log too figure out if deepsleep had the same problem.

Pycom seems to deliver fixes fast, if you can describe the error well. The only problem is to locate the error.

[livius]

@travist
I have Wipy2 with old firmware 1.7.2.b1 still running since 18.09.2017.
But it work alone without any expansion board and without deepsleep functionality (it work 24/7).
It use only onewire - 3x DS18B20
and i2c devices like bmp180, bh1750, pcf8574
and 4x MG996R
and wifi

[TravisT]

@livius

That is funny, because I had this work for 2 months before having issues. Now they seem sporadic. I am worried a sensitivity is now showing itself. But we will see after more testing.

thanks