FiPy randomly deleting program / factory resetting on power off/on
-
I have a mission-critical issue which MUST be solvable, since it is a software issue.
I have invested a lot in Pycom and hope to use these boards for a remote solar powered sensor monitoring station. It needs to be rock solid and absolutely bulletproof for my high-performance application.
I have been facing a repeated problem wherein the battery dying (power off to board) and then being recharged next daytime (power on) has caused the board to go from program loaded before powerdown to BLANK after new powering, with default empty main.py and boot.py - i.e., all my files/flash memory ERASED.
I read elsewhere here, from an old comment, that power off during file I/O could do this. This would be unacceptable, as my board is continuously logging pulses to SD card. Is this the cause of my issue?
My power supply is just fine. It's a 12v battery that will get discharged probably to around 7v before my 12v to 5v converter doesn't output enough to power board. Level converter powers FiPy through USB output.
How can I resolve this issue? I cannot have the Pycom board resetting once deployed far away in the wilderness, and becoming totally dysfunctional!
In my final design I will have a bigger battery to try to get 24/7 power. But that's just masking the issue: I need to have a Pycom board that can lose power anytime and load up anew once battery is charged with the exact same code on it.
Can anyone help? I really do not want to abandon Pycom for this project given the effort and money I've spent on this solution.
-
@PaulM You might want to look into firmware version 1.19. At the moment it is still in beta, but it adds the option to use LittleFS instead of FAT as the primary filesystem of the internal flash. This prevents data corruption even in the case of power loss during write operations.
-
@paulm What stops you from using the esp-idf and it's Arduino package do program the board at C level? The WiPy is "just" an ESP32 board with PSRAM. The other devices add RF circuitry with specified chips at documented GPIO pins.
The design decision of Pycom was to makes board based on MicroPython. You might not like that or Python at all, but you do not have to use it. If you need suggestions on how to write C code, either stand-alone or as extension to MicroPython, look into the sources.
-
@egimbernat Thank you for sharing. It would be an ideal solution for me to place my code into the raw immutable firmware level. Your skill level surpasses mine greatly in this respect. To be honest, I love the idea of these boards, but I despite this inefficient and unprofessional MicroPython joke. It feels like a toy - not the real deal. Easy to brake and MUCH harder to read than C. Where the hell is memcpy/Pointers/free? I wish I could write raw C into the immutable firmware of Pycom's boards, which I like from a hardware point of view.
Any chance of Pycom making a guide on how to program your own brownout-proof immutable firmware, whether in C or MicroPython?
As a developer of a system that needs to be rock-solid for professional grade 24/7 usage, I am emotionally scarred by the main.py deletion event and terrified to rely upon it, even if I clear out all file writes. It just simply cannot happen in the real-world.
If I omit ALL file write calls, is my fear irrational? Could it then be guaranteed 50 million random powerdowns mid-code-execution wouldn't do this, as long as it is not touching flash/sd card files?
-
@egimbernat The partition table has two firmware section, which are alternately used only if you upload the firmware with OTA. If you upload the firmware with the Pycom loader, always the first partition is used for storing the firmware. Which one is booted depends on the OTA state. If you never used OTA, the first partition is booted. Using OTA the partitions alternate, and with safe boot just the other partition will be booted.
https://docs.pycom.io/chapter/gettingstarted/programming/safeboot.html
P.S.: The Pinout sheets are wrong about safe boot.
P.S.2: There is still a hiccup in this mechanism. If by using OTA the second partition is the boot partition, using the serial firmware loader, which will place the new image into the first partition, seems to have no effect. Still the second partition will be started. See
https://github.com/pycom/pycom-micropython-sigfox/issues/81
-
I have also experienced this issue on my product, I opted to monitor the battery voltage and cut I/O operations when is below the secure minimum, similar as PySense/PyTrack boards on a ADC pin, and, to ensure that for some unexpected reason the board lost power, I freeze the Python code in custom firmware, that way, when the board starts with previous firmware due to corrupt I/O operation use the freezed Python code instead of a blank ones.
For this happen, is probably that you need to flash at least 3 times the board for erasing the old firmwares.Hope this helps.
-
@paulm The actual firmware uses a FAT file system. When power is lost during a write operation to a file or with a file open for writing, then almost certainly the file system will be corrupt. I had that quite a few times. Even if the file system is not cleared during the next boot, the content may not be useful. So avoiding file write when power is low is surely a good strategy.
Besides the file system I had never a problem with removing power from a pycom or other flash based device. For battery operated devices it is still highly recommended to control the battery and shut down properly on low battery, even if it only protects the battery.
-
@robert-hh I really appreciate your contribution of expertise! One last question: is the instability of randomly unpowering the FiPy (as to cause factory reset) limited to doing so SOLELY when it is busy with File I/O?
In other words, could I simplify this whole design by still letting the FiPy run down to minimum voltage and randomly shutting off, BUT turning off File I/O completely once we are below say 11.2V, to guarantee there are no file operations underway when it dies? That's assuming it won't ever reset to factory when depowered not during a file operation (maybe this assumption is wrong).
Would this ensure it does not factory reset, or is it still not the "proper" way?
-
@paulm It's certainly a good idea do use the arduino for power control. The Arduino can check the supply power and signal the FiPy, when the power is low. The Fipy then can finish whatever it is doing and go to deep sleep. The Arduino then can switch to a low power mode, and cycle through sleep phases and wake-up to check whether power is restored. The pure atmel chip can go below 1 µA, but the support circuitry will draw much more. Like the voltage regulator of the Arduino with 6 mA, a SD card at 3-5 mA, .... For that, even pull-up resistors, GPIO modes etc count. A single 4.7k pullup at an GPIO out forced to 0 will add 1 mA to the total consumption. Reducing the sleep mode power is a tedious task.
As a first step, you have to define the power margins for switching off and on again. These are related to the battery type you use and the remaining charge you need for power control. So for a 12V lead battery, you would define e.g. the off level to 10.8 V, and the on level to 11.4 V.
You might also consider using separate low power voltage regulators for the Arduino and the FiPy with an enable input, and switch off the power for the unused unit. I made good experience with Pololu modules, in your case e.g. a D24V10F5 device, which has a shutdown current of about 2 µA.
Obviously you can change the roles and let the Fipy check the power and send the other device to sleep. It is all a matter of energy needed in the sleep period and for the short wakeup phases. Even if the FiPy uses a lot of power to start, the total energy balance counts.
-
@robert-hh Thank you for your reply. The extra complication is unfortunate, but please help me get through it as it seems necessary.
I am using an external Arduino Uno anyways as a separate microcontroller to log the exact microsecond of high speed pulses (0-500Hz) from 4 channels, 24/7. The Arduino 1. writes pulses to SD card, and 2. sends them to Pycom FiPy over UART.
I'm only using the Pycom to upload to web over 4G (it is not stable enough to be my primary pulse logger), as well as aggregate some other less-critical, slower variables, such as pressure, battery voltage, and an external temperature probe.
If I connected my battery voltage meter instead to the Arduino to detect a low battery voltage and send a safe shutdown command to FiPy over UART, would this solve the problem?
How would you recommend I handle the power-back-up factor in that case?
Thank you in advance for your thoughts.
-
@paulm You definitely have to monitor the supply power and shutdown actively the FiPy, before it browns out. You can use the internal ADC and send the device to deep sleep, or use an external power monitoring circuit.
Personally I would prefer an external circuit, like a dedicated device as the LM3710, or a small micro like an ATTiny, which you could also program to restart the FiPy once sufficient power is back.
