External Flash lose files

[tuftec]

@robert-hh thanks for the pointers.i will rework my code to see if I can get it to work reliably with lfs.
I will provide feedback on my progress.

As an aside, how do you use editor and other frozen bits that you typically add to your builds? Is there a user document somewhere?

Thanks again.

[robert-hh]

@tuftec The editor is here: https://github.com/robert-hh/Micropython-Editor with a README and a doc-file.
I typically import it in boot.py or main.py with:
from pye_mp import pye
and call it with:
pye(file_name)

The upysh module is here (more or less): https://github.com/micropython/micropython-lib/tree/master/upysh
I usually import it with:
from upysh import *
which creates a list of names with commands. Then there is a man command, which lists all available commands, something like ls or ls("name"), rm, mv, mkdir, rmdir, cat, cp, head.., Unix like commands.

Note: Loading files to the editor is not fast. Your DipApp.py takes a while to load. And the file size is limited by RAM. With SPIRAM, that should not be the limiting issue. Not all keystrokes of the editor are available with all terminal emulators. It works best with Linux and picocom, or tio, or screen. On Windows, you may use Putty or TeraTerm.

[tuftec]

@robert-hh thanks

[robert-hh]

@serafimsaudade @Gijs @tuftec OK. This thread is overloaded now with the discussion of three different topics:

a) Flash wear. For that I made a PR: https://github.com/pycom/pycom-micropython-sigfox/pull/516 Tests are still running well, with two devices beyond 4 Million writes of a single short file. All well. I will continue up to 10 Million writes.

b) During that wear levelling test, I had core dumps at about every 50_000 writes. They were related to the heartbeat flash. I made a PR which changed that heartbeat timing a little bit. After that, the crashes disappeared, at least in that test. I am not 100% confident that the change cured the initial problem, which could also be a Core-0/Core-1 collision. But the change is also not intrusive and saves a few clock cycles. PR here: https://github.com/pycom/pycom-micropython-sigfox/pull/525

c) There was a core dump when one attempted to close a file twice using littlefs. Even if that is a bad practice, it should not crash the firmware. I copied over the mechanism used by Damien George to ignore the succeeding closes. PR here: https://github.com/pycom/pycom-micropython-sigfox/pull/524

Maybe some of that will eventually find it's way into the official build.

[tuftec]

@robert-hh Thanks. All good. I have tidied up my code and incorporated your suggestions. Seems to be working now with reliable lte and lfs. I will leave it run for some time to confirm. I can now move on to the next level of test and debug. Tanks again.

[robert-hh]

@tuftec Test finished.
The WiPy3 stopped as planned after 10 Million write cycles without problems using LFS with wear leveling ENABLED. I had to reboot it a few times manually since it lost WIFi connection. That relates to my AP.
I stopped the test with LoP4 after 7.5 Million cycles, and with FiPy after 4 Million cycles. None of them had an issue with writing. The FiPy was in the game, because it had Core dumps caused by some interaction with the heartbeat flash every ~100_000 cycles. After changing code, that went well for 3 Million cycles.

[tuftec]

@robert-hh good to hear. Thanks for the update.
I am continuing, although slowly, with my developments. No other issues to speak of at this point. Peter.

[Géza Husi]

@tuftec, @robert-hh : May I ask that have you found any issues (except the LED/Heartbeat topic) when enabled wear-leveling ? E.g. corrupted FS, reduced power-loss resilience capability, increased size of the files ? Thanks in advance !

[robert-hh]

@Géza-Husi I had a test running with that setting on three devices:

• LoPy4, 7.4 Million cycles, ~350_000 cycles/day
• FiPy, 4 Million cycles, ~200_000 cycles/day
• Genuine ESP32 with WiPy3 firmware, 10 Million cycles, ~180_000 cycles/day

The test consisted of writing the same 2k = 1 block sized file over and over again. The test code was set up to continue the test after a reboot.
Reboots had to be forced by pushing reset about every seconds day due to WiFi loss. The devices were not connected to a computer and I looked into the REPL once a day using telnet.
At no time there were file system corruptions. No file increase, no noticeable decrease in free file space. At the ESP32 with 1024 blocks I see still 1007 free blocks. 11 are used by the file system. 6 blocks used by the super block structure. That adds to 1024. So LFS performed really well.
It seemed that the ESP32 got a little bit slower over time from about 200_000 cycles/day down to 180_000 cycles/day. But the first number were taken when connected to a PC with REPL open, the second when running detached with just a power supply at USB, talking to no one.
No clue why the LoPy4 was so much faster. Most likely a faster flash chip.

[Géza Husi]

@robert-hh Many thanks for this detailed analysis and testing. The results are promising, based on your tests and also information from others I think it is clear that enabling wear-leveling does not have any significant negative effect.