External Flash lose files

robert-hh

@serafimsaudade @gijs @tuftec I have a long term test running with the updated firmware, one WiPy3-alike and one LoPy4. The WiPy3 is at 2.5 Million writes, the LoP4 at 2 Million write. The LoPy4 is faster (for whatever reason), so it will overtake the WiPy in the next days. I plan to run in up to 10 Million cycles. That will take another 3 weeks.
During the test I ran into core dumps, caused by the heartbeat LED flashing. So I have a FiPy running just for assessing that bug. I made an issue report for that: https://github.com/pycom/pycom-micropython-sigfox/issues/518
It looks like a race condition between LED "On" and LED "Off", when due to the heavy load especially during file close the "On" cycle is delayed such that it collides with the "Off" Cycle. ~~But I am pretty confident that I found a very simple bug fix. Similar to the one here by just changing a number.~~ Note: Did not fix it. :-(

serafimsaudade

@robert-hh

I also seeing some random coredumps. I create this thread.

The heartbeat LED is disable and I also turn off the LED on boot.py:

pycom.heartbeat(False)  # disable the heartbeat LED
pycom.rgbled(0x000000)

Best Regard,

robert-hh

@serafimsaudade The core dump you showed is about different situation that the one I am hunting. No relation to the RGB led.

tuftec

@robert-hh thanks for your help with the LFS and flash issues. I still plan to do some more testing and share the results, but for now I am focusing on trying to sort out some LTE reliability issues. Once that is bedded down, I will get back onto the flash life issues.
Peter.

tuftec

@robert-hh I have tested my code with the firmware build that has lfs 2.03.
it crashes at the same sport.

crash dump follows


10:30:18: date/time =  20 12 26 9 26
PurgeTimer =  0  PurgePeriod =  4
backed up nv data
closed file1
10:30:20: abort() was called at PC 0x40099b38 on core 1

ELF file SHA256: 0000000000000000000000000000000000000000000000000000000000000000

Backtrace: 0x40091f2c:0x3ffe2c10 0x40092179:0x3ffe2c30 0x40099b38:0x3ffe2c50 0x40085476:0x3ffe2c80 0x40084f89:0x3ffe2ca0 0x4000bec7:0x3ffe2cc0 0x400f8b7a:0x3ffe2ce0 0x400f9cf6:0x3ffe2d00 0x400f9dbc:0x3ffe2d20 0x4010d081:0x3ffe2d40 0x40107e49:0x3ffe2d70 0x4010424d:0x3ffe2d90 0x401042dd:0x3ffe2db0 0x40111c03:0x3ffe2dd0 0x40107f5c:0x3ffe2e70 0x4010424d:0x3ffe2ea0 0x401042dd:0x3ffe2ec0 0x40111c03:0x3ffe2ee0 0x40107f5c:0x3ffe2f80 0x4010424d:0x3ffe2fe0 0x401042dd:0x3ffe3000 0x40111c03:0x3ffe3020 0x40107f5c:0x3ffe30c0 0x4010424d:0x3ffe3130 0x4010427a:0x3ffe3150 0x400e7369:0x3ffe3170 0x400e75f9:0x3ffe3210 0x400e6041:0x3ffe3230

Rebooting...
10:30:25: SYSTEM RESTART?
10:30:27: start HW init
disable the LTE modem for now
[AT] 668231296 AT+CFUN?
[AT-OK] +31 ```

I will attempt to create test version of my code that can repeatedly produce the crash for you to test.

tuftec

@robert-hh I have attempted to trim back my application so that you can run it at your end.
It will consistently crash.
Copy the 6 files onto a device (change the extension for DipConfig and NVvar to be .json) and it should run.
Still a lot of extra stuff in there but you should be able to find your way.
The program uses the .json files which it opens and reads/writes to.

This code works fine on FAT but crashes with LFS.

@peterp @Gijs your assistance here would also be appreciated.

NVvar.py main.py DipConfig.py DipBoot.py DipApp.py boot.py

robert-hh

@tuftec Thanks. I'll give that a try.
The long term test of simple writes are still running. both 3.5 Million on a WiPy3(-alike) and a LoPy4. I will stop it on the LoPy4 soon and only continue it on the WiPy. Then I can deal with your test.

The heartbeat related crash did not re-appear after a code change now for ~1 Million cycels on a FIpy, where would not earlier never for more than 130_000 cycles, ~50_000 in average.

robert-hh

@tuftec Maybe you should open a new thread for this discussion, because this one was initially about flash wear.

tuftec

@robert-hh possibly. My interest is in wear levelling to get the best life from the flash. If the lfs is not reliable then it is kind of pointless.

robert-hh

@tuftec about the test files: did you run them with or w/o REPL (P16 low or high)?

robert-hh

@tuftec More questions:
a) The imports are case senstitive. Tha got lost on download and had to be adapted
b) DipConfig.py must be DipConfig.json and NNvar.py must be NVvar.json.

I started with erasing a FiPy module first.
Then I uploaded a flash image with lfs2.3 enabled.
Then I uploaded your files, name corrected.
Then, after reboot, the code starts, flashes blue-yellow-blue & stops soon after without any error message. What is the code expecting?

Edit: OK. Got it. P16 must be pulled high to start the code and get the crash

robert-hh

@tuftec I found an inconsistency in DipApp.py:
At line 1044, you close nvfp. The file seems to be closed already, and so lfs goes haywire about it. FAT seems not to care.
Edit. There are quite a few single fp.close() and nvfp.close() calls in the app. As far as I could see, these are not required, since all open's are followed by close calls. Anyhow, it's better to use the with statement, which will take carte of the close, even in case of an error.

Edit2: Using a simple test case, double close is not an issue. mmhh? So it's not clear why it fails here.

robert-hh

@tuftec OK. I can replicate the crash with a simple test at global level, pasted into repl or executed as a small module:

f = open("test", "w")
f.write("Nonsense")
f.close()
f.close()

The same happens is you write for instance f = None instead global f.
What cause the crash in your code is:
a) defining fp and nvfp as global in line 40, or more to say, defining variables as global at module level is strange.
b) trying to close files twice. Even without the global definition you would get an error at the second close, albeit not a crash.

tuftec

@robert-hh thanks for looking at this.
Yes, i suspected the multiple closes might be an issue. I might need to find a way to test a file for open before closing. It does seem a bit pointless though. There are multiple entry points in some areas and i need to ensure the files are closed correctly bwfore going to deepsleep.
Yes, i struggled with the global local thing. I need to be able to access variables at the highest level and even between modules (DipBoot and DipApp as examples) without the inefficiencies of having to pass parameters.
Maybe I can just catch the close error (assuming there is one and it doesnt just crash).

As long as this is known and then well documented we can probably work around it. Although it would be nice if the behaviour was the same as for FAT.

What would yousuggest next? Clearly I need to use lfs to maximise flash life.

Thanks.

tuftec

@robert-hh i do not have enough experience with micropython. Is there a defined technique for how to share variables between modules. I have a timer that I share in an attempt to work out runtime for instance. I also found that I had to make some variables global so that my program would work correctly after coming out of deepsleep.
I am probably doing something wrong but making variables truely global seemed to stop micropython from complaining.

robert-hh

@tuftec You may work with global variables. Or you can use classes and its bound variables. It is just a matter of preference. Only you have to avoid double close of files. Even if the latter should not hurt, there seem to be a special problem with micropython here, which as to be addressed separately.

About files:

remove the isolated close statements. You do not need them.
for cases when you open a file for reading/writing, better use something like:

			with open('DipConfig.json','w') as f:
			    fp.write(json.dumps(dipconfig))
			time.sleep(1)

I have only seen one open where the close did not immediately follow, and that is in line 657. That one may be obsolete.
There is an open with a following load in line 830. It looks as of you could close the file there (or better use the with.... structure, which will take care of the close).

Yes, I understand that you have to communicate with a timer module. So the global variables. But structure of the whole thing seems overly complicated, with redundant definitions and imports.

tuftec

@robert-hh thanks for the pointers.i will rework my code to see if I can get it to work reliably with lfs.
I will provide feedback on my progress.

As an aside, how do you use editor and other frozen bits that you typically add to your builds? Is there a user document somewhere?

Thanks again.

robert-hh

@tuftec The editor is here: https://github.com/robert-hh/Micropython-Editor with a README and a doc-file.
I typically import it in boot.py or main.py with:
from pye_mp import pye
and call it with:
pye(file_name)

The upysh module is here (more or less): https://github.com/micropython/micropython-lib/tree/master/upysh
I usually import it with:
from upysh import *
which creates a list of names with commands. Then there is a man command, which lists all available commands, something like ls or ls("name"), rm, mv, mkdir, rmdir, cat, cp, head.., Unix like commands.

Note: Loading files to the editor is not fast. Your DipApp.py takes a while to load. And the file size is limited by RAM. With SPIRAM, that should not be the limiting issue. Not all keystrokes of the editor are available with all terminal emulators. It works best with Linux and picocom, or tio, or screen. On Windows, you may use Putty or TeraTerm.

tuftec

@robert-hh thanks

robert-hh

@serafimsaudade @Gijs @tuftec OK. This thread is overloaded now with the discussion of three different topics:

a) Flash wear. For that I made a PR: https://github.com/pycom/pycom-micropython-sigfox/pull/516 Tests are still running well, with two devices beyond 4 Million writes of a single short file. All well. I will continue up to 10 Million writes.

b) During that wear levelling test, I had core dumps at about every 50_000 writes. They were related to the heartbeat flash. I made a PR which changed that heartbeat timing a little bit. After that, the crashes disappeared, at least in that test. I am not 100% confident that the change cured the initial problem, which could also be a Core-0/Core-1 collision. But the change is also not intrusive and saves a few clock cycles. PR here: https://github.com/pycom/pycom-micropython-sigfox/pull/525

c) There was a core dump when one attempted to close a file twice using littlefs. Even if that is a bad practice, it should not crash the firmware. I copied over the mechanism used by Damien George to ignore the succeeding closes. PR here: https://github.com/pycom/pycom-micropython-sigfox/pull/524

Maybe some of that will eventually find it's way into the official build.

Explore Pybytes | Official Documentation | Report a Firmware Bug/Issue | GitHub

External Flash lose files

Pycom on Twitter