External Flash lose files
-
After done all this tests I'm very concern about the flash sanity.
It is possible the flash has reach the write limit cycles ?PS: last year I have install +- 150 units.
Best Regards,
-
@serafimsaudade That depends on how many write operations are performed on the flash. The typical figure a vendor guaratees is 100000 erase/write cycles. But practically more write cycles are possible, and it should not affect all devices at the same time. Additionally, LFS has mechanisms for wear leveling, such that all blocks of the flash are written equally, not a few blocks more that others, like it happens with the FAT file system.
Taking the number of 10**5 in a year, that would mean ~274 write events of a single block every day - or one every 5 minutes. Can you estimate how often your devices write to flash?
-
Well, after doing a double check into the code. This 11 wipy write to flash +- every 10sec.
Best Regards,
-
@serafimsaudade That's indeed quite often, about 3 Million writes/year. A few more questions:
- how many byte do you write every time (average)
- do you close the file after every write? If yes, data will be written, if no, data will be buffered and only written, if the buffer is full or the file is closed.
- which WiPy do you use, WiPy2 or WiPy3? That matters with respect to the size of the file system.
I do not remember the details of the LFS implementation, whether they do some kind of journalling or not. In any case the fault safe property should cause more write operations.
-
@serafimsaudade I understood first that you have 11 devices in total and all failed at the same time. However, if 11 out of 150 fail in a short range of time, then a hardware problem could be the cause.
I do not see an easy way of verification that hypothesis. On a quick search I did not find test code for the flash chip.
You could obviously use a Arduino or espressif software environment and create a small program that tries writing and reading the flash. Or you open a device and replace the flash chip. -
@robert-hh said in External Flash lose files:
@serafimsaudade That's indeed quite often, about 3 Million writes/year. A few more questions:
- how many byte do you write every time (average)
- do you close the file after every write? If yes, data will be written, if no, data will be buffered and only written, if the buffer is full or the file is closed.
- which WiPy do you use, WiPy2 or WiPy3? That matters with respect to the size of the file system.
I do not remember the details of the LFS implementation, whether they do some kind of journalling or not. In any case the fault safe property should cause more write operations.
I write +- 200 bytes;
I close the file after every write.
I'm using the WiPy3.Best Regards
-
This 11 wipy3 are the only with 10 sec write.
The other +-139 wipy3, write to flash every 60sec.Best Regards,
-
@serafimsaudade OK. So if it is a flash wear problem, you have about 6 years time. Besides that....
Doing a quick calc. The WiPy3 flash file system is 4 MB, or 1024 (or about 10^3) blocks of 4 k. With a well distributed wear leveling, one could make at least 10^8 block writes. That number has to be divided by the number of block writes caused by you file operation to get the number of safe file write. I do not know this number. To get that, I could add debug messages to the LFS. For that, I have another question:
- How is the actual file write done. Do you write a single short file every time, or do you append the data to an ever increasing file. Or do you write to alternate files, ....
-
Sorry I mistype the file size, The wipy3 write to flash an file with +- 2K bytes.
"How is the actual file write done. Do you write a single short file every time, or do you append the data to an ever increasing file. Or do you write to alternate files, ...."
I open always the same file, write with no append, and close the file.
Best Regards,
-
@serafimsaudade Since the block size is 4k, writing 200 bytes or 2k bytes does not make a difference.
-
@robert-hh So I wrote a little python script:
# def run(): b = bytearray(2038) f = open("logdata", "w") f.write(b) f.close()And I added a few printf statements to sflash_diskio_littlefs.c, which print a line every time erase or write is called. Then I called up the little script a few times. This is the output:
>>> wtest.run() *** block erase 0 *** *** block prog 0 *** *** block erase 235 *** *** block prog 235 *** *** block erase 1 *** *** block prog 1 *** >>> wtest.run() *** block erase 236 *** *** block prog 236 *** *** block erase 0 *** *** block prog 0 *** >>> wtest.run() *** block erase 237 *** *** block prog 237 *** *** block erase 1 *** *** block prog 1 *** >>> wtest.run() *** block erase 238 *** *** block prog 238 *** *** block erase 0 *** *** block prog 0 *** >>> wtest.run() *** block erase 239 *** *** block prog 239 *** *** block erase 1 *** *** block prog 1 *** >>> wtest.run() *** block erase 240 *** *** block prog 240 *** *** block erase 0 *** *** block prog 0 ***The result is confusing. It shows some kind of wear leveling, the block with the changing numbers from 235 to 240 (going on to 1024 as I noticed and then skipping to 2), but also, that alternating block 0 and block 1 are written/erased. These are so called "superblocks". According to the spec, they should change after a while. I will let that run to see if it happens. If not, then that is a good candidate for flash failure.
P.S.: If you worry about me killing a device. No problem. It is an open one, on which I already had replaced a flash chip. And I have spare chips here.
-
-
@serafimsaudade No change so far after 5000 writes. I changed the scheme writing to 128 different file names. And then alternative superblock numbers were used. Small section from the log:
47 ------------------------------------- *** block erase 427 *** *** block prog 427 *** *** block erase 0 *** *** block prog 0 *** 48 ------------------------------------- *** block erase 428 *** *** block prog 428 *** *** block erase 1 *** *** block prog 1 *** 49 ------------------------------------- *** block erase 429 *** *** block prog 429 *** *** block erase 0 *** *** block prog 0 *** 50 ------------------------------------- *** block erase 430 *** *** block prog 430 *** *** block erase 1 *** *** block prog 1 *** 51 ------------------------------------- *** block erase 431 *** *** block prog 431 *** *** block erase 0 *** *** block prog 0 *** 52 ------------------------------------- *** block erase 432 *** *** block prog 432 *** *** block erase 1 *** *** block prog 1 *** 53 ------------------------------------- *** block erase 433 *** *** block prog 433 *** *** block erase 0 *** *** block prog 0 *** 54 ------------------------------------- *** block erase 434 *** *** block prog 434 *** *** block erase 829 *** *** block prog 829 *** 55 ------------------------------------- *** block erase 435 *** *** block prog 435 *** *** block erase 828 *** *** block prog 828 *** 56 ------------------------------------- *** block erase 436 *** *** block prog 436 *** *** block erase 829 *** *** block prog 829 *** 57 ------------------------------------- *** block erase 437 *** *** block prog 437 *** *** block erase 828 *** *** block prog 828 *** 58 ------------------------------------- *** block erase 438 *** *** block prog 438 *** *** block erase 829 *** *** block prog 829 *** 59 ------------------------------------- *** block erase 439 *** *** block prog 439 *** *** block erase 828 *** *** block prog 828 *** 60 --------------------------But it seem to be just another two block numbers.
-
@serafimsaudade There seem to be a discussion about superblock wear: https://github.com/littlefs-project/littlefs/issues/376
But this is from early 2020, and the Pycom firmware uses littlfs 1.x, which is before April 2019, when 2.0 was published.Version 2 improved wear leveling and prevents wear of the superblocks. See: https://github.com/littlefs-project/littlefs/releases/tag/v2.0.0
P.S.: At 3300 file writes still only 2 superblock pairs are used, and not even at the same ratio. Blocks (0, 1) and (828, 829) are at a ratio of about 2:1
-
@serafimsaudade Using a set of 512 different file names, the number of superblock pairs went up to 8. In a sequence of about 10000 file write, the following frequencies of these block pairs occured:
Block Frequency (0,1) 1120 (44,45) 2028 (755,756) 1200 (817,818) 1170 (828,829) 1000 (881,882) 1147 (943,944) 1145 (1004,1005) 1147Interestingly, one block pair was used about twice as often as the others, like in the previous test with 128 different file names.
-
Tks very much for the work you have been doing.
Checking the results you have post so far. The small numbers of superblock pairs used suggest that the flash will wear up must faster than expected?Best Regards,
-
@serafimsaudade said in External Flash lose files:
The small numbers of superblock pairs used suggest that the flash will wear up must faster than expected?
Yes, that's my fear. The interesting thing is, that even if the chip is told to withstand 100000 cycles, it lasted several million cycles. But that is not something one can trust on. Technically there are only a few option:
- Using a more robust memory type. That can be a cf card, which have wear leveling embedded. Or something a F-RAM module.
- Using LFS version 2. I will check, a) if lfs2 behaves different, and b) how much work it is to replace the existing LFS1 by LFS2. As far as I recall, the file systems are not compatible. So there is no OTA update, but at least the hardware stays the same. The Micropython.org version of MicroPython uses LFS2, so I can use that one for testing.
-
@serafimsaudade Some results;
a) Using LFS2 just means to replac four files of teh source by the newer version. But the file system changes.
b) Even then, block level wear leveling has to be enabled. And that flag exists in V1 too. So I go back to V1 and test, whether that works in V1 too. -
@serafimsaudade So it looks like the option of block level wearout protection works for V1 too. I tested it with a setting of 100. But I imageine that this setting eats up superblocks (and space) permanently. So a level of 10000 could be more appropriate.
Edit: The file & line to be changed: littlefs/sflash_diskio_littlefs.c, line 48
Edit2: Looking in to the code, the actual LFS version used by pycom is 2.0, not 1.x as I assumed earler. The other version tried is 2.3. So not much difference then. It was just the non-enabled block-wear protection.
-
I have been watching this thread with interest.
What does it all mean in terms of realistic flash file system life.
I have an application where I am updating the data in a file (2K bytes) every minute.
I am now concerned that the flash will wear out relatively quickly.
I was using LFS but had to switch back to FAT due to some peculiarities with the way my code was functioning.
I feel that I am going to need wear levelling otherwise the memory is likely to wear out in a few years. I need my devices to have a life expectancy of at least 10 years.
Any ideas on the best approach to ensure maximum flash life????@peterp & @Gijs do you have any information that you can share on flash life, filesystem types, wear levelling and how to optimise its use??
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login