External Flash lose files
-
Hi,
I have install in 2019 11 wipy 3. And used flash to save some configs. Everything has runs ok until December 2020, when all the 11 wipy became offline. I bring them to the office to check the problem.
The firmware installed on the devices didn't find the configs files saved into the flash.
So I checked the flash on a FTP client, and seemed formated.
I sended the configs files again over FTP and reboot the wipy and the firmware didn't boot because of missing config files. Then I opened the flash over FTP client and the flash has empty again.
All the 11 wipy got the same problem. This is every concern.I'm afraid that the external memory flash are faulty on all the 11 wipy.
I used the LFS on all devices.Best regards,
Luís Santos -
@serafimsaudade Please tell, which firmware version you are using and whether you made any changes to the device.
It is unlikely that all eleven devices have the same hardware fault. That looks more like a software problem. -
Hi @robert-hh
I'm using the version 1.20.2.RC9 from the github, because I compile custom frozen files.
Tomorrow I will program one the faulty wipy using the update manager and the official firmware from pycom, and check if the flash retain the config files.Best regards.
-
Hi Again.
I got an update about my problem.
I updated the firmware of one the fault wipy using the pycom update manager to the latest release V1.20.2.R4.
For this test I didn't compile the firmware from gitbhub with custom fronzen files.
Over FTP I copy the software into the flash. When I reset the pycom the software fail to start returning an error. Checking the flash it was empty.Best regars.
-
@serafimsaudade What I understand:
- you updated the firmware with the pycom updater to V1.20.2.R4. Did you use the USB interface for that? Did you check the "erase flash" option to have a well defined initial state?
- You uploaded you application file with ftp.
And then, after reset, the flash is empty. That looks like a corrupted file system. Therefore it is important to start the re-build with a full flash erase.
-
I use an USB/uart converter.
I check the "erase flash".
Yes I uploaded the application with FTP.I also did another test with the pycom-fwtool-cli:
pycom-fwtool-cli -p /dev/ttyUSB0 erase_all Running in FTDI mode Erasing the board can take up to 40 seconds. Board erased successfully in 15.07 seconds pycom-fwtool-cli -p /dev/ttyUSB0 flash -t WiPy-1.20.2.r4.tar.gz Running in FTDI mode Script Version: 2.1 Flash operation successful.And the result is the same.
Best Regards,
-
@serafimsaudade After the two steps you shoed in the last post the flash is indeed empty, beside a skeleton boot.py and main.py.
-
After done all this tests I'm very concern about the flash sanity.
It is possible the flash has reach the write limit cycles ?PS: last year I have install +- 150 units.
Best Regards,
-
@serafimsaudade That depends on how many write operations are performed on the flash. The typical figure a vendor guaratees is 100000 erase/write cycles. But practically more write cycles are possible, and it should not affect all devices at the same time. Additionally, LFS has mechanisms for wear leveling, such that all blocks of the flash are written equally, not a few blocks more that others, like it happens with the FAT file system.
Taking the number of 10**5 in a year, that would mean ~274 write events of a single block every day - or one every 5 minutes. Can you estimate how often your devices write to flash?
-
Well, after doing a double check into the code. This 11 wipy write to flash +- every 10sec.
Best Regards,
-
@serafimsaudade That's indeed quite often, about 3 Million writes/year. A few more questions:
- how many byte do you write every time (average)
- do you close the file after every write? If yes, data will be written, if no, data will be buffered and only written, if the buffer is full or the file is closed.
- which WiPy do you use, WiPy2 or WiPy3? That matters with respect to the size of the file system.
I do not remember the details of the LFS implementation, whether they do some kind of journalling or not. In any case the fault safe property should cause more write operations.
-
@serafimsaudade I understood first that you have 11 devices in total and all failed at the same time. However, if 11 out of 150 fail in a short range of time, then a hardware problem could be the cause.
I do not see an easy way of verification that hypothesis. On a quick search I did not find test code for the flash chip.
You could obviously use a Arduino or espressif software environment and create a small program that tries writing and reading the flash. Or you open a device and replace the flash chip. -
@robert-hh said in External Flash lose files:
@serafimsaudade That's indeed quite often, about 3 Million writes/year. A few more questions:
- how many byte do you write every time (average)
- do you close the file after every write? If yes, data will be written, if no, data will be buffered and only written, if the buffer is full or the file is closed.
- which WiPy do you use, WiPy2 or WiPy3? That matters with respect to the size of the file system.
I do not remember the details of the LFS implementation, whether they do some kind of journalling or not. In any case the fault safe property should cause more write operations.
I write +- 200 bytes;
I close the file after every write.
I'm using the WiPy3.Best Regards
-
This 11 wipy3 are the only with 10 sec write.
The other +-139 wipy3, write to flash every 60sec.Best Regards,
-
@serafimsaudade OK. So if it is a flash wear problem, you have about 6 years time. Besides that....
Doing a quick calc. The WiPy3 flash file system is 4 MB, or 1024 (or about 10^3) blocks of 4 k. With a well distributed wear leveling, one could make at least 10^8 block writes. That number has to be divided by the number of block writes caused by you file operation to get the number of safe file write. I do not know this number. To get that, I could add debug messages to the LFS. For that, I have another question:
- How is the actual file write done. Do you write a single short file every time, or do you append the data to an ever increasing file. Or do you write to alternate files, ....
-
Sorry I mistype the file size, The wipy3 write to flash an file with +- 2K bytes.
"How is the actual file write done. Do you write a single short file every time, or do you append the data to an ever increasing file. Or do you write to alternate files, ...."
I open always the same file, write with no append, and close the file.
Best Regards,
-
@serafimsaudade Since the block size is 4k, writing 200 bytes or 2k bytes does not make a difference.
-
@robert-hh So I wrote a little python script:
# def run(): b = bytearray(2038) f = open("logdata", "w") f.write(b) f.close()And I added a few printf statements to sflash_diskio_littlefs.c, which print a line every time erase or write is called. Then I called up the little script a few times. This is the output:
>>> wtest.run() *** block erase 0 *** *** block prog 0 *** *** block erase 235 *** *** block prog 235 *** *** block erase 1 *** *** block prog 1 *** >>> wtest.run() *** block erase 236 *** *** block prog 236 *** *** block erase 0 *** *** block prog 0 *** >>> wtest.run() *** block erase 237 *** *** block prog 237 *** *** block erase 1 *** *** block prog 1 *** >>> wtest.run() *** block erase 238 *** *** block prog 238 *** *** block erase 0 *** *** block prog 0 *** >>> wtest.run() *** block erase 239 *** *** block prog 239 *** *** block erase 1 *** *** block prog 1 *** >>> wtest.run() *** block erase 240 *** *** block prog 240 *** *** block erase 0 *** *** block prog 0 ***The result is confusing. It shows some kind of wear leveling, the block with the changing numbers from 235 to 240 (going on to 1024 as I noticed and then skipping to 2), but also, that alternating block 0 and block 1 are written/erased. These are so called "superblocks". According to the spec, they should change after a while. I will let that run to see if it happens. If not, then that is a good candidate for flash failure.
P.S.: If you worry about me killing a device. No problem. It is an open one, on which I already had replaced a flash chip. And I have spare chips here.
-
-
@serafimsaudade No change so far after 5000 writes. I changed the scheme writing to 128 different file names. And then alternative superblock numbers were used. Small section from the log:
47 ------------------------------------- *** block erase 427 *** *** block prog 427 *** *** block erase 0 *** *** block prog 0 *** 48 ------------------------------------- *** block erase 428 *** *** block prog 428 *** *** block erase 1 *** *** block prog 1 *** 49 ------------------------------------- *** block erase 429 *** *** block prog 429 *** *** block erase 0 *** *** block prog 0 *** 50 ------------------------------------- *** block erase 430 *** *** block prog 430 *** *** block erase 1 *** *** block prog 1 *** 51 ------------------------------------- *** block erase 431 *** *** block prog 431 *** *** block erase 0 *** *** block prog 0 *** 52 ------------------------------------- *** block erase 432 *** *** block prog 432 *** *** block erase 1 *** *** block prog 1 *** 53 ------------------------------------- *** block erase 433 *** *** block prog 433 *** *** block erase 0 *** *** block prog 0 *** 54 ------------------------------------- *** block erase 434 *** *** block prog 434 *** *** block erase 829 *** *** block prog 829 *** 55 ------------------------------------- *** block erase 435 *** *** block prog 435 *** *** block erase 828 *** *** block prog 828 *** 56 ------------------------------------- *** block erase 436 *** *** block prog 436 *** *** block erase 829 *** *** block prog 829 *** 57 ------------------------------------- *** block erase 437 *** *** block prog 437 *** *** block erase 828 *** *** block prog 828 *** 58 ------------------------------------- *** block erase 438 *** *** block prog 438 *** *** block erase 829 *** *** block prog 829 *** 59 ------------------------------------- *** block erase 439 *** *** block prog 439 *** *** block erase 828 *** *** block prog 828 *** 60 --------------------------But it seem to be just another two block numbers.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login