External Flash lose files

[robert-hh]

@serafimsaudade There seem to be a discussion about superblock wear: https://github.com/littlefs-project/littlefs/issues/376
But this is from early 2020, and the Pycom firmware uses littlfs 1.x, which is before April 2019, when 2.0 was published.

Version 2 improved wear leveling and prevents wear of the superblocks. See: https://github.com/littlefs-project/littlefs/releases/tag/v2.0.0

P.S.: At 3300 file writes still only 2 superblock pairs are used, and not even at the same ratio. Blocks (0, 1) and (828, 829) are at a ratio of about 2:1

[robert-hh]

@serafimsaudade Using a set of 512 different file names, the number of superblock pairs went up to 8. In a sequence of about 10000 file write, the following frequencies of these block pairs occured:

Block     Frequency
(0,1)       1120
(44,45)     2028
(755,756)   1200
(817,818)   1170
(828,829)   1000
(881,882)   1147
(943,944)   1145
(1004,1005) 1147

Interestingly, one block pair was used about twice as often as the others, like in the previous test with 128 different file names.

[serafimsaudade]

@robert-hh

Tks very much for the work you have been doing.
Checking the results you have post so far. The small numbers of superblock pairs used suggest that the flash will wear up must faster than expected?

Best Regards,

[robert-hh]

@serafimsaudade said in External Flash lose files:

The small numbers of superblock pairs used suggest that the flash will wear up must faster than expected?

Yes, that's my fear. The interesting thing is, that even if the chip is told to withstand 100000 cycles, it lasted several million cycles. But that is not something one can trust on. Technically there are only a few option:

• Using a more robust memory type. That can be a cf card, which have wear leveling embedded. Or something a F-RAM module.
• Using LFS version 2. I will check, a) if lfs2 behaves different, and b) how much work it is to replace the existing LFS1 by LFS2. As far as I recall, the file systems are not compatible. So there is no OTA update, but at least the hardware stays the same. The Micropython.org version of MicroPython uses LFS2, so I can use that one for testing.

[robert-hh]

@serafimsaudade Some results;
a) Using LFS2 just means to replac four files of teh source by the newer version. But the file system changes.
b) Even then, block level wear leveling has to be enabled. And that flag exists in V1 too. So I go back to V1 and test, whether that works in V1 too.

[robert-hh]

@serafimsaudade So it looks like the option of block level wearout protection works for V1 too. I tested it with a setting of 100. But I imageine that this setting eats up superblocks (and space) permanently. So a level of 10000 could be more appropriate.

Edit: The file & line to be changed: littlefs/sflash_diskio_littlefs.c, line 48

Edit2: Looking in to the code, the actual LFS version used by pycom is 2.0, not 1.x as I assumed earler. The other version tried is 2.3. So not much difference then. It was just the non-enabled block-wear protection.

[tuftec]

I have been watching this thread with interest.
What does it all mean in terms of realistic flash file system life.
I have an application where I am updating the data in a file (2K bytes) every minute.
I am now concerned that the flash will wear out relatively quickly.
I was using LFS but had to switch back to FAT due to some peculiarities with the way my code was functioning.
I feel that I am going to need wear levelling otherwise the memory is likely to wear out in a few years. I need my devices to have a life expectancy of at least 10 years.
Any ideas on the best approach to ensure maximum flash life????

@peterp & @Gijs do you have any information that you can share on flash life, filesystem types, wear levelling and how to optimise its use??

[robert-hh]

@tuftec The experience of @serafimsaudade indicates that.
But enabling block wear protection requires just setting that value from 0 to e.g. 100000. That's what the flash supplier guarantees.
With an update every minute you will be at 100000 write cycles after 70 days. With the numbers seen by @serafimsaudade it would be ~6 years.

[tuftec]

@robert-hh So what are you saying? With a FAT file system, does it keep writing the file to the same place or will it exercise some degree of levelling? If not it will wear out that block of flash in 70 days!!!!!

If I change back to LFS, assuming my original crashing problem does not resurface, can I then get wear levelling to increase this to potentially 6 years?

How do I make the changes to ensure the levelling is activated correctly? Is this something that can be done at run time or does it need a recompile of pycom os (I have never done this before and not keen on starting)?

[tuftec]

I have current product in the field that uses a PIC micro at its heart. This product writes data back to flash every 2 mins. I have a sizable number of units in the field, in remote applications that have been operating fault free for 6 years. Many of them have not been powered down or reset for over 2 years.
I do not do anything special to preserve the flash memory on these units. They just work.
I had expected to have a similar experience with the FiPy devices that I am planning to deploy into similar applications.

[robert-hh]

@tuftec With respect to flash wear, FAT is bad. The FAT file system uses a file allocation table (FAT) at a fixed location, and the master directory also at a fixed location. The FAT is written whenever a file is created, or extended. The directory is written on every action that changes a file, at least on file close. So in your case, that FAT may not be changed on every file write, but the directory is. With FAT, the file data may be written at different places, but maybe not. With a block size of 4 K and a file size of 2k, it may as well be that the data is always written to the same block.
LFS has at the moment at least wear leveling for the file data, but not for the superblock.
The change to enable block level wear leveling requires re-compiling the code. But the file system may stay.

[robert-hh]

@tuftec said in External Flash lose files:

I have current product in the field that uses a PIC micro at its heart.

Even if flash vendors ensure at least 100000 write cycles, the actual numbers that can be achieved is several million. It depends a lot on the operating conditions. So 6 years of operation with an update every 2 minutes means ~ 1.6 Million cycles. The devices of @serafimsaudade broke after ~3 Million cycles.

[robert-hh]

@tuftec said in External Flash lose files:

If I change back to LFS, assuming my original crashing problem does not resurface, can I then get wear levelling to increase this to potentially 6 years?

When chanhing back to LFS, and since you have to re-compile the code anyhow, you may consider updating LFS from the actual version 2.0 to the recent version. Maybe the problem will disappear then.
P.S.: I am not overly happy with LFS. FAT is faster and seems more reliable under normal conditions. However, with FAT you will end up with a corrupted file system on a power fail during writes.

[tuftec]

@robert-hh power fail corruption is less of an issue for me since i have permanent solar/battery power. A short flash life is a major drama however.
I might be able to reconfigure my code to use the nvram methods but I am not sure that will improve the situation.

Thanks.

[robert-hh]

@tuftec NVRAM is also flash based, but is told to implement protection against flash wear.

[serafimsaudade]

@robert-hh said in External Flash lose files:

@serafimsaudade So it looks like the option of block level wearout protection works for V1 too. I tested it with a setting of 100. But I imageine that this setting eats up superblocks (and space) permanently. So a level of 10000 could be more appropriate.

Edit: The file & line to be changed: littlefs/sflash_diskio_littlefs.c, line 48

Edit2: Looking in to the code, the actual LFS version used by pycom is 2.0, not 1.x as I assumed earler. The other version tried is 2.3. So not much difference then. It was just the non-enabled block-wear protection.

Tks @robert-hh
I will compile an new fw with the change.
Do you think with this change the wipy3 that have flash wearout will be reusable?

Best Regards,

[Lourenço Frade]

Greetings Robert. I'm working with @serafimsaudade in this issue, and we believe there may be a way to reuse the flash if we change the starting address of the starting address (sflash_diskio.h, line 26) by 128kbytes (essentially, avoiding the first two super block addresses, and initiating the block cycle from superblock 2). The problem is we are uncertain whether this would shift ALL the flash, including the flash RAM the firmware partition, or if the address defined is just relative to the volatile memory.

[robert-hh]

@serafimsaudade said in External Flash lose files:

Do you think with this change the wipy3 that have flash wearout will be reusable?

If the problem was flash wearout, then this single change will most likely not bring the broken device to live again, because block 0 and 1 are defective. One possibility I see is to shift the file system a little bit up in memory. If that applies to bloc 0 and 1, four block should be sufficient. For that, you could to modify esp32/fatfs/src/drivers/sflash_diskio.h, lines 24 to 28 as follows:

#define SFLASH_FS_SHIFT                 4
#define SFLASH_BLOCK_COUNT_8MB          MICROPY_PORT_SFLASH_BLOCK_COUNT_8MB - SFLASH_FS_SHIFT
#define SFLASH_FS_SECTOR_COUNT_8MB      ((SFLASH_BLOCK_SIZE * SFLASH_BLOCK_COUNT_8MB) / SFLASH_FS_SECTOR_SIZE)
#define SFLASH_START_ADDR_8MB           (0x00400000 + SFLASH_BLOCK_SIZE * SFLASH_FS_SHIFT)
#define SFLASH_START_BLOCK_8MB          (SFLASH_START_ADDR_8MB / SFLASH_BLOCK_SIZE)
#define SFLASH_END_BLOCK_8MB            (SFLASH_START_BLOCK_8MB + (SFLASH_BLOCK_COUNT - 1)) - SFLASH_FS_SHIFT

Edit: The symbol SFLASH_BLOCK_COUNT is not defined at all. But since SFLASH_END_BLOCK_8MB and SFLASH_END_BLOCK_4MB are not used at all, it's not a problem. I guess, the line should be:

#define SFLASH_END_BLOCK_8MB            (SFLASH_START_BLOCK_8MB + (SFLASH_BLOCK_COUNT_8MB - 1))```

[robert-hh]

@Lourenço-Frade Hello. I just suggested the same, of shifting the file system 4 blocks up. As far as I understand, this is just the file system which will be moved, as it is in the upper area of the flash. With my suggested change, the size is also accordingly reduced. So it should not affect any other objects in flash.

[serafimsaudade]

@robert-hh

Shifting the file system seems to make it.
I got one wipy3 up and running. :)
I will start testing the rest.

Best Regards,