External Flash lose files

[serafimsaudade]

@robert-hh

Ok, Tks Robert, I will try that.

Best Regards,

[tuftec]

Hi Guys, Thanks for that. Some really good insight.
I do not have experience in re-building Pycom.
Has anyone built a very recent version 1.20.2.r3 or later that has the enhanced lfs in it. Last time I tried to use lfs, it didn't work very well for me.

Peter.

[robert-hh]

@tuftec For which board. And do you needPyBytes support?

[tuftec]

No Pybytes and FiPy.
Thanks

[robert-hh]

@tuftec it's here: https://github.com/robert-hh/Shared-Stuff.git
Changes to the recent Dev version:

• use lfs 2.3 with block level wear-leveling enabled
• support UART inverted mode
• enforce guard time for the RGB LED
• some minor tweaks, which may not apply for FiPy

In addition, the frozen modules contain the upysh module (see: https://github.com/micropython/micropython-lib/tree/master/upysh) and the onboard editor pye (see: https://github.com/robert-hh/Micropython-Editor)

Edit: LFS2.3 clears all files. So save them beforehand.

[tuftec]

@robert-hh Thanks. I will give it a try.

[tuftec]

@robert-hh unfortunately the LFS 2.3 build you created does not work for me.
It dumps core when trying to close 2 x open JSON files, as does the previous LFS build that I tried.

It appears to fail between the first and second file close.
You made a comment about LFS being slower. Is there a chance that maybe the second close interferes with the first (that has not completed)????

I am going to have to ultimately run LFS to get the flash life that I need.

Any clues further ideas.

I will try to add some delays to see what happens.

[robert-hh]

@tuftec I started on Saturday another long term test running (writing million of times a single file). The test is now at ~800_000 writes. During that test, I see regular core dumps. Not at fixed intervals. I'm on it to find a reason. Alternatively I could go back to LFS2.03 with wear leveling enabled.
So you have seen the same error with the previous LFS? Then going back might not solve the problem.

P.S.: I used to prefer FAT because it is more reliable in normal operations. However crashes during files writes have a high chance of total file system corruption.

[tuftec]

@robert-hh adding 500ms delays before and during the file close sequence made no improvement.

It appears to fail at the second close.

The first file is only open for reading.
The second file has been open for writing.

There is a distinct possibility that the second file was closed at an earlier stage in the process before this close is called. But this should not cause a crash.

Any ideas.

[tuftec]

@robert-hh Yes, I have seen core dumps with previous versions of LFS. That is why I moved to FAT again. But I need the improved file system life.

[tuftec]

@peterp do you have any thoughts on this one? Solving this one would make the system so much more reliable.

[robert-hh]

@tuftec Does your error happen every time you try?

[tuftec]

@robert-hh Yes. Happens every single time.

[robert-hh]

@tuftec Can you post the trace dump? Then I can loop the code with the elf file.

[tuftec]

@robert-hh I will try to capture this tomorrow when I have some time.

[robert-hh]

@tuftec I have looked at four trace dumps in my tests, and all happen during a file close operation. Somewhere down the line, a ISR Watchdog timer fires after entering a critical section. So it looks like a lock-up condition.
At littlefs/vsf_littlefs_file.c, line 104-105 looks interesting. There, the semaphore is taken and then the close call proceeds, which finally ends up in the ISR Watchdog event. The ISR is caused by the rmt driver, which is called for the heartbeat LED!

[robert-hh]

@tuftec Could you try to run the test with heartbeat LED off?

[tuftec]

@robert-hh the heartbeat led is disable in my application already. I will try to capture and send the dump a little later. Thanks.

[robert-hh]

@tuftec Can you also send the shortest version of the script, that causes the bug? I have a FiPy here running w/o problems.
Edit: Do you use the RGB led for anything else?

[tuftec]

@robert-hh I use the heartbeat led for various status indications.
I do not currently have a cut-down test version of the script currently.

Here is the crash dump.

backed up nv data
18:08:23: closed file1
abort() was called at PC 0x40099b38 on core 1

ELF file SHA256: 0000000000000000000000000000000000000000000000000000000000000000

Backtrace: 0x40091f2c:0x3ffe2bf0 0x40092179:0x3ffe2c10 0x40099b38:0x3ffe2c30 0x40085476:0x3ffe2c60 0x40084f89:0x3ffe2c80 0x4000bec7:0x3ffe2ca0 0x400f882a:0x3ffe2cc0 0x400f93c5:0x3ffe2ce0 0x400fa076:0x3ffe2d00 0x400fa13c:0x3ffe2d20 0x4010d399:0x3ffe2d40 0x40108161:0x3ffe2d70 0x40104565:0x3ffe2d90 0x401045f5:0x3ffe2db0 0x40111f4b:0x3ffe2dd0 0x40108274:0x3ffe2e70 0x40104565:0x3ffe2ea0 0x401045f5:0x3ffe2ec0 0x40111f4b:0x3ffe2ee0 0x40108274:0x3ffe2f80 0x40104565:0x3ffe2fe0 0x401045f5:0x3ffe3000 0x40111f4b:0x3ffe3020 0x40108274:0x3ffe30c0 0x40104565:0x3ffe3130 0x40104592:0x3ffe3150 0x400e7351:0x3ffe3170 0x400e75e1:0x3ffe3210 0x400e6029:0x3ffe3230

Rebooting...
18:08:26: 

Peter.