lfs_file_truncate cause file corruption

[ajaybhargav]

Hi @geky,

LFS has been acting strange. There is no error during read/write but file data itself is corrupt with lot of 0xFF in between correct data. I dumped the file (using lfs API, not raw SPI) on console, so you can see reading file is working fine. But the data inside is corrupted. See below:

The actual data... and then bunch of 0xFF

after 3220 bytes I get the correct data again:

The correct data starts with header, AIS/AHTT and ends with length of data written which is 0x70 in this case as highlighted above. We have only 4 files in the filesystem to store data logs. The current error case, size of files are are as follows. both normal_pri.bin and normal_sec.bin are corrupt.
0byte critical_pri.bin
232bytes critical_sec.bin
36804bytes normal_pri.bin
37180bytes normal_sec.bin
What I also observed is file also contains information or root record (metablock?) how can that be possible? it should not be part of file at all. see the file actually starts with this:

Any help would be appreciated.

Regards,
Ajay Bhargav

[lsilvaalmeida]

Hello @ajaybhargav. I think understand your "problem". In fact, I think that is the expected behaviour. In littlefs, a file takes at least one block (if not an inline file). So your file really end with 0x70 (see the last word befor the Fs, in little endian). After the end of your file, the rest of the block is "empty" that mens, normally, that all bits are set to 1.

The others files are in others blocks with the same structure.

I hope this helps.

[ajaybhargav]

Hi @lsilvaalmeida I think you missed the point. I mentioned clearly the dump images i posted is actual data of file not SPI block data. So I do not expect RAW SPI block data to show up in files. I know the data i write ends with 0x70 (I mentioned that) but I do not expect to see 0xFF in it. What file data should show is continuous stream of
[header][data][len][header][data][len]
like this... but something else is happening.

I did another test of writing data to two log files at 5 sec and 10sec interval respectively. The corruption ended up getting me two files with same file size and with same data. Which is not expected. as both files have different frequency of writing so cannot have same size.

[ajaybhargav]

I must explain what I do here, I store logs in file system. The logs are appended to a log file with following structure:
{ [header] [CRC] [Data] } [Len of data in {}]
The data is appended one after the other. Now when I retrieve the logs, I delete them from file on successful retrieval. Based on number of logs retrieved I truncate the file to length of data read. I am able to read 100 to 150 logs.. sometimes even more. But retrieval stops after some time (not fixed count). The moment it stops, I end up with a corrupted file. I have no idea why file truncate causes file to corrupt. Maybe it has to do something with the way file is organized during truncation.

[Johnxjj]

Can you give a simple code example to locate the problem quickly.

[ajaybhargav]

Here is a sample code.
lfs.zip

Zip contain 2 file:
lfs_write.c -> generates packet
Command line to test:
./lfs_write <packets> [format]
packets: Num of packets, default 50
format: optional, format lfs before use.

lfs_read.c -> read, verify and delete packet using truncate.
Command line:
./lfs_read

Problem starts anywhere from 100 packets. try it once or twice and you will get corrupt len error.

Edit:
Wrong files attached earlier. Updated now.

[ajaybhargav]

Did you get a chance to test this?

[ajaybhargav]

There some issue in truncation logic when file goes beyond block size. As a workaround I changed my logging code, So instead of writing in a single file. I write into multiple files while keeping the file size under 4K (block size). I do not see any issue while doing truncate on file. Please fix this issue.

[ryankurte]

We were observing what appears to be the same issue, where truncate is used on long (file >> block size) files sometimes the dirent or the whole file system end up corrupted.

After a bunch of fumbling around, increasing the cache and lookahead sizes both from 16 bytes to 128 bytes seems to have alleviated this in our case (although we have not exhaustively tested this yet).

cc. @webworxshop

[Indumathi1412]

I am facing file size resets, (file truncates) after file block size reached. How it can be solved?
My file open parameters are
lfs_file_opencfg(&lfs, &fileinit, filename,
(LFS_O_RDWR | LFS_O_CREAT | LFS_O_APPEND), &buff);

[ajaybhargav]

Here is a sample code.
lfs.zip

Zip contain 2 file:
lfs_write.c -> generates packet
Command line to test:
./lfs_write <packets> [format]
packets: Num of packets, default 50
format: optional, format lfs before use.

lfs_read.c -> read, verify and delete packet using truncate.
Command line:
./lfs_read

Problem starts anywhere from 100 packets. try it once or twice and you will get corrupt len error.

Edit:
Wrong files attached earlier. Updated now.

@ryankurte I have provided sample code to test. It might be old.

@Indumathi1412 I am keeping filesize under 4K (block size for SPI flash I used) and that workaround is working well for me as of now. Try to do the same and see if it works.

[Indumathi1412]

Thank you @ajaybhargav I kept 1K file size, beyond the size made file truncates but my physical memory address keeps on increasing, after it reaches max address, file corruption occurs as no more space is available. My requirement is not to immediately read the log data. Recent 10K(10 files each 1K size) log data is enough. Is it possible to write, the file from the start address. I tried lfs_remove() same behaviour. I am using Nor Flash IC.

[hgspbs]

truncate extends from wrong block

t.patch.txt