Reading MF4 with errors

[markwaterbury]

What is the intent of asammdf when it comes to handling invalid files (files where MDF Validator shows errors)? Is it expected that asammdf may throw an error, or does it attempt to read the file, despite potential corruption?

Asking because I have a file with some errors (shown below), and asammdf is not able to read the file, but other tools I have tried can open it (likely with some loss of data).

This is the error from asammdf when calling MDF():

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python\Python311\Lib\site-packages\asammdf\mdf.py", line 398, in __init__
    self._mdf = mdf_v4.MDF4(name, channels=channels, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python\Python311\Lib\site-packages\asammdf\blocks\mdf_v4.py", line 393, in __init__
    self._read(mapped=True, progress=progress)
  File "C:\Python\Python311\Lib\site-packages\asammdf\blocks\mdf_v4.py", line 652, in _read
    data_blocks = list(data_blocks_info)  # load the info blocks directly here
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python\Python311\Lib\site-packages\asammdf\blocks\mdf_v4.py", line 2002, in _get_data_blocks_info
    id_string, block_len = COMMON_SHORT_uf(stream, addr)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
struct.error: unpack_from requires a buffer of at least 4561008 bytes for unpacking 16 bytes at offset 4560992 (actual buffer size is 4492448)

#4 from the validator is what is causing the error.

Adding the following code to _get_data_blocks_info() in mdf_v4 allows the invalid block to be skipped, and the rest of the file can be read without issue.

Add this:

if addr > self.file_limit:
    logger.warning(f"Data block address {addr:X} is outside the file size {self.file_limit}")
    break

Here:

asammdf/src/asammdf/blocks/mdf_v4.py

Line 2001 in 15b1032

Is this an acceptable solution to allow recovery of a file that is broken? I recognize that this covers just one of many possible cases where the address to a block might point outside the file.

[JulienGrv]

I think it would be nice to be able to continue on errors when possible but it should be an opt-in feature (not default) gated behind an option (name examples: warn_on_error or recover_on_error) or using warnings when such error is encountered.

[markwaterbury]

@danielhrisca should I open an issue for this? Basically, I am wondering if asammdf throwing an error on this file is expected behavior, or if a change like the code I posted is worth including to try and read the file, essentially ignoring the faulty blocks?

[danielhrisca]

@markwaterbury please try the development branch code

from asammdf import MDF, set_global_option, get_global_option

set_global_option("raise_on_incomplete_blocks", False)

m = MDF(r'myfile.mf4')