zipfile use on MDF

[carguaga]

First is first: let me thank you (Daniel and the participants) for the fast and efficient mf4 interpreter implementation. I am impressed on how well it performs when compared with the ones provided by other popular SWs.

I am currently benchmarking access times for several options, and I found a problem when trying to access a zipped mf4.

If I run:

mf4 = MDF(zipfile.ZipFile('XXX_BackUp_22-02-21_081529_445.zip'))

I get the following error:

Traceback (most recent call last):
  File "ford.py", line 66, in <module>
    mf4 = MDF(zipfile.ZipFile('KUGA_BackUp_22-02-21_081529_445.zip'))
  File "/usr/local/lib/python3.7/dist-packages/asammdf/mdf.py", line 256, in __init__
    name = original_name = Path(name)
  File "/usr/lib/python3.7/pathlib.py", line 1003, in __new__
    self = cls._from_parts(args, init=False)
  File "/usr/lib/python3.7/pathlib.py", line 658, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/usr/lib/python3.7/pathlib.py", line 642, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not ZipFile

I can call with no issues zipfile.ZipFile('XXX_BackUp_22-02-21_081529_445.zip'), or mf4 = MDF('XXX_BackUp_22-02-21_081529_445.mf4).

My asammdf version details:

Name: asammdf
Version: 7.0.7
Summary: ASAM MDF measurement data file parser
Home-page: https://github.com/danielhrisca/asammdf
Author: Daniel Hrisca
Author-email: [email protected]
License: LGPLv3+
Location: /usr/local/lib/python3.7/dist-packages
Requires: canmatrix, lz4, numexpr, numpy, pandas, typing-extensions
Required-by:

Thanks in advance,

Carlos

[danielhrisca]

There is a bug that the zipfile support was deleted from the code base, but you can give the zip name directly instead

mf4 = MDF('XXX_BackUp_22-02-21_081529_445.zip')

[carguaga]

Effectively this worked. I little slower than I'd have wished, but works.

Thanks, Daniel

[danielhrisca]

It has to unzip the file first. How big is it?

[carguaga]

I am currently benchmarking with a 300 MB uncompressed MF4, while the final system will probably use 50 MB files. The file must be transferred over ethernet from the DAQ PC to the target HW, where it is processed with python. I am evaluating the pros/cons of compressing the file before transmission, either using compressed MDF4, zip file, or converting to other data format as mat.

[danielhrisca]

I think you can get better performance by compressing using lz4 with a faster compression level on the DAQ side and decompressing on the HW target before feeding it to the MDF processing script.

[carguaga]

Thanks for the hint. I will study that option too.