How to generate a new mdf file size-efficiently?

[sirlagg]

Hi, I am working on a script that makes modifications (corrections) to existing data. At the moment it changes units and applies a factor to the data (multiplying the data by a factor). At the moment, the function I have come up with the following function (this works):

def applyFactor2(yop, signal_dict):
    # yop  = MDF object
    # signal_dict = { (group_index, channel_index): (float factor, str unit),  ... }
    # v2 that operates with the full group at once in order to safe space reusing the same timebase
    yop2 = MDF(version=yop.version)

    for g_idx, group in enumerate(tqdm(yop.groups, leave=False)):
        acq_name = group.channel_group.acq_name # String
        acq_source = group.channel_group.acq_source # Source object
        channel_names = [channel.name for channel in group.channels] # List of strings
        channel_addresses = [(g_idx, c_idx) for c_idx in range(len(channel_names))] # List of tuples
        try:
            signals = yop.select([(None, g_idx, c_idx) for c_idx in range(len(group.channels))], raw=True, copy_master=False) # List of Channel objects;
        except:
            continue
        
        for address in channel_addresses:
            if address in signal_dict:
                ch = signals[address[1]]
                factor, unit = signal_dict[(ch.group_index, ch.channel_index)]
                ch.unit = unit
                if ch.conversion and ch.conversion.conversion_type == 1:
                    ch.conversion.a = ch.conversion.a * factor
                    ch.conversion.b = ch.conversion.b * factor
                else:
                    ch.samples = ch.samples * factor
        yop2.append(signals, acq_name=acq_name, acq_source=acq_source, common_timebase=True)

    yop2.header.start_time = yop.header.start_time
    return yop2

The idea here is to use the same timebase for the whole group so it saves space (in v1 I was not doing so and the resulting file was much bigger).

However, I am seeing that the resulting file is still significantly bigger than the original (125MB vs 73MB). Checking it in CANape, I can see that in the new file, there is a "t" signal for each Message that is not present in the original file; which makes me guess that the increased size comes from here. Is it there a better way of doing this? Is it there a better way of doing the "append"? Many thanks in advance.

[sirlagg]

I have made some modifications to the function so that the master is not appended. This has helped a lot reducing the size, however it is still bigger than originally (94MB vs 73MB). If a file is re-processed with this method, the size marginally changes. I have also noticed that if I simply load the original file and then save it with compression=2, I am still reducing the file size. All of these make me think there must be something going on during the iteration+append that ends up generating a different (larger) mdf file.

def applyFactor2(yop, signal_dict):
    # yop  = MDF object
    # signal_dict = { (group_index, channel_index): (float factor, str unit),  ... }
    # v2 that operates with the full group at once in order to safe space reusing the same timebase
    yop2 = MDF(version=yop.version)

    for g_idx, group in enumerate(tqdm(yop.groups, leave=False)):
        acq_name = group.channel_group.acq_name # String
        acq_source = group.channel_group.acq_source # Source object
        channel_names = [channel.name for channel in group.channels] # List of strings
        channel_addresses = [(g_idx, c_idx) for c_idx in range(len(channel_names))] # List of tuples
        try:
            signals = yop.select([(None, g_idx, c_idx) for c_idx in range(len(group.channels))], raw=True, copy_master=False) # List of Channel objects;
        except:
            continue
        
        for address in channel_addresses:
            if address in signal_dict:
                ch = signals[address[1]]
                factor, unit = signal_dict[(ch.group_index, ch.channel_index)]
                ch.unit = unit
                if ch.conversion and ch.conversion.conversion_type == 1:
                    ch.conversion.a = ch.conversion.a * factor
                    ch.conversion.b = ch.conversion.b * factor
                else:
                    ch.samples = ch.samples * factor
        
        # Original
        # print(g_idx, yop.masters_db[g_idx], signals[0].name) # Debug line
        # yop2.append(signals, acq_name=acq_name, acq_source=acq_source, common_timebase=True)

        # Append all signals but the master
        signals.pop(yop.masters_db[g_idx])
        yop2.append(signals, acq_name=acq_name, acq_source=acq_source, common_timebase=True)

        # Remove all signals names 't'. Same result as removing the master signal but very slightly slower
        # signals = [ch for ch in signals if ch.name != 't']
        # yop2.append(signals, acq_name=acq_name, acq_source=acq_source, common_timebase=True)

    yop2.header = yop.header
    return yop2