MDF3 channel error

[zhangyk99]

Hello, @ ihedvall,
I encountered an error while creating the MFD3 file channel.

If the number of channels created exceeds a certain limit, data recording will encounter problems, and data errors are only related to the number of channels. For example, using one time channel, 200 double channels with a size of 8 bytes, and 200 string channels with a size of 64 bytes may result in garbled or missing data during recording.
I am attaching a test code to illustrate the above issues.
May I ask if there is a specific quantity limit when creating MDF3 channels, or if it is possible to support recording data from nearly 5000 data channels simultaneously.

TEST(MDF3Error, test) {
    uint16_t channelNum = 200;
    string filePath = "D:/test/mdf3_test.mdf";
    ifstream f(filePath);
    if (f.good()) {
        f.close();
        ASSERT_EQ(remove(filePath.c_str()), 0);
    }
    auto writer = MdfFactory::CreateMdfWriterEx(MdfWriterType::Mdf3Basic);
    writer->Init(filePath);

    auto *header = writer->Header();
    ASSERT_TRUE(header != nullptr);

    auto *dg3 = writer->CreateDataGroup();
    auto *cg3 = mdf::MdfWriter::CreateChannelGroup(dg3);
    {
        auto *master = mdf::MdfWriter::CreateChannel(cg3);
        master->Name("Time");
        master->Description("Time channel");
        master->Type(ChannelType::Master);
        master->DataType(ChannelDataType::FloatLe);
        master->DataBytes(4);
        master->Unit("s");
    }
    for (uint32_t i = 0; i < channelNum; i++) {
        auto *channel = mdf::MdfWriter::CreateChannel(cg3);
        channel->Name("channel_double_" + to_string(i));
        channel->Description("float");
        channel->Type(ChannelType::FixedLength);
        channel->DataType(ChannelDataType::FloatLe);
        channel->DataBytes(sizeof(float));
    }
    for (uint32_t i = 0; i < channelNum; i++) {
        auto *channel = mdf::MdfWriter::CreateChannel(cg3);
        channel->Name("channel_string_" + to_string(i));
        channel->Description("string");
        channel->Type(ChannelType::FixedLength);
        channel->DataType(ChannelDataType::StringAscii);
        channel->DataBytes(64);
    }

    writer->InitMeasurement();
    writer->StartMeasurement(TimeStampToNs());
    for (size_t sample = 0; sample < 10; ++sample) {
        auto cn_list = cg3->Channels();
        cn_list[0]->SetChannelValue(0.01 * static_cast<double>(sample));
        for (uint32_t i = 0; i < channelNum; i++)
            cn_list[i + 1]->SetChannelValue(11.1 * static_cast<double>(sample));
        for (uint32_t i = 0; i < channelNum; i++)
            cn_list[i + channelNum + 1]->SetChannelValue(std::to_string(sample));

        auto ns70 = TimeStampToNs();

        writer->SaveSample(*cg3, ns70);
        this_thread::sleep_for(10ms);
    }
    writer->StopMeasurement(TimeStampToNs());
    writer->FinalizeMeasurement();

    MdfReader oRead(filePath);
    oRead.ReadEverythingButData();
    const auto *file = oRead.GetFile();
    DataGroupList dg_list;
    file->DataGroups(dg_list);
    for (auto *dg: dg_list) {
        ChannelObserverList observer_list;
        auto cg_list = dg->ChannelGroups();
        for (const auto *cg: cg_list) {
            CreateChannelObserverForChannelGroup(*dg, *cg, observer_list);
        }
        oRead.ReadData(*dg);

        double value = 0;
        uint32_t count = 0;
        bool valid = true;
        std::vector<double> values_exp{};
        std::vector<double> values_read{};
        for (const auto &channel: observer_list) {
            if (count == 0) {
                count++;
                continue;
            } else if (count < channelNum + 1) {
                size_t samples = channel->NofSamples();
                values_exp.clear();
                values_read.clear();
                bool fail = false;
                for (size_t sample = 0; sample < samples; ++sample) {
                    values_exp.push_back(11.1 * static_cast<double>(sample));
                    channel->GetEngValue(sample, value);
                    values_read.push_back(value);
                    if(11.1 * static_cast<double>(sample) - value > 0.00001)
                        fail = true;
                }
                if(fail)
                    EXPECT_EQ(values_read, values_exp);

            } else {
                size_t samples = channel->NofSamples();
                values_exp.clear();
                values_read.clear();
                bool fail = false;
                for (size_t sample = 0; sample < samples; ++sample) {
                    values_exp.push_back(sample);
                    channel->GetEngValue(sample, value);
                    values_read.push_back(value);
                    if(value == sample)
                        fail = true;
                }
                if(fail)
                    EXPECT_EQ(values_read, values_exp);
            }
            count++;
        }
    }
}

Extract some running results:

error: Expected equality of these values:
values_read
Which is: { 1.7219155529623352e-41, 1.7793687899996527e-41, 1.815242030686368e-41, 1.8366818971905377e-41, 1.8869885120597987e-41, 1.9012817563959118e-41, 1.9443016192506837e-41, 1.9873214821054556e-41, 2.0304814748066599e-41, 2.0735013376614318e-41 }
values_exp
Which is: { 0, 11.1, 22.2, 33.3, 44.4, 55.5, 66.6, 77.7, 88.8, 99.899999999999991 }

error: Expected equality of these values:
values_read
Which is: { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }
values_exp
Which is: { 0, 11.1, 22.2, 33.3, 44.4, 55.5, 66.6, 77.7, 88.8, 99.899999999999991 }

C:\Users\79933\Documents\GitHub\PostMortemLogger\tests\PMLRecordTest.cpp(1171): error: Expected equality of these values:
values_read
Which is: { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }
values_exp
Which is: { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }

There are probably several types of errors mentioned above, and only one truncation should be selected for each type.

[ihedvall]

@zhangyk99
I will look into the bug.

1. Note that MDF (3/4) isn't good on storing string values.
2. The MDF3 only have a 32-bit file pointers which limit the file size. This might be a limit (< 4GB file size).
3. The 5000 channels might be a problem (see [2]) for MDF 3. MDF 4 should handle this better.
4. You have many string signals. Check if you can define them as enumerates instead.
5. The byte size for a CG (3) record is a 16-bit value.

[zhangyk99]

Greatly appreciate your help. I’ll check the CG record sizes and refactor the string signals as suggested.

[zhangyk99]

At the same time, I also tested channels that create a single double type, with an upper limit of approximately 2000 channels.

[ihedvall]

@zhangyk99
I can validate that this is a bug. I have not found the issue yet. The string sample values are correct while the first 8 float sample values are correct. The remaining float samples are 0. There is no obvious error in the code so I think this might take some day(s) to find.

[ihedvall]

@zhangyk99
Thanks for finding this bug. MDF 4 and MDF 3 have a mismatching bit and byte offset setting. I hope that the changes will work for you. The error occurred at byte offset > 8192 bytes.

Your secondary problem with 5000 channels should in principle work with MDF 3 but watch out for the 32-bit file position problem.
Switch to MDF4 in that case.

Please verify that my changes work in your application.

[zhangyk99]

@ ihedvall
Thank you for your help. I will conduct testing in my project and provide you with timely feedback on the results.

[zhangyk99]

@ ihedvall
I used testwrite 3 to test the generation of 5000 double channel files and read them without any problems, but when using asammbf 8.2.5 to read the corresponding generated files, there is still a problem with garbled data. I am not sure if it is a problem with the files being written or with asammdf reading data.

Asammbf 8.2.5 download link:
https://canlogger.csselectronics.com/files/asammdf/v825dev5/asammdfgui-v8.2.5dev5-windows.zip

[ihedvall]

@zhangyk99
It's most likely a problem in the MDF library. Its a problem around the "additional byte offset" interpretation. It takes a day to figure out the correct offset.

[ihedvall]

@zhangyk99
I fixed the "additional byte offset" problem but the number of record bytes in the CG block is a 16-bit value. In the test case above the limit is below ~950 channels so 5000 channels will not work in MDF 3. It's either switch to MDF 4 or limit the number of channels. Strings takes a lot of bytes in a sample record.

I have not checked in the fixes. It's bad exception when writing to many channels that I needs to fix. The write should just fail.

[zhangyk99]

@ ihedvall
Thank you for fixing the issue and restoring it in a timely manner.
At present, my project needs to meet the upper limit of setting 5000 double channels simultaneously, while only a few string channels need to be set. Previously, setting the same number of two channels in unit testing was only for the sake of simplicity. I hope you can appropriately expand the upper limit of the number of double channnels when making modifications.
Finally, thank you again for your help.