read CanMessage types from the MDF file

[sy950915]

Hello, I noticed in your library that when writing to an MDF file, we can directly pass the CanMessage type. I was wondering if it's also possible to directly read CanMessage types from the MDF file?

[ihedvall]

@sy950915
There is no read support for the CAN message wrapper. It's possible to add it but it's little bit tricky as many third party loggers stores the bits and bytes position differently for the composite channels (sub-channels).

The end-user should only subscribe on the CAN_DataFrame channel and of course the timestamp. Maybe a special subscriber type that inherit from the normal channel observer and have a new "GetCanMessage" function with sample index as input. Note that no subscription should be needed for the composite channel as they points into the CAN_DataFrame (the CG record actually).

I suppose the same requirement would be for the LIN, FlexRay, MOST and ETH message wrapper. The new subscriber needs to optimize how to get message data out of the CAN_DataFrame byte array. This should minimize the memory consumption.

[sy950915]

@ihedvall Yes, there is indeed a requirement for reading LIN, FR, and ETH data. Recently, I have been using mdflib to read and write MDF files. The progress has been relatively smooth, but I have encountered a few issues, such as:

When writing CAN error frames, after saving the MF4 file and importing it into CANoe 16, CANoe reports an error and does not recognize this information.

When writing ETH_Frame, CANoe also fails to recognize it.

[ihedvall]

@sy950915
I don't have the Vector CANoe software why it's tricky to trace the error. If you have the possibility to check the error messages from CANoe and forward them, I can check why it is not recognizing the CAN errors.

The ETH and LIN are new protocols that may not be tested on so many reader platforms. I have noticed that the Vector MDF Validator complains on some issues.

[sy950915]

@ihedvall

Yes, indeed. Currently, I've also observed some peculiar behavior when CANoe parses MDF files. The error shown in the screenshot above is quite strange—whenever I create LIN/ETH data groups in the MDF file, CANoe even fails to parse CAN data. I've also uploaded the MDF file I generated for reference.

Additionally, I'd like to ask: Can MDF files be read in chronological order? After all, when writing data, it's always sequential based on timestamps.

[sy950915]

@ihedvall

2.zip

[ihedvall]

@sy950915
I have analyzed the 2.MF4 file and there is a big error that the DG2 (CAN data) do not have its Variable Length Signal Data (VLSD) stored in neither a SD block nor a VLSD CG block. So the message data bytes are missing.

I checked with my test code and that code works OK. It's little bit of bad timing as I'm changing the code regarding the configuration but I don't think I accidentally corrected this bug. In the code below, the second if statement should be true.

I suspect that you is running multiple DG threads (LIN, CAN and normal) at the same time. The Storage Type is set in the writer. I suspect there is a design issue when introducing multiple DG threads. The Storage Type belong to the DG block not globally to the writer object.

This design bug was introduced with the new multiple DG storage option.

 const bool mandatory_only = writer_.MandatoryMembersOnly();
  const IChannel* cn_data_byte = nullptr; // Need to update the VLSD Record ID
  if (IChannelGroup* cg_data_frame = dg_block.CreateChannelGroup(
          MakeGroupName("DataFrame"));
      cg_data_frame != nullptr) {
    cg_data_frame->PathSeparator('.');
    cg_data_frame->Flags(CgFlag::PlainBusEvent | CgFlag::BusEvent);
    CreateTimeChannel(*cg_data_frame,"t");
    CreateCanDataFrameChannel(*cg_data_frame);
    cn_data_byte = cg_data_frame->GetChannel("CAN_DataFrame.DataBytes");
    CreateSourceInformation(*cg_data_frame);
  }

  if (writer_.StorageType() == MdfStorageType::VlsdStorage && cn_data_byte != nullptr) {
    // Need to add a special CG group for the data samples

    if (auto* cg_samples_frame = dg_block.CreateChannelGroup("");
        cg_samples_frame != nullptr) {
      cg_samples_frame->Flags(CgFlag::VlsdChannel);
      cn_data_byte->VlsdRecordId(cg_samples_frame->RecordId());
    }
  }

[sy950915]

@ihedvall
Yes, exactly. I was indeed writing multiple configurations simultaneously, but I just identified the root cause. When creating the CAN data, calling MWriter->StorageType(MdfStorageType::VlsdStorage) resolved the issue mentioned earlier. Now I've successfully generated an MDF file containing CAN, LIN, and ETH data, and it can be properly replayed in CANoe.

[sy950915]

@ihedvall
When reading an MDF file, is it possible to retrieve all data in strict chronological order? For example, can get_channel_group_value return channel group values sorted by timestamp?

[ihedvall]

@sy950915
I'm not sure if I understand your question. The master channel i.e. the time channel must be stored in chronological order. This means that the sample index is in chronological order. There is a IChannelObserver::GetChannelValue(sample index..) and a GetChannelSamples() function.

I'm not sure, but where did you find the get_channel_group_value() function?

[sy950915]

Ah, I see. The get_channel_group_value was just a hypothetical example I used. What I really meant is to treat the MDF file as a data repository and retrieve the data row by row (in a sequential, line-by-line manner).

[ihedvall]

@sy950915
There is currently 2 ways to do this.

1. Instead of adding channel observers, you can add one Sample Observer instead. It has a callback function that returns sample, record Id (=Channel group) and record bytes. In the callback that you need to write, you need to convert the record bytes to channel values/engineering values. The problem is that for CG VLSD data have its own record ID, so it will be messy. The Channel Observer is actually a Sample Observer. The channel object have all the functions for converting record bytes to values.
2. Use the ReadVLSDData() function. This is a 3-step read instead of the 2-step read. First read the configuration. Then setup the subscription (channel observers) but set the ReadVLSDData property to false. Now ReadData as usual and extract the sample-to-offset list. The last is to fetch the VLSD data by supplying a callback function. Almost similar to the [1] but it is more flexible and easier. This method is mainly used for large files typical with video stream or pictures as samples.

Both method uses minimum of memory, well at least it's not the reader problem. The first method is fast and simple if you know the storage format but for general read you might do all the tricks as the Channel Observer do. There is a partial read function as well.

The MDF must be read sequential i.e. row-by-row. If you want to read column-by-column, then you need to convert the MDF file to something as Parquet files. This is a Data Lake application but this is a story for another day.

[sy950915]

@ihedvall
Thank you for the provided solutions. Today I tried the first approach, but it felt a bit messy in practice. I'm now planning to try the second approach, but I didn't see an example using the ReadVLSDData() function. Would it be possible for you to provide a usage example?

[ihedvall]

I took a code snippets from a unit test. It starts as a normal channel observer read but before calling the ReadData() function, set the frame channel to not read any VLSD data bytes(frame_observer->ReadVlsdData(false)). This read will be fast as you don't need to read most of the data bytes. Note that the callback function below, doesn't do anything useful.

Next is to fetch the sample index to VLSD offset list. Make a callback function that is called for each offset and CAN data bytes in the ReadVlsdData.

This method will do the job but you need some knowledge about the MDF file. This function is not so fast as you read the file twice. It very useful for ultra-large files with VLSD data blobs.

      MdfReader oRead(test_file);
      EXPECT_TRUE(oRead.ReadEverythingButData()) << oRead.ShortName();

      const auto *header = oRead.GetHeader();
      ASSERT_TRUE(header != nullptr);

      auto *last_dg = header->LastDataGroup();
      ASSERT_TRUE(last_dg != nullptr);

      auto channel_groups = last_dg->ChannelGroups();
      ASSERT_FALSE(channel_groups.empty());

      auto *channel_group = channel_groups[0];
      ASSERT_TRUE(channel_group != nullptr);

      auto *time_channel = channel_group->GetChannel(kLargeTime);
      ASSERT_TRUE(time_channel != nullptr);

      auto *counter_channel = channel_group->GetChannel(kLargeFrameCounter);
      ASSERT_TRUE(counter_channel != nullptr);
      auto *frame_channel = channel_group->GetChannel(kLargeFrameData);
      ASSERT_TRUE(frame_channel != nullptr);

      auto time_observer = CreateChannelObserver(*last_dg,
                                                 *channel_group, *time_channel);
      auto counter_observer = CreateChannelObserver(*last_dg,
                                                    *channel_group, *counter_channel);
      auto frame_observer = CreateChannelObserver(*last_dg,
                                                  *channel_group, *frame_channel);
      frame_observer->ReadVlsdData(false);

      EXPECT_TRUE(oRead.ReadData(*last_dg)) << oRead.ShortName();
      auto offset_list = frame_observer->GetOffsetList();

      //      size_t sample_count = 0;
      std::function callback = [&](uint64_t offset, const std::vector<uint8_t> &buffer) {
        const bool offset_exist = std::any_of(offset_list.cbegin(),offset_list.cend(),
                                              [&] (uint64_t off) {
          return off == offset;
        });
        EXPECT_TRUE(offset_exist) << "Offset: " << offset << std::endl;
      };

      EXPECT_TRUE(oRead.ReadVlsdData(*last_dg, *frame_channel, offset_list, callback)) << oRead.ShortName();

[sy950915]

@ihedvall

Thank you. When using the following code, I noticed that the RecordId() of CAN_DataFrame.DataBytes is the same as the RecordId() of other CAN_DataFrame entries, which prevents me from correctly retrieving the CAN_DataFrame.DataBytes data in the callback function.

TSAPI(s32) mdf_test(const char* mdffilename)
{
MdfReader reader(mdffilename);
reader.ReadEverythingButData();
const auto* header1 = reader.GetHeader();

uint64_t nof_sample_read = 0; // Testing abort of reading just 10 samples
for (int idx = 0; idx < 1/*header1->DataGroups().size()*/; idx++)
{

    auto* last_dg1 = header1->DataGroups()[idx];
    ISampleObserver sample_observer(*last_dg1);
    sample_observer.DoOnSample = [&](uint64_t sample1, uint64_t record_id,
        const std::vector<uint8_t>& record) -> bool {

        auto CGList = last_dg1->ChannelGroups();
        for (int chnidx = 0; chnidx < CGList.size(); chnidx++)
        {
            auto CNList = CGList[chnidx]->Channels();
                
            for (int cidx = 0; cidx < CNList.size(); cidx++)
            {
                std::string values;
                if (CNList[cidx]->RecordId() == record_id)
                {
                    bool valid = sample_observer.GetChannelValue(*CNList[cidx], sample1,
                        record, values);
                    std::cout << "Name:" << CNList[cidx]->Name() << " Value:" << values << std::endl;
                }
            }
        }
        return true;
    };
    reader.ReadData(*last_dg1);
    // 10 samples
    sample_observer.DetachObserver();
}
reader.Close();
return 0;

}

1.zip

[ihedvall]

@sy950915
You thinking little bit wrong. The Record ID belong to the Channel Group why all channels within the group will have the same Record ID. You can move your check of ID, to outside of the for loop.

Now to your real problem. The Data Bytes channel is actual marked as a Variant Length Signal Data so even if its data type is a Byte Array, it actually stores and 64-bit offset into the Signal Data. I haven't checked your file but I assume that it uses the CG VLSD option. This means that the data bytes are stored in another Channel Group with another Record ID .

The IChannel object have a property called VlsdRecordId() that points to the CG VLSD Record ID. To summarize, the Channel Observer first check if the Record ID match the VLSD Record ID and if not, basically run your for loop but skipping the Data Byte channel. If it match the VLSD Record ID, the record bytes includes number of bytes followed by the number of bytes.

Yes, it's a mess but that's how the MDF works. Getting the sample value in 2 separate callback calls, you need to rethink your design. You may spy on the ChannelObserver::OnSample() callback function.

[ihedvall]

@sy950915
I converted this issue to a discussion. I have added the bugs/improvement to separate issues instead.

[sy950915]

@ihedvall

After reviewing your response, I modified the code and confirmed that the issue was indeed caused by the DataBytes being VLSD. After adjusting the DataBytes type, I was able to read the data normally. However, when attempting to read VLSD-type DataBytes, I encountered the following issues and am unsure whether they are bugs:

The actual data in DataBytes is [0,0,0,0,0,0,0,0], but the values I read are [0,1,2,3,4,5,6,7].

Under the same conditions, when I tried reading directly using mdf::ChannelObserverList, the values obtained were also [0,1,2,3,4,5,6,7].

Here is my test code, and the test file is still the 1.mf4 file uploaded previously.

Could you help clarify whether this is a bug or if I’m missing something in the implementation?

TSAPI(s32) mdf_test(void* AObj,const char* mdffilename/, PMDFReadCallbackList CallList/)
{
MdfReader reader(mdffilename);
reader.ReadEverythingButData();
const auto* header1 = reader.GetHeader();

uint64_t nof_sample_read = 0; // Testing abort of reading just 10 samples
const auto start_open = MdfHelper::NowNs();
for (int idx = 0; idx < header1->DataGroups().size(); idx++)
{

    auto* last_dg1 = header1->DataGroups()[idx];
    ISampleObserver sample_observer(*last_dg1);
    sample_observer.DoOnSample = [&](uint64_t sample1, uint64_t record_id,
        const std::vector<uint8_t>& record) -> bool {
        bool IsContinu = true;
        auto CGList = last_dg1->ChannelGroups();
        for (int chnidx = 0; chnidx < CGList.size(); chnidx++) {

            auto CNList = CGList[chnidx]->Channels();
            s32 ACNSize = CNList.size();
            std::vector<const char*> ANames(ACNSize, nullptr); 
            std::vector<double> AValues(ACNSize, 0.0);          
            for (int cidx = 0; cidx < ACNSize; cidx++) 
            {
                auto cn = CNList[cidx];
                uint64_t _record_id = cn->RecordId();
                bool parse_record = _record_id == record_id;
                if (cn->VlsdRecordId() > 0 && record_id == cn->VlsdRecordId())
                {
                    parse_record = true;
                }
                if (!parse_record)
                    continue;
                const auto array_size = cn->ArraySize();
                if (cn->Type() == ChannelType::VariableLength ) {
                    if (!(cn->VlsdRecordId() > 0 && record_id == cn->VlsdRecordId()))
                        continue;
                    std::vector<uint8_t> AValueList;
                    auto name = cn->Name();
                    char* nameCopy = new char[name.size() + 1];
                    strcpy_s(nameCopy, name.size() + 1, name.c_str());
                    ANames[cidx] = nameCopy;
                    bool valid = sample_observer.GetChannelValue(*cn, sample1,
                        record, AValueList);
                    /*for (uint64_t array_index = 0; array_index < array_size; ++array_index) {
                        const auto sample_index = static_cast<size_t>((sample1 * array_size) + array_index);
                        bool valid = cn->GetChannelValue(record, AValueList, array_index);
                    }*/
                    
                }
                else
                {
                    double values = 0;
                    std::vector<uint8_t> AValueList;
                    bool valid = sample_observer.GetChannelValue(*cn, sample1, record, AValueList);
                    valid = sample_observer.GetChannelValue(*cn, sample1, record, values);
                    if (valid) {
                        auto name = cn->Name();
                        char* nameCopy = new char[name.size() + 1];
                        strcpy_s(nameCopy, name.size() + 1, name.c_str());
                        ANames[cidx] = nameCopy;
                        AValues[cidx] = values;
                    }
                }
                
            }
            
           /* if (CallList->FCallbackSignal) {
                CallList->FCallbackSignal(AObj, ANames.data(), AValues.data(), ACNSize, &IsContinu);
            }*/
            for (auto namePtr : ANames) {
                if (namePtr) {
                    delete[] namePtr;
                }
            }
        }
        
        return IsContinu;
    };
    reader.ReadData(*last_dg1);
    // 10 samples
    sample_observer.DetachObserver();
}
reader.Close();
return 0;

}

[ihedvall]

@sy950915
The correct Data Bytes in the input file {0.1.2.3.4.5.6.7} for each sample. There is something wrong when reading in the data bytes. You read in the bytes but shouldn't the bytes be saved somewhere (missing line AValues[cidx] = AValueList ?).

[sy950915]

@ihedvall
No, that probably isn't the reason. I need to confirm with you: in the callback sample_observer.DoOnSample = [&](uint64_t sample1, uint64_t record_id, const std::vector<uint8_t>& record), does record return the actual data? And then, do we extract the real DataBytes from record?

Currently, the record returned in the callback is {0.1.2.3.4.5.6.7}, and the DataBytes I parsed is also {0.1.2.3.4.5.6.7}. I checked the memory values directly in the debugger, so there's no need to store the values separately. I suspect there might still be an issue with how I'm using it somewhere.

[sy950915]

@ihedvall
Hello, I suddenly thought of a question: when I'm reading data, is there a way to determine whether the channel group is a CAN/LIN/FR/ETH channel group? Of course, what I want to confirm is whether there is a sufficiently universal method to make this determination.

[ihedvall]

@sy950915
Both yes and no. It's the SI (Source Information) block that is attached to the CG block. It holds information about the Bus Type. It's can also inform about any Protocol as J1939. It works for all protocols but not for CAN. This what I'm working with currently.

[sy950915]

@ihedvall
Apologies, the issue might have been on my end. The data read is actually correct. However, when I imported the written data into CANoe 16, CANoe parsed the CAN FD DataBytes as {0,0,0,0,0,0,0,0}, while the actual data written was {0,1,2,3,4,5,6,7}. Could this discrepancy be caused by differences between MDF 4.10 and MDF 4.20 versions?