Possible data loss during MF4 finalization (mdflib / multithreaded writer)

[simonhyliko]

Hi,

I'm currently doing my first internship, working solo on developing a CAN data collection application. I'm encountering an issue with missing data when finalizing MF4 files using mdflib.
Since this is my first experience with a multithreaded application and file handling at this level, it's quite possible my approach isn’t optimal, especially regarding thread synchronization and file finalization, so I’m very open to suggestions.

---

Application architecture

CAN Reader Thread

• Reads raw CAN frames from a non-blocking Linux RAW socket (select + read).
• Pushes them onto a thread-safe queue (ThreadSafeQueue<CanFrame>).

DBC Decoder Thread

• Pops raw frames from the queue (wait_and_pop with timeout).
• Decodes signals using DBC definitions (via dbcppp).
• Creates a structure containing decoded signals (CanMessage).
• Calls the Mf4Writer component directly to save the data.

MF4 Writer Component (called by Decoder Thread)

• Uses a std::mutex to guard access to the mdf::MdfWriter instance.
• Configures channel groups based on DBC metadata (one CG per CAN message ID).
• Initializes the MdfWriter and calls InitMeasurement().
• Calls StartMeasurement() when the first sample arrives, using std::chrono::steady_clock for time anchoring.
• For each decoded message (CanMessage):
  • Computes absolute (uint64_t ns) and relative (double s) timestamps.
  • Sets channel values.
  • Calls mfd_writer_->SaveSample(*channel_group, timestamp_ns).

File rotation

• When a size limit (e.g. 15 MB) is reached (under mutex lock):
  • Calls close_current_file() → performs StopMeasurement() and FinalizeMeasurement() on the old writer.
  • Creates a new MdfWriter instance for the next file.

---

Shutdown / finalization sequence

Graceful shutdown (e.g., SIGINT)

1. Main thread detects shutdown request.
2. Stops CAN reader thread first and waits for it to join.
3. Waits for the raw frames queue to become empty.
4. Stops DBC decoder thread and waits for it to join.
5. Calls the Mf4Writer stop() method, which internally calls close_current_file() (→ StopMeasurement() + FinalizeMeasurement()).

I've added delays before stopping the writer, but it doesn’t solve the issue.

---

Observed problem

Consistently, the last ~1 % of the frames sent to the Mf4Writer are missing from the resulting MF4 file.
This occurs both during:

• Graceful shutdown
  e.g., if I send 2300 frames and stop the app, the MF4 file contains ~2278 samples.
  If I send 100, I get 99.

---

Question

Is there a known buffering mechanism within mdflib (beyond standard file I/O buffering) that might require an explicit flush before calling StopMeasurement() / FinalizeMeasurement(), especially when SaveSample() calls occur very close to the finalization call?
Or could there be a timing issue where FinalizeMeasurement() returns before all buffered data is guaranteed to be written to disk?

[ihedvall]

@simonhyliko
You are right that there is an internal queue of CAN messages.
The internal queue is thread-safe so you don't need to create an own queue.
The queue solves the problem of the pretrig time and the compression option.
The internal queue is created when calling the InitMeasurement() function.
The main issue is that the queue assumes that all messages are put onto the queue in time order.
When your application calls the StartMeasurement(start time) and a later call with StopMeasurement(stop time), all samples between start time - pretrig time and the stop time. are stored to the file.
After the start, the internal queue is periodically stored to the disc.

The weakness of this design is that if a late sample is added to the queue (SaveSample()), it might be put after an older sample.
When stopping, samples that are older that the stop time are not saved onto the disc.
It actually stop if the next sample is older
You can test this by switching the MDF writer type to MdfWriterType::MdfConverter.
The converter type should store all samples added by SaveSample().

Some comments.

1. The SaveSample() function should be thread-safe so you might skip the external queue.
2. You do have a problem to synchronize the calls to the SaveSample() in time order.
3. Do not use a steady clock. Always use a system clock (UTC). The steady clock is related to the start of the computer and not to the UTC time. This might be your original problem.
4. Just a hint that if you embed the DBC file, most reader applications can decode the raw CAN messages.
5. During file rotation, you should create the new writer before stopping the old writer. Old stop time is new new start time.
6. During shutdown, try to call the stop and finalize measurement before stopping the MDF write thread. The MDF writer only have one thread.
7. You can use several DG blocks in parallel. One per message. It generates one internal queue per message.

[simonhyliko]

Thanks a lot for your quick and detailed reply, it really helped me understand what was going on !!!
I did have a timestamp issue indeed, and after switching everything to std::chrono::system_clock and reworking the shutdown sequence, the missing samples problem is gone 😄
I also applied most of the changes you mentioned.
Really appreciate your reactivity and the clear explanations, it made a big difference

[ihedvall]

@simonhyliko
Just for your information.

I am currently developing a CAN to MQTT application that is similar to your application (ihedvall/cantomqtt).
The CAN messages are decoded by one or more DBC files, into DBC signals that in turn, are forwarded to an MQTT broker.

The Bus Master is a GUI application that simulates automotive buses (ihedvall/busmaster).
It is used to test the above applications and also to create as configuration GU(I.

The Bus Message Lib is a simple serialization and bus interface library.
Its purpose is to publish bus messages to a bus (shared memory or TCP/IP) that one or more application can subscribe on.

The above is little bit tricky to understand but might be considered if your application will be used by a customer.
You can connect through linked in if you need some more information or a meeting of the above (MS Teams) .

Best Regards
Ingemar Hedvall

[simonhyliko]

Thanks a lot for sharing those projects, they look very interesting.
In my case, the current design focuses on storing CAN data as MF4 files, compressing them, and sending them over HTTP to minimize bandwidth usage, so it’s a bit different from an MQTT-based setup i guess.
The CAN network is connected through a PCAN-USB interface on the telematics unit, that’s how I simulate frame transmission.
I’ll definitely have a look at your tools. I’m sure they have a lot to teach me !
Ive just sent you a connection request on LinkedIn!