Targeting embedded

[tuna-f1sh]

Have you considered having compile options for targeting embedded devices? I've not looked too deep but main blockers I see are the dynamic libs:

• ZLIB: Could miniz but dropped in? Or it seems like it would be fairly simple to compile without support for compression and remove dependency.
• EXPAT: I think this could actually compile for a embedded target without too much overhead. Is this a critical component or just used for header/meta data?

Just thinking out loud at the moment, thought I would mention here in case others have explored. Since MFD is a powerful logging format, it seems like this would be nice as a Zephyr module.

[ihedvall]

@tuna-f1sh
It already exist a project item for this in the link below.
Embedding MDF

The current MDF (C++) Library can handle most reads and writes for MDF files. It is a rather bulky design that uses threads and conditions normally only available in OS for computers. The idea is to make a writer only library with limited functionality.

There exist 2 types of writers, bus logger or sample value loggers. The bus loggers are simple while the sample values loggers are more complex. ZLIB is used only for compression and it is the ASAM standard that defines that library. Other libraries may work but needs to be tested.

Looking at the CSS Electronics embedded solution as an example. It do not support compression and only used for bus logging. The initial plan was to use Rust but C/C++ can also be used as language. The MDF writes to a file so any Real-Time OS is not needed. For incoming samples and timestamp may need RTOS.

Limitations ?

1. Writer only?
2. MDF 4 only (4.1)
3. No Array support.
4. Compression?
5. Scaling limitations
6. Expat XML is used for reading meta-data but not needed for writing the same.

[tuna-f1sh]

Ah apologies, I did a search but seems GitHub projects are the only thing not searched!

Thanks for explaining the threading currently being used. Yes, targeting a writer only with basic functionality was my plan too. I think compression could be included with miniz but probably best to start without.

Do you have a link to the CSS Electronics embedded solution or is it private? I came across https://github.com/CSS-Electronics/mdf4-converters - seems to be the only C++ project and using an old lib. I know the new one uses mdflib.

I’ve also been considering a Rust bindings crate with minimum interface for writing, since there are a few pure Rust crates but none as fully featured as your library; haven’t found one that does writing even. Didn’t consider it for embedded as think it might be a leap to do a binding crate and nostd - could be an interesting goal though.

[ihedvall]

@tuna-f1sh
The CSS embedded is a private repository but can serve as an example. The CSS logger have some disadvantages.

1. Only bus logger support.
2. Do not compress (DZ blocks).
3. Don't finalize the MDF file.

The simplest solution is to keep the C++ language but removing read support. Making an embedded library that is in between the CSS and the current MDF library, could be a first approach.

I would keep the design with an internal sample queue with an write-to-file thread. The internal queue is needed for the pre-trig support and for compression that saves 4MB chunks.

The current library also support saving to multiple DG blocks at the same time. This means one queue/thread per DG block. Note that this requires DL/DT(DZ) handling. Technical not compression (DT->DZ) but the same mess.

I'm currently adding support for meta data and it is a big change. An embedded system only need basic MD comment support, excluding the ASAM harmonized object support.

There are also 2 companion libraries that could be of interest. The Sparkplug B (MQTT) pub/sub library could be used when the devices have Ethernet connections. The utility library have some event logging support including the syslog protocol. Need to investigate if these libraries already are supported. At least syslog should exist.

The internal sample queue should maybe be exposed. The RTOS should have some sort of queues already. Maybe some investigations needed?

At some point, the (embedded) MDF file needs to be copied to another system.

Time plan:

1. Need to finalize the meta-data support. This became to be much more work than expected. It's 20+ new interface classes and documentation is 50% of the work. I still have 3-4 weeks work with this.
2. Finalize the bus logger interface as it is required in the embedded system. This is mainly moving existing code why I think 2-3 weeks of work.
3. Start to make an embedded MDF C++ library. Creating a general library first so the major part of the library can be tested in windows/Linux.
4. Moving the library onto a real embedded system with RTOS (Zephyr?).

[tuna-f1sh]

Yes I see that there are more important things to sort out and that time plan makes sense. Glad I started the discussion though for ideas moving forwards.

A Zephyr module is a nice target because it's increasingly becoming the de-facto standard in embedded RTOS. It has nice support for threading so don't see this as an issue. It also has OS abstraction so it's possible a POSIX interface could be made to make it as generic as possible. Could then be tested on Linux. Or some syscalls that the host application has to define.

Vaguely related I created a simple buildroot package https://gist.github.com/tuna-f1sh/5e6398bf63a275dd36b7144eab5a1718 . One things are a bit more stable, might be worth making that more official.

[ihedvall]

@tuna-f1sh
I have made [1] and [2] points of the above list. Do you have any idea of a suitable embedded hardware for the Zephyr application? My next project is to work with CAN and MQTT in a Debian based embedded device. The managed to use the MDF library as it is but I assume they will be interested to an embedded version.

[tuna-f1sh]

The SAMx70x series is a powerful MCU used in automotive a lot.

It's Cortex M7 with a good amount of flash and ROM and RAM, MCAN, HSMCI (high speed memory card interface) and RTC so a good platform for logging . It's well supported in Zephyr.

I've also been testing the ESP32C3/C6 recently. They are cheap but also powerful with a lot of peripherals.

The nice thing about Zephyr is it will be portable to any capable targets, including native_sim. It allows running locally as a binary using Linux syscalls.

[tuna-f1sh]

Can help testing and integration with Zephyr as module.

[tuna-f1sh]

@ihedvall replying to #147 (comment) here to keep context.

1. Compression needs ZLIB. Is there a good replacement for this in the embedded world?

I haven't used it but came across miniz in my research for #129. I would suggest compress is a stage 2 goal, unless you think it needs to be considered from the outset.

2. The storage need some sort of abstraction layer instead of using the FILE* interface. The C++ uses the std::streambuf interface as abstraction interface. Maybe you have some experience in this issue?

Since you mentioned Zephyr, I would suggest following their File System API. It allows the App to bring their own file system.

Doesn't have to use it directly. I probably have a file.h with declaration of the required file IO functions - taking inspiration from Zephyr and matching if possible so it drops in. I guess one only needs a sub-set: fs_open, fs_write, fs_flush, fs_close? Maybe fs_stat?

Then the App can define the functions to be linked at the mdflib at link time. Could have some weak definition place holders for testing.

3. XML is used for meta-data. The C++ have a large number of meta-data interfaces for meta-data. As only writing of XML strings is required, no external XML parsing is needed.

Agreed.

4. The so-called multi DG saving is requested but require temporary storage of samples i.e. requires memory.

Maybe make it optional? Or is there a configuration for size of temporary storage so that the lib could be tuned for the target?

5. Compression support also require at least 4-8 MB of memory / DG block. Similar to [4].

A good reason to ignore for initial development and then add as an optional feature later.

Also I agree with the limitations, particularly threading. With polling, one can bring their threading platform of choice to perform the polling or otherwise.