Duneuro Optimization 1 : input files for duneuro

[tmedani]

Hello,

This is in the todo list and we can discuss it here, low priority but important.

right now, the input for duneuro are ASCII files,
From brainstorm, in the case of the isotropic model, we use the *.msh
file for the geometry and *.cond for the conductivity and in the case of the anisotropy, we use the Cauchy files (.knw and .geo).

The w/r of these files takes ~10 to 30 min in the case of the millions of nodes and elements, which is around ~ 5% to ~20% of the computation time.

This is working, but we can improve by using binary input files.

This will need change on the duneuro c++ code (@juangpc ;) nice challenge)

Before that, we need to discuss which format to use as a standard between us and the duneuro team.

In SimBio the vista file format is used, https://www.mrt.uni-jena.de/simbio/index.php?title=File_Formats

Best
Takfarinas

[martcous]

Thank you Takfa, I am a bit of an outsider on this project: is there a technical specification for the ASCII duneuro input data files and the Vista format? The link you provided seems to only describe the Simbio .par parameter file.

If there are open formats used in other software such as Vista in Simbio I agree it would be best if this can be adopted, perhaps there are even open source IO functions we could reuse either in Matlab or C++.

[tmedani]

Hi @martcous,

Thanks for your feedback.

Indeed there is no much online documentation about the file format that we are using and even on the vista file.

I found some of these files and few pdf files that explain some part here
https://sourceforge.net/projects/neurofem-simbio/files/

I will check with the German group if there are any other links for documentation.

Furthermore, I have checked about the vista files and indeed there is already some function in cpp and mex already
https://github.com/fieldtrip/fieldtrip/tree/master/external/simbio

but I think we need to add them within the i/o functions of duneuro, somewhere here:

https://gitlab.dune-project.org/duneuro/duneuro/-/tree/master/duneuro/io

[juangpc]

Hi everybody.
Few comments.

• Let's do this.

• Before I start. Few considerations.

  -> We've used binary files so far in the project to avoid rounding errors while converting floating-point values back and forth. John pointed out this problem at an early stage and although you can use text files and make the rounding errors pretty much disappear if yo do things nicely, I agree that it is more elegant to use binary files and that's what we did. Nobody has pointed out that we're using binary files because of efficiency, just wanted to remark this for completion.

  -> I think that if in one part of the project we're using binary files, we should try to use them in every other part, just for ease of mind, or perhaps, just to make it easier to remember how things work in the project. Additionally, I think that when some info is stored into a file that is not intended for human use, it really makes sense to use binary, especially when storing floating-point values. So again, let's do it.

  -> When reading or writing info from or into a file you're tapping into some library functions first and pretty soon you'll hit an operating system api. The difference between r/w binary data or character data is not that relevant. The most important aspect is the bandwidth of the device you're using. Yes, sometimes if you're r/w big chunks of data you can increase efficiency but you can do that both with binary or with characters. So as an initial start, the increase of efficiency I don't think will come strictly from changing from text to binary files.

  -> Having said that.... Matlab is an interpreted language. Mening excruciatingly slow in reading/writing operations. I haven't looked at the code yet, but I think tha it would make sense to try to write/read big blobs of data to increase efficiency and if really needed I can do a c-mex matlab function to read or write the data from a binary file and have it stored in Matlab's heap.

  -> being that we're thinking of taping into c-mex api. I think it is, at least, worth conversation to somehow allow memory sharing between the different operations/functions in the duneuro process. Or somehow giving Matlab access to that program's mem range. This would allow to completely avoiding the write to file and read from file processes. This would substantially increase efficiency I'm talking like 10x to 100x increases. Typical hard drive bandwidth is between 100-500mb/s up to 3gb/s in a fancy one through pci port. However memcpy operations are 5-20gb/s.

• In the past @ftadel has patiently listened to my ideas of using more c-mex api in bst. And rejected them. Yes, for reasons I understand and support. So let's keep talking.

Sorry. Wordy mondy.

[ftadel]

In Matlab, reading/writing floating points in text fields is indeed terribly slower than binary files (because of the conversion, a lack of optimization, and also simply because the files are much bigger). I've been facing this problem many time.
Indeed, using a binary files instead would make it a lot faster, at least for the Matlab part.
Adding more I/O file formats is usually not too complicated, with no risk for the rest of the API.

@juangpc
How feasible would that be to write a C-mex caller for duneuro instead of going though a system call?
(a mex-file that would receive the mesh as a Matlab matrix and return the leadfield as a Matlab matrix instead of saving it all to the hard drive)

In the case where:

1. this is something you can do in a limited amount of time
2. the recompilation of the mex-file is super well documented
3. the total expected time in gain is significant with respect to the processing time (it makes a significantly different experience from the user point of view)
   => then it could also be a good idea (but more invasive, possibly more unstable, with larger coding requirements and more complicated distribution/maintenance :)

[juangpc]

Great!

In Matlab, reading/writing floating points in text fields is indeed terribly slower than binary files (because of the conversion, a lack of optimization, and also simply because the files are much bigger). I've been facing this problem many time.
Indeed, using a binary files instead would make it a lot faster, at least for the Matlab part.
Adding more I/O file formats is usually not too complicated, with no risk for the rest of the API.

Aha. I didn't know that. Thank you.

How feasible would that be to write a C-mex caller for duneuro instead of going though a system call?

For the duneuro part, linux and darwin will work pretty much out of the box. windows... 😟 complicated.

1. Yes. I can do this. Well, I don't want to underestimate this. Anywhere between 1 day and a week. for next biweekly?.
2. I will do my best.
3. Let me start here. I can do an initial assessment.

[tmedani]

Hello all,

Nice to read all these relevant comments and learn these new things here :).

If I can add some comments, regarding the previous steps of this project,
Duneuro offers a matlab (and python) interface with mex files while all the data are passed by memory (matlab matrix input and output) without using any input and output files.

http://duneuro.org/ ==.at the section documentation

However, this is working only on Linux ... with @juangpc we have spent a lot of time trying to generate the mex files for windows... at the end nothing :( ... with more persistence, we were able to generate the binaries :)

Furthermore, in order to generate these Mex files, which are related to the OS and the Matlab version, the users must compile the Duneuro code on their computer... which is not easy for most of the brainstorm users.
Also, @juangpc did a super job that summarize the tedious steps of the compilation in only few lines here

The ideal case is to have cross-versions and cross-platforms mex files for the users without asking them to compile the duneuro core code... as it's the case for SimBio in fieldtrip.

I don't know how much work these needs and how long it will take.

Regarding the current version, with the binaries, the main change that we need is adding new binaries input files for duneuro,
and of course, we need to define the file format, then write these files from matlab and add a cpp code to duneruo in order to read these files.

@juangpc has successfully added the code that writes the duneuro output as binaries and it was slightly faster than the text files.

Changing only the input files will not affect much the actual version of the brainstoem code.

[ftadel]

I'd vote against optimizations that would require 1) the users to compile something on their computers or 2) doesn't work on windows. It's probably better to have something slower but easier to deploy.