Use boost shared memory to access an in-memory `fst` table from multiple processes

[MarcusKlik]

See here, boost allows for the creation of memory shared between processes. For fst that could mean that a single in-memory fst table can be shared between different processes. First between R processes, for example by node in a parallel cluster (package parallel). Later, also between different fst clients, such a an R and Python client. With shared memory there is virtually no overhead in accessing a table from multiple processes.

A notable advantage is that we can run custom R methods on the same fst table, something that is not possible now (R is single threaded at it's core).

[renkun-ken]

I tried boost shared memory between R sessions and C++ program before. I'm curious about how you allow R functions to work with vectors in shared memory without creating a new R vector which requires copying the data?

Here's a very naive implementation with R's native serialization: https://github.com/renkun-ken/sharedata.

[MarcusKlik]

Hi @renkun-ken, interesting question!

Unfortunately, R decides which memory it allocates to store it's vectors and tables, so we can't use boost shared memory to directly share R objects (hence the clone_object method in your package :-))

The idea is that we can hold the contents of a fst file directly in memory. So in fact, that would be an in-memory (compressed) table using the fst format.

The fst package can provide an interface to that in-memory object and perform the same operations on it as would be possible with a fst file (but faster in most cases). The biggest advantage would be that the table would require less memory than a regular R table, and many operations can be performed using multiple threads (so in fact operations could be faster than with a regular table).

Such a fst table could also be stored in shared memory. If you have that, you can access the table from different R sessions, but perhaps even more important, also from different programming languages and applications!

When we have a Python client for fstlib for example, you can use a shared-memory fst table to access the data both from R and from Python without any copying...

As you probably know, on Windows boost shared memory is implemented through the file system, so there wouldn't be an advantage of using boost shared memory in term of speed on that platform (you can just as well use a fst file, it's basically the same thing if I'm not mistaken :-)).

Perhaps you would like to share some of your experience on boost shared memory once the in-memory fst table is implemented? Having a shared memory table would provide similar functionality as a plasma object store that's being developed as part of Apache Arrow (but much simpler).

greetings

[xiaodaigh]

Wow! Exciting stuff. This would make fst even better!