Implementation of Map and C++/Rust interop concerns

[ruvi-d]

Hello All,

I've been looking to the possible options to implement the Ballerina Map type and would like to share my opinions for discussion here.

Target features / requirements

I summarized the target feature set here [1]

High level design

• Implement the BalMap object in the runtime (Rust) and expose the creation, deletion and functionality via C-Style methods to LLVM IR codegen logic
• In the runtime side we can create a BalMap class that contains a Rust HashMap as an underlying data structure and expose the features listed in [1] via class methods.
• This is the same approach taken for the current Array implementation
• We have also discussed this option with related to the String type implementation and come to consensus (at least no objections) about this option [2]
• The other option is to implement all the functionality in LLVM IR code using minimum runtime features like maloc and free. But this is not recommended due to the significantly high effort required to effectively reinvent the wheel[3].

Concerns related to the limitations of C++/Rust interoperability

1. Lack of support for generics
   • Because Rust Foreign Function Interface (FFI) uses a C ABI style interface[4], we can't use generic functions to create containers for various primitive types
   • e.g. For array[int], array[float], etc; we need to duplicate the code for each primitive type. i.e. array_int_new, array_float_new, etc
2. No type checking safeties
   • Since data structures need to be passed as basic pointers (void*), we will have to forgo some of Rust's safety guarantees.

If our Runtime was to be written in C++, then we could use generics to simplify the interfaces and pass object references and get type checking safeties. Vice versa, it would also work better if both the compiler and runtime was in Rust.

References:

[1] : https://gist.github.com/ruvi-d/18c10fe382dc923fc3e0ebc32a6ee738
[2] : https://ballerina-platform.slack.com/archives/C01LNSAHK18/p1612811412012200
[3] : https://mapping-high-level-constructs-to-llvm-ir.readthedocs.io/en/latest/appendix-a-how-to-implement-a-string-type-in-llvm/index.html
[4] : http://jakegoulding.com/rust-ffi-omnibus/

[ruvi-d]

One more point.

I am not entirely clear on the best approach to release memory that is allocated on the runtime side. I am hoping that Ayonam might have a recommendation. My initial thoughts are that this problem can be simplified with the use if smart pointers. Though we will need to have the compiler and runtime written in the same lang to support it.

[manuranga]

I understanding is, calling std::mem::forget and std::mem::drop manually should take care of the memory, but we need to test it out.

ie: when we crate a rust object, before returning it to llvm side we call forget and at a later time, ballerina garbage collector will call drop to clean it.

[manuranga]

Lack of support for generics

Even in C++, we'll not be able to call template functions directly form generated code. We'll have to do manual template specialization using wrappers. It will be similar in rust as well. We might be able to automate it by writing a macro to generate it, in either case.

No type checking safeties

Ya, we'll not be able to leverage Rust's native safety advantage specially at the boundary (ie: layer that interface with generated code). But hopefully the deeper parts like stdlib/langlib will be benefited. We should localize all our unsafe keyword usages just to the runtime layer.

[ruvi-d]

Even in C++, we'll not be able to call template functions directly form generated code. We'll have to do manual template specialization using wrappers. It will be similar in rust as well. We might be able to automate it by writing a macro to generate it, in either case.

Yes, I think you are correct. I was trying a few experiments, but there doesn't seem to be any option to call C++ generics.

Thanks @manuranga for the feedback.