[FEA] Interchange protocol between processes on the same device.

[trivialfis]

A related issue: #11462

We would like to transfer a cuDF dataframe between a JVM process and a Python process without data copy . This is primarily used in PySpark environment where Spark can execute user-defined Python functions in JVM processes. The current solution in Spark is to serialize the dataframe into arrow format and perform an inter-process transfer on host. This is not efficient for both memory usage and computation time. Our proposed solution at the moment is to use the CUDA IPC mechanism for transferring metadata between 2 processes without actually copying data between host and device or between different processes. The message sent between 2 processes in our proposed method is a JSON document that describes some properties of the dataframe along with an encoded CUDA IPC handle.

Describe the solution you'd like
We would like to add two roundtrip methods for generating an IPC message that describes the dataframe and reconstructs the dataframe from that message at the C++/C level along with wrappers in java/python.

df = cudf.DataFrame({"a": [1, 2, 3]})
msg = df.to_ipc()

# in a different process but on the same CUDA device
df = cudf.DataFrame.from_ipc(msg)

As for a quick design of the message, for the lack of a better term, we can jsonify the __dataframe__ protocol along with the use of encoded CUDA IPC handle. The message sent between processes can be something similar to:

{
  "columns": [
    {
      "describe_categorical": {
        "is_dictionary": true,
        "is_ordered": true,
        "mapping": {
          "a": 0
        }
      },
      "describe_null": [
        3,
        null
      ],
      "dtype": [
        0,
        64,
        "i",
        "="
      ],
      "buffers": [{"ipc_handle": "aeb6df622e4d6ed84dac526c8815c52c5bb2a34855a765270d6e1a502dc6574b"}],
      "metadata": null,
      "null_count": null,
      "offset": 0,
      "size": 128
    },
    {
      ...
    }
  ]
}

We might be able to reuse some logic in the df_protocol.py module for implementing this feature.

Describe alternatives you've considered

• Suggested by @jakirkham , I looked into the serialize and deserialize methods. They are good starting points but not suitable for transferring ownership between 2 different language environments due to the use of pickle in type serialization.
• Suggested by @shwina , I looked into the dataframe protocol, which is very close to what we want, except for its Python only interface and the use of a pointer.
• mapInArrow method from PySpark, which still requires a full data transfer.

[shwina]

Thanks for writing this up, @trivialfis! Firstly, I'm not super familiar with CUDA IPC and any associated pitfalls, so I'll cc @trxcllnt and @kkraus14 here, who may be able to comment more on the suggested approach and API.

Regarding the suggestion to use a 'jsonified' interchange protocol for messaging, I do have some concerns. Firstly, the dataframe interchange protocol has the narrower goal of enabling interchange between Python libraries. The semantics may not always match 1-to-1 with IPC.

Secondly, and relatedly, I'm concerned that we would effectively be introducing a new IPC format, while others like Feather already exist. Would supporting Feather solve this problem using a more standard approach?

We recently removed the bit of functionality around the Arrow IPC Format from cudf. I don't believe Spark was supporting that, and on the Python side, it was largely unused. More importantly, my understanding was that for GPU-resident data, the implementation wasn't even truly zero-copy (@trxcllnt can confirm). But perhaps we could consider bringing back (and fixing) that code?

[trivialfis]

@shwina Thank you for the reply.

Would supporting Feather solve this problem using a more standard approach?

I looked into arrow IPC, which allows writing and reading data efficiently, and reading can be optimized with memory mapped file. But still, we will have to go through device -> host then host -> device along with read-write to socket/mmap/pipe/file for transferring data from jvm to python process. Feel free to correct me if I'm wrong, the wording of the document seems a little bit vague.

Secondly, and relatedly, I'm concerned that we would effectively be introducing a new IPC format

I think it will remain proprietary and doesn't need to be stable for some period of time since we are transferring CUDA IPC mem handle. Anyone who wants to read the data must understand how to use CUDA API to obtain the pointer. For the "jsonify df protocol", that's because the protocol has complete information about the dataframe and I would like to avoid reinventing the wheel.

my understanding was that for GPU-resident data, the implementation wasn't even truly zero-copy

If it's close to zero-copy then it's probably good enough. The best case scenario is there's a way to let another process view the data in the current process without incurring significant overhead in data transfer using only arrow. At the moment, CUDA IPC handle (not arrow CUDA writer, but the IPC API provided by CUDA toolkit) is the only way I can think of: https://numba.pydata.org/numba-doc/latest/cuda/ipc.html

[kkraus14]

The previous Arrow IPC code that was recently removed didn't go through host memory, it was entirely in GPU. That being said it did require a lightweight serialization to pack the data of a dataframe into a single buffer that could be IPCed instead of having to IPC each underlying buffer. Here's the spec: https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format. One issue with this is that it expects the Schema to be inline with the data and it doesn't really make sense for the Schema to be in GPU memory.

The dataframe interchange protocol referenced by @shwina currently uses pointers within a process and has a pointer per underlying buffer in the dataframe. This could be an arbitrarily large number of buffers depending on the number of columns and their types. You could imagine building something similar to this and using CUDA IPC handles instead of pointers as well as serializing all of the Python class hierarchy and information surrounding it and using some form of side channel to move that between processes. Looks like you've roughly sketched this out already.

Ideally, something like this would get handled in C/C++ instead of in Python so that the implementation can be reused across languages.

[trivialfis]

@kkraus14 Thank you for the comment!

The previous Arrow IPC code that was recently removed didn't go through host memory, it was entirely in GPU.

Could you please share some insight on how it's implemented under the hood?

something like this would get handled in C/C++ instead

I agree, also might be easier to maintain due to memory management.

[kkraus14]

Could you please share some insight on how it's implemented under the hood?

You can see the implementation here:

• https://github.com/rapidsai/cudf/blob/v22.06.01/cpp/src/comms/ipc/ipc.cpp
• https://github.com/rapidsai/cudf/blob/v22.06.01/cpp/include/cudf/ipc.hpp

[trivialfis]

Thank you for the reference. I will dig into arrow.

Also, this issue is more a question of whether this feature is welcomed for cuDF. I can help work on it if we agree on the design.

[trivialfis]

I looked into cuDF and arrow today. I think the arrow IPC should be sufficient for now. Will work on a wrapper tomorrow.

cc @wbo4958

[jakirkham]

cc @pentschev @madsbk (in case either of you have thoughts on this process interchange use case)