Passing opened files from Python to Rust

[wiktor-k]

Hi,

I've got a Rust API exposed to Python (say decrypt_file function) and I'd like to call it passing Python's _io.BufferedReader object like so:

with open('file', 'rb') as f:
   decrypt_file(f)

It would be good if I could receive that reader as std::io::Read but so far I haven't found any way to do that.

The problem is passing data that is potentially big but could be read sequentially on the Rust side.

I've workarounded it by building special functions that pass filenames to Rust and then open the file in Rust but then I need to expose special functions for bytes etc.

Is there a better way to do it? I've searched the issues but didn't get any useful results.

[adamreichold]

I think there are two issues: Passing files and passing I/O objects like BufferedReader (which also manage an internal buffer).

If you are able to avoid the BufferedReader layer, you could pass file descriptors which are basically integers to Rust and use File::from_raw_fd.

Otherwise, you could just take &PyAny and assume that it has a read Python method which you can call via PyAny::call1. You could then try to implement a custom wrapper like

struct PyReader<'a>(&'a PyAny);

impl Read for PyReader<'_> { ... }

which would however imply quite a bit of overhead to copy the bytes return by the Python read into the &mut [u8] buffer passed into the Rust read.

[wiktor-k]

Okay, while not ideal both approaches have their strengths: I guess raw file descriptors would have the best performance but the call site wouldn't look as nice as this:

with open('file', 'rb') as f:
   decrypt_file(f)

On the other hand the PyAny/read trick looks nice to the caller but suffers from needless copies.

I was looking for any kind of abstraction that'd at the same time work for both files and in memory buffers without sacrificing performance.

Nonetheless your suggestions give me some ideas for implementation. Thanks!

(I'll leave the question un-answered for a week or so and if no others answers appear close it).

[adamreichold]

the call site wouldn't look as nice as this:

I think

with open('file', 'rb', buffering=0) as f:
    decrypt_file(f.fileno())

would work. Just have to make sure to std::mem::forget the Rust-side File to avoid it closing the file descriptor on Drop. (Or better yet wrap it in ManuallyDrop.)

On the other hand the PyAny/read trick looks nice to the caller but suffers from needless copies.

I fear this unavoidable if you want to interface with std::io::Read as the API just work differently w.r.t. ownership.

I was looking for any kind of abstraction that'd at the same time work for both files and in memory buffers without sacrificing performance.

If the files are large enough to offset the overhead of setting it up, memory maps would be an operating system provided abstraction allowing this, i.e. mmap files and always pass bytes into the Rust function.

But yes, if there was a simple and performant way, it should definitely be part of PyO3 itself.

[wiktor-k]

Wow, thanks for the detailed explanations! Very much appreciated 🙏

But yes, if there was a simple and performant way, it should definitely be part of PyO3 itself.

Yeah, agreed. Although I'm not sure if this is possible so I'm not filing an issue.

[tliron]

After finding this discussion, I also found out that a few people made libraries to provide a convenient Rust wrapper.

I like this one.

[wiktor-k]

Looks promising! Thanks for sharing 🙇‍♂️