Zero copy IPC with shared_memory

Feature request

support dump into/load from shared_memory.buf

Motivation

zero copy IPC without pickle

Your contribution

testing

[tammela]

Adding a memoryview based loader would be nice.
This PR describes a potential use case: vllm-project/vllm#24469
The file(s) could initially be loaded into a shared memory area from a network backed fs and then loaded by all the participating processes using memory views

[Narsil]

Having memoryview reads is only stabilized in Python 3.11 ABI. And there's an issue because memoryview cannot be guaranteed to be immutable, so the Rust view of it is that any u8 could be modified.

https://docs.rs/pyo3/latest/pyo3/buffer/struct.PyBuffer.html#method.as_mut_slice

Reading from that requires unsafe. It's not technically too different from mmap being unsafe but some platforms allow making mmap safe (by basically not sending new data to safetensors once the file is open). Surface does not seem huge but I'd rather mention it. (Also with recent pytorch versions, the unsafety is handled within pytorch itself, and not within safetensors since we're using torch primitives for that.

Now, since the original issue is network backed FS, I can create a PR with memory view so you can see if it brings any help, but my guess is that it won't.

The issue seems to be within the network mounting part. The entire thing is how many network fetch are actually issued, and do they multiplex correctly the calls so that few network calls are issued. The entire point of a MMAP interface should be about lettitng the kernel (and the actual hardware) worry about making those calls efficiently.

[tammela]

Now, since the original issue is network backed FS, I can create a PR with memory view so you can see if it brings any help, but my guess is that it won't.

I wrote a PoC and it barely improved the peak memory usage (-5%). The problem was that I still had to create the bytearrays out of the view. If I created views on top of argument view the shared memory would need to live until all safetensors files were loaded, making things worse.

Have you ever experimented using madvise(..., MADV_SEQUENTIAL) before generating the tensor https://github.com/huggingface/safetensors/blob/main/bindings/python/src/lib.rs#L651? This hints the kernel to issue read aheads in this particular area