posix-ipc dependency

[bdpedigo]

Hi @william-silversmith - I'm using cloud-volume in a docker build in Python 3.12, and noticing that most of the build time is taken up by building posix-ipc from source. It looks like they don't have wheels for Python 3.12 and nothing has been updated on PyPI since 2022, although I see some activity on their build process so perhaps they are working on that.

Regardless, I wanted to ask whether this is something you'd consider adding as an optional dependency, or if it's core to a lot of cloud-volume?

[william-silversmith]

Hi Ben,

It's currently used for multi-process IO on Linux (it provides shared memory). This logic was written a long time ago (when python 2.7, 3.4, and 3.5 were still supported). As of Python3.8, there's a SharedMemory module available in the Python base library: https://docs.python.org/3/library/multiprocessing.shared_memory.html so it might be possible to simply substitute that.

[william-silversmith]

I made a little progress here, but it'll take some more time to iron it out.

#659

[bdpedigo]

thanks @william-silversmith! I hope this isn't too much work or at least is something you wanted to do anyway... I should say also that for my use case I'm literally just using cloud-volume for downloading meshes which I don't think would be affected by anything about shared memory here, unless I'm missing something. so if this proves difficult I could also try to patch the dependency changes myself or something like that

[william-silversmith]

I did some more work on this and got it kinda working, but there are weird error messages e.g. when closing a python interpreter if you don't delete your array reference first. Not sure how to make this better yet..