Pillow to NumPy without copying data

[HonzaCustomInk]

I'm trying to see if there is a way to go from a Pillow image to a numpy array without duplicating the data.

I have found some resources elsewhere, but none of them gave me an answer:

• Somebody else's question on SO, without an answer - https://stackoverflow.com/questions/56391994/pil-image-to-numpy-array-without-copy
• numpy docs, which basically says this is not possible ("No copy is performed if the input is already an ndarray with matching dtype and order"), although my reading indicates there is more to this (memoryview, various buffer types, array interfaces and their version) - https://docs.scipy.org/doc/numpy/reference/generated/numpy.asarray.html
• Another PIL issue that might be relevant - WIP Allow new-style buffer interfaces to be read/write #2980
• Numpy issue discussing interop with Pillow, from about this comment on - ENH: Add support for PEP 3118 buffers with offsets (non-contiguous) numpy/numpy#7467 (comment)

What did you do?

I took https://en.wikipedia.org/wiki/Barack_Obama#/media/File:President_Barack_Obama.jpg (any file should work) and this MWE

import psutil # pip install psutil
import numpy as np
from PIL import Image

process = psutil.Process()

print(f"Start - {psutil.Process().memory_info().rss / 1024 / 1024:.02f}")
pil_image = Image.open("President_Barack_Obama.jpg")
print(f"After open - {psutil.Process().memory_info().rss / 1024 / 1024:.02f}")
pil_image.load()
print(f"After load - {psutil.Process().memory_info().rss / 1024 / 1024:.02f}")
np_image = np.asarray(pil_image)
print(f"After np.asarray - {psutil.Process().memory_info().rss / 1024 / 1024:.02f}")

and got

Start - 25.93
After open - 26.59
After load - 61.24
After np.asarray - 113.98

meaning both PIL and numpy allocate memory (~35 and ~50 mb respectively). I've also tried using np.array (somehow takes 80 mb instead of 50 mb) instead of np.asarray, specifying the dtype for the numpy arrays (uint8, int32 (one long per pixel)), to no avail.

What did you expect to happen?

I hoped that the some mechanism could be used to share the memory between PIL and numpy. For my purposes, a solution that would work for RGBA 8-bit images would be good enough, but a generic solution is welcome, I've stumbled upon this a couple of time already.

What actually happened?

Both numpy and PIL allocated memory

What are your OS, Python and Pillow versions?

• OS: MacOS 10.14.6 (Mojave)
• Python: Python 3.7.4 (default, Jul 9 2019, 18:13:23)
  [Clang 10.0.1 (clang-1001.0.46.4)] on darwin
• Pillow: 6.1.0
• Numpy: 1.17.2

Thanks!

[radarhere]

This is a duplicate of #3336. In that issue, it is suggested that imageio might be able to accomplish this - #3336 (comment)

[HonzaCustomInk]

Thanks for answering. For posterity: I've tried doing the same thing with imageio, and while it does indeed convert to a numpy array without overhead once loaded, it internally uses Pillow (it literally calls frame = np.array(frame, dtype=dtype) in pillow.py:637, where frame is of type PIL.JpegImagePlugin.JpegImageFile), so the memory consumptions is even worse (instead of the above mentioned 35 mb for pure PIL to 115 mb with imageio). So this does not really sound like a solution.

import psutil
import numpy as np
import imageio

process = psutil.Process()

print(f"Start - {psutil.Process().memory_info().rss / 1024 / 1024:.02f}")
iio_image = imageio.imread("President_Barack_Obama.jpg")
print(f"After imread - {psutil.Process().memory_info().rss / 1024 / 1024:.02f}")
np_image = np.asarray(iio_image)
print(f"After np.asarray - {psutil.Process().memory_info().rss / 1024 / 1024:.02f}")

yields

Start - 29.49
After imread - 145.06
After np.asarray - 145.07

Since you mentioned #3336, I take it that there's no convert to numpy without a copy with PIL right now. Feel free to close and thanks!

[wiredfool]

You might try #2980. I think that will help the issue.

[HonzaCustomInk]

Installed via pip install "git+https://github.com/wiredfool/Pillow.git@buffer-updates", it seems python is picking it up correctly

>>> import PIL
>>> PIL.__version__
'6.2.0.dev0'

but I still get the same results.

Start - 23.40
After open - 24.70
After load - 59.54
After np.asarray - 113.02

[wiredfool]

The point of that PR was to enable r/w access by numpy without copying, so it's possible that it only works the other way.

[HonzaCustomInk]

That sounds correct.. do you see anything wrong with my code (edited for brevity)?

pil_image = Image.open("President_Barack_Obama.jpg")
pil_image.load()
np_image = np.asarray(pil_image)

it seems to me it only does "r/w access by numpy without copying".

[wiredfool]

Looking at the Numpy source, it looks like the asarray function will use the existing storage if the array_like first parameter is already an ndarray, but if it's a subclass or other type, it will be copied. There is another function asanyarray which will pass through subclasses of ndarray, but a PIL image isn't an ndarray.

np.frombuffer https://docs.scipy.org/doc/numpy/reference/generated/numpy.frombuffer.html looks like a better fit for the numpy side, it will use a buffer exported from something else. You'd need to reshape it to a 2d array, but that would at least get it into numpy without copying on their side. However, on the pillow side, it calls https://github.com/python-pillow/Pillow/blob/master/src/PIL/Image.py#L704 __array_interface__, which renders the image to bytes. In theory, there's nothing preventing(*) the data element from being a mmaped buffer, at least for smaller images that are continuous in memory. Larger images have a zoned memory allocation which makes it more complicated.

(*) "Nothing" other than implementing support for r/o mmaps and a buffer interface for core images.

On the Pillow side, it looks like there's at least one copy using tobytes, but you could potentially drop a copy on the numpy side with frombuffer rather than fromarray.

[HonzaCustomInk]

Hey @wiredfool, thanks for looking into this and sorry for the delay.

Larger images have a zoned memory allocation which makes it more complicated.

Those are exactly the images where one'd like to be able to avoid the memory overhead.

I think this is an interesting thing to pursue for PIL overall, but I can see that it would require non-trivial effort. I have been able to work-around for my specific use-case, but would love to see any changes in this direction in the future.

[radarhere]

If you have anything further that you think is specific to this issue, let us know, but for now, I'm closing this as a duplicate.

[kwikwag]

@radarhere This is not entirely a duplicate of #3336. That is not only about sharing memory, it's about preventing copies and redundant memory allocations. Specifically, when a new Image object is created, _imaging.fill() is called, resulting in more than half of the runtime involved in instantiating the object. Second, I am not convinced there is no memory copy going on.

Consider the following example code (run in IPython). Creating an image from a NumPy buffer should take an equal amount of time, regardless of its size. Specifically, I would like to understand why the call to ImagingDecoder.decode is taking so long.

import numpy as np
from PIL import Image

for size in ((128, 128), (1024, 1024)):
    data = (np.random.uniform(size=(size[1], size[0], 3)) * 255).astype(np.uint8)

    print('Size:', size)
    %timeit Image.frombuffer('RGB', size, data, 'raw', 'RGB', 0, 1)
    import cProfile
    with cProfile.Profile() as prof:
        for _ in range(1000):
            Image.frombuffer('RGB', size, data, 'raw', 'RGB', 0, 1)
    prof.print_stats()

Outputs:

Size: (128, 128)
18.3 µs ± 146 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
         30002 function calls in 0.025 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     3000    0.001    0.000    0.002    0.000 Image.py:2875(_check_size)
     1000    0.001    0.000    0.007    0.000 Image.py:2896(new)
     1000    0.002    0.000    0.023    0.000 Image.py:2935(frombytes)
     1000    0.001    0.000    0.025    0.000 Image.py:2973(frombuffer)
     1000    0.001    0.000    0.002    0.000 Image.py:393(_getdecoder)
     2000    0.001    0.000    0.001    0.000 Image.py:512(__init__)
     1000    0.001    0.000    0.002    0.000 Image.py:543(_new)
     1000    0.002    0.000    0.013    0.000 Image.py:807(frombytes)
        1    0.000    0.000    0.000    0.000 cProfile.py:117(__exit__)
     1000    0.003    0.000    0.003    0.000 {built-in method PIL._imaging.fill}
     1000    0.000    0.000    0.000    0.000 {built-in method PIL._imaging.raw_decoder}
     1000    0.000    0.000    0.000    0.000 {built-in method builtins.getattr}
     7000    0.001    0.000    0.001    0.000 {built-in method builtins.isinstance}
     6000    0.000    0.000    0.000    0.000 {built-in method builtins.len}
     1000    0.000    0.000    0.000    0.000 {method 'copy' of 'dict' objects}
     1000    0.009    0.000    0.009    0.000 {method 'decode' of 'ImagingDecoder' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
     1000    0.000    0.000    0.000    0.000 {method 'setimage' of 'ImagingDecoder' objects}


Size: (1024, 1024)
1.54 ms ± 30.1 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
         30002 function calls in 1.236 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     3000    0.002    0.000    0.003    0.000 Image.py:2875(_check_size)
     1000    0.002    0.000    0.686    0.001 Image.py:2896(new)
     1000    0.003    0.000    1.233    0.001 Image.py:2935(frombytes)
     1000    0.002    0.000    1.236    0.001 Image.py:2973(frombuffer)
     1000    0.002    0.000    0.004    0.000 Image.py:393(_getdecoder)
     2000    0.002    0.000    0.002    0.000 Image.py:512(__init__)
     1000    0.002    0.000    0.003    0.000 Image.py:543(_new)
     1000    0.002    0.000    0.544    0.001 Image.py:807(frombytes)
        1    0.000    0.000    0.000    0.000 cProfile.py:117(__exit__)
     1000    0.678    0.001    0.678    0.001 {built-in method PIL._imaging.fill}
     1000    0.001    0.000    0.001    0.000 {built-in method PIL._imaging.raw_decoder}
     1000    0.001    0.000    0.001    0.000 {built-in method builtins.getattr}
     7000    0.001    0.000    0.001    0.000 {built-in method builtins.isinstance}
     6000    0.001    0.000    0.001    0.000 {built-in method builtins.len}
     1000    0.000    0.000    0.000    0.000 {method 'copy' of 'dict' objects}
     1000    0.537    0.001    0.537    0.001 {method 'decode' of 'ImagingDecoder' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
     1000    0.001    0.000    0.001    0.000 {method 'setimage' of 'ImagingDecoder' objects}