Image interpolation method for downscaling (nearest neighbour, antialias)

[Animenosekai]

I was testing different resizing algorithms and I noticed that the Nearest Neighbour algorithm is way faster than Antialiasing.

I was just wondering why ImageHash used Image.ANTIALIAS over Image.NEAREST for something that will be processed by the program (we don't really care about how the image look if we have the features)

[JohannesBuchner]

Can you show a few example images and what their anti-aliased & nearest down-sampled (8x8pixel) image looks like?

[Animenosekai]

Here are the tests:

Timing

I've made some timing tests with the following snippet of code:

from os.path import dirname, abspath
from os import listdir
from time import time
from PIL import Image
DATA_DIR = dirname(dirname(abspath(__file__))) + "/data/"
data = []

for image in listdir(DATA_DIR):
    try:
        data.append(Image.open(f"{DATA_DIR}{image}"))
    except: pass
for image in data:
    image.load()
start = time()
for image in data:
    image.resize((8, 8), Image.NEAREST)
print("Time taken (Image.NEAREST):", time() - start, "sec.")
start = time()
for image in data:
    image.resize((8, 8), Image.ANTIALIAS)
print("Time taken (Image.ANTIALIAS):", time() - start, "sec.")

data[0].save("image_1_noresize.png")
data[0].resize((8, 8), Image.NEAREST).save("image_1_nearest.png")
data[0].resize((8, 8), Image.ANTIALIAS).save("image_1_antialias.png")
data[4].save("image_5_noresize.png")
data[4].resize((8, 8), Image.NEAREST).save("image_5_nearest.png")
data[4].resize((8, 8), Image.ANTIALIAS).save("image_5_antialias.png")
data[33].save("image_34_noresize.png")
data[33].resize((8, 8), Image.NEAREST).save("image_34_nearest.png")
data[33].resize((8, 8), Image.ANTIALIAS).save("image_34_antialias.png")

Results

This are the results printed out:

Time taken (Image.NEAREST): 0.02369093894958496 sec.
Time taken (Image.ANTIALIAS): 0.367678165435791 sec.

I've intentionally preloaded all of the images to have a fair test (the first resize() doesn't need to load the image in memory)

The images are all from the Anime Faces dataset: https://www.kaggle.com/soumikrakshit/anime-faces

There is a total of 21551 64x64 RGB images (.png with no alpha)

Image Results (8x8)

Image	Original (64x64)	ANTIALIAS	NEAREST
1
2
3 --> Image 432 in the dataset

Sorry I couldn't find the name of the first images because it kind of picked them randomly and I tried to find them using a really basic and simple image difference algorithm:

class ImageHash():
    def __init__(self, image, r=None, g=None, b=None) -> None:
        self.path = image
        self.original = Image.open(image)
        image = self.original.resize((SIZE, SIZE), Image.NEAREST)
        data = image.getdata()
        self.r = ([d[0] for d in data] if r is None else r)
        self.g = ([d[1] for d in data] if g is None else g)
        self.b = ([d[2] for d in data] if b is None else b)

    def difference(self, image):
        r_diff = mean([abs(value - image.r[index]) for index, value in enumerate(self.r)]) / 255
        g_diff = mean([abs(value - image.g[index]) for index, value in enumerate(self.g)]) / 255
        b_diff = mean([abs(value - image.b[index]) for index, value in enumerate(self.b)]) / 255
        return mean([r_diff, g_diff, b_diff])

But couldn't manage to find them

Computer Specs

MacBook Air (M1, 2020)
OS: macOS Big Sur 11.0.1
CPU: Apple M1
RAM: 16 Go
Python 3.9.1 | packaged by conda-forge | (default, Dec 9 2020, 01:07:47) [Clang 11.0.0 ] on darwin
PIL 8.0.1

[JohannesBuchner]

Yes, looking at the 8x8 pixels, the nearest method introduces some dark pixels and more noise. It looks like it is not a small effect.

Probably the hashes of the two methods are even different.

[Animenosekai]

Yea I think that they will be different but isn't it accentuating the features?

Like, if a human needed to see these images it would definitely be worth using ANTIALIAS (which is Lanczos) but here a computer is just gonna compute it.

Or maybe add a parameter where you can define the resizing algorithm used?

[JohannesBuchner]

Could you also compare the speed of dhash/ahash/phash with the two methods? I am not sure this is the slowest part.

[Animenosekai]

Tests

Here is the script used to test with the different algorithm:

from time import time
from PIL import Image
from imagehash import average_hash, dhash, phash

img = Image.open("image_1_noresize.png")
img.load()

start = time()
for _ in range(100000):
    average_hash(img)
print("Took", time() - start, "seconds to create 100000 hashes with average_hash")

start = time()
for _ in range(100000):
    dhash(img)
print("Took", time() - start, "seconds to create 100000 hashes with dhash")

start = time()
for _ in range(100000):
    phash(img)
print("Took", time() - start, "seconds to create 100000 hashes with phash")

Results

With ANTIALIAS:

Took 3.2136831283569336 seconds to create 100000 hashes with average_hash
Took 2.847418785095215 seconds to create 100000 hashes with dhash
Took 6.5547778606414795 seconds to create 100000 hashes with phash

With NEAREST:

Took 1.4975638389587402 seconds to create 100000 hashes with average_hash
Took 1.1021497249603271 seconds to create 100000 hashes with dhash
Took 4.1882407665252686 seconds to create 100000 hashes with phash

I changed the resize method directly in the source code between each test

Using cProfile you can see that with ANTIALIAS:

tottime: 6.032 sec {method 'resize' of 'ImagingCore' objects}

With NEAREST:

tottime: 0.193 sec {method 'resize' of 'ImagingCore' objects}

[cooperdk]

I just determined that using Image.antialias makes imagehash NOT produce the same phash values as goimagehash in many cases.
With Image.bilinear the values are mostly the same. I have one image which is off by a hamming dist of 2.

[RandomNameUser]

Having messed around with image resizing in Python, I have a few comments:

Using NEAREST to generate the hash sounds like a bad idea. NEAREST is very sensitive to minor image shifts, as it will just pick a pixel value more or less at random, depending on which pixel is used. So the hashes for similar but slightly shifted images will be very different, which is (IMHO) not the goal of the hashing. So a better algorithm is needed.

A common approach to speed up image scaling in Python is to do it in two steps: scale down the image at 2x, 4x or 8x the target size using NEAREST and then do a final step using ANTIALIAS. The result will be not exactly the same as a full antialias, but much faster and significantly closer than the NEAREST.

In addition to that PIL has a special method to downscale images quickly: thumbnail. Thumbnail is most effective if it is used before the image data is loaded, as it can tell the file loader to only load the needed data for the smaller image. This works very well for JPEG, less well for other formats.

I did run some tests (code at the end). I used the same test images as @Animenosekai, but I did not preload the data. I did warm the file system cache by loading all images once and discarding them.

Here is an example image and its 8x8 versions for comparison:

Original	Nearest	Anti-Alias	Nearest+AA	Thumbnail

In this view the differences between AA, N+AA and TH seem to be almost invisible, but the hashes do find differences. I tested it with dhash, and while nearest has an average distance of 16 (clearly unacceptable) to AA, both N+AA and TH have 3.65, which is much better but still noticeable. This probably really only matters in cases where there are old, existing hashes to compare to, for new projects and databases I wouldn't expect to see a difference in detection rate for either of these algorithms.

Interestingly I got fairly different performance numbers than @Animenosekai. In my tests without preloading data and with calculating the hash (from the scaled-down image) the differences between the algorithms were very small:

Mode	Time
Image.ANTIALIAS	8.469 sec.
Image.NEAREST	8.167 sec.
Image.NEAREST+ANTIALIAS	7.996 sec.
Image.THUMBNAIL4	8.418 sec.

So, not very exciting. 😒

However, the test faces are only 64x64 pixels, which is very small. I tested it with some larger (~3000x4000) JPEG test images, and got more interesting results:

Mode	Time	Speedup	Avg. Distance
Image.ANTIALIAS	23.155 sec.
Image.NEAREST	16.357 sec.	1.42	16.54
Image.NEAREST+ANTIALIAS	14.079 sec.	1.64	3.56
Image.THUMBNAIL4	6.757 sec.	3.43	4.78

So THUMBNAIL is 3x faster than AA, and about 2x faster than NEAREST.

So at this point I'm not sure what I would recommend. My use case is more like the second test: large images on disc. In that case THUMBNAIL makes a big difference, so I would love seeing it in imagehash.

Just my $.02...

Test Code:

from os import listdir
from time import time
from PIL import Image
import imagehash
DATA_DIR = "testimages_64/"
DATA_DIR = "testimages_4k/"
imagefiles = []

for image in listdir(DATA_DIR):
    try:
        imagefiles.append(f"{DATA_DIR}{image}")
    except: pass

print("Warming filesystem cache...")
for i in imagefiles:
    image = Image.open(i)
    image.load()
print("Done. Starting measurements...")

nhashes = []
start = time()
for i in imagefiles:
    image = Image.open(i)
    image = image.resize((8, 8), Image.NEAREST)
    nhashes.append(imagehash.dhash(image))
ntime = time() - start
print(f"Time taken (Image.NEAREST): {ntime:.3f} sec.")

ahashes = []
start = time()
for i in imagefiles:
    image = Image.open(i)
    image = image.resize((8, 8), Image.ANTIALIAS)
    ahashes.append(imagehash.dhash(image))
atime = time() - start
print(f"Time taken (Image.ANTIALIAS): {atime:.3f} sec.")

nahashes = []
start = time()
for i in imagefiles:
    image = Image.open(i)
    image = image.resize((8*4, 8*4), Image.NEAREST).resize((8, 8), Image.ANTIALIAS)
    nahashes.append(imagehash.dhash(image))
natime = time() - start
print(f"Time taken (Image.NEAREST+ANTIALIAS): {natime:.3f} sec.")

def preresize(img, box):
    factor = 1
    while img.size[0] > box[0] * factor and img.size[1] > box[1] * factor:
        factor *= 2
    if factor > 1:
        img.thumbnail((img.size[0] / factor, img.size[1] / factor), Image.NEAREST)
    return img

thhashes = []
start = time()
for i in imagefiles:
    image = Image.open(i)
    image = preresize(image, (8*4, 8*4))
    image = image.resize((8,8), Image.ANTIALIAS)
    thhashes.append(imagehash.dhash(image))
ttime = time() - start
print(f"Time taken (Image.THUMBNAIL4): {ttime:.3f} sec.")

# Calc average distances
ndist = nadist = thdist = 0
for aa, n, na, th in zip(ahashes, nhashes, nahashes, thhashes):
    ndist += aa - n
    nadist += aa - na
    thdist += aa - th
ndist /= len(ahashes)
nadist /= len(nahashes)
thdist /= len(thhashes)

print(f"Speedup: N {atime/ntime:.2f}, dist {ndist:.2f}")
print(f"Speedup: NA {atime/natime:.2f}, dist {nadist:.2f}")
print(f"Speedup: Th4 {atime/ttime:.2f}, dist {thdist:.2f}")


def save_img(name, fname):
    image = Image.open(fname)
    image.save(name + "_orig.png")
    image.resize((8, 8), Image.NEAREST).resize((64,64), Image.NEAREST).save(name + "_nearest.png")
    image.resize((8, 8), Image.ANTIALIAS).resize((64,64), Image.NEAREST).save(name + "_antialias.png")
    image.resize((8*4, 8*4), Image.NEAREST).resize((8, 8), Image.ANTIALIAS).resize((64,64), Image.NEAREST).save(name + "_nearest+aa.png")
    image = Image.open(fname)
    preresize(image, (32, 32)).resize((8, 8), Image.ANTIALIAS).resize((64,64), Image.NEAREST).save(name + "_thumbnail.png")

save_img("1", imagefiles[0])

[JohannesBuchner]

I guess if the user uses the Thumbnail4 method to pass the imagehash functions a file in the right size, the resize functions within will be noop.

Is that right? In that case we can recommend users to use Thumbnail4.

[RandomNameUser]

Yes, that's an option. At least pillow's resize checks and doesn't resize if the size is the same. The only price you pay is making a copy of the index image. Given that it's small that's probably not a very big deal.

So yeah, recommending something like th4 may make sense for the large JPEG on disc use case. Maybe it would make sense to add it as a utility function to imagehash, to make using it simpler? It's not a lot of code, but I know people (me included ;) are lazy...

[JohannesBuchner]

If so, we should also include information about the ICC profiles, see #110 , so that these are correctly handled.

If you can prepare a short script, that would be great.

I would prefer to include it in the README as a section "How to best read image files" so the code is in plain sight.

[JohannesBuchner]

A subtlety is that some hashes do hashsize x hashsize, some do (hashsize + 1) x hashsize, but we can spell that out.