Support for perceptual hash snapshots with archived output files

[lmmx]

Hey Frank 👋

I'm working with a program that generates images, and I'd like to verify outputs remain consistent by comparing perceptual hashes (fast, compact) while keeping the actual image files archived for manual inspection when needed.

Current approach

I'm using czkawka to generate perceptual hashes of image outputs:

import czkawka as cz
from pathlib import Path

finder = cz.ImageSimilarity()
output_path = my_black_box(42)
output_hash = finder.hash_image(output_path)

assert output_hash == snapshot("8LS0tOSwvLQ")

This works - the perceptual hashes are compact (e.g., "8LS0tOSwvLQ"), suitable for inline snapshots, and allow for similarity comparison unlike cryptographic hashes.

What I'm trying to achieve

I'd like something like:

output_path = my_black_box(42)
assert output_path == external("phash:")

where inline-snapshot would:

1. Compute and snapshot the hash (what gets checked on test runs)
2. Archive the image file, named by its hash (e.g., 8LS0tOSwvLQ.png)

Why both hash and file?

My program does page dewarping optimization, which is slow. On test runs, I want to compare against the hash string without loading images. When a test fails or output changes, I need the original image available for visual comparison to understand what regressed.

The hash serves as a proxy for the expected output while the archived file provides the reference for investigation.

Proposed implementation: new storage protocol

Looking at how hash: works (storing files based on content hash), I think a phash: protocol would fit naturally:

external("phash:8LS0tOSwvLQ.png")  # filename derived from perceptual hash

How it would work

1. User writes: assert output_path == external("phash:")
2. inline-snapshot computes the perceptual hash of the image
3. Fills in: assert output_path == external("phash:8LS0tOSwvLQ.png")
4. Stores the image at that perceptual-hash-derived location

On subsequent test runs:

• Compute perceptual hash of new output
• Compare against 8LS0tOSwvLQ from the snapshot string
• If match: test passes (no file I/O needed after initial hash computation)
• If different: show diff and offer to update

The archived image serves as reference for manual inspection when outputs change.

Implementation approach

Since protocols determine where/how files are named (hash:, uuid:) while formats determine how data is serialized (.txt, .json, .bin), this fits the protocol pattern. I need custom naming logic based on perceptual hash, but standard image storage.

I'm thinking it’d be best to implement this as a storage protocol similar to HashStorage, computing perceptual hashes instead of SHA256. The main question is whether this belongs in inline-snapshot core or as a third-party extension - I'll start with a standalone implementation and we can discuss integration if it proves generally useful.

One alternative would be parameterizing HashStorage to accept different hash functions, but I think that would complicate the implementation without much benefit since perceptual hashing has specific requirements (image inputs, tolerance handling eventually).

I'm planning to start with a third-party extension (e.g., inline-snapshot-phash) since:

• It requires a new dependency
• A standalone implementation can better validate/iterate on the approach to see if it’s of value
• It can serve as a reference for other custom storage protocols

If it starts looking broadly useful, I'd love to contribute to inline-snapshot proper ☺️

[15r10nk]

I think this is a great idea 😃 .
I haven't added StorgeProtocol to the documented API yet, as I wanted to learn more about possible use cases before defining a more or less final API for it. I've been waiting for someone like you to come up with new use cases.

I would definitely prefer it to be implemented in an additional package, and I'm always happy to welcome new contributors.

The current API should be improved a little for this:

• StorageProtocol.sync_used_externals should return something that does not expose as many internal types as it currently does. A list of ExternalLocation that should be deleted should suffice, and perhaps the name of the method should also be changed.

• We need a way to register new storage protocols (and replace the default_storages function).

• I think a base class for HashStorage and PHashStorage would also be useful (PHashStorage would only need to implement new_location).

• Let me know if you think more is needed.

Would you like to work on a PR for these changes to inline-snapshot, or would you just like to implement the plugin?

[lmmx]

I don't understand why the PNG file path I receive in my storage class's new_location method appears to have been hashed, losing its file suffix:

(Pdb++) p file_path
PosixPath('/tmp/inline-snapshot-n1v6noil/tmp-path-bec195a3-9a6c-4a9a-bf72-7e7fa967c830')

I hotfixed that by just copying the file and hashing it

tmp_with_ext = file_path.with_suffix(".png")
shutil.copy(file_path, tmp_with_ext)

That then lets my hasher work, and I get a passing snapshot test 🎉

or at least... a passing creation

            else:
>               raise ValueError(f"'{name}' is missing a suffix")
E               ValueError: 'phash:AAAAAAAAAAA' is missing a suffix

../inline-snapshot/src/inline_snapshot/_external/_external_location.py:81: ValueError
================================================================================================================== short test summary info ===================================================================================================================
FAILED demo/demo_test.py::test_blue_square - ValueError: 'phash:AAAAAAAAAAA' is missing a suffix
============================================================================================================== 1 failed, 2 deselected in 0.08s ===============================================================================================================
(inline-snapshot-phash) louis 🌟 ~/dev/inline-snapshot-phash $ ls .inline-snapshot/
phash
(inline-snapshot-phash) louis 🌟 ~/dev/inline-snapshot-phash $ ls .inline-snapshot/phash/
AAAAAAAAAAA

It looks like I need to give it a suffix after all...

More work needed, but exciting to see it come together even in a rudimentary way! 😸

[15r10nk]

red_square should be no Path but an object which you want to store externally. Some image class ...

You can then define a handler for this object type and inline-snapshot will use this handler and the given suffix (see https://15r10nk.github.io/inline-snapshot/latest/external/register_format/)

you can also check if your datatype works with a builtin handler first.

[15r10nk]

ahh found your code. I think you have a problem in your FormatHandler.
This should always be true for any value:

tmp_path=...

format.encode(value,tmp_path)
assert value == format.decode(tmp_path)

This is not the case with your implementation, because you return a different path (the tmp_path).

https://github.com/lmmx/inline-snapshot-phash/blob/4ca43bef77dd9ff3496cbb6c7484e3bb8550c1ac/src/inline_snapshot_phash/_format.py#L29-L32

Format and Storage should also work independent from each other. Which means that your PerceptualHashStorage should work with any Format which generates .png files for example.

I had not the time to look at all your code yet. Maybe this helps you already.

[lmmx]

No worries Frank, thanks I'll follow that up!

Initially I tried using a Path, I've changed it to bytes now just while it's at MVP stage.

Ideally it would allow hashes for PNG and JPEG copies of the same image to be saved just once (I suspect an extension-specific Format would not) but I'm happy to relax that requirement.

[lmmx]

So snapshot creation now works and once made the comparison succeeds. Unfortunately I didn't solve I worked out why the tests produce a teardown failure (due to state().missing_values incrementing in create mode) and filed a bug report:

• External base __eq__ causes test failure in teardown after new snapshot created #314

That explained why the teardown errored despite the "if you ran with create please ignore this" message.

V glad to have working snapshot creation now! 🙂 🎉

[15r10nk]

We should also change how == works currently for external snapshots. It currently loads the value from disc and compares it. StorageProtocol needs an API to customize the comparison.

[lmmx]

I see it:

inline-snapshot/src/inline_snapshot/_external/_external_base.py

Lines 40 to 80 in a108d2d

	def __eq__(self, other):
	"""Two external objects are equal if they have the same value"""
	external_type = (
	"external" if type(self).__name__ == "External" else "external_file"
	)
	__tracebackhide__ = True
	if isinstance(other, Outsourced):
	self._location.suffix = other._location.suffix
	other = other.data
	if isinstance(other, ExternalBase):
	raise UsageError(
	f"you can not compare {external_type}(...) with {external_type}(...)"
	)
	if isinstance(other, GenericValue):
	raise UsageError(
	f"you can not compare {external_type}(...) with snapshot(...)"
	)
	first_comparison = not self._was_compared
	self._was_compared = True
	if self._is_empty():
	self._assign(other)
	state().missing_values += 1
	if state().update_flags.create:
	return True
	return False
	value = self._load_value()
	result = value == other
	if not result and first_comparison:
	self._assign(other)
	state().incorrect_values += 1
	return True
	return result

Yep that explains how to make a tolerance-based comparison for phash (where near-matches could pass within a threshold). That's a cleaner design than trying to handle it in the format layer. Awesome 😄

[15r10nk]

I think there might also be an usability problem.

Lets say you have code like this:

def test_1():
   assert create_image1() == external("phash:1238abe.png")

def test_2():
   assert create_image2() == external("phash:1238abe.png")

and both create_image function return similar images but not the same and the result of create_image2 is never saved because it is similar to create_image1.

You are not able to watch the diff to the last run when the result of create_image2 changes. You only see the diff to the result of create_image1.

Maybe this is exactly what is wanted when you use this, maybe not. I don't know.

[lmmx]

def test_1():
   assert create_image1() == external("phash:1238abe.png")

def test_2():
   assert create_image2() == external("phash:1238abe.png")

If both create_image function return similar images but not the same and the result of create_image2 is never saved because it is similar to create_image1.

Right, that’s our potential failure mode. Yes, that's exactly the behaviour I want. Using perceptual hashing to deliberately avoid snapshotting multiple times when outputs are perceptually identical.

Traditionally snapshot testing assumes deterministic processes with identical outputs. This approach snapshots based on a property (perceptual similarity) rather than exact identity. This opens the door to ‘a property within a tolerance’. The same pattern could apply to semantic similarity via embeddings (imagine an emb: protocol for strings that are synonyms) etc.

If 10 functions generate a red square (PNG, JPG, different sizes), they share one perceptual hash, so one archived image serves all tests. The hash comparison passes without redundant storage. It’s not that you’re "never saving" your outputs, you're actively accepting saved values as sufficient when they fall within the perceptual similarity category.

It’s assumed when you do so that, were the snapshot test to break (e.g. a blue triangle) then you would still be able to compare, because you coulddiscern the qualitative difference from any of those actual files (the PNG, the JPG, etc).

The trade-off is that when create_image2() changes, you diff against whichever test first generated that hash (e.g., create_image1()'s archive), not the last run of create_image2(). For my use case (page dewarping where outputs should be perceptually stable), that's acceptable - only perceptual change warrants investigation and I don’t want to blow up my repo size with lots of fixture outputs if I can help it, the deduplication is the point (the library I built the czkawka package on is an image deduplication tool so it is typically the point of the hashing).

You want to avoid re-saving unless there's a meaningful perceptual change. This matches insta's behaviour where context can change between runs, but snapshots only update when the actual content needs creation/fixing.

• YAML info key, it’s mentioned briefly in docs
• For an example see the snapshots in my polars-genson repo’s genson-cli crate, integration tests of a CLI for JSON schematisation and normalisation.

The trade-off is what you identified yes but I think you’d probably find it agreeable [rather than a failure mode] if you reframe to thinking about it as a file that is an acceptable equivalent under some different definition of what it means to “have this output saved on disk”. Namely the definition of “being an identical image file” changes to “looks close enough”. It’s a sort of compression, or like squinting your eyes at something. Like imagine comparing if two photocopies/scans are “the same”. If our Python code is a photocopier, we do not need to update our snapshots on each run if we photocopied the same document.

You are not able to watch the diff to the last run when the result of create_image2 changes. You only see the diff to the result of create_image1.

Maybe this is exactly what is wanted when you use this, maybe not. I don't know.

When create_image2() changes, you diff against whichever test first generated that hash (e.g., create_image1()'s archive), not the last run of create_image2(). For page dewarping where outputs should be perceptually stable, that's acceptable - any perceptual change is worth investigating regardless of which test's archive it diffs against.

I’d present this to users as there being an upside (deduplication) and a downside (one-to-many fixture-to-snapshot relationship).

I’d like this for my test suite for page dewarping (flattening curved book pages from photos) so as to avoid two failure modes:

• Mass review when a tweak to the optimisation changes outputs slightly (but imperceptibly to the viewer)
• Mass acceptance without understanding what changed (checking one and approving the rest because it looks "about the same" - i.e. manual work that a perceptual hash can automate)

The first obviously leads to the second when a developer tries to save time, but defeats the purpose of snapshot testing.

The phash approach separates "did perceptual quality change?" (the test) from "what exactly changed?" (manual inspection of archived images).

RE: per-test vs per-hash storage: to be more precise, I am proposing that perceptual hashing be used for content-based addressing. Files with the same phash are treated as the same, like git's SHA256 but for perceptual equivalence. It should be understood (by us and users) that this form of storage will make perceptually similar files indistinguishable and thus the same on disk.

If you wanted one image per test, you'd override new_location or equivalent and that would be a form of deliberate duplication. I'm open to adding a "store per-test in phash-named directories" option if use cases emerge, but starting with per-hash storage matches the immediate goal: this is more like insta's context behavior (doesn't update on non-breaking context changes) than traditional exact-match snapshots.

[15r10nk]

Thank you for the explanation.

Right, that’s our potential failure mode. Yes, that's exactly the behaviour I want.

👍