Tests with `external` fail by default in the CI

[zvyn]

The default of --inline-snapshot=disable when a CI is detected causes external to fail with an inline_snapshot._external._storage._protocol.StorageLookupError.

This is to be expected since disable disables all snapshot "magic".

But maybe --inline-snapshot=report would be a better default for CI for this reason?

[15r10nk]

No, it should work with disable. Did you use the hash or uuid storage? A script where I can reproduce this error would also be useful.

[zvyn]

Sorry, I clearly underestimated how hard this would be to reproduce. With https://github.com/zvyn/demo-inline-snapshot-issues-292 I got the issue after creating everything in backend in the repo root and then moving it into backend with executing pytest with --inline-snapshot=create in between.

In that case external would still look in the old path even after deleting and re-creating the __inline_snapshot__-folder when run with disable but it passes with report:

location = ExternalLocation(storage='uuid', stem='90200275-a3ee-43d4-8d0b-124efbc00ef7', suffix='.json', filename=PosixPath('/tmp/issue/demo/tests/test_bug.py'), qualname='test_something')

The test-file is actually at /tmp/issue/backend/demo/tests/test_bug.py

I coudn't figure out how I managed to trigger this behavior in a CI-Pipeline. The script there does the equivalent of:

python3 -m venv venv  # Python 3.11 but does not seem to matter
venv/bin/pip install -r backend/requirements.txt
venv/bin/pytest -vvv --reuse-db --cov --junitxml=report.xml -c backend/pyproject.toml --inline-snapshot=report

The the issue comes down to report and disable looking at different places under very specific and hard to reproduce circumstances.

[15r10nk]

I can reproduce your failure when I rename the backend folder to backend2

But the removing __pycache__ fixed the problem for me.

> rm __pycache__/ tests/__pycache__/ -r

demo-inline-snapshot-issues-292/backend2/demo on  main [✘?] via 🐍 v3.8.3 
> uv run pytest --inline-snapshot=disable
========================================= test session starts ==========================================
platform linux -- Python 3.13.6, pytest-8.4.2, pluggy-1.6.0
django: version: 5.2.6, settings: demo.settings (from ini)
rootdir: /home/frank/projects/tmp/demo-inline-snapshot-issues-292/backend2
configfile: pyproject.toml
plugins: django-4.11.1, inline-snapshot-0.28.0, anyio-4.10.0
collected 1 item                                                                                       

tests/test_bug.py .                                                                              [100%]

========================================== 1 passed in 0.03s ===========================================

I think the problem is that the path which I use is stored inside the ".pyc" files. This path is not changed after you moved the code (and cache). This is also the reason why you can not reproduce this in CI (I think you do not commit the __pycache__ folder).

Can you make sure that you:

• have committed the external files
• that you can reproduce the error locally with --inline-snapshot=disable
• also after removing all __pycache__ folders
• Are you maybe moving your code around during your CI build? Maybe as artifacts between different jobs?

[15r10nk]

Hello @zvyn, can you check whether the problem is caused by moving files in the CI?

[zvyn]

Hi @15r10nk,

Looking back at this I had committed the fixture as __inline_snapshot__/test_folder/test_file_name/<uuid>.json at first and added a file with the same content as __inline_snapshot__/test_folder/__module__/<uuid>.json later.

Now deleting the __module__ variant will pass all tests with --inline-snapshot=report but fail with a StorageLookupError with --inline-snapshot=disable.

Only keeping the __module__ variant works with both.

I made sure this isn't some caching fluke by running git clean -fx path/to/tests/ in between test runs.

Unfortunately I can't really tell you how I ended up with the two file locations for the same external snapshot. I use the same external(<uuid>) twice in the same test file.

[15r10nk]

Thank you, I think I can reproduce your problem.
I assume you renamed your function or moved your external with the specific id in your tests (which is supported), I will fix this.

What is not supported is to use the external with the same id twice like this.

from inline_snapshot import external                                    
                                                                            
def test_1():                                                          
    assert "a" == external("uuid:9f4e5531-3710-4a4d-a11c-c6a3ddca7335.txt")

def test_2():                                                          
    assert "a" == external("uuid:9f4e5531-3710-4a4d-a11c-c6a3ddca7335.txt")

you should change your code to something like this or generate different uuids

from inline_snapshot import external                                    
                                                                            
expected = external("uuid:9f4e5531-3710-4a4d-a11c-c6a3ddca7335.txt")                                                                            

def test_1():                                                          
    assert "a" == expected

def test_2():                                                          
    assert "a" == expected

I will add a check that uuids are not used twice.

But this problem is unrelated to your issue.

[15r10nk]

I guess your tests are not in "tests/" in your project root.

setting test-dir to your test folder should help in this case.

[tool.inline-snapshot]
test-dir="demo/tests"