Reconsider our choice of using _test directory as the default

[willingc]

🚀 Feature

Use test as the default directory for tests.

Also, placing test directory outside of src would prevent the test directory to be shipped as part of the wheel.

Motivation

Brings this into a more standard pattern as recommended by Scientific Python.

See #76 for additional context.

[jni]

Is there a write up you can point to about this @willingc?

• using test as the directory within src is (I think?) obviously bad because now that's part of your public API, but
• not shipping it with wheels may be desirable or undesirable? I like that in the old days I could do pytest -m skimage and it would run the test suite of the installed skimage. Haven't done that in a while but it feels valuable.

I don't feel too strongly about this, I just like to have a good paper trail documenting why certain decisions are made.

[Czaki]

On EuroPython talk from a Fedora packaging person (https://ep2025.europython.eu/session/bridging-the-gap-viewing-your-project-through-a-downstream-packager-s-lens), she asked to ship test in sdits (but not required in wheel) as it will simplify life of persons who try to automate packaging upstream package updates.

[willingc]

Using test at the same level as src is explained here: https://learn.scientific-python.org/development/guides/pytest/#basic-test-structure and also checked by their repo checker.

[TimMonko]

I didn't initially reply to these comments, not to ignore them, but out of ignorance. I didn't really understand what the big deal is about wheel or not, or why anyone up or down stream would want to run my tests (and I still don't know!). But, I was just triaging through Scientific Python stuff and this was one of the changes to make so I wanted to try it out, certainly we can keep current organization but I have a few rough experiences where I've observed what I think makes #83 is helpful.

1. _tests naming causes some IDEs to by default not detect the tests in a project. We've probabaly mostly forgotten about this because with experience its an easy fix. But as a noob, it took me hours to figure out how to run my tests in my own plugin, where it would work in other repos I cloned. Note, naming it tests works without issue.
2. Having tests and src mirror each other in dir tree to me is nicer to understand than nesting tests inside src. Ultimatey, it's helpful to have test and modules in parallel at the same depth from the root.
3. Coverage also is hard to set up (or change) when tests is nested, because it will also count coverage for/against the test files -- without modification otherwise to something like pyproject
4. At one point, trying to exclude tests from my built package required editing MANIFEST.in. This is now obsolete, but there are many google searched materials that would lead one to trying this still.

All legit things I faced as a plugin dev, and I think all experiences that would be improved with tests in root rather than src

[jni]

Apologies for missing this discussion while on holidays.

@willingc I have read the SP recommendation but to me it reads like they are trying to provide firm guidance, which is appreciated by beginners, rather than define a best practice, such as using src:

You should make a folder called “tests” in your repository (probably alongside your src or module folder, but rarely they are placed inside the module folder).

(I don't see a value judgement in that sentence or the later paragraph, but I might be missing something.

@TimMonko Thanks for your comments, which do help me to understand the rationale.

_tests naming causes some IDEs to by default not detect the tests in a project.

That's indeed terrible! But something that could probably be fixed by an update to our pyproject.toml?

Ultimately, it's helpful to have test and modules in parallel at the same depth from the root.

Unit tests mirroring the same hierarchy as the source code is indeed very helpful — and much more easy to maintain if the tests are shipped in the same folder as the code, as moving packages around moves the corresponding tests with them.

Coverage also is hard to set up (or change) when tests is nested, because it will also count coverage for/against the test files

This one in particular is interesting because Ned Batchelder, author of coverage.py, specifically recommends including tests in coverage measurements: https://nedbatchelder.com/blog/202008/you_should_include_your_tests_in_coverage.html

At one point, trying to exclude tests from my built package required editing MANIFEST.in.

This is the big one for me — given the recommendation (above) from packaging folks to include tests in source distributions, my only blocker for the move is that building the sdist for the package should include the tests directory. If we can satisfy that, then everything above are weak opinions, and I'm happy to be outvoted. (@TimMonko "why anyone up or down stream would want to run my tests" for packaging folks, this can help them be sure that a package works with their particular configuration of numpy, numba, compilers, etc. Perhaps you wrote your tests on your machine and they passed with the Windows compiled version of NumPy, but they will fail when running on Linux ARM machines. Packaging folks care a lot about the packages they distribute working across systems.) (Though it's funny, conda-forge I think decided early on that they wouldn't have the bandwidth to do that for all packages so when I was wrapping up napari for conda-forge, we couldn't get the tests to pass and John Kirkham was like "just check that it imports". I'm pretty sure it's like that to this day. 😂)

[willingc]

Unit tests mirroring the same hierarchy as the source code is indeed very helpful — and much more easy to maintain if the tests are shipped in the same folder as the code, as moving packages around moves the corresponding tests with them.

Choosing a test layout - 2 options. Either option: close to the code or a separate, similarly structured hierarchy, tests directory works for projects.

I do lean toward a separate directory for tests, as it tends to simplify large code refactors and restructures where you don't want to be changing code and modifying tests, with the exception of new tests, in the same PR. By not mixing code and tests into the same PRs during a major refactor, you ensure backwards compatibility.

For a plugin template, I would lean toward implementing the more popular approach of a separate, though similar hierarchy, tests directory.

Shipping an sdist with a test directory, either approach, is fine.

[jni]

For a plugin template, I would lean toward implementing the more popular approach of a separate, though similar hierarchy

Thanks @willingc. As I ruminated on this throughout the day, I did think that choosing the most common default is better for beginners.