Recursive glob fails to exclude files from sdist

[zahlman]

The documentation claims:

Changed in version 3.8: Include and exclude patterns can now use recursive glob patterns (**).

However, using flit-core 3.12, I try using this clause in pyproject.toml:

[tool.flit.sdist]
exclude = ["**/*.swp"]

and use build programmatically to invoke flit-core (via a wrapper script):

def build_from_sdist(src, dst):
    builder = ProjectBuilder(src)
    # delete = False for debugging
    with TemporaryDirectory(delete=False) as sdist_build_dir:
        print("THE DIRECTORY IS AT:", sdist_build_dir) # for debugging
        ProjectBuilder(src).build('sdist', sdist_build_dir)

and discover that the resulting sdist still contains those .swp files (corresponding to source files I currently have open in Vim; they are named with a leading .). Although notably it does not include e.g. .pyproject.toml.swp.

Specifying an explicit, complete path to one of the "temporary" files (still leaving everything open in Vim while running the build script) does cause that file to be excluded from the sdist.

[takluyver]

Thanks! I think the issue is with the leading .. The expansion happens here:

flit/flit_core/flit_core/sdist.py

Line 46 in 13792b5

for path in sorted(glob(osp.join(basedir, pattern), recursive=True)):

For the glob module, wildcards don't match files starting with . by default, and that's documented. There's an include_hidden parameter, but only from Python 3.11 onwards.

It looks like switching to pathlib's glob method might fix that. But it would also be nice to have a more efficient filter implementation that doesn't rely on expanding each pattern into matching files and directories first (this probably seemed more reasonable before ** recursive matching). But that might get tied up in the differences between Python versions.

Maybe we could also be smarter, either about .swp files specifically, or about when .hidden files are included. 🤔

In the meantime, I would guess that you can work around this by making the pattern **/.*.swp, assuming these files are always named with a leading . .