packaging: pip install from git/sdist ships no data/*.json -> FileNotFoundError: udtools/data/upos.json

[EphraimMeiri]

Summary

Installing the udtools package from source / a git tag produces a wheel that contains none of the data/*.json files the validator needs, so validation fails immediately with FileNotFoundError: .../udtools/data/upos.json. Only the pre-built PyPI wheel works.

Reproduction

pip install "git+https://github.com/UniversalDependencies/tools.git@r2.18#subdirectory=udtools"
echo -e "# sent_id = 1\n# text = X\n1\tX\tx\tNOUN\t_\t_\t0\troot\t_\t_\n" | python -m udtools ...   # or via Validator()

Result:

FileNotFoundError: [Errno 2] No such file or directory: '.../site-packages/udtools/data/upos.json'

Cause

pyproject.toml declares:

[tool.setuptools.package-data]
udtools = ["data/*.json"]

but there is no udtools/src/udtools/data/ directory in the repository (the data/*.json files appear to be generated from the UD docs at release time and bundled only into the published PyPI wheel). When building from the repo, the glob matches nothing, so the resulting wheel/sdist ships without upos.json, feats.json, deprels.json, edeprels.json, etc.

Impact: it's not possible to pip install a working validator from source or from a release tag — only the published PyPI artifact functions.

Secondary issue: version not bumped per tag

The r2.18 tag's pyproject.toml still declares version = "0.2.7", identical to the older PyPI 0.2.7 release — even though the code differs (e.g. the level2.py Warning import is present on the tag but absent in the PyPI 0.2.7 wheel; see companion issue). This makes the two indistinguishable by version.

Suggestions

• Either commit the generated data/*.json into the package source, or document/automate the generation step so a from-source build includes them (e.g. a build hook).
• Bump the package version for each release/tag so installed builds are distinguishable.

[dan-zeman]

The JSON data are actually present in the repository but they are in the top folder (data/*.json), outside udtools. I confirm that a copy is created (udtools/src/udtools/data/*.json) when releasing udtools to PyPI.

There are three ways of using the validator that should currently work:

• Install the package from PyPI (pip install udtools). Static, JSON data not updated.
• Do not install it as a Python package, just git clone git@github.com:UniversalDependencies/tools.git, then run validate.py from the root folder of the repository. For many years this was the only way, and it is still available as an option.
• git clone the tools repo as in the previous option, then install it as an editable package sitting in the cloned repo: pip install --no-cache-dir --editable tools/udtools. This way it is possible to update the JSON data whenever needed by calling git pull.

The file pyproject.toml is a copy of the one used during the last release on PyPI, so that's how its version number is to be interpreted.

UD releases and PyPI releases of udtools are independent cycles. The r2.18 tag denotes the UD release and can be found in all treebank repositories + docs and tools. Here it merely marks the commit that was at the head when the UD release package was archived on Lindat; the code in udtools can be, and the JSON data almost certainly is, different from any version released on PyPI.