Add custom checkers to `twine check`

[bhrutledge]

Caveat: I have very little experience with proposing and implementing substantial new features/APIs in open-source projects.

There have been more than a few requests to add functionality to twine check, but those have been mostly flagged as duplicates of #430, which is currently blocked on pypa/packaging#147, which hasn't seen much movement.

#843 contains a proposal to add custom "checkers" via setuptools entry_points, which would allow functionality to be added by independently-maintained packages. Twine already allows for plugin sub-commands via entry_points, but those would have to implement argument parsing and reading packages from disk, and be run as a separate command from twine check.

The purpose of this issue is to gather requirements and propose an API for custom checkers. For an initial (but insufficient) overview of the current API, see the autodoc for twine.commands.check.

Some initial ideas/questions:

• Currently, the only check is for long_description, implemented in a private method. That should eventually be refactored to match the new API, but it could be refactored earlier to help establish the new API.
• What are some examples of custom checks that people would like to write? Can we/they share example implementations to drive the API design?
• What attributes of a package do people want to check? Presumably metadata, which is available via PackageFile, along with the original filename.
• What lessons can we learn from other tools that offer plugin functionality, e.g. flake8?
• How should/must the new API be documented before it's implemented?
• I'm assuming we want/need to maintain the API of check() and main() in twine.commands.check.

[bhrutledge]

@sigmavirus24 What do you think of this initial writeup? Any edits before I publicize it (e.g. to folks who submitted check feature requests)?

[bhrutledge]

@sigmavirus24 Just noticed the 👍 (can't decide if the absence of notifications for those are good or bad). Is there anything you'd like to add, e.g. lessons learned from flake8?

[webknjaz]

Looks reasonable. Although, I don't know if the checkers would need custom args (if they did, we'd also be talking about a config file, I suppose).

[bhrutledge]

@webknjaz Can you give an example of a checker that would need custom args? And/or can you propose how some of twine check features would be implemented in a checker?

[webknjaz]

Not really, I was just commenting on the original mentions of having to implement arg parsing which doesn't seem necessary right now.

[bhrutledge]

cc @di @ssbarnea @gaborbernat @henryiii

Y'all have shown some interest in improving twine check. Do you have any thoughts on this proposal?

[ssbarnea]

I am not very keen on having plugins for these checks as it would mean that we would need to modify each project running 'twine check --strict' in order to enable to newly added checks. I do not want to have to add them to benefit from newer checks. If we can avoid that downside the better. If we add new check only to strict, we should be fine.

Regarding where these checks are implemented, i do not care much, but if they are in another project, it should be a direct dependency of twine, not an optional one.

[di]

I think the proposal is functionally sound, but I'm not sure I see the demand for third-party plugins for twine check. I think the main goal of the command is determining "will PyPI accept/reject this distribution & why" and to that end, we probably want to be authoritative about the results and not leave that for others to implement.

Assuming pypa/packaging#147 eventually happens (I see no objections there, just and lack of people to work on it) I'm having a hard time seeing what else users would want twine check to do, and thus a justification for this feature.

[henryiii]

I think I'll likely be a little repetitive of @di and @ssbarnea if I write a full response. I don't think there are an unlimited number of checks for metadata that need to be done. There are a set of things that are needed (either for PyPI, or for consistency), but there isn't a need for an arbitrary number of things, like flake8. And, honestly, I think pre-commit has been a better solution than flake8 - instead of providing an extension system for running checks, it's an infrastructure for running anything, so there are lots of great (and modifying) checks that are not in flake8. I would see the same thing here - if you really want to add a custom check, you can write a custom checker and then run it on your wheel files, adding it to pre-commit or CI or wherever. Extending something based on what is installed in the current environment has caused issues for pytest (flaky is registered by multiple extensions, which causes issues for pypa/build).

I'm very much in favor of twine check becoming more powerful, but I don't think it needs to be come externally extensible. A good design for internal use is nice, but it doesn't seem to be that helpful to be able to add twine-<stuff> extensions to add checks. If there's really a need for a package-metadata version of flake8, it doesn't have to be part of twine, IMO.

For completeness, here's a few things I can think of that would be nice to check:

• Extra quotes. Sometimes package names and such get an extra set of quotes (at least if placed in setup.cfg, this can be common) - could be checked when making packages too
• Python-Requires missing (would need to be twine check only)
• Missaligned Python-Requires and wheel tags (py2 + 3+ for Python requires) (twine check only)

For "extendable" checks, the only think I would think of would be validating md or rst markup, but that can be done in the original file with a normal checker, usually.

[CyrilRoelandteNovance]

So I proposed #843 and tried to introduce the plugin system because I felt like it could be useful. To be honest, I'm fine with dropping the "plugin" architecture and having all checkers live in twine.

What are some examples of custom checks that people would like to write? Can we/they share example implementations to drive the API design?

• Make sure a license is specified
• Make sure the "license" field is not used when a classifier is enough
• Make sure "python-requires" is specified
• Check for the presence of "Programming Language :: Python ::" classifiers
• Make sure a homepage/author is specified, as it's often useful for users/packagers to be able to contact authors to report bugs/submit patches

How should/must the new API be documented before it's implemented?

I think it should be kept simple:

def checker(filename: str) -> Tuple[List[str], List[str]]:

The checker would take the filename of a tarball or wheel and return a list of warnings/errors.