What's the idea?

We'd like to find more issues like the following before conda-pypi is widely adopted:

• conda pypi install huggingface-hub fails with a TypeError #150

We'd also like to gain the ability to arbitrarily test and package from PyPI across the three major operating systems we support (Windows, macOS and Linux).

What should be done?

To help in our QA efforts, we propose adding a new GitHub action that does the following:

• GitHub Action should be a separate workflow and only run when it's manually triggered
• The GitHub Action should accept a list of packages available on PyPI
• For each package, attempt to install this package into a separate environment
• If installation succeeds, verify the package installed correctly by running a python interpreter and importing the package
• For any failures, print the exception message so it can be inspected later

The actual implementation should be done inside of pytest so that we utilize our existing testing infrastructure. But, we need a way to dynamically parameterize these tests so that we can pass in arbitrary package names. Ideally, it would be best if there was a way for us to pass these in on the command line in the GitHub Action runners.

The test that installs and tests these packages should not be run during the normal test suite (i.e. we only want to manually trigger these). To achieve this, we should define a new pytest marker.

Additional context

The following shows how you can add new command line options to pytest that can then be passed in to tests:

In conftest.py:

def pytest_addoption(parser):
    parser.addoption(
        "--values",
        action="append",
        help="Values to parametrize tests with",
    )


def pytest_generate_tests(metafunc):
    if "value" in metafunc.fixturenames:
        values = metafunc.config.getoption("--values")
        if values:
            metafunc.parametrize("value", values)

In your test:

def test_dynamic(value):
    assert value