Malformed .in results in uncaught exception

[msftcangoblowm]

Lets see how pip-compile performs given a purposefully malformed .in file. Normally would expect a pip-compile specific Exception and a non-zero exit code.

malformed-pip.in

>=24pip\n

Which is obviously malformed. Since a .in is user input, it can contain errors.

pip has vendored package, packaging. When packaging fails to parse a requirement, it produces this Exception chain

pip._vendor.packaging._tokenizer.ParserSyntaxError
pip._vendor.packaging.requirements.InvalidRequirement
pip._internal.exceptions.InstallationError

pip-compile does not gracefully catch the InstallationError. Instead just bombs with exit code 1 and the below traceback.

Environment Versions

OS: Linux distro Void Linux LXDE
Python version: Python 3.9.16
pip version: pip 24.3.1
pip-tools version: pip-compile, version 7.4.1

Steps to replicate

echo ">=24pip\n" > malformed-pip.in
pip-compile --allow-unsafe --no-header -o malformed-pip.txt malformed-pip.in

Expected result

pip-compile should catch the Exception in a try-except block. Exception is reraised with a pip-compile specific Exception. The entire traceback does not get shown.

Taken from the traceback, handling of the Exception doesn't happen here.

piptools/scripts/compile.py", line 383, in cli
constraints.extend(

Actual result

Traceback (most recent call last):
File "[venv base path]/lib/python3.9/site-packages/pip/_vendor/packaging/requirements.py", line 36, in init
parsed = _parse_requirement(requirement_string)
File "[venv base path]/lib/python3.9/site-packages/pip/_vendor/packaging/_parser.py", line 62, in parse_requirement
return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
File "[venv base path]/lib/python3.9/site-packages/pip/_vendor/packaging/_parser.py", line 71, in _parse_requirement
name_token = tokenizer.expect(
File "[venv base path]/lib/python3.9/site-packages/pip/_vendor/packaging/_tokenizer.py", line 142, in expect
raise self.raise_syntax_error(f"Expected {expected}")
File "[venv base path]/lib/python3.9/site-packages/pip/_vendor/packaging/_tokenizer.py", line 167, in raise_syntax_error
raise ParserSyntaxError(
pip._vendor.packaging._tokenizer.ParserSyntaxError: Expected package name at the start of dependency specifier
>=24pip
^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "[venv base path]/lib/python3.9/site-packages/pip/_internal/req/constructors.py", line 362, in _parse_req_string
return get_requirement(req_as_string)
File "[venv base path]/lib/python3.9/site-packages/pip/_internal/utils/packaging.py", line 45, in get_requirement
return Requirement(req_string)
File "[venv base path]/lib/python3.9/site-packages/pip/_vendor/packaging/requirements.py", line 38, in init
raise InvalidRequirement(str(e)) from e
pip._vendor.packaging.requirements.InvalidRequirement: Expected package name at the start of dependency specifier
>=24pip
^

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "[venv base path]/bin/pip-compile", line 8, in
sys.exit(cli())
File "[venv base path]/lib/python3.9/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "[venv base path]/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "[venv base path]/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "[venv base path]/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "[venv base path]/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "[venv base path]/lib/python3.9/site-packages/piptools/scripts/compile.py", line 383, in cli
constraints.extend(
File "[venv base path]/lib/python3.9/site-packages/piptools/_compat/pip_compat.py", line 77, in parse_requirements
yield install_req_from_parsed_requirement(parsed_req, isolated=isolated)
File "[venv base path]/lib/python3.9/site-packages/pip/_internal/req/constructors.py", line 480, in install_req_from_parsed_requirement
req = install_req_from_line(
File "[venv base path]/lib/python3.9/site-packages/pip/_internal/req/constructors.py", line 405, in install_req_from_line
parts = parse_req_from_line(name, line_source)
File "[venv base path]/lib/python3.9/site-packages/pip/_internal/req/constructors.py", line 379, in parse_req_from_line
req: Optional[Requirement] = _parse_req_string(req_as_string)
File "[venv base path]/lib/python3.9/site-packages/pip/_internal/req/constructors.py", line 376, in _parse_req_string
raise InstallationError(msg)
pip._internal.exceptions.InstallationError: Invalid requirement: '>=24pip': Expected package name at the start of dependency specifier
>=24pip
^ (from line 1 of /tmp/pytest-of-chaosmonkey/pytest-24/test_compile_malformed_in_mix_0/requirements/malformed-pip.in)

[msftcangoblowm]

Here is the code excerpt. Since the unexpected Exception is not caught, the exit code is not 2.

The return variable name is metadata. The try-except block doesn't catch the exception, so it looks like metadata gets assigned!

When metadata.requirements is accessed, somewhere, a Generator is executed and raises the Exception, resulting in a nasty repetitive traceback.

/piptools/scripts/compile.py", line 383

        elif is_setup_file:
            setup_file_found = True
            try:
                metadata = build_project_metadata(
                    src_file=Path(src_file),
                    build_targets=build_deps_targets,
                    attempt_static_parse=not bool(build_deps_targets),
                    isolated=build_isolation,
                    quiet=log.verbosity <= 0,
                )
            except BuildBackendException as e:
                log.error(str(e))
                log.error(f"Failed to parse {os.path.abspath(src_file)}")
                sys.exit(2)

            if not only_build_deps:
                constraints.extend(metadata.requirements)

Notice the yield in piptools._compat.pip_compat.parse_requirements. So dealing with a Generator, that throws an Exception. Looks like and was mistaken for an object property, metadata.requirements

File "[venv base path]/lib/python3.9/site-packages/piptools/scripts/compile.py", line 383, in cli
constraints.extend(
File "[venv base path]/lib/python3.9/site-packages/piptools/_compat/pip_compat.py", line 77, in parse_requirements
yield install_req_from_parsed_requirement(parsed_req, isolated=isolated)

[msftcangoblowm]

Assign metadata.requirements to a variable within try, so the exception can be caught and tested for.

elif is_setup_file:
            setup_file_found = True
            try:
                metadata = build_project_metadata(
                    src_file=Path(src_file),
                    build_targets=build_deps_targets,
                    attempt_static_parse=not bool(build_deps_targets),
                    isolated=build_isolation,
                    quiet=log.verbosity <= 0,
                )
                gen_reqs = metadata.requirements
                lst_reqs = list(gen_reqs)
            except BuildBackendException as e:
                log.error(str(e))
                log.error(f"Failed to parse {os.path.abspath(src_file)}")
                sys.exit(2)
            except pip._internal.exceptions.InstallationError as e:
                log.error(str(e))
                log.error(f"Failed to parse {os.path.abspath(src_file)}")
                sys.exit(2)
            if not only_build_deps:
                constraints.extend(lst_reqs)

then write a pytest test with the malformed .in file ">=24pip\n"

[msftcangoblowm]

Generator is a curse word!

There are three cases in the if-elif-else block calling the Generator that can produce an Exception. The Exception can occur on every constraints.extend(gen_reqs) call.

piptools/scripts/compile.py lines: 366, 383, and 390

The solution is to not do that.

parse_requirements and metadata.requirements must be brought into ONE try-except block. Rather than three.

I get it. The code is easy to read, but that comes at the expense of not being DRY.