dist-info data ignored in favour of file found in the package

[llimeht]

Between pip-licenses 5.5.0 and 5.5.5, the licence output from the SasView wheel changed; I suspect this is due to now finding lowercase filenames (#208) accidentally meaning that other files were found.

SasView wheel correctly declares:

License: BSD-3-Clause
License-File: LICENSE.TXT

and sasview-[version].dist-info/licenses/LICENSE.TXT exists, with v5.5.0 returning that file:

$ pip-licenses -v
pip-licenses 5.5.0

$ pip-licenses -p sasview --with-license-file --format=json
[
  {
    "License": "BSD License",
    "LicenseFile": ".../.venv/lib/python3.13/site-packages/sasview-[VERSION].dist-info/licenses/LICENSE.TXT",
    "LicenseText": "Copyright (c) 2009-2025, ...

However, the sasview wheel also includes the file sas/docs/_sources/user/license.rst.txt (sasview ships its documentation inside the wheel as it needs to be displayed at runtime and while offline). That's an rst file with other text in, not just the licence text. That's not the version we want pip-licenses to return, but suddenly v5.5.5 has started doing so:

$ pip-licenses -v
pip-licenses 5.5.5

$ pip-licenses -p sasview --with-license-file --format=json
[
  {
    "License": "BSD License",
    "LicenseFile": ".../.venv/lib/python3.13/site-packages/sas/docs/_sources/user/license.rst.txt",
    "LicenseText": ".. _license:\n\n...

My expectation (and how pip-licenses used to behave) is that pip-licenses would give me the licence file defined in the wheel metadata, not some other file that happened to be in the wheel which could be entirely unrelated data, or as in this case, the correct data but in an unwanted rst format.

[stefan6419846]

Yes, this is most likely related to allowing lowercase filenames as well.

There are multiple factors coming into play here:

• How importlib.metadata provides the list of files.
• How pip-licenses evaluates them.
• What the users expect (apart from unexpected breaking changes).

pip-licenses only returns the first match per file, although there have been requests for returning all files (see #71). From a license review point-of-view, returning and looking at all license files would indeed be the correct approach to detect and correctly attribute "hidden" licenses (from my experience, django is a package which heavily relies on this).

pip-licenses internally does the following:

• Retrieve the list of files using importlib.metadata.Distribution.files.
  • Read the files from the RECORD file of the dist-info directory.
  • Read the files from the installed-files.txt file of the egg-info directory.
  • Read the files from the SOURCES.txt file of the egg-info directory.
• Filter this list with regex patterns to extract the first license, notice and author file which actually exists.

Thus the returned file depends on the order of the entries inside the RECORD file (as far as I understand, there is no requirement to have them sorted in any specific order), which for sasview 6.1.3 defines

• sas/docs/_sources/user/license.rst.txt in line 1413,
• sas/docs/doctrees/user/license.doctree in line 1890,
• sas/docs/user/license.html in line 2041 and
• sasview-6.1.3.dist-info/licenses/LICENSE.TXT in line 2324.

With changing the matching logic to lowercase, this will now exit earlier for sasview.

[llimeht]

Looking through a few dozen installed modules here, the RECORD file is asciibetical sorted - that's not mandated by the spec but I would assume the reproducible builds folks provided patches to make it so in both the metadata-producing and zip-file-producing code. It's then our dumb luck that the distributable is called sasview while the module is sas, and so the file in the metadata is last not first.

I can see that for the use of pip-licences in an auditing sense, it would be desirable to mine all licences; at the same time, pip-licenses has a data presentation layer that dumps the data into file formats that are for human readable presentation not for auditing (which is what we wanted in sasview - we wanted pretty formatted licenses of everything in the bundle to put in the About box, and we were quite happy not to have to write something like pip-licenses to achieve that goal!).

In the absence of being able to deal with all licence files, is it reasonable to prioritise the dist-info metadata rather than all licence files? I had expected that pip-licenses was doing something vaguely like (knowing that it is vastly more complicated than this with wheels, eggs, editable installs, multiple historical standards, and packages just making up their own thing) :

import importlib.metadata
import fnmatch

info = importlib.metadata.metadata("sasview")
files = importlib.metadata.files("sasview")

[f for f in files if fnmatch.fnmatch(f, "sasview-*.dist-info/licenses/" + info["License-File"])]
# [PackagePath('sasview-6.1.2.dev102+g18d7612ed.d20251111.dist-info/licenses/LICENSE.TXT')]

and then poking at the file list level only if the metadata could not be found via the defined metadata interface.

To balance the audit use-case, could there be a --all-licenses or --use-metadata cli option control the search, that also being extensible in a future state where pip-licenses supports multiple licence files?

[stefan6419846]

Hardcoding to *.dist-info should not even be required here. From my experience, we will most likely always deal with path-based distributions and filtering can take advantage of this by inserting something like

if not abs_path.is_relative_to(Path(pkg._path)):
    continue

into get_pkg_included_file. AFAIK pkg._path will always be the path to the metadata directory.

[reactive-firewall]

Sorry for the delay for feedback on my end; I just wanted to provide an update of where this stands, due to ongoing work on #316 causing longer delays than I initially hoped/wanted.

While there are significant changes going on in the background (namely #316 as part of refactoring the code for Version 6.x series), this (#330) is indeed some kind of regression (likely related to some technical debt around changes in GHI #208 and PR #297 and will interact with plans for GHI #309).

Several comments on this: First I've added this as a specific use-case to track as part of #71 and related GHI #242. Second, I plan to add SasView to some new integration testing (along with GHI #309) in CI/CD to support this edge-case of module and file names 'multi-match/sort-discrepancies' (not sure what to call it) either-way. Third, I like the idea of being able to possibly sort the multiple matches, and will take a look at how reasonable it is to implement and, more importantly, maintain.