Basic skeleton to start pkg_resources migration

[uranusjr]

This implements the basic shim structure I mentioned in #7413 (comment). I moved the lru_cache() shim to utils.compat since it’s useful for get_environment().

One very trivial change is made in self_outed_check to demostrate how the end result would look like in essence.

I think (hope?) this would be enough for someone to run with the idea and start refactoring. I think the logical next refactor would be to change AbstractDistribution to return BaseDistribution instead, and modify the prepare code. After that, usages in the new resolver can switch.

[deveshks]

Thank you for this. I think this will help one of my PRs #8054 as well (also pointed out in #8054 (comment))

Should I should hold on to making changes to that PR until this gets matured and merged?

[uranusjr]

I think it works either way. pkg_resources usages in that part of the code is local, and can be rewritten relatively easily no matter the order. I’ll leave the decision to @pradyunsg.

[pradyunsg]

LGTM!

[pypa-bot]

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

[uranusjr]

Rebased to include changes from #8054. I chose to keep get_distribution() and get_installed_distributions() in pip._internal.utils.misc as wrappers to pip._internal.metadata for now to avoid cluttering the patch. We should remove usages of them gradually and eventually always call pip._internal.metadata directly instead.

[uranusjr]

@deveshks You’ll need to take a look to make sure I got everything right from #8054.

[pfmoore]

We should remove usages of them gradually and eventually always call pip._internal.metadata directly instead.

Maybe we should add this as a comment in the code, to guide future maintainers in refactoring? A note in an issue comment is likely to get forgotten, whereas a note in the code won't.

[uranusjr]

Good idea, I added the following comment to both functions:

# TODO: Refactor the call site to make it call get_environment() directly,
# and operate on the wrapper PipMetadataDistribution instead.

[uranusjr]

This seems to rebase cleanly to master, hopefully the test suite agrees. I want to get this merged soon and start working on the transition. The plan is to push this out for Python 3 in the 20.3 release (but keeping the old implementation in --use-deprecated so things won’t break as bad), and remove the old implementation when we drop Python 2 support.

[uranusjr]

So obviously that didn’t happen. Since we’ll want 21.0 to be a minimal change release, I’m adding this to 21.1, which should be plenty of time to get this worked into a mergable state.

[uranusjr]

Note for future self. With python/importlib_metadata#111 resolved, pip should have all the needed parts to remove pkg_resources entirely. This finds locations of all distributions matching name in a list of prefixes:

for path in MetadataPathFinder._search_paths(name, search_paths):
    distribution = PathDistribution(path)  # Build a distribution class at `path` for inspection.

The _search_paths() function is private, but importlib-metadata promises to keep it steady for pip. pip needs its own Distribution wrapper class to bind a distribution to the right path, which should not be too difficult.

[pfmoore]

From #9540:

Things are blocked by #8114, I’m waiting for a good timing to merge it.

I'm not sure what "good timing" you want here, this has no conflicts and 2 approvals so IMO, it's good to go. I'm grateful you didn't land it just before 21.0 (and even more so that you didn't land it between 21.0 and 21.0.1!) but I think the 21.0 cycle is more or less done now. Let's wait a week in case anyone screams about 21.0.1, but unless they do, I'd say you can merge this any time from then.

[uranusjr]

I'm not sure what "good timing" you want here

At this point, a couple of more days to make sure 21.0.1 does not contain catastrophic issues that require an emergency release.

[uranusjr]

A few weekdays have passed and I think it’s safe to assume the 21.0 release if free of major issues. I’ll merge this later today and start working on migration.