UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf3 in position 1970: invalid continuation byte

[Vicentelles]

Environment

• pip version: 10.0.1
• Python version: 3.7
• OS: Windows 10

Description

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf3 in position 1970: invalid continuation byte

Expected behavior

Uninstall a local package seamlessly
How to Reproduce

1. Get package from '...' vrt is a local installed package
2. Then run 'pip uninstall vrt
3. An error occurs.

Output

Exception:
Traceback (most recent call last):
  File "c:\users\vicen\appdata\local\programs\python\python37\lib\site-packages\pip\_internal\basecommand.py", line 228, in main
    status = self.run(options, args)
  File "c:\users\vicen\appdata\local\programs\python\python37\lib\site-packages\pip\_internal\commands\uninstall.py", line 68, in run
    auto_confirm=options.yes, verbose=self.verbosity > 0,
  File "c:\users\vicen\appdata\local\programs\python\python37\lib\site-packages\pip\_internal\req\req_install.py", line 660, in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "c:\users\vicen\appdata\local\programs\python\python37\lib\site-packages\pip\_internal\req\req_uninstall.py", line 355, in from_dist
    for path in uninstallation_paths(dist):
  File "c:\users\vicen\appdata\local\programs\python\python37\lib\site-packages\pip\_internal\req\req_uninstall.py", line 50, in unique
    for item in fn(*args, **kw):
  File "c:\users\vicen\appdata\local\programs\python\python37\lib\site-packages\pip\_internal\req\req_uninstall.py", line 67, in uninstallation_paths
    r = csv.reader(FakeFile(dist.get_metadata_lines('RECORD')))
  File "c:\users\vicen\appdata\local\programs\python\python37\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 1408, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "c:\users\vicen\appdata\local\programs\python\python37\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 1405, in get_metadata
    return value.decode('utf-8') if six.PY3 else value
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf3 in position 1970: invalid continuation byte

[Vicentelles]

Uninstalled deleting the vrt and vrt-0.0.1.dist_info folders

Pip install ./ works well
but ...

pip uninstall vrt

gives again the same error

[pkkj]

I hit a similar issue though the callstack is somehow different.

Environment

• pip version: 10.0.1
• Python version: 3.5
• OS: Windows 10
• Default codepage: cp936 (Simplified Chinese)

Exception:
Traceback (most recent call last):
  File "c:\python35\lib\site-packages\pip\_internal\basecommand.py", line 141, in main
    status = self.run(options, args)
  File "c:\python35\lib\site-packages\pip\_internal\commands\install.py", line 347, in run
    use_user_site=options.use_user_site,
  File "c:\python35\lib\site-packages\pip\_internal\req\__init__.py", line 42, in install_given_reqs
    auto_confirm=True
  File "c:\python35\lib\site-packages\pip\_internal\req\req_install.py", line 863, in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "c:\python35\lib\site-packages\pip\_internal\req\req_uninstall.py", line 314, in from_dist
    'installed-files.txt').splitlines():
  File "c:\python35\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 1402, in get_metadata
    return value.decode('utf-8') if six.PY3 else value
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc6 in position 608: invalid continuation byte

I doubt that this is caused by inconsistent encoding for writing and reading meta data files, including "install-files.txt".

For example, in req_install.py, no encoding is specified when writing. In my machine, the installed-files.txt is encoded in GB2312 (Simplified Chinese)

            inst_files_path = os.path.join(egg_info_dir, 'installed-files.txt')
            with open(inst_files_path, 'w') as f:
                f.write('\n'.join(new_lines) + '\n')

However, the script try to decode the file by using UTF8 when installed-files.txt (where exception happens):

    def get_metadata(self, name):
        if not self.egg_info:
            return ""
        value = self._get(self._fn(self.egg_info, name))
        return value.decode('utf-8') if six.PY3 else value

I do some tests. If I converted installed-files.txt to UTF8, I won't hit such UnicodeDecodeError

[chrahunt]

The specific line mentioned above is here.

[chrahunt]

The fix here should be to write the installed-files.txt encoded as utf-8 (at least in Python 3), to match the behavior on the reading side.