UnicodeDecodeError when calling iter_entry_points

[alanhamlett]

Using Python 3.7 with built-in pkg_resources when calling pkg_resources.iter_entry_points('pygments.lexers') a UnicodeDecodeError is raised:

Traceback (most recent call last):
  File "D:\Software\Install\NoGreenInstall\Python37\lib\site-packages\pkg_resources\__init__.py", line 2698, in get_entry_map
    ep_map = self._ep_map
  File "D:\Software\Install\NoGreenInstall\Python37\lib\site-packages\pkg_resources\__init__.py", line 2669, in __getattr__
    raise AttributeError(attr)
AttributeError: _ep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\Users\Administrator\.vscode-insiders\extensions\wakatime.vscode-wakatime-1.2.13\wakatime-master\wakatime\stats.py", line 121, in guess_lexer_using_filename
    lexer = custom_pygments_guess_lexer_for_filename(file_name, text)
  File "c:\Users\Administrator\.vscode-insiders\extensions\wakatime.vscode-wakatime-1.2.13\wakatime-master\wakatime\stats.py", line 307, in custom_pygments_guess_lexer_for_filename
    for lexer in _iter_lexerclasses():
  File "c:\Users\Administrator\.vscode-insiders\extensions\wakatime.vscode-wakatime-1.2.13\wakatime-master\wakatime\packages\pygments\lexers\__init__.py", line 233, in _iter_lexerclasses
    for lexer in find_plugin_lexers():
  File "c:\Users\Administrator\.vscode-insiders\extensions\wakatime.vscode-wakatime-1.2.13\wakatime-master\wakatime\packages\pygments\plugin.py", line 52, in find_plugin_lexers
    for entrypoint in iter_entry_points(LEXER_ENTRY_POINT):
  File "D:\Software\Install\NoGreenInstall\Python37\lib\site-packages\pkg_resources\__init__.py", line 645, in iter_entry_points
    entries = dist.get_entry_map(group)
  File "D:\Software\Install\NoGreenInstall\Python37\lib\site-packages\pkg_resources\__init__.py", line 2701, in get_entry_map
    self._get_metadata('entry_points.txt'), self
  File "D:\Software\Install\NoGreenInstall\Python37\lib\site-packages\pkg_resources\__init__.py", line 2408, in parse_map
    for group, lines in data:
  File "D:\Software\Install\NoGreenInstall\Python37\lib\site-packages\pkg_resources\__init__.py", line 3036, in split_sections
    for line in yield_lines(s):
  File "D:\Software\Install\NoGreenInstall\Python37\lib\site-packages\pkg_resources\__init__.py", line 2268, in yield_lines
    for ss in strs:
  File "D:\Software\Install\NoGreenInstall\Python37\lib\site-packages\pkg_resources\__init__.py", line 2627, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "D:\Software\Install\NoGreenInstall\Python37\lib\site-packages\pkg_resources\__init__.py", line 1408, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "D:\Software\Install\NoGreenInstall\Python37\lib\site-packages\pkg_resources\__init__.py", line 1405, in get_metadata
    return value.decode('utf-8') if six.PY3 else value
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x91 in position 40: invalid start byte

[jaraco]

The problem here is that the content of the entry_points.txt file for some package on the system has non-UTF-8 in it. You'll have to track down which package is supplying those entry points and determine how they're getting encoded with another encoding. If you know which package might have this invalid encoding, it should be easy enough to replicate by installing that package and then running list(pkg_resources.iter_entry_points('pygments.lexers')), perhaps by using pip-run:

$ pip-run pygments-mathematica -- -c "import pkg_resources; print(list(pkg_resources.iter_entry_points('pygments.lexers')))"                                 
Collecting pygments-mathematica
  Downloading https://files.pythonhosted.org/packages/9a/ca/552bbee919077d41528875d12bd592f195ced28185f807e21dce4d81dfb0/pygments_mathematica-0.3.3-py2.py3-none-any.whl
Collecting Pygments>=2 (from pygments-mathematica)
  Using cached https://files.pythonhosted.org/packages/13/e5/6d710c9cf96c31ac82657bcfb441df328b22df8564d58d0c4cd62612674c/Pygments-2.3.1-py2.py3-none-any.whl
Installing collected packages: Pygments, pygments-mathematica
Successfully installed Pygments-2.3.1 pygments-mathematica-0.3.3
[EntryPoint.parse('MathematicaLexer = mathematica:MathematicaLexer'), EntryPoint.parse('xonsh = xonsh.pyghooks:XonshLexer'), EntryPoint.parse('xonshcon = xonsh.pyghooks:XonshConsoleLexer')]

There may be an underlying bug in setuptools that has caused this package to have invalid encodings in the entry-points metadata, but as reported the issue appears to be with the package not meeting the expectations for metadata format.

[alanhamlett]

Running:
list(pkg_resources.iter_entry_points('pygments.lexers'))

Produces output:
[]

As seen in this comment.

[jaraco]

That seems to contradict the claim from the original post on this ticket. There must be something different about the environment in which the error occurred. Do you have any way to invoke commands on that environment? I could put together a command or set of commands that could help you track down the issue. Or you could just search for all files called entry_points.txt (possibly in zip files) and see which one or ones have non-UTF-8 characters in them.

[alanhamlett]

There must be something different about the environment

Ah yes, the list command was run in Python2 but the original error came from Python3. There must be both installed. I don't have access to invoke commands on the system, I'm just relying on help from @ZainChen to run the commands on his machine and report the output.

[gandarez]

Both python2 and python3 produce the same result as you can see in the image below

[gandarez]

any update here?

[gandarez]

any update?

[jaraco]

I don't see a problem stated except an unexpected UnicodeDecodeError that I was unable to replicate. Can someone state the problem and expected behavior and steps to replicate?