Unhandled UnicodeDecodeError

[devl00p]

Hello,

I'm using httpx 0.18.1 and I have regularly bumped into UnicodeDecodeError like the following one :

$ python
Python 3.8.10 (default, May 05 2021, 15:36:36) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import httpx
>>> httpx.get("https://osumarion.osu.edu/search.html?search=%22%3E%3Cachieving+breakthrough+discoveries&id=259").text
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/devloop/.local/share/virtualenvs/test/lib/python3.8/site-packages/httpx/_models.py", line 1315, in text
    self._text = "".join([decoder.decode(self.content), decoder.flush()])
  File "/home/devloop/.local/share/virtualenvs/test/lib/python3.8/site-packages/httpx/_decoders.py", line 285, in decode
    return self.decoder.decode(data)
  File "/usr/lib64/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 27315: invalid continuation byte

httpx 0.18.1
httpcore 0.13.6
os: Linux

What about adding a errors="ignore" parameter in that _models.py file ?

Regards

[tomchristie]

Not sure. From digging into this particular case it looks like the content is actually encoded with Windows-1252, although the Content-Type header incorrectly indicates UTF-8.

(I had to use chardet.detect(r.content) in order to determine what the correct encoding was here.)

From a little bit of digging in, I think requests uses errors="replace" here. On the plus side that means you don't see the exception. On the negative side it masks the fact that the content will be decoded incorrectly.

[devl00p]

I think most of us (writing crawlers) are most interested in the HTML tags than in the content.

Having a valid HTML webpage even with malformed text encoding is still better than having nothing.

Too me, using errors="replace" wouldn't be a serious problem but it is just my own opinion

[tomchristie]

Actually, agreed on this after some discussion on #1657

The proposed change in #1791 switches the decoding style to errors="replace".