Response content is omitted if it is erroneous

[sardok]

Long story short

Some http response content could be broken, in this case caller should be able to allow access to raw response content.

Expected behaviour

Disable the gzip decompression on demand and return the raw data.

Actual behaviour

Changes due to set env variable, see below.

Steps to reproduce

import aiohttp
import asyncio
run = asyncio.get_event_loop().run_until_complete
resp = run(aiohttp.request('get', 'http://www.rofu.de'))
#FlowControlStreamReader protocol  <aiohttp.client_proto.ResponseHandler object at 0x7f3155d75fd0>
run(resp.read())
#b''

when AIOHTTP_NO_EXTENSIONS=1

import aiohttp
import asyncio
run = asyncio.get_event_loop().run_until_complete
resp = run(aiohttp.request('get', 'http://www.rofu.de'))
#FlowControlStreamReader protocol  <aiohttp.client_proto.ResponseHandler object at 0x7f41ad5fc6a0>
len(run(resp.read()))  # blocks and raises TimeoutError 

Your environment

python 3.5.2
aiohttp 2.0.7
ubuntu 16.04

[fafhrd91]

this would hard to do with default parsers. you may try to use custom class

[asvetlov]

Hmm, example works on master but with AIOHTTP_NO_EXTENSIONS=1 only. Cythonized version just prints nothing.
@fafhrd91 would you take a look?

[fafhrd91]

It is probably because buffering happen in http parser. I do t see any way how we can implement this properly. We would need buffer each response in memory just in case if of error. We may introduce something like compression=true, that is it, no switching to raw mode on error.

[asvetlov]

Sorry, I don't follow what compression=true should do.

[fafhrd91]

that is for raw content access. we can not return raw content on error. developer has to instruct aiohttp with something like compression=true or even raw_content=true, then we can skip all chunking and compression decoding

[asvetlov]

Make sense

[shturman]

Just found one more example with similar behavior:
https://www.howweelearn.com/ads.txt
non empty response if AIOHTTP_NO_EXTENSIONS=1 and empty with HttpResponseParserC