async request / socket ? #16590

pn2new · 2025-01-15T15:36:56Z

pn2new
Jan 15, 2025

I am trying to establish a connection to a server, and would then like to keep the connection open and read the response line by line, in a non-blocking (async) way. The url uses https://.

So far, I tried to use requests, which could work but does not provide an option for line by line reading, and is not async. Looking into the code of request, I think the line by line reading could be added. However, this still is not async.

Is there a library that would allow me to do that without having to re-invent the wheel? What I am trying to achieve is something like this (works on "non-micro" python), but as mentioned everything should be async:

def poll_ntfy_messages():
    url = f"{SERVER}/{TOPIC}/json?since=60m"
    try:
        response = requests.get(url, stream=True)
        for line in response.iter_lines():
            if line:
                print(line)
        response.close()
    except Exception as e:
        print("Error polling messages:", e)

Answered by Carglglz

Jan 16, 2025

I think you need to specify HTTP/1.1 version for the ClientSession e.g. get.py example returns Transfer-Encoding: chunked,

import sys

# ruff: noqa: E402
sys.path.insert(0, ".")
import aiohttp
import asyncio


URL = sys.argv.pop()

if not URL.startswith("http"):
    URL = "http://micropython.org"

print(URL)


async def fetch(client):
    async with client.get(URL) as resp:
        assert resp.status == 200
        return await resp.text()


async def main():
    async with aiohttp.ClientSession(version=aiohttp.HttpVersion11) as client:
        html = await fetch(client)
        print(html)


if __name__ == "__main__":
    asyncio.run(main())

printing headers

 $ micropython examples/get.p…

View full answer

bixb922 · 2025-01-15T17:47:55Z

bixb922
Jan 15, 2025

Have you taken s look at aiohttp in micropython-lib?

4 replies

pn2new Jan 16, 2025
Author

I am trying to get this running with aiohttp. Reading the response in one piece works:

async def async_ntfy():
    url = f"{SERVER}/{TOPIC}/json?since=600m&poll=1"
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            html = await response.text()
            print(html)

asyncio.run(async_ntfy())

However, I do not manage to read the response line-by-line, i.e. in a "chunked" way. On my linux box, with the same server, this works (code in my initial post). So this is not an issue with the server, but the client side.

Here is what I have tried (on my ESP32):

async def async_ntfy():
    url = f"{SERVER}/{TOPIC}/json?since=600m&poll=0"
    async with aiohttp.ClientSession() as session:
        pprint(session)
        async with session.get(url, headers = { "TE": "chunked" }) as response:
            print("Status:", response.status)
            print("Content-type:", response.headers['Content-Type'])
            pprint(response.headers)
            # async for chunk in response.content.???
            #     print(chunk)
            pprint(response)
            html = await response.text()
            print(html)

asyncio.run(async_ntfy())

The output I get is (I added some print statements to the aiohttp/__init__.py file to see what is going on):

## request headers
{'User-Agent': 'compat', 'TE': 'chunked', 'Connection': 'close'}

sline b'HTTP/1.1 200 OK\r\n'
line=b'Server: nginx\r\n'
line=b'Date: Thu, 16 Jan 2025 14:03:38 GMT\r\n'
line=b'Content-Type: application/x-ndjson; charset=utf-8\r\n'
line=b'Connection: close\r\n'
line=b'Access-Control-Allow-Origin: *\r\n'
line=b'\r\n'

chunked: False

Somehow the server does not respond with "Transfer-Encoding: chunked"... But as said, in the "desktop python" code above, this works, with the same server on the other end...

Carglglz Jan 16, 2025

I think you need to specify HTTP/1.1 version for the ClientSession e.g. get.py example returns Transfer-Encoding: chunked,

import sys

# ruff: noqa: E402
sys.path.insert(0, ".")
import aiohttp
import asyncio


URL = sys.argv.pop()

if not URL.startswith("http"):
    URL = "http://micropython.org"

print(URL)


async def fetch(client):
    async with client.get(URL) as resp:
        assert resp.status == 200
        return await resp.text()


async def main():
    async with aiohttp.ClientSession(version=aiohttp.HttpVersion11) as client:
        html = await fetch(client)
        print(html)


if __name__ == "__main__":
    asyncio.run(main())

printing headers

 $ micropython examples/get.py
http://micropython.org
b'GET / HTTP/1.1\r\nUser-Agent: compat\r\nHost: micropython.org\r\nConnection: close\r\n\r\n'
b'Server: nginx/1.10.3\r\n'
b'Date: Thu, 16 Jan 2025 19:01:28 GMT\r\n'
b'Content-Type: text/html; charset=utf-8\r\n'
b'Transfer-Encoding: chunked\r\n'
b'Connection: close\r\n'
b'Vary: Accept-Encoding\r\n'
b'X-Frame-Options: SAMEORIGIN\r\n'
b'Strict-Transport-Security: max-age=15768000\r\n'
b'\r\n'
<!DOCTYPE html>

...

Not sure if you can use Transfer-Encoding in the headers, see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding

Answer selected by pn2new

pn2new Jan 16, 2025
Author

Thanks, adding version=aiohttp.HttpVersion11 indeed was the solution to get a chunked response! I did not see this because the response I got was b'HTTP/1.1 200 OK\r\n'.

Maybe you can also help me with iterating over the result by chunk?
I tried some async for constructs, but these all did not work.
What worked was something like below, but this seems awkward.

async def async_ntfy():
    url = f"{SERVER}/{TOPIC}/json?since=60m"
    async with aiohttp.ClientSession(version=aiohttp.HttpVersion11) as session:
        async with session.get(url) as response:
            # this does not work
            # async for line in response.content:
            #     print(f"line={line}")
            while True:
                line = await response.content.readline() 
                # extend the above by getting length of response, and then response, and then skip '\r\n'
                print(f"line={line}")

Carglglz Jan 17, 2025

Maybe you can also help me with iterating over the result by chunk?

I think your solution is good, as long as the response content is not encoded.
The alternative is to use await response.read(chunk_size) see https://docs.aiohttp.org/en/stable/client_quickstart.html#streaming-response-content

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MicroPython

async request / socket ? #16590

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

MicroPython

async request / socket ? #16590

Uh oh!

pn2new Jan 15, 2025

Replies: 1 comment · 4 replies

Uh oh!

bixb922 Jan 15, 2025

Uh oh!

Uh oh!

pn2new Jan 16, 2025 Author

Uh oh!

Carglglz Jan 16, 2025

Uh oh!

pn2new Jan 16, 2025 Author

Uh oh!

Carglglz Jan 17, 2025

pn2new
Jan 15, 2025

Replies: 1 comment 4 replies

bixb922
Jan 15, 2025

pn2new Jan 16, 2025
Author

pn2new Jan 16, 2025
Author