urequests: Extra response headers at the end of res.text with Pico W

[canuckdev]

Using latest urequests with Pico W and finding that extra response headers are being appended to the res.text attribute which results in the data not being able to be parsed as JSON.

import urequests as requests
import config
import putils

postHeaders = { 'Content-Type': 'application/json', 'Accept': 'application/json' }
alsaHost = 'pi9'
method = 'POST'
url = 'http://' + alsaHost + ':1780/jsonrpc'
postData = '{"id":0,"jsonrpc":"2.0","method":"Server.GetStatus"}'

try:
    wlan = putils.wlan_connect(config.ssid, config.password)
    res = requests.request(method, url, data=postData, headers=postHeaders)
    if res.status_code != 200:
        print('{} {} {}'.format(res.status_code, method, url))
    else:
        print(res.text)
except OSError as e:
    print('OSError in reqUrl {} {}'.format(method, url))

The result:

{"id":0,"jsonrpc":"2.0","result":{"server":{"groups":[{"clients":[{"config":{"instance":1,"latency":0,"name":"","volume":{"muted":false,"percent":33}},"connected":true,"host":{"arch":"aarch64","ip":"192.168.1.126","mac":"b8:27:eb:99:b6:1e","name":"pi4","os":"Debian GNU/Linux 12 (bookworm)"},"id":"b8:27:eb:99:b6:1e","lastSeen":{"sec":1699725481,"usec":49965},"snapclient":{"name":"Snapclient","protocolVersion":2,"version":"0.26.0"}},{"config":{"instance":1,"latency":0,"name":"","volume":{"muted":false,"percent":33}},"connected":true,"host":{"arch":"aarch64","ip":"192.168.1.144","mac":"e4:5f:01:f5:48:88","name":"pi8","os":"Debian GNU/Linux 12 (bookworm)"},"id":"e4:5f:01:f5:48:88","lastSeen":{"sec":1699725480,"usec":546206},"snapclient":{"name":"Snapclient","protocolVersion":2,"version":"0.26.0"}}],"id":"77eaeda3-1297-437d-e8cf-d492a07bf103","muted":false,"name":"","stream_id":"default"}],"server":{"host":{"arch":"aarch64","ip":"","mac":"","name":"pi9","os":"Debian GNU/Linux 12 (bookworm)"},"snapserver":{"controlProtocolVersion":1,"name":"Snapserver","protocolVersion":1,"version":"0.26.0"}},"streams":[{"id":"default","properties":{"canControl":false,"canGoNext":false,"canGoPrevious":false,"canPause":false,"canPlay":false,"canSeek":false},"status":"playing","uri":{"fragment":"","host":"","path":"/tmp/snapfifo","query":{"chunk_ms":"20","codec":"flac","name":"default","sampleformat":"48000:16:2"},"raw":"pipe:////tmp/snapfifo?chunk_ms=20&codec=flac&name=default&sampleformat=48000:16:2","scheme":"pipe"}}]}}}HTTP/1.0 200 OK
Server: Snapcast
Content-Type: application/json
Content-Length: 0

Looking at the output there are 4 response headers appended to the JSON response:

HTTP/1.0 200 OK
Server: Snapcast
Content-Type: application/json
Content-Length: 0

Running the equivalent code in python3 gives the expected response of only the JSON output in the res.text attribute.

canuckdev

[projectgus]

@canuckdev Thanks for being patient while someone got back to you about this.

It looks like you're talking to a SnapCast JSON HTTP endpoint, is that right? It seems either its embedded webserver is not sending a valid HTTP/1.0 response or MicroPython requests isn't parsing the response properly. However it's hard to tell which without a SnapCast server to query for a sample response.

If you're able to, could you please build up your POST request via a curl command and send it with the -v --http1.0 options appended? The verbose output might show enough to figure out what part of the response is confusing MicroPython requests module.

[canuckdev]

Thanks for getting back to me. Here is zip which includes a script for the POST request with stdout in test.out and stderr in test.err

canuckdev
test.zip

[projectgus]

Thanks for that, @canuckdev. I am unfortunately none the wiser, as this HTTP response looks OK:

*   Trying 192.168.1.37:1780...
* Connected to pi9 (192.168.1.37) port 1780 (#0)
> POST /jsonrpc HTTP/1.0
> Host: pi9:1780
> User-Agent: curl/7.81.0
> Content-Type: application/json
> Accept: application/json
> Content-Length: 52
> 
} [52 bytes data]
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: Snapcast
< Content-Type: application/json
< Content-Length: 1525
< 
{ [1525 bytes data]
* Closing connection 0

The first thing the requests module does after sending the request is parse out the status line (HTTP/1.0 200 OK), and then it's discarded so I don't see how it can be printed after the body:
https://github.com/micropython/micropython-lib/blob/master/python-ecosys/requests/requests/__init__.py#L146

Of course, maybe requests is sending a malformed POST compared to the curl. I don't suppose you're in a position to capture the network packets from a faulty request/response? (Can do this on the Pi end with something like sudo tcpdump -i any -w requests.pcap tcp port 1780, then perform the request and Ctrl-C to exit.)

Also, just to double check, did you install urequests by mpremote mip install urequests or via some other method?

(In recent versions urequests is a minimal wrapper around requests.)