Skip to content

test_post[http] fails in some systems #905

New issue
New issue
Open
Open
test_post[http] fails in some systems#905
@sanvila

Description

@sanvila
sanvila
opened on Jan 19, 2025

Hello. As reported here:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1087334

while building the Debian package for vcr.py version 7.0.0 on AWS virtual machines of type m7a.medium or r7a.medium, I get the following error:

E BrokenPipeError: [Errno 32] Broken pipe

Complete error message here:

vcr.py.txt

Incidentally, those AWS instances have 1 CPU, but we don't know if that's relevant or not (it may be the case that it fails because they are "too slow").

An error like this one started to happen simultaneously on packages vcr.py and pytest-httpbin, so we suspected of the common build-dependency python3-urllib3.

Now pytest-httpbin does not fail anymore but vcr.py still fails, so maybe there is indeed a problem in vcr.py.

Thanks.

Activity

cjwatson

cjwatson commented on Jan 28, 2025

@cjwatson
cjwatson
on Jan 28, 2025

I spent some time investigating this today. It's possible I'm missing something, but this seems to be a complex interaction of several different problems.

I'm seeing failures in tests/integration/test_urllib3.py::test_post (which has been around for a while) and tests/unit/test_stubs.py::TestVCRConnection::test_body_consumed_once_* (which were introduced in #851). In all cases, the error is essentially BrokenPipeError because the httpbin test server is closing the connection before the test has finished sending the request body. (This is likely to be somewhat racy, which is why we're only seeing this in certain circumstances.)

strace reveals that httpbin is closing the connection with a response along these lines (I've decoded it from the raw sendto calls for readability):

HTTP/1.1 501 NOT IMPLEMENTED
Date: Tue, 28 Jan 2025 14:08:45 GMT
Server: Pytest-HTTPBIN/0.1.0
Content-Type: text/html; charset=utf-8
Content-Length: 159
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Connection: Close

<!doctype html>
<html lang=en>
<title>501 Not Implemented</title>
<h1>Not Implemented</h1>
<p>Chunked requests are not supported for server WSGIServer/0.2</p>

The relevant code is httpbin.core.before_request, which denies chunked requests unless it's running under gunicorn, to work around various bugs. It's thus hard to see how VCR.py can test chunked transfer requests using pytest-httpbin without some work elsewhere.

I tried hacking httpbin.core.before_request locally to run request.environ["wsgi.input_terminated"] = 1 instead of aborting, and the tests hung. I guess that's a relative of the flask problem mentioned in that comment in httpbin, but I wasn't able to track down the details.

While debugging this, I also found that the test_body_consumed_once_* unit tests are broken in a different way: because they don't send a Content-Type header, werkzeug doesn't bother to read the request body (I think). The simplest change there is probably to add headers={"Content-Type": "application/x-www-form-urlencoded"} to the request calls in _test_body_consumed_once and change the body iterators in the test methods from 1234567890 to a=1234567890 (and similar). I wasn't able to fully test this fix due to the problems above, though.

I don't know how to get much further here, because trying to test chunked requests using pytest-httpbin just seems kind of doomed at the moment. I also don't think this is really a new issue: kevin1024/pytest-httpbin#33 and (probably) kevin1024/pytest-httpbin#59 seem to be more or less the same thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions
      test_post[http] fails in some systems · Issue #905 · kevin1024/vcrpy