Skip to content

Uncaught socket exception during timeout handling.Β #210

Open
@maximelb

Description

@maximelb

❓ I'm submitting a ...

  • 🐞 bug report
  • 🐣 feature request
  • ❓ question about the decisions made in the repository

🐞 Describe the bug. What is the current behavior?
If a client goes away while a request timeout is being processed, the socket.error exception produced in /cheroot/server.py _conditional_error() bubbles up uncaught all the way to the thread worker. This generates a stack trace regardless of logging config.

❓ What is the motivation / use case for changing the behavior?
This is producing STDERR errors in our container log, preventing us from handling errors properly. I also suspect given enough occurrence it may lead to thread pool starvation although I have not experimented with it.

πŸ’‘ To Reproduce
This is hard to reproduce, I suspect timing is heavily involved. We are seeing it occur in our environment when requests timeout. These requests are coming from a load balancer.

πŸ’‘ Expected behavior
The exception should be caught and treated as a simple failure and simply trigger completion and cleanup of the client's request (the client simply has gone away).

πŸ“‹ Details
Here is the traceback we're seeing:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/site-packages/cheroot/workers/threadpool.py", line 114, in run
    conn.communicate()
  File "/usr/local/lib/python2.7/site-packages/cheroot/server.py", line 1286, in communicate
    self._conditional_error(req, '408 Request Timeout')
  File "/usr/local/lib/python2.7/site-packages/cheroot/server.py", line 1334, in _conditional_error
    req.simple_response(response)
  File "/usr/local/lib/python2.7/site-packages/cheroot/server.py", line 1115, in simple_response
    self.conn.wfile.write(EMPTY.join(buf))
  File "/usr/local/lib/python2.7/site-packages/cheroot/makefile.py", line 78, in write
    data_mv[bytes_sent:bytes_sent + SOCK_WRITE_BLOCKSIZE],
  File "/usr/local/lib/python2.7/site-packages/cheroot/ssl/pyopenssl.py", line 144, in send
    *args, **kwargs
  File "/usr/local/lib/python2.7/site-packages/cheroot/ssl/pyopenssl.py", line 91, in _safe_call
    raise socket.error(errnum)
error: -1

πŸ“‹ Environment

  • PyOpenSSL version: 19.0.0
  • Cheroot version: 6.5.4
  • CherryPy version: 17.4.2
  • Python version: 2.7.13
  • OS: Linux Alpine

πŸ“‹ Additional context
I believe a simple try/except/pass for the socket.error in the context of https://github.com/cherrypy/cheroot/blob/v6.5.5/cheroot/server.py#L1276 should fix the issue without any negative side-effects since the server is already dealing with a client that has gone away.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is brokenhelp wantedSomebody help us, please!reproducer: presentThis PR or issue contains code, which reproduce the problem described or clearly understandable STR

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions