Skip to content

Preventing or Cleaning Up Dead HTTP Processes #618

@davefilip

Description

@davefilip

Rene,

I'm looking for advice how how to clean up old, dead HTTP daemon tasks.

I have, in particular, one RPi 3B -- 2.4 GHz only -- that will start accumulating "dead" HTTP requests, e.g.:

15 02036240 block    httpd@2036240
16 01E39580 block    httpd@1e39580

This usually happens after I see this in the system log:

Oct 25 09:06:07.70 httpd: Receive failed

I've found in CHTTPDaemon::ParseRequest() where this happens, which returns HTTPUnknownError back to CHTTPDemon::Worker(), so as a "brute force" I tried to do a CTask::Terminate() after decrementing s_nInstanceCount in CHTTPDaemon::Worker() when it sees this error, but that caused more problems (I was hoping it would work its way up the stack, and clean things up). I won't waste your time as to why that didn't work, since it seems like a brute force / dirty solution.

Nonetheless, if this were Linux (or any other Unix flavor), since I have the address of the process, I would simply 'kill -9' that process. However, as you and I have discussed in the past, Circle will only terminate a Task "gently" by itself, as only the running task can call CTask::Terminate().

I should note that sometimes when I see 'httpd: Receive failed' in the log, it appears to clean itself up and that the https task will terminate on its own, sometimes after several seconds. Other times, it will not. So I suspect it has to do with something stalling in the Worker when it realizes that there is a problem (perhaps hanging on the delete m_pSocket, just a guess?).

So you could argue that network problems should be resolved first, and I believe I have made some progress by turning off Bluetooth on about 10 Raspberry Pis, since it shares the same frequency (and RPi 3Bs only support 2.4 GHz WiFi). But I would also argue that bad client requests and/or network errors should never bring down a server. Which is what happens when I have ten (10) of these, and then I have to reboot in order to get rid of the dead httpd Tasks.

Based on the logs, I estimate that about 1.2% of the requests are becoming dead, whereas 98.8% are successful.

[Yes, I could also increase the worker count and the process count, but that only prolongs the problem without solving it.]

So let me know if you have any advice on where and how I can clean up these dead https processes, or else prevent them from getting "stuck" in a block state to begin with.

Any feedback or advice will be appreciated, and I will accept "Simply put, can't do that in Circle", if that is the correct answer.

Cheers,

Dave.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions