Description
Describe the bug
I was going to use configuration with fluentd and in_http plugin behind nginx (where nginx is just a proxy providing security). But faced with this issue in nginx logs while testing:
2020/08/04 13:27:05 [error] 436#436: *214 readv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.122.1, server: localhost, request: "POST /test HTTP/1.1", upstream: "http://172.18.0.2:9880/test", host: "192.168.122.73"
It appears in around 15% cases of transmission. (15% of nginx log entries are the same as mentioned above). But despite this, all events pass through fluentd. I mean: they successfully leave it through output plugin. (tested on tens of events)
Here below is the simplest as possible configuration where the issue appears.
To Reproduce
Touch fluentd (behind nginx) with curl several times:
curl -X POST -d '{"foo":"bar"}' -H 'Content-Type: application/json' http://$nginx_ip/test
Expected behavior
Error entries should not be in nginx logs
Your Environment
- fluentd 1.11.1
- nginx/1.19.1
- Ubuntu 18.04.2 LTS
- 4.15.0-51-generic
- Docker version 19.03.6, build 369ce74a3c
Your Configuration
Fluentd official container: fluentd.conf
<source>
@type http
</source>
<match **>
@type null
</match>
Nginx official container: just edit "location /" in default.conf and do "kill -HUP"
location / {
proxy_pass http://fluentd:9880;
}
Your Error Log
192.168.122.1 - - [04/Aug/2020:13:27:04 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:05 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
2020/08/04 13:27:05 [error] 436#436: *214 readv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.122.1, server: localhost, request: "POST /test HTTP/1.1", upstream: "http://172.18.0.2:9880/test", host: "192.168.122.73"
192.168.122.1 - - [04/Aug/2020:13:27:06 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:07 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:08 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:08 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:09 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
2020/08/04 13:27:10 [error] 436#436: *226 readv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.122.1, server: localhost, request: "POST /test HTTP/1.1", upstream: "http://172.18.0.2:9880/test", host: "192.168.122.73"
192.168.122.1 - - [04/Aug/2020:13:27:10 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:11 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
Additional context
It can be somehow related with #2963