graphite-api-blueflood-finder container timing out (empty response) #765
Description
Hello Team,
Have a very specific issue here, and was told you may be able to help! For context, I am working on statsd integration with Rackspace Metrics:
https://support.rackspace.com/how-to/integrate-statsd-with-rackspace-metrics/
I'm having no problem with this integration, and am successfully pumping statsd statistics into metrics intake. I can also confirm I can manually pull this data out via 'curl'. However, at this time we don't support visualization of this data through our "intelligence" product, and so the recommended method for visualization is grafana. I was given by the following article by the product manager to accomplish this:
https://github.com/rackerlabs/blueflood/wiki/Grafana-3.0-with-Blueflood-datasource
The second step is to spin up a blueflood-finder+graphite-api container in order to use Rackspace metrics as the back end. It's within this container that I am having the issue:
https://github.com/rackerlabs/docker-graphite-api-blueflood-finder
As previously mentioned, from within the container I can retrieve the metrics information, authenticating with the variables in the container environment:
root@730791559fcc:/# env
RAX_USERNAME=<scrubbed>
HOSTNAME=730791559fcc
GPG_KEY=<scrubbed>
TENANT_ID=<scrubbed>
PYTHON_VERSION=2.7.12
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
BLUEFLOOD_QUERY_URL=https://global.metrics.api.rackspacecloud.com
PWD=/
LANG=C.UTF-8
PYTHON_PIP_VERSION=9.0.1
SHLVL=1
HOME=/root
RAX_APIKEY=<scrubbed>
GRAFANA_URLS=<scrubbed>
_=/usr/bin/env
root@730791559fcc:/# curl -sX GET -H "X-Auth-Token: $API_TOKEN" -H "Content-Type: application/json" "$BLUEFLOOD_QUERY_URL/v2.0/<scrubbed>/metrics/search?query=*" |python -m json.tool
[
{
"metric": "statsd.bad_lines_seen"
},
{
"metric": "sample.gauge"
},
{
"metric": "statsd.packets_received"
},
{
"metric": "statsd.timestamp_lag"
},
{
"metric": "statsd.metrics_received"
}
]
However, when querying the graphite-api it times out with the following:
root@730791559fcc:/# netstat -pntl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 30/python2
root@730791559fcc:/# curl -vv "http://localhost:8888/metrics/find?query=*"
* Hostname was NOT found in DNS cache
* Trying ::1...
* connect to ::1 port 8888 failed: Connection refused
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8888 (#0)
> GET /metrics/find?query=* HTTP/1.1
> User-Agent: curl/7.38.0
> Host: localhost:8888
> Accept: */*
>
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
I've enabled debug logging within the app, and captured the following during the request:
root@730791559fcc:/# tail -fn0 /var/log/blueflood-graphite-finder.log
2016-12-12 23:43:04 DEBUG blueflood_finder TenantBluefloodFinder.query: *
2016-12-12 23:43:04 INFO blueflood_finder BluefloodClient.find_nodes_from_bf: <FindQuery: * from * until *>
2016-12-12 23:43:36 DEBUG blueflood_finder Blueflood Finder v31
2016-12-12 23:43:36 DEBUG blueflood_finder BF enum thread started:
2016-12-12 23:43:36 DEBUG blueflood_finder BF finder submetrics enabled: False
gunicorn only logs the following, with nothing in the access log:
root@730791559fcc:/# tail /var/log/gunicorn-error.log
[2016-12-12 23:31:31 +0000] [30] [CRITICAL] WORKER TIMEOUT (pid:88)
[2016-12-12 23:31:31 +0000] [94] [INFO] Booting worker with pid: 94
Please let me know if there is any other information I can provide, and thank you in advance for your help!