nginx and uwsgi deployment

[bllacksheep]

Hi there,

I'm wondering if the process of serving the app via nginx/uwsgi has been documented anywhere? I know there's a quick start run through on httpd. I've been trying for a few days on and off and I can't even seem to get it to serve directly from the uwsgi server. Via nginx I get 502 socket file cannot be created. And via uwsgi I just get internal server error.

It's likely I'm doing something dumb but is the app set up to just plugin the uwsgi.ini into the server as long as it's pointing to the .wsgi file and the projectdir? It's a flask based app correct? I've run through a bit of the documentation and I should be able to serve it at a bear minimum with uwsgi.

I can run it fine with the ceph-dash.py file so all dependencies are met and the likes.

Any help or tuts you can direct me to that apply to the structure of this app and how it should be served sort of like a step by step so's I know I'm not fecking it up.

Trying to start with uwsgi:

uwsgi --http :5000 --chdir /etc/nginx/sites-enabled/ceph-dash/ --wsgi-file /etc/nginx/sites-enabled/ceph-dash/cephdash.wsgi --callable application --master --uid nginx --gid nginx

Server starts:

As you can see I added print statements to sys.path and application within the .wsgi to check it was loading and it finds what it needs apparently. But mountpoint is still empty.

*** Operational MODE: single process *** ['/etc/nginx/sites-enabled/ceph-dash/app/', '.', '', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib/python2.7/site-packages'] <Flask 'app'> WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0xe0b6d0 pid: 26487 (default app) *** uWSGI is running in multiple interpreter mode *** spawned uWSGI worker 1 (and the only) (pid: 26487, cores: 1)

Request gives internal server error:

[pid: 26487|app: 0|req: 1/1] 10.0.2.2 () {36 vars in 614 bytes} [Sun Aug 28 11:07:06 2016] GET / => generated 291 bytes in 49 msecs (HTTP/1.1 500) 2 headers in 84 bytes (1 switches on core 0)

Also tried with the .ini file:

uwsgi --ini uwsgi.ini

chdir           = /etc/nginx/sites-enabled/ceph-dash
wsgi-file       = /etc/nginx/sites-enabled/ceph-dash/cephdash.wsgi
enable-threads  = true
master          = true
processes       = 4
#socket          = /var/run/ceph-dash.socket
http        = :5000
#chmod-socket    = 664
vacuum          = true
uid             = nginx
gid             = nginx
daemonize       = /var/log/uwsgi/ceph-dash.log

What am I doing wrong? Pretty sure it's going to come down to either wrong file in wrong place or uwsgi not pointing to the correct place.

Cheers

[Crapworks]

Hi @Lighiche

to be honest, I was never running ceph-dash with nginx, only with apache. So I can't really point in the right direction. The error message also doesn't tell me much about what could be wrong.

To answer your question: Yes, it's a flask based app and should work with any wsgi capable webserver.

Are there any other logfiles or error messae that could hint on the problem?
Were you using the wsgi file provided under ceph-dash/contrib/wsgi/cephdash.wsgi? Because in the ini file you point to ceph-dash/cephdash.wsgi.

If you've copied the file from the contrib folder from to the toplevel dir, you need to change the line.

sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))

to

sys.path.insert(0, os.path.join(os.path.dirname(__file__), '.'))

Regards,
Christian

[ac-obair]

Hi there, thanks for the reply. This is absolutely everything I have and everything I know to try.

I copied files out of the contrib folder to:

/opt/ceph-dash/

1. uwsgi.ini
   • chdir = /etc/nginx/sites-enabled/ceph-dash/
   • wsgi-file = /etc/nginx/sites-enabled/ceph-dash/contrib/wsgi/cephdash.wsgi
2. uwsgi.params

I've changed the lines slightly to reflect that I had moved the .wsgi config file up to a higher level I've since moved it back to contrib/wsgi/cephdash.wsgi and it's the same issue.

I have the cloned dir in this folder /etc/nginx/sites-enabled/ceph-dash/

The only other error log I get is when I place nginx infront of uwsgi, I blanked out the hostname not that it really matters.

curl localhost:5000
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.6.3</center>
</body>
</html>

2016/08/29 09:58:17 [crit] 29117#0: *3 connect() to unix:///var/run/ceph-dash.sock failed (2: No such file or directory) while connecting to upstream, client: 127.0.0.1, server: <hostname>, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///var/run/ceph-dash.sock:", host: "localhost:5000"

I already have nginx on these boxes serving something for me and there's a whole HA load balancing thing going on a step above on the network so I'd prefer not to then complicate it more with apache ya know. If I write a small helloworld flask app, I can serve that via flask and uwsgi

with flask:

• Running on http://0.0.0.0:5000/
• Restarting with reloader
  127.0.0.1 - - [29/Aug/2016 10:13:30] "GET / HTTP/1.1" 200 -

with uwsgi:

uwsgi --http :5000 --wsgi-file hellworld.py --callable app --processes 4 --threads 2 --stats 127.0.0.1:9191

[1] 29315
# *** Starting uWSGI 2.0.13.1 (64bit) on [Mon Aug 29 10:18:48 2016] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-4) on 29 August 2016 09:55:26
os: Linux-3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016
nodename: prdceph-mon00
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /opt/ceph-dash
detected binary path: /usr/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your processes number limit is 7282
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :5000 fd 4
uwsgi socket 0 bound to TCP address 127.0.0.1:44674 (port auto-assigned) fd 3
Python version: 2.7.5 (default, Aug 18 2016, 15:58:25)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
Python main interpreter initialized at 0x187cfb0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 415280 bytes (405 KB) for 8 cores
*** Operational MODE: preforking+threaded ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x187cfb0 pid: 29315 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 29315)
spawned uWSGI worker 1 (pid: 29320, cores: 2)
spawned uWSGI worker 2 (pid: 29321, cores: 2)
spawned uWSGI worker 3 (pid: 29323, cores: 2)
spawned uWSGI worker 4 (pid: 29324, cores: 2)
*** Stats server enabled on 127.0.0.1:9191 fd: 18 ***
spawned uWSGI http 1 (pid: 29326)

curl localhost:5000
[pid: 29324|app: 0|req: 1/1] 127.0.0.1 () {28 vars in 295 bytes} [Mon Aug 29 10:18:56 2016] GET / => generated 11 bytes in 3 msecs (HTTP/1.1 200) 2 headers in 79 bytes (1 switches on core 0)

I've even

tried using --http-socket in uwsgi and changing the upsteam in nginx to be a listening port

cephdash.conf

upstream uwsgi {
    #server unix:///var/run/ceph-dash.sock;
     server 0.0.0.0:3000;
}

uwsgi.ini
http-socket = :3000

nginx error.log
2016/08/29 10:39:25 [error] 29489#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: prdceph-mon00, request: "GET / HTTP/1.1", upstream: "uwsgi://0.0.0.0:3000", host: "localhost:5000"

Response is still 502 bad gateway. At this point it's obvious to me that uwsgi can't serve the app I just can't figure out why.

So

For clarity AS ROOT starting with uwsgi --ini uwsgi.ini

[uwsgi]

# ceph-dash-related settings
# the base directory (full path)
chdir           = /etc/nginx/sites-enabled/ceph-dash/
# ceph-dash's wsgi file
wsgi-file       = /etc/nginx/sites-enabled/ceph-dash/contrib/wsgi/cephdash.wsgi
enable-threads  = true
# process-related settings
# master
master          = true
# maximum number of worker processes
processes       = 4
# the socket (use the full path to be safe
#socket          = /var/run/ceph-dash.sock
http-socket    = 0.0.0.0:3000
# with appropriate permissions
chmod-socket    = 664
# clear environment on exit
vacuum          = true
uid             = nginx
gid             = nginx
# Run in the background as a daemon
daemonize       = /var/log/uwsgi/ceph-dash.log

On port 3000 I get

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>

And the only log output for this is:

*** Starting uWSGI 2.0.13.1 (64bit) on [Mon Aug 29 10:48:49 2016] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-4) on 29 August 2016 09:55:26
os: Linux-3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016
nodename: prdceph-mon00
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /opt/ceph-dash
detected binary path: /usr/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
setgid() to 994
setuid() to 996
chdir() to /etc/nginx/sites-enabled/ceph-dash/
your processes number limit is 7282
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 0.0.0.0:3000 fd 3
Python version: 2.7.5 (default, Aug 18 2016, 15:58:25)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
Python main interpreter initialized at 0x2556c70
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 363800 bytes (355 KB) for 4 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x2556c70 pid: 29533 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 29533)
spawned uWSGI worker 1 (pid: 29538, cores: 1)
spawned uWSGI worker 2 (pid: 29539, cores: 1)
spawned uWSGI worker 3 (pid: 29540, cores: 1)
spawned uWSGI worker 4 (pid: 29541, cores: 1)
[pid: 29541|app: 0|req: 1/1] 127.0.0.1 () {24 vars in 255 bytes} [Mon Aug 29 10:49:06 2016] GET / => generated 291 bytes in 98 msecs (HTTP/1.1 500) 2 headers in 84 bytes (1 switches on core 0)

[bllacksheep]

Running wsgi with nginx for dummies

So ever wanted to know how to fix all the problems?

FIX FOR 500 RESPONSE CODE

Nginx is NOT starting any wsgi processes for you, I had to start this server manually. In order to get this to work from uwsgi the full path MUST be given to .wsgi file.

All files owned by nginx in /etc/nginx/sites-enabled

WORKING

---

uwsgi server doing the http business

uwsgi --socket 0.0.0.0:5000 --protocol=http --wsgi-file /etc/nginx/sites-enabled/ceph-dash/contrib/wsgi/cephdash.wsgi

WORKING

---

uwsgi --ini uwsgi.ini

[uwsgi]
chdir           = /etc/nginx/sites-enabled/ceph-dash
wsgi-file       = /etc/nginx/sites-enabled/ceph-dash/contrib/wsgi/cephdash.wsgi
enable-threads  = true
master          = true
processes       = 4
http-socket     = 0.0.0.0:5000      # uwsgi server doing the http business
vacuum          = true
uid             = nginx
gid             = nginx
daemonize       = /tmp/ceph-dash.log

FIX FOR BAD GATEWAY 502 Bad Gateway (nginx can't find uwsgi server)

start the uwsgi server and make sure the next fix 503 is in place

FIX FOR BAD GATEWAY 503 (uwsgi server reachable but not able to process request)

The point of wsgi protocol (natively supported by nginx) is that it can proxy_pass requests off to the uwsgi server, ** in a wsgi binary format ** the http is handled by the listening nginx port 5000.

WORKING

---

uwsgi --ini uwsgi.ini

[uwsgi]
chdir           = /etc/nginx/sites-enabled/ceph-dash
wsgi-file       = /etc/nginx/sites-enabled/ceph-dash/contrib/wsgi/cephdash.wsgi
enable-threads  = true
master          = true
processes       = 4
socket          = :3000             # curl: (52) Empty reply from server, NOT handling http business
vacuum          = true
uid             = nginx
gid             = nginx
daemonize       = /tmp/ceph-dash.log

Now start nginx with this config

upstream uwsgi {
    server 127.0.0.1:3000;    # native wsgi protocol being passed here 
}

server {
    listen 5000;              # handling all the http business
    server_name what.ever.com 
    charset     utf-8;
    location / {
        uwsgi_pass  uwsgi;
        include     /etc/nginx/sites-enabled/ceph-dash/contrib/nginx/uwsgi_params;
    }
}

The response for this is successful

curl -I localhost:5000
HTTP/1.1 200 OK
Server: nginx/1.6.3
Date: Wed, 31 Aug 2016 13:03:07 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 7794
Connection: keep-alive

FIX FOR USING .SOCK FILE

If uwsgi can't create the socket the server won't start. You can quickly find out if it started with pgrep uwsgi. If it returns nothing then it's not running you can also check manually if the socket created.

NOTES

• uwsgi could not create a socket in /var/run because of the user I was starting it with
• uwsgi could create a socket with socket = /tmp/ceph-dash.sock however programs starting a socket in /tmp can only be seen by the program that created it. So nginx can't connect to it.
• I created a dir /var/run/uwsgi and chown'd it with the user I was running uwsgi with i.e. cephuser. Now uwsgi could create it but nginx still couldn't access it to pass to because it now does not have permission. To get around this I just set the chmod-socket permissions to 777 and restarted uwsgi, it worked.

It's not ideal and there's probably a better way to either host that server with the same user running wsgi or have a group with shared membership access.

WORKING

---

[uwsgi]
chdir           = /etc/nginx/sites-enabled/ceph-dash
wsgi-file       = /etc/nginx/sites-enabled/ceph-dash/contrib/wsgi/cephdash.wsgi
enable-threads  = true
master          = true
processes       = 4
socket          = /var/run/uwsgi/ceph-dash.socket
chmod-socket    = 777
vacuum          = true
uid             = nginx
gid             = nginx
#daemonize       = /var/log/uwsgi/ceph-dash.log
daemonize       = /tmp/ceph-dash.log

And comfirmation:

/var/run/uwsgi/
srw-rw-r--. 1 cephuser cephuser     0 Aug 31 14:11 ceph-dash.sock

Finally change the nginx upsteam block and restart nginx and voilà, pay dirt.

upstream uwsgi {
    server unix:////var/run/uwsgi/ceph-dash.sock;
}

[Crapworks]

Hi @Lighiche

thank you for the detailed solution! Is this problem fixed for you and can I close this issue? I would also like to create a FAQ section in the README and link to this issue so that people who have the same problems can easily find it. Ok for you?

Regards,
Christian

[bllacksheep]

This is fine can I have a link to it? Also sorry for late reply I didn’t get this.

…

On 7 Jan 2017, at 17:31, Christian Eichelmann ***@***.***> wrote: Hi @Lighiche <https://github.com/Lighiche> thank you for the detailed solution! Is this problem fixed for you and can I close this issue? I would also like to create a FAQ section in the README and link to this issue so that people who have the same problems can easily find it. Ok for you? Regards, Christian — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHTweOyvG5yFNWGuHaq9wKycfCXmUeumks5rP8wAgaJpZM4Ju5CS>.

[Crapworks]

Hi,

thanks!
https://github.com/Crapworks/ceph-dash#user-content-faq

Regards,
Christian