couchdb failed to start

[tpatzig]

We need to analyze, why couchdb was dead, thus chef-server not working.
In the messages logs we found that:

2017-05-29T07:31:59.317993+00:00 admin systemd[1]: Stopping LSB: start and stop CouchDB database server...
2017-05-29T07:31:59.338323+00:00 admin sudo:     root : TTY=unknown ; PWD=/ ; USER=couchdb ; COMMAND=/usr/bin/couchdb -d
2017-05-29T07:31:59.339378+00:00 admin sudo: pam_unix(sudo:session): session opened for user couchdb by (uid=0)
2017-05-29T07:31:59.343429+00:00 admin systemd[1]: Starting user-494.slice.
2017-05-29T07:31:59.343752+00:00 admin systemd[1]: Created slice user-494.slice.
2017-05-29T07:31:59.345068+00:00 admin systemd[1]: Starting User Manager for UID 494...
2017-05-29T07:31:59.347081+00:00 admin systemd[1]: Starting Session c509078 of user couchdb.
2017-05-29T07:31:59.347526+00:00 admin systemd-logind[5588]: New session c509078 of user couchdb.
2017-05-29T07:31:59.347761+00:00 admin systemd[1]: Started Session c509078 of user couchdb.
2017-05-29T07:31:59.385396+00:00 admin systemd: pam_access(systemd-user:account): access denied for user `couchdb' from `systemd-user'
2017-05-29T07:31:59.385713+00:00 admin systemd: pam_sss(systemd-user:account): Access denied for user couchdb: 10 (User not known to the underlying           authentication module)
2017-05-29T07:31:59.386047+00:00 admin systemd[13071]: Failed at step PAM spawning /usr/lib/systemd/systemd: Operation not permitted
2017-05-29T07:31:59.386850+00:00 admin systemd[1]: Started User Manager for UID 494.
2017-05-29T07:31:59.420540+00:00 admin chef-server[17198]: [2017-05-29T07:31:59+00:00] ERROR: Connection refused connecting to localhost:5984 for /chef/      _design/id_map/_view/name_to_id, retry 1/5
2017-05-29T07:32:00.397468+00:00 admin sudo: pam_unix(sudo:session): session closed for user couchdb
2017-05-29T07:32:00.399711+00:00 admin couchdb[13062]: Stopping CouchDB..done
2017-05-29T07:32:00.401305+00:00 admin systemd-logind[5588]: Removed session c509078.
2017-05-29T07:32:00.403345+00:00 admin systemd[1]: Stopped User Manager for UID 494.
2017-05-29T07:32:00.403629+00:00 admin systemd[1]: Stopping user-494.slice.
2017-05-29T07:32:00.403844+00:00 admin systemd[1]: Removed slice user-494.slice.
2017-05-29T07:32:00.404043+00:00 admin systemd[1]: Starting LSB: start and stop CouchDB database server...
2017-05-29T07:32:00.427714+00:00 admin couchdb[13099]: Starting CouchDB..done
2017-05-29T07:32:00.445208+00:00 admin sudo:     root : TTY=unknown ; PWD=/ ; USER=couchdb ; COMMAND=/usr/bin/couchdb -d
2017-05-29T07:32:00.445933+00:00 admin sudo: pam_unix(sudo:session): session opened for user couchdb by (uid=0)
2017-05-29T07:32:00.447407+00:00 admin systemd[1]: Starting user-494.slice.
2017-05-29T07:32:00.447672+00:00 admin systemd[1]: Created slice user-494.slice.
2017-05-29T07:32:00.448808+00:00 admin systemd[1]: Starting User Manager for UID 494...
2017-05-29T07:32:00.450634+00:00 admin systemd[1]: Starting Session c509079 of user couchdb.
2017-05-29T07:32:00.450896+00:00 admin systemd-logind[5588]: New session c509079 of user couchdb.
2017-05-29T07:32:00.451133+00:00 admin systemd[1]: Started Session c509079 of user couchdb.
2017-05-29T07:32:00.452302+00:00 admin systemd: pam_access(systemd-user:account): access denied for user `couchdb' from `systemd-user'
2017-05-29T07:32:00.452579+00:00 admin systemd: pam_sss(systemd-user:account): Access denied for user couchdb: 10 (User not known to the underlying           authentication module)
2017-05-29T07:32:00.452852+00:00 admin systemd[13126]: Failed at step PAM spawning /usr/lib/systemd/systemd: Operation not permitted
2017-05-29T07:32:00.453427+00:00 admin systemd[1]: Started User Manager for UID 494.
2017-05-29T07:32:00.462583+00:00 admin sudo: pam_unix(sudo:session): session closed for user couchdb
2017-05-29T07:32:00.464347+00:00 admin couchdb[13117]: Stopping CouchDB..done
2017-05-29T07:32:00.464844+00:00 admin systemd[1]: Started LSB: start and stop CouchDB database server.
2017-05-29T07:32:00.466260+00:00 admin systemd-logind[5588]: Removed session c509079.
2017-05-29T07:32:00.468286+00:00 admin systemd[1]: Stopped User Manager for UID 494.
2017-05-29T07:32:00.468522+00:00 admin systemd[1]: Stopping user-494.slice.
2017-05-29T07:32:00.468727+00:00 admin systemd[1]: Removed slice user-494.slice.
2017-05-29T07:32:04.421235+00:00 admin chef-server[17198]: [2017-05-29T07:32:04+00:00] ERROR: Connection refused connecting to localhost:5984 for /chef/      _design/id_map/_view/name_to_id, retry 2/5

[matelakat]

@vuntz mentioned it might be related to logrotate.