Failures on RHEL 7

[redhatromero]

Stuck with rabbitmq failing....

Before I get into the details, a couple of things I needed to do on a fresh Centos 7 install:

1. I had only done a base install, but "private-chef-ctl reconfigure" failed until I installed httpd, then ran a system update and rebooted.
2. I was then getting a hanging behavior here:

* link[/opt/opscode/init/couchdb] action create
    - create symlink at /opt/opscode/init/couchdb to /opt/opscode/embedded/bin/sv
    - configure service runit_service[couchdb]
* link[/opt/opscode/service/couchdb] action create
    - create symlink at /opt/opscode/service/couchdb to /opt/opscode/sv/couchdb

1. After some digging, it became apparent that I was missing the "erlang" package, which was not available at the CentOS 7 repository. I added the EPEL 7 repo file, then installed "erlang".... that seemed to get me past this portion.
2. Now I'm stuck here:

Recipe: private-chef::rabbitmq
  * runit_service[rabbitmq] action enableRecipe: <Dynamically Defined Resource>
  * directory[/opt/opscode/sv/rabbitmq] action create (up to date)
  * template[/opt/opscode/sv/rabbitmq/run] action create (up to date)
  * directory[/opt/opscode/sv/rabbitmq/log] action create (up to date)
  * directory[/opt/opscode/sv/rabbitmq/log/main] action create (up to date)
  * template[/opt/opscode/sv/rabbitmq/log/run] action create (up to date)
  * template[/opt/opscode/sv/rabbitmq/log/config] action create (up to date)
  * link[/opt/opscode/init/rabbitmq] action create (up to date)

    - configure service runit_service[rabbitmq]

Recipe: private-chef::rabbitmq
  * execute[/opt/opscode/bin/private-chef-ctl start rabbitmq] action run
================================================================================
Error executing action `run` on resource 'execute[/opt/opscode/bin/private-chef-ctl start rabbitmq]'
================================================================================


Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /opt/opscode/bin/private-chef-ctl start rabbitmq ----
STDOUT: warning: rabbitmq: unable to open supervise/ok: file does not exist
STDERR:
---- End output of /opt/opscode/bin/private-chef-ctl start rabbitmq ----
Ran /opt/opscode/bin/private-chef-ctl start rabbitmq returned 1


Resource Declaration:
---------------------
# In /opt/opscode/embedded/cookbooks/private-chef/recipes/rabbitmq.rb

 79:   execute "#{opc_ctl} start rabbitmq" do
 80:     retries 20
 81:   end
 82:



Compiled Resource:
------------------
# Declared in /opt/opscode/embedded/cookbooks/private-chef/recipes/rabbitmq.rb:79:in `from_file'

execute("/opt/opscode/bin/private-chef-ctl start rabbitmq") do
  action "run"
  retries 0
  retry_delay 2
  command "/opt/opscode/bin/private-chef-ctl start rabbitmq"
  backup 5
  returns 0
  cookbook_name :"private-chef"
  recipe_name "rabbitmq"
end

Running handlers:
[2014-11-21T12:15:45-05:00] ERROR: Running exception handlers
Running handlers complete

[2014-11-21T12:15:45-05:00] ERROR: Exception handlers complete
[2014-11-21T12:15:45-05:00] FATAL: Stacktrace dumped to /opt/opscode/embedded/cookbooks/cache/chef-stacktrace.out
Chef Client failed. 2 resources updated in 51.528969427 seconds
[2014-11-21T12:15:45-05:00] ERROR: execute[/opt/opscode/bin/private-chef-ctl start rabbitmq] (private-chef::rabbitmq line 79) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/opscode/bin/private-chef-ctl start rabbitmq ----
STDOUT: warning: rabbitmq: unable to open supervise/ok: file does not exist
STDERR:
---- End output of /opt/opscode/bin/private-chef-ctl start rabbitmq ----
Ran /opt/opscode/bin/private-chef-ctl start rabbitmq returned 1
[2014-11-21T12:15:45-05:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

1. I installed "librabbitmq", "librabbitmq-devel", and "rabbitmq-server", rebooted, then reinstalled "private-chef (11.1.6). Ditched this version and downloaded 12, since I am getting to the same place.
2. Version 12 is the same.... hangs here:

* directory[/opt/opscode/sv/rabbitmq/log/main] action create
    - create new directory /opt/opscode/sv/rabbitmq/log/main
    - change mode from '' to '0755'

  * template[/opt/opscode/sv/rabbitmq/log/run] action create
    - create new file /opt/opscode/sv/rabbitmq/log/run
    - update content in file /opt/opscode/sv/rabbitmq/log/run from none to 0e09b8
        --- /opt/opscode/sv/rabbitmq/log/run    2014-11-21 13:07:03.303442999 -0500
        +++ /tmp/chef-rendered-template20141121-1933-1l5gq9c    2014-11-21 13:07:03.305442999 -0500
        @@ -1 +1,4 @@
        +#!/bin/sh
        +exec chpst -U opscode -u opscode \
        +  svlogd -tt /var/log/opscode/rabbitmq
    - change mode from '' to '0755'

  * template[/opt/opscode/sv/rabbitmq/log/config] action create
    - create new file /opt/opscode/sv/rabbitmq/log/config
    - update content in file /opt/opscode/sv/rabbitmq/log/config from none to e3b0c4
        (no diff)
    - change mode from '' to '0644'

  * link[/opt/opscode/init/rabbitmq] action create
    - create symlink at /opt/opscode/init/rabbitmq to /opt/opscode/embedded/bin/sv


    - configure service runit_service[rabbitmq]  * link[/opt/opscode/service/rabbitmq] action create
    - create symlink at /opt/opscode/service/rabbitmq to /opt/opscode/sv/rabbitmq

Just hangs right here....

[redhatromero]

Sorry about the formatting...

[redhatromero]

So... I tried forcing "rabbitmq-server" into action:

[root@chefent ent-chef]# service rabbitmq-server status
Redirecting to /bin/systemctl status  rabbitmq-server.service
rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; disabled)
   Active: active (running) since Fri 2014-11-21 15:04:12 EST; 31s ago
  Process: 2819 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmqctl wait /var/run/rabbitmq/pid (code=exited, status=0/SUCCESS)
  Process: 2792 ExecStartPre=/bin/sh -c /usr/lib/rabbitmq/bin/rabbitmqctl status > /dev/null 2>&1 (code=exited, status=2)
 Main PID: 2818 (beam)
   CGroup: /system.slice/rabbitmq-server.service
           ??2818 /usr/lib64/erlang/erts-5.10.4/bin/beam -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/ra...
           ??2913 inet_gethost 4
           ??2914 inet_gethost 4

Nov 21 15:04:10 chefent.jordanje.mac rabbitmqctl[2819]: pid is 2818 ...
Nov 21 15:04:12 chefent.jordanje.mac rabbitmq-server[2818]: RabbitMQ 3.1.5. Copyright (C) 2007-2013 GoPivotal, Inc.
Nov 21 15:04:12 chefent.jordanje.mac rabbitmq-server[2818]: ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
Nov 21 15:04:12 chefent.jordanje.mac rabbitmq-server[2818]: ##  ##
Nov 21 15:04:12 chefent.jordanje.mac rabbitmq-server[2818]: ##########  Logs: /var/log/rabbitmq/rabbit@chefent.log
Nov 21 15:04:12 chefent.jordanje.mac rabbitmq-server[2818]: ######  ##        /var/log/rabbitmq/rabbit@chefent-sasl.log
Nov 21 15:04:12 chefent.jordanje.mac rabbitmq-server[2818]: ##########
Nov 21 15:04:12 chefent.jordanje.mac rabbitmq-server[2818]: Starting broker... completed with 0 plugins.
Nov 21 15:04:12 chefent.jordanje.mac rabbitmqctl[2819]: ...done.
Nov 21 15:04:12 chefent.jordanje.mac systemd[1]: Started RabbitMQ broker.

BTW:
I reached this page from here:
https://tickets.opscode.com/browse/CHEF-3838
His file... /opt/chef-server/embedded/cookbooks/runit/recipes/default.rb
in Chef12, is... /opt/opscode/embedded/cookbooks/runit/recipes/default.rb. In this file I did:

case node['platform_family']
when 'rhel', 'redhat', 'centos'

and

else
    include_recipe 'build-essential'
    # `rpmdevtools` is in EPEL repo in EL <= 5
    include_recipe 'yum-epel' if node['platform_version'].to_i == 7

    packages = %w{rpm-build rpmdevtools tar gzip}
    packages.each do |p|
      package p
    end

    if node['platform_version'].to_i >= 7
      package 'glibc-static'

Still fails at trying to start "rabbitmq"

[redhatromero]

Ran:

yum install glibc-static rpmdevtools

No change...

[redhatromero]

looks to me like it IS running:

rabbitmq 2818 1 0 15:04 ? 00:00:04 /usr/lib64/erlang/erts-5.10.4/bin/beam -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@chefent -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@chefent.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@chefent-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@chefent-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@chefent"
root 5396 1860 0 15:42 pts/0 00:00:00 /bin/bash /usr/bin/chef-server-ctl reconfigure
root 5399 5397 0 15:42 pts/0 00:00:04 /opt/opscode/embedded/bin/ruby /opt/opscode/embedded/bin/chef-solo -c /opt/opscode/embedded/cookbooks/solo.rb -j /opt/opscode/embedded/cookbooks/dna.json

[redhatromero]

I give for now... I will back the system down to 6.5 or 6.6.

[kbaltrinic]

I am having the exact same set of issues trying to install Jenkins on Centos 7 using the Ops code Jenkins cookbook with the 'war' option (which uses runit to run the war). Similarly The ops-code chef server cookbook, though it appears to use its own runit cookbook, has a detailed post on the same problem Chef Server runit install fails on CentOS 7 #1955.

The short of it being that from 6.x to 7 Centos switched from using upstart to systemd as its service manager. So it seems that the solution is to update the runit cookbook configure runit to use systemd on Cento >= 7.

[redhatromero]

Dang it...!!! I would have never thought to check for that, I guess because it appeared to be alive (??? have to look back at my notes).

Thanks though, for the feedback. It will be a little while before I try it again being that I could afford more time since I was being pressed for a working solution.

[jtimberman]

While we do need to get systemd support added to this cookbook so runit can work properly on CentOS/RHEL 7, this is being addressed in the Chef Server's cookbooks under this PR.

[slyness]

Since the PR referenced (chef-cookbooks/enterprise-chef-common#25) above is merged has this issue been resolved?

[kbaltrinic]

@slyness No I do not think that this issue is resolved. The enterprise-chef-common PR is specific to enterprise-chef which does not use the runit cookbook (but does much the same thing). The runit cookbook itself appears to still broken vis-a-vis Centos 7. I am still unable to install Jenkins on Centos 7 via the jenkins cookbook due to this issue, and I imagine the original reporter is still unable to install RabbitMQ for the same reason.

[marcusl33beach]

Just tried install chef on CentOS7 and ran into the same issue. Has there been any progress on this ?