command "crm node online machine1" takes just over 2 minutes since Ubuntu 24.04 upgrade

[KoenDG]

On ubuntu 24.04, crmsh is version 4.6.0-1ubuntu2.

Corosync: 3.1.7-1ubuntu3

Pacemaker: 2.1.6-5ubuntu2

From my (anonymized) configuration /etc/corosync/corosync.conf:

transport: udp
...
interface: {
  bindnetaddr: 192.168.0.0
}
...
nodelist {
    node {
        ring0_addr: 192.168.0.50
        name: machine1
        id: 1
    }
    node {
        ring0_addr: 192.168.0.51
        name: machine2
        id: 2
    }
    node {
        ring0_addr: 192.168.0.52
        name: machine3
        id: 3
    }
}

However, my /etc/hosts looks like this:

10.0.0.1 machine1
10.0.0.2 machine2
10.0.0.3 machine3

The corosync/pacemaker cluster communicates with their nodes over 192.168.0.0/24 and other services over a different internal network. This allows for clean iptable rules, separating traffic.

Am I right in thinking that what is happening is that first the software sees "machine1", checks /etc/hosts, sees 10.0.0.1 and tries connecting to the node via that IP? (which will be blocked by the firewall in my case)

And then after 2 minutes it gives up and tries ring0_addr? And if so, is there a way to rely on ring0_addr instead?

Or is something else going on?

I'm now waiting 2 minutes for an IP address resource to swap to a different host, that's my issue here.

[nicholasyang2022]

Am I right in thinking that what is happening is that first the software sees "machine1", checks /etc/hosts, sees 10.0.0.1 and tries connecting to the node via that IP? (which will be blocked by the firewall in my case)

Yes, only the primary IP, listed in /etc/hosts is supported. See https://documentation.suse.com/sle-ha/15-SP7/html/SLE-HA-all/cha-ha-requirements.html#sec-ha-requirements-other

[KoenDG]

Strange, because it does work, but only after 2 minutes 15 seconds. It's very consistent that way. And traffic to the ports is only allowed on the network of the ring0_addr. So that would suggest it does eventually try the ring0_addr

Forgive my ignorance, but if the ring0_addr isn't the one being used for communication between nodes... why have it at all?

It seems like a mistake to rely solely on /etc/hosts. There may be good reasons why a company wants to isolate certain kinds of traffic. Being forced to use /etc/hosts or pay a 2 minutes 15 seconds waiting fee is... not conducive to productive work.

EDIT: Just did a test allowing the ports traffic over the other network. Checked that packets could be sent... crm node online still taking 2 minutes 15 seconds. What can be going on here?

EDIT2: I tried getting an strace output.

These are the last logged lines before it hangs for 2 minutes:

openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=968, ...}) = 0
lseek(5, 0, SEEK_SET)                   = 0
read(5, "127.0.0.1       localhost\n\n# The"..., 4096) = 968
close(5)                                = 0
openat(AT_FDCWD, "/dev/null", O_RDWR|O_CLOEXEC) = 5
pipe2([6, 7], O_CLOEXEC)                = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
vfork()                                 = 2172641
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
close(7)                                = 0
close(5)                                = 0
read(6, "", 50000)                      = 0
close(6)                                = 0
wait4(2172641,

And the first lines after the 2 minutes are these:

wait4(2172641, [{WIFEXITED(s) && WEXITSTATUS(s) == 255}], 0, NULL) = 2172641
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2172641, si_uid=0, si_status=255, si_utime=0, si_stime=0} ---
getpid()                                = 2172592
getpid()                                = 2172592
uname({sysname="Linux", nodename="mymachine", ...}) = 0
getpid()                                = 2172592
newfstatat(AT_FDCWD, "/var/log/crmsh/crmsh.log", {st_mode=S_IFREG|0664, st_size=682, ...}, 0) = 0
newfstatat(AT_FDCWD, "/var/log/crmsh/crmsh.log", {st_mode=S_IFREG|0664, st_size=682, ...}, 0) = 0
lseek(3, 0, SEEK_END)  

Maybe this is useful info?

EDIT3: I just checked the process being spawned and it's trying to SSH into the other hosts... why??? I didn't configure that.

[KoenDG]

And it looks like this is where it's happening: https://github.com/ClusterLabs/crmsh/blob/4.6.0/crmsh/user_of_host.py#L98

And it looks hardcoded.

So now I'm going to have to look in to ssh client settings to drop this timeout, I guess?

It's a timeout... and yet still my cluster is correctly updated with my intended changes...

Am I then to assume it updates the local host, and this just syncs automatically via corosync, port 5404,5405,5406 and what have you?

[nicholasyang2022]

Can you get a backtrace, by pressing ctrl + c, during the 2 minutes 15 seconds wait?

[KoenDG]

Is there a particular log file I should check for a message?

I'm seeing only the command being cancelled and nothing more.

EDIT: For clarity, I've currently mitigated my issue somewhat by setting an ssh config for the root user, timing out connections very quickly if they do not respond.