The parameter "full_command" in "vrrp_track_process" section not working

[Asbjorn88]

Linked #1265

Describe the issue
The parameter "full_command" in "vrrp_track_process" section not working (do not match the given process).

To Reproduce
I have a similar problem from issue #1265, but using different nginx processes.

I've tried it in different ways:

vrrp_track_process nginx{
    process "/usr/sbin/nginx" "-c /etc/angie/nginx.conf"
    full_command
    delay 3
    quorum 1
}

vrrp_track_process nginx-bu {
    process "/usr/sbin/nginx" "-c /etc/angie/nginx-bu.conf"
    full_command
    delay 3
    quorum 1
}

vrrp_track_process nginx-web {
    process "/usr/sbin/nginx" "-c /etc/angie/nginx-web.conf"
    full_command
    delay 3
    quorum 1
}

and so:

vrrp_track_process nginx{
    process /usr/sbin/nginx -c /etc/nginx/nginx.conf
    full_command
    delay 3
    quorum 1
}

vrrp_track_process nginx-bu {
    process /usr/sbin/nginx -c /etc/nginx/nginx-bu.conf
    full_command
    delay 3
    quorum 1
}

vrrp_track_process nginx-web {
    process /usr/sbin/nginx -c /etc/nginx/nginx-web.conf
    full_command
    delay 3
    quorum 1
}

and so:

vrrp_track_process nginx{
    process "/usr/sbin/nginx" "-c" "/etc/angie/nginx.conf"
    full_command
    delay 3
    quorum 1
}

vrrp_track_process nginx-bu {
    process "/usr/sbin/nginx" "-c" "/etc/angie/nginx-bu.conf"
    full_command
    delay 3
    quorum 1
}

vrrp_track_process nginx-web {
    process "/usr/sbin/nginx" "-c" "/etc/angie/nginx-web.conf"
    full_command
    delay 3
    quorum 1
}

Expected behavior
If "full_command" was set in "vrrp_track_process" section, then the process "/usr/sbin/nginx" "-c" "/etc/angie/nginx-bu.conf", "/usr/sbin/nginx" "-c" "/etc/angie/nginx-web.conf", "/usr/sbin/nginx" "-c" "/etc/angie/nginx.conf" should match.

Keepalived version
Keepalived v2.1.5

Distro (please complete the following information):
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=8.10
x86_64

Details of any containerisation or hosted service (e.g. AWS)
on hosted

Configuration file:

vrrp_track_process nginx {
    process "/usr/sbin/nginx" "-c /etc/nginx/nginx.conf"
    full_command
    delay 3
    quorum 1
}

vrrp_track_process nginx-bu {
    process "/usr/sbin/nginx" "-c /etc/nginx/nginx-bu.conf"
    full_command
    delay 3
    quorum 1
}

vrrp_track_process nginx-web {
    process "/usr/sbin/nginx" "-c /etc/nginx/nginx-web.conf"
    full_command
    delay 3
    quorum 1
}

vrrp_instance web {
    state MASTER
    interface "bond0.404" 
    virtual_router_id 254
    priority 100
    advert_int 1
    preempt_delay 1

    virtual_ipaddress {
      10.63.0.17/29 dev bond0.404 
      10.63.11.14/28 dev bond0.410 
      10.63.11.78/28 dev bond0.411 
      10.63.11.30/28 dev bond0.412 
      10.63.11.46/28 dev bond0.413 
      10.63.11.62/28 dev bond0.414 
      10.63.11.94/28 dev bond0.415
      10.63.11.110/28 dev bond0.416 
      10.63.11.126/28 dev bond0.417 
      10.63.11.142/28 dev bond0.418 
      10.63.11.158/28 dev bond0.419 
      10.63.11.174/28 dev bond0.421 
      10.63.11.190/28 dev bond0.422 
      10.63.11.206/28 dev bond0.423 
      10.63.11.222/28 dev bond0.424
      10.63.11.238/28 dev bond0.425 
      10.63.11.254/28 dev bond0.426 
      10.63.12.14/28 dev bond0.427 
    }
    track_process {
         nginx-web
   }
}


vrrp_instance BU {
    state BACKUP
    interface "bond0.402"
    virtual_router_id 253
    priority 99
    advert_int 1
    preempt_delay 1

    virtual_ipaddress {
      10.63.0.9/29 dev bond0.402
    }
    track_process {
         nginx-bu
   }
}

System Log entries

Oct 10 17:52:44 smr1-bal-n3 systemd[1]: Starting LVS and VRRP High Availability Monitor...
Oct 10 17:52:44 smr1-bal-n3 Keepalived[255579]: Starting Keepalived v2.1.5 (07/13,2020)
Oct 10 17:52:44 smr1-bal-n3 Keepalived[255579]: Running on Linux 5.15.0-312.187.5.3.el8uek.x86_64 #2 SMP Sun Sep 21 08:53:14 PDT 2025 (built for Linux 4.18.0)
Oct 10 17:52:44 smr1-bal-n3 Keepalived[255579]: Command line: '/usr/sbin/keepalived' '-D'
Oct 10 17:52:44 smr1-bal-n3 Keepalived[255579]: Opening file '/etc/keepalived/keepalived.conf'.
Oct 10 17:52:44 smr1-bal-n3 Keepalived[255580]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Oct 10 17:52:44 smr1-bal-n3 Keepalived[255580]: Starting VRRP child process, pid=255581
Oct 10 17:52:44 smr1-bal-n3 systemd[1]: Started LVS and VRRP High Availability Monitor.
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: Registering Kernel netlink reflector
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: Registering Kernel netlink command channel
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: Opening file '/etc/keepalived/keepalived.conf'.
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: (web) Warning - preempt delay will not work with initial state MASTER - clearing
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: Assigned address 10.63.0.18 for interface bond0.404
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: Assigned address 10.63.0.11 for interface bond0.402
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: (BU) entering FAULT state (tracked process nginx-bu quorum not achieved)
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: (web) entering FAULT state (tracked process nginx-web quorum not achieved)
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: (web) entering FAULT state
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: (BU) entering FAULT state
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: Registering gratuitous ARP shared channel
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: (web) removing VIPs.
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: (BU) removing VIPs.
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: VRRP sockpool: [ifindex( 14), family(IPv4), proto(112), fd(11,12)]
Oct 10 17:52:44 smr1-bal-n3 Keepalived_vrrp[255581]: VRRP sockpool: [ifindex( 16), family(IPv4), proto(112), fd(13,14)]

Active process:

ps -auxfw | grep nginx
root      287465  0.0  0.0  12220  1188 pts/0    S+   18:10   0:00              \_ grep --color=auto nginx
root        9731  0.0  0.0  32716  2520 ?        Ss   15:34   0:00 nginx: master process #1 [/usr/sbin/nginx -c /etc/nginx/nginx.conf]
nginx       9732  0.0  0.0  92640 33928 ?        S    15:34   0:01  \_ nginx: worker process #1
nginx       9733  0.0  0.0  92640 34768 ?        S    15:34   0:01  \_ nginx: worker process #1
nginx       9734  0.0  0.0  92640 34756 ?        S    15:34   0:01  \_ nginx: worker process #1
nginx       9735  0.0  0.0  92640 33928 ?        S    15:34   0:02  \_ nginx: worker process #1
nginx       9736  0.0  0.0  92640 34764 ?        S    15:34   0:01  \_ nginx: worker process #1
nginx       9737  0.0  0.0  92640 33924 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9738  0.0  0.0  92640 33924 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9739  0.0  0.0  92640 33924 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9740  0.0  0.0  92640 34760 ?        S    15:34   0:01  \_ nginx: worker process #1
nginx       9741  0.0  0.0  92640 34768 ?        S    15:34   0:01  \_ nginx: worker process #1
nginx       9742  0.0  0.0  92640 34760 ?        S    15:34   0:01  \_ nginx: worker process #1
nginx       9743  0.0  0.0  92640 33924 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9744  0.0  0.0  92640 33924 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9745  0.0  0.0  92640 33924 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9746  0.0  0.0  92640 33924 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9747  0.0  0.0  92640 33924 ?        S    15:34   0:00  \_ nginx: worker process #1
root        9753  0.0  0.0  66576  2664 ?        Ss   15:34   0:00 nginx: master process #1 [/usr/sbin/nginx -c /etc/nginx/nginx-web.conf]
nginx       9754  0.0  0.0 126500 35408 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9755  0.0  0.0 126500 35728 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9756  0.0  0.0 126500 35628 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9757  0.0  0.0 126500 35696 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9758  0.0  0.0 126500 35628 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9759  0.0  0.0 126500 35844 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9760  0.0  0.0 126500 35628 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9761  0.0  0.0 126500 35504 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9763  0.0  0.0 126500 35628 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9764  0.0  0.0 126500 35596 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9767  0.0  0.0 126500 35312 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9769  0.0  0.0 126500 35448 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9771  0.0  0.0 126500 35480 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9773  0.0  0.0 126500 35612 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9775  0.0  0.0 126500 35592 ?        S    15:34   0:00  \_ nginx: worker process #1
nginx       9778  0.0  0.0 126500 35592 ?        S    15:34   0:00  \_ nginx: worker process #1
root      183130  0.0  0.0  33744  2444 ?        Ss   17:12   0:00 nginx: master process #1 [/usr/sbin/nginx -c /etc/nginx/nginx-bu.conf]
nginx     183131  0.0  0.0  93668 33316 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183132  0.0  0.0  93668 34404 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183133  0.0  0.0  93668 33316 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183134  0.0  0.0  93668 34400 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183135  0.0  0.0  93668 34388 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183136  0.0  0.0  93668 34404 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183137  0.0  0.0  93668 34400 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183138  0.0  0.0  93668 34404 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183139  0.0  0.0  93668 34400 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183140  0.0  0.0  93668 34400 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183141  0.0  0.0  93668 34400 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183142  0.0  0.0  93668 34400 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183143  0.0  0.0  93668 34404 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183144  0.0  0.0  93668 34404 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183145  0.0  0.0  93668 34400 ?        S    17:12   0:00  \_ nginx: worker process #1
nginx     183146  0.0  0.0  93668 34400 ?        S    17:12   0:00  \_ nginx: worker process #1

[pqarmitage]

Each command line argument needs to be parsed by keepalived as a separate word, so, for example,
process "/usr/sbin/nginx" "-c /etc/nginx/nginx-web.conf"
will not work, but
process "/usr/sbin/nginx" "-c" "/etc/nginx/nginx-web.conf"
and
process /usr/sbin/nginx -c /etc/nginx/nginx-web.conf
will work.

I note that the location of the nginx configuration files in the examples in the original report varies between /etc/nginx and /etc/angie, and I assume that the problem is not related to that.

I will need to install nginx to test this properly, and will provide an update once I have done so.

[pqarmitage]

I have now got to the bottom of this issue, and just need to work out how to resolve it.

Currently, if nginx starts when keepalived is already running, then for keepalived to detect the nginx process, the process line needs to be:
process /usr/sbin/nginx -c /etc/nginx/nginx-web.conf
or equivalent (i.e. depending on the actual command line used to start nginx).

However, nginx changes the reported command line (see the output of ps auxfw | grep nginx ) when it starts up, and it changes it to:
nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx-web.conf
and so the keepalived configuration needed to detect the process if nginx starts before keepalived is:
process "nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx-web.conf"

In the former case /usr/sbin/nginx -c /etc/nginx/nginx-web.conf is formed from an array of pointers to three separate strings. In the latter case nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx-web.conf is a single string with embedded spaces (hence the need for the "s).

I think what we will need, in addition to the process option, is a running_process option, so your configuration would look like:

vrrp_track_process nginx-web {
    process /usr/sbin/nginx -c /etc/nginx/nginx-web.conf
    running_process "nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx-web.conf"
    full_command
    delay 3
    quorum 1
}

There are some complications that need further thought. For example, what if nginx and keepalived start at the same time. When keepalived starts and checks the currently running processes it might see either the process or the running_process version. Likewise, if nginx starts after keepalived is running, it is possible, depending on the scheduling of processes, that keepalived sees either the original process string or the updated string.

I have done some further testing with nginx to see what happens. On my system there is one nginx master process and 16 worker processes. When nginx starts after keepalived, keepalived sees 17 processes matching the process config. When keepalived starts after nginx it sees just one process, and that matches the running_process config. I don't think this is a problem but I need to see what happens when keepalived reloads.

I have looked at the nginx code to see how it changes its apparent cmdline, and it is horrible. It looks as though what it is doing is that same as libbsd's setproctitle() function, and the Linux kernel has specific code to handle this change of cmdline. The problem is that /proc/PID/cmdline is then not correct. For the worker processes, it shows:
"nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
and for the master process:
"nginx: master process /usr/bin/nginx -c /home/quentin/keepalived/nginx/nginx-web.conf" which looks correct, except that each command line parameter should be a separate string, rather than the whole being one string.

Also /proc/PID/environ is 3086 empty strings for the worker processes, and for the master process it is
/nginx/nginx-web.conf
followed by 3064 empty strings.

I should have a patch in a day or two to support the running_process functionality - I have a working first cut of it now.

[pqarmitage]

@Asbjorn88 Sorry for the delay in producing a patch - there are some further complications I have come across.