chef recipes are forcing lockd to port 32768 which leads to occasional NFS mount failures.

[gwolski]

APC 3.11.2 and 3.13.2
I've been struggling with lockd failures when I try to do an NFS mount at boot time of my parallelcluster machines. One in a few hundred machines fails to NFS v3 mount disks....

I get the following error message in my /var/log/messages files:

Aug 25 10:31:49 ip-10-6-12-13 kernel: lockd_up: makesock failed, error=-98
Aug 25 10:31:50 ip-10-6-12-13 92_tsi_mount_disks_withdiags.sh[4215]: mount.nfs: mount(2): Address already in use
Aug 25 10:31:50 ip-10-6-12-13 92_tsi_mount_disks_withdiags.sh[4215]: mount.nfs: Address already in use

Some chef recipe that I cannot find on my AMI is forcing the lockd to use port 32768.
You can see that in the file:

cat /etc/nfs.conf
# Generated by Chef for [hostname redacted] # Local modifications will be overwritten.
#
# This is a general configuration for the
# NFS daemons and tools
[general]
pipefs-directory=/var/lib/nfs/rpc_pipefs

[gssd]
use-gss-proxy=1

[lockd]
port=32768
udp-port=32768

It is also set in the /etc/modprobe.d/options_lockd.conf and /etc/syctl.d/ in two files
99-chef-fs.nfs.nlm_tcpport.conf:fs.nfs.nlm_tcpport = 32768
99-chef-fs.nfs.nlm_udpport.conf:fs.nfs.nlm_udpport = 32768

Redhat has an article on this problem. https://access.redhat.com/solutions/2200331- it is mentioned this is for RHEL 7, so it is old, but it was updated in 2024 and smells exactly like my problem.

I have not been able to figure out what is actually locking port 32768, so I have just modified the boot up of my machine to change everything to dynamic, which is the default RHEL/Rocky experience:

# make sure lockd uses a dynamic port to avoid address collisions
[[ ! -e /etc/modprobe.d/options_lockd.conf.fcs ]] && cp /etc/modprobe.d/options_lockd.conf /etc/modprobe.d/options_lockd.conf.fcs
sudo sed -i '/^\[lockd\]/,/^$/s/^port=/#port=/' /etc/nfs.conf
sudo sed -i '/^\[lockd\]/,/^$/s/^udp-port=/#udp-port=/' /etc/nfs.conf
sudo sed -i -e 's/nlm_tcpport=[0-9]*/nlm_tcpport=0/g' -e 's/nlm_udpport=[0-9]*/nlm_udpport=0/g' /etc/modprobe.d/options_lockd.conf
# Apply sysctl changes dynamically
sudo sysctl fs.nfs.nlm_tcpport=0
sudo sysctl fs.nfs.nlm_udpport=0
modprobe -v -r lockd
modprobe -v lockd

I would ask that you remove this static assignment.

I am running with this new setup and will report back with an update if this is working.

[gwolski]

I've been running now for two weeks. Thousands of machines have started, no more lockd failures. Would really ask that you take out the static assignment of lockd ports. Not sure why that would be a good idea anyway.....

[nchao-aam]

My team has been experiencing this same issue. The static port assignment is likely a security measure to make it easier to seal up a firewall:

https://github.com/aws/aws-parallelcluster-cookbook/blame/921f2d8d73ef9f3a86fb330dc96e5c21e422385e/cookbooks/nfs/attributes/default.rb#L33

# Default options are taken from the Debian guide on static NFS ports

Reference to Debian NFS guide mentioned in the comment: https://wiki.debian.org/SecuringNFS

Our workaround is to change the ephemeral port range on the node so that we can still maintain that security standard. We noticed the ephemeral port range starts at 32768:

cat /proc/sys/net/ipv4/ip_local_port_range
>> 32768   60999

I suspect some background process is running during boot time while the NFS mounts are being created and that process is taking the first port in that range, 32768.

We added the following to our user-data before the NFS mounts occur to shift the ephemeral port range to be what the IANA recommends for dynamic port ports. This should prevent lockd from having any port collision with background processes that use the ephemeral port range.

echo "net.ipv4.ip_local_port_range = 49152 65535" >> /etc/sysctl.conf
sysctl -p /etc/sysctl.conf

We just applied this change yesterday to one of our parallel clusters and are still monitoring it.

[gwolski]

So now I understand why it would be a good idea, i.e. lock down ports for security group reasons.
But it looks like 32768 is a bad port to use as a fixed port as it is at the start of the ephemeral port range on RHEL 8, 9, Ubuntu, alinux (according to Google). Oddly, the IANA website shows 32768 reserved for filenet-tms - not sure then how all these OSes chose 32768 as the start for ephemeral start if 32768 is spoken for?

I don't know what the "right" answer would be, someone smarter than me needs to make that call.

@nchao-aam - can you share how you applied the ephemeral port range change to your user-data or provide me a link showing an example? I have not done that yet. Your solution feels a bit more elegant than mine. Thank you.

[nchao-aam]

So now I understand why it would be a good idea, i.e. lock down ports for security group reasons. But it looks like 32768 is a bad port to use as a fixed port as it is at the start of the ephemeral port range on RHEL 8, 9, Ubuntu, alinux (according to Google). Oddly, the IANA website shows 32768 reserved for filenet-tms - not sure then how all these OSes chose 32768 as the start for ephemeral start if 32768 is spoken for?

I don't know what the "right" answer would be, someone smarter than me needs to make that call.

@nchao-aam - can you share how you applied the ephemeral port range change to your user-data or provide me a link showing an example? I have not done that yet. Your solution feels a bit more elegant than mine. Thank you.

This is what we included in our user-data to change the ephemeral port range:

echo "net.ipv4.ip_local_port_range = 49152 65535" >> /etc/sysctl.conf
sysctl -p /etc/sysctl.conf

However, we found that this has not fixed the issue and are opting to go with your workaround for now.