Question/Bug : mount error 32

[DeuxCatPotes]

When i try to preupgrade :

====> * target_userspace_creator
        Initializes a directory to be populated as a minimal environment to run binaries from the target system.
Process Process-595:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 74, in _do_run
    actor_instance.run(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/actors/__init__.py", line 289, in run
    self.process(*args)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/actor.py", line 58, in process
    userspacegen.perform()
  File "/usr/lib/python2.7/site-packages/leapp/utils/deprecation.py", line 42, in process_wrapper
    return target_item(*args, **kwargs)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 1274, in perform
    scratch_reserve=reserve_space) as overlay:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.py", line 582, in create_source_overlay
    with mounts.pop('/') as root_mount:
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/mounting.py", line 362, in __enter__
    self.mount()
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/mounting.py", line 337, in mount
    self._create()
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/mounting.py", line 353, in _create
    details=None)
MountError: Mount operation with mode loop from /var/lib/leapp/scratch/diskimages/root_ to /var/lib/leapp/scratch/mounts/root_ failed: Command ['mount', '-o', 'loop', '/var/lib/leapp/scratch/diskimages/root_', '/var/lib/leapp/scratch/mounts/root_'] failed with exit code 32.
2024-11-13 12:22:48.699 ERROR    PID: 11513 leapp.workflow.TargetTransactionFactsCollection: Actor target_userspace_creator has crashed: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 74, in _do_run
    actor_instance.run(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/actors/__init__.py", line 289, in run
    self.process(*args)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/actor.py", line 58, in process
    userspacegen.perform()
  File "/usr/lib/python2.7/site-packages/leapp/utils/deprecation.py", line 42, in process_wrapper
    return target_item(*args, **kwargs)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 1274, in perform
    scratch_reserve=reserve_space) as overlay:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.py", line 582, in create_source_overlay
    with mounts.pop('/') as root_mount:
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/mounting.py", line 362, in __enter__
    self.mount()
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/mounting.py", line 337, in mount
    self._create()
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/mounting.py", line 353, in _create
    details=None)
MountError: Mount operation with mode loop from /var/lib/leapp/scratch/diskimages/root_ to /var/lib/leapp/scratch/mounts/root_ failed: Command ['mount', '-o', 'loop', '/var/lib/leapp/scratch/diskimages/root_', '/var/lib/leapp/scratch/mounts/root_'] failed with exit code 32.



=========================================================================================================
Actor target_userspace_creator unexpectedly terminated with exit code: 1 - Please check the above details
=========================================================================================================

Debug output written to /var/log/leapp/leapp-preupgrade.log

Some other information :

df -h 
Sys. de fichiers                                        Taille Utilisé Dispo Uti% Monté sur
devtmpfs                                                  4,8G       0  4,8G   0% /dev
tmpfs                                                     4,8G       0  4,8G   0% /dev/shm
tmpfs                                                     4,8G    9,1M  4,8G   1% /run
tmpfs                                                     4,8G       0  4,8G   0% /sys/fs/cgroup
/dev/mapper/centos_dcp--cos7--hard--v1--0-root            5,0G    323M  4,7G   7% /
/dev/mapper/centos_dcp--cos7--hard--v1--0-usr              15G    3,2G   12G  21% /usr
/dev/sda1                                                1014M    203M  812M  20% /boot
/dev/mapper/centos_dcp--cos7--hard--v1--0-home            4,0G     43M  4,0G   2% /home
/dev/mapper/centos_dcp--cos7--hard--v1--0-srv             2,0G     33M  2,0G   2% /srv
/dev/mapper/centos_dcp--cos7--hard--v1--0-var              35G    3,9G   32G  12% /var
/dev/mapper/centos_dcp--cos7--hard--v1--0-var_log         4,0G    1,1G  3,0G  27% /var/log
/dev/mapper/centos_dcp--cos7--hard--v1--0-tmp            1014M    553M  462M  55% /tmp
/dev/mapper/centos_dcp--cos7--hard--v1--0-var_log_audit  1014M     74M  941M   8% /var/log/audit
tmpfs                                                     966M       0  966M   0% /run/user/12043

I do not understand what can make that issue occur ? My best option is missing space somewhere ?

[pirat89]

@DeuxCatPotes Hi \o can you share output for following commands?

• cat /etc/fstab
• mount
• rpm -qa '*leapp*'
• cat /etc/system-release

Also, as I see that you are using Centos 7.9, I expect you are using downstream modification of leapp-repository - Elevate. I am aware they have several modifications so I cannot be sure whether we could reproduce this at all. In case that you are using Elevate, I suggest you to contact the vendor for the help also.

If I try to make a blind shot, I do not think it's a space problem. I would check any "non-standard" mount options as we are aware of some issues around that. I would check whether there is any system log that could explain what happened (selinux?). As you have the backtrace, I would try to add a breakpoint into the code to pause the execution of the related actor when the problem happens, so all created files (e.g. disk images) would be still present on the disk and you could try to experiment with the mount operation outside of leapp. Also I suggest to check during the execution of leapp e.g. number of opened files. I would expect a different err msg in case of hitting some limits but I cannot rule out this possibility. Also I would be a little bit afraid of the persistent /tmp. I know it's possible to upgrade even having /tmp on a partition instead of having using tmpfs, but who knows 🤷 (from what I wrote above, I would give this a least significance as a possible root cause).

Also check logs under /var/log/leapp/leapp-preupgrade.txt during the execution of the target_userspace_creator actor, whether there are not any suspicious warnings or command output before.

[DeuxCatPotes]

Hi thx for feedback.
Some information :

I m not the guy who create that virtual machine.

cat /etc/fstab

/dev/mapper/centos_dcp--cos7--hard--v1--0-root /                       	xfs     defaults        0 0
UUID=804f5f76-d67f-4aa6-b15e-464a987230d6 /boot                   	xfs     nodev,nosuid    0 0
/dev/mapper/centos_dcp--cos7--hard--v1--0-home /home                   	xfs     nodev,nosuid  0 0
#/dev/mapper/centos_dcp--cos7--hard--v1--0-opt /opt                    	xfs     nodev,nosuid        0 0
/dev/mapper/centos_dcp--cos7--hard--v1--0-srv /srv                    	xfs     nodev,nosuid        0 0
/dev/mapper/centos_dcp--cos7--hard--v1--0-tmp /tmp                    	xfs     nodev,nosuid 0 0
/dev/mapper/centos_dcp--cos7--hard--v1--0-usr /usr                    	xfs     nodev         0 0
/dev/mapper/centos_dcp--cos7--hard--v1--0-var /var                    	xfs     nodev,nosuid 0 0
/dev/mapper/centos_dcp--cos7--hard--v1--0-var_log /var/log              xfs     nodev,nosuid 0 0
/dev/mapper/centos_dcp--cos7--hard--v1--0-var_log_audit /var/log/audit  xfs     nodev,nosuid 0 0
/dev/mapper/centos_dcp--cos7--hard--v1--0-var_tmp /var/tmp              xfs     nodev,nosuid 0 0
/dev/mapper/centos_dcp--cos7--hard--v1--0-swap swap                     swap    defaults        0 0

mount

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime,hidepid=2)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=4934292k,nr_inodes=1233573,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,rdma)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/centos_dcp--cos7--hard--v1--0-root on / type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/centos_dcp--cos7--hard--v1--0-usr on /usr type xfs (rw,nodev,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18348)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,relatime,seclabel)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/sda1 on /boot type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/centos_dcp--cos7--hard--v1--0-home on /home type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/centos_dcp--cos7--hard--v1--0-srv on /srv type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/centos_dcp--cos7--hard--v1--0-var on /var type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/centos_dcp--cos7--hard--v1--0-var_log on /var/log type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/centos_dcp--cos7--hard--v1--0-var_tmp on /var/tmp type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/centos_dcp--cos7--hard--v1--0-tmp on /tmp type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/centos_dcp--cos7--hard--v1--0-var_log_audit on /var/log/audit type xfs (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/12043 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=989088k,mode=700,uid=12043,gid=10000)

rpm -qa 'leapp'

leapp-upgrade-el7toel8-deps-0.20.0-2.0.5.el7_9.noarch
leapp-0.17.0-1.0.2.el7_9.noarch
leapp-upgrade-el7toel8-0.20.0-2.0.5.el7_9.noarch
leapp-deps-0.17.0-1.0.2.el7_9.noarch
python2-leapp-0.17.0-1.0.2.el7_9.noarch

cat /etc/system-release

Oracle Linux Server release 7.9

Can the nosuid prevent the upgrade ? I think not, previous vm migrated also had nosuid flags.

Maybe i can try to setup a tmpfs ?

Also that vm is not currently in centos 7 I already migrate it one time to ol7. At that day i already did more or less 10 vm without that issue.

Hope you can help me @pirat89 , and thx for your time.

[DeuxCatPotes]

Ok @pirat89 i think i find the root cause. I have to go but i will update tomorrrow for a proper feedback and maybe a better warning

[DeuxCatPotes]

Hello computer sea friends,

So what occurs during my try to upgrade.

The current not working step is a mount loop step (as log show).
But what happened :

• when the loop option is used the loop kernel module is load and unload.
• but on the current device the kernel option modules_disabled was set to 1 preventing the load.

I think a test can be done to check the current value :
sysctl kernel.modules_disabled

kernel.modules_disabled = 1

or

kernel.modules_disabled = 0

A grep can also display clue about where the value is set.

thx for your time.
I let you close the issue @pirat89 when you read that.

[pirat89]

@DeuxCatPotes thank you for the investigation of the issue! I think that it makes sense to check the value on failed mount or at least add a suggestion to the error msg for user to check the value as a possible root cause. I am keeping the issue opened to cover it.