Prevent NVMe-oF boot devices from being removed by disconnect-all

[johnmeneghini]

On systems that boot from SAN over NVMe-TCP or NVMe-FC, it is quite easy to make the system inoperable by issuing a 'nvme disconnect-all". For instance, after issuing the disconnect on a system which boots over NVMe-FC, the console reports the following:

[  103.827323] block nvme1n1: no available path - failing I/O 
[  103.827337] nvme1n1p4: writeback error on inode 67250623, offset 90112, sector 59489176 
[  103.827342] block nvme1n1: no available path - failing I/O 
[  103.827346] nvme1n1p4: writeback error on inode 67250623, offset 94208, sector 59489184 
[  103.832144] XFS (nvme1n1p2): Block device removal (0x20) detected at fs_bdev_mark_dead+0x43/0x60 (fs/xfs/xfs_super.c:1179).  Shutting down filesystem. 
[  103.832152] XFS (nvme1n1p2): Please unmount the filesystem and rectify the problem(s) 
[  103.832200] block nvme1n1: no available path - failing I/O 
[  103.832204] XFS (nvme1n1p4): log I/O error -5 
[  103.832206] XFS (nvme1n1p4): Filesystem has been shut down due to log error (0x2). 
[  103.832207] XFS (nvme1n1p4): Please unmount the filesystem and rectify the problem(s). 

[johnmeneghini]

Adding @martin-gpy @tbzatek for more information. This issue was reported to Red Hat and I think this is an upstream feature request.

[igaw]

The same problem exists with any FS mounted from an 'attached' storage. As far I am aware the general consensus here is, just don't do it. ip down, ifconfig etc. don't prevent you to shutdown an networking interface which is serving a mounted FS.

I am not convinced that adding even more logic to the ill defined 'connect-all' and 'disconnect-all' command is not going to make things better.

Anyway, I assume you are also attending the ALPSS conference; we can discuss this it there.

[tbzatek]

There are numerous occassions when you can shoot yourselves in the foot. Preventing that is often a battle with a wind. This bugreport has been sitting downstream for a while yet it was treated as low priority and seemingly not really viable for spending much time on it.

One idea was to somehow mark the nbftX network interfaces as private in NetworkManager, but after some other related discussions and the general upstream tendency to avoid network interface renaming this idea didn't make much sense after all.

Another idea was to use nvme-cli application strings to tag nbft-initiated connections, so they're excluded from 'disconnect-all' and such. No attempt to test this in practice was ever made.

Cc: @mwilck

[hreinecke]

I thought that's why we did introduced the '--context' options, precisely to identify which application/context created a particular connection.
What we can / might do here is to treat 'nvme disconnect-all' to mean 'disconnect all devices with no context set' (which would cover all manually created devices). And then to have a --force option to disconnect all device irrespective of the context.

[martin-belanger]

I thought that's why we did introduced the '--context' options, precisely to identify which application/context created a particular connection.

Unfortunately, the context is an attribute of the root object (i.e. struct nvme_root::application), which does not persist between invocations of nvme-cli. NBFT boot devices are created by one invocation of nvme-cli, while disconnet-all is performed by another invocation. And the context gets lost between them.

For this whole thing to work, the context would need to be saved somewhere (sysfs maybe, but how exactly?) and be an attribute of the controller so that we can tell which application created each controller.

[johnmeneghini]

I really need martin-gpy to reply to this. This request comes from NetApp.

See: https://issues.redhat.com/browse/RHEL-41068

[igaw]

From the RH bug report:

Consider "iscsiadm --mode node --logoutall=all", does that do the same thing on iSCSI?

And this is exactly my point. We have added way too much policy magic (local configuration) into connect-all and now we start to get the same discussions with disconnect-all. I am not convinced this is the right way to go any further. We need to figure out a better way to deal with this.

[johnmeneghini]

On 9/25/25 12:16 PM, Daniel Wagner wrote: *igaw* left a comment (linux-nvme/nvme-cli#2913) <#2913 (comment)> From the RH bug report: Consider "iscsiadm --mode node --logoutall=all", does that do the same thing on iSCSI?

And the answer to this question, from the iscsi maintainer, was: No, if the record is marked startup==onboot then logoutall never touches it

And this is exactly my point. We have added way too much policy magic (local configuration) into |connect-all| and now we start to get the same discussions with |disconnect-all|. I am not convinced this is the right way to go any further. We need to figure out a better way to deal with this.

I'm not disagreeing with you... disconnect-all is a mess. I'm just reporting what many customers and partners (not just NetApp) are saying. As NVMe-oF gets adopted, and more and more people use it, invariably what happens is people compare it to iSCSI and FCP. This is another one of those areas where NVMe/TCP is different from iSCSI... and people don't like it! So if you can't shoot yourself in the foot by mistakenly disconnecting your boot device with iSCSI or FCP, then people are going to expect the same from NVMe/TCP+FCP. We will need to address these expections, somehow. /John

[igaw]

Understood, though it like keep the discussion on technical ground. My point we have added horrible code in the connect logic and there seems no end in adding even more. It reached the point where it can't be maintained properly. Testing is almost impossible. This is the reasons why I push back.

Yes, this needs to be addressed and it might be that connect-all and disconnect-all have to go.

I need to familiarize myself with the iscsi setup. Anyway, is my assumption correct, that there is a daemon which holds state?

[mwilck]

Consider "iscsiadm --mode node --logoutall=all", does that do the same thing on iSCSI?
And the answer to this question, from the iscsi maintainer, was:

No, if the record is marked startup==onboot then logoutall never touches it

Well, "marked startup==onboot" requires that a node record exists in the iSCSI database, and that it has been assigned this property. The equivalent of this would be a matching entry in config.json with a respective property (not sure if there's a "startup mode" property in config.json, but could surely be added if not).

But neither for iSCSI nor NVMe do such entries necessarily exist ¹. iSCSI boot systems will run iscsiadm -m fw -l to set up sessions, which does not create node records. The user (or system software) needs to run iscsiadm -m discovery -t fw manually to creatte those. Only then can the user modify parameters like startup. Same for NVMe, connections initiated from the NBFT don't necessarily have entries in config.json.

However, for iSCSI I can confirm that iscsiadm -m node --logoutall=all does not disconnect sessions for which no node records exist (aka iBFT sessions). This is a distinctive difference wrt nvme disconnect-all, which disconnects everything.

We could make the semantics of disconnect-all match those of iscsiadm by disconnecting only those connections that have entries in config.json (and startup mode other than onboot). In practice, that would turn disconnect-all into a no-op on many systems, because more often than not users don't bother to have config.json entries for their connections.

Footnotes

1. And it's debatable wheter such entries should exist, because of the dynamic / variable nature of the iBFT and NBFT entries. ↩