Add support for `nvme top` (monitor feature)

[igaw]

There are use cases where the user wants to monitor the system changes. nvme-cli currently shows a snapshot. A simple way to achieve this in shell is to do:

watch -n1 nvme show-topology

Though sometimes shell scripting is adding an additional layer of problems, e.g. when creating logs for other tools to consume #2189

@hreinecke pointed out in https://lore.kernel.org/linux-nvme/ee0d329a-334b-429c-a916-f03c64bbb323@suse.de/

Well, that might be tricky. The current 'tree' structure is build
once when the program starts up. Any changes to that structure after
that are not tracked, and there (currently) are no hooks for updating
it. So having a 'monitor' function will get tricky.

To do that we would either need a udev event receiver (using uevents
to update the tree structure) or looking into fanotify()/inotify().
Then the tree structure would always be up-to-date and things like
'monitor' will be possible.
Will be hell for the python bindings, mind 🙁 

[ikegami-t]

Just created the PR: #2989.

[igaw]

@shroffni is also working on this feature. FWIW, he is going to present/discuss this during Linux Plumbers next week in Japan. In case you are also attending LPC you could discuss this topic directly :)

[shroffni]

@ikegami-t Yes I will be presenting nvme-top next week at LPC. If you're planning to attend the event then lets meet in person and discuss. You may also drop me an email at nilay@linux.ibm.com if you want to sync in advance.

[ikegami-t]

Thank you so much for your suggestion! Just checked the detailed schedule then I am not sure at this moment if I can join the meeting (and seems a little bit difficult since on the weekday day time but I would like to join if possible). Anyway if I can join I will let you know. After the evenet if you can share the presentation material I will be happy (but it is okay also without sharing if it is difficult). If you need any help to implement the feature I can do it (but I am not sure how much I can do help you sorry). By the way I live in Tokyo so easy to go there;-)

[shroffni]

Yeah I think presentation and video shall be available on the LPC event site few days after the event.

[ikegami-t]

Noted and will do check them. Thank you.

[jk-ozlabs]

Subscribing out of interest, but let me know if I can help out with any of the MI considerations!

[shroffni]

For the reference, here I have listed down community feedback received for nvme-top
presentation during LPC-2025:

1. Grafana Export

   • Having a Grafana export was viewed as a very useful and valuable addition to the tool.

2. Polling vs. Event-Based Updates

   • Relying on inotify/fsnotify or uevents for regular parameter updates may introduce
     complexity and risk missing kernel updates. A poll-based approach is recommended for
     commonly changing parameters such as queue depth, statistics, controller state, and
     I/O policy.

   • Exception: topology changes (e.g., adding or removing namespaces or controllers).
     For topology updates, listening to kernel events is necessary so that nvme-top can
     rebuild the topology using nvme-cli and update the dashboard in real time.

3. UI Framework

   • ncurses is not required; a simple termios-based approach is sufficient to build an
     effective and responsive dashboard.

4. Handling Large Topologies

   • In environments with many namespaces per subsystem, fitting all subsystems, namespaces,
     controllers, and nsheads on a single page may not be practical.
     One suggestion is to add an interactive menu to toggle between different views or tables.
   • Refer the htop implementation if simplifying or scaling the dashboard becomes challenging.

5. Controller Summary Enhancements
   In addition to displaying the controller state, the controller summary should include:

   • Number of controller resets,
   • Number of reconnect attempts (especially for NVMe-oF),
   • Potentially other reliability-related counters.

6. libnvme API Improvements

   • It was agreed upon to enhance existing libnvme APIs (targeted for the 3.x release) to
     return current values rather than cached ones.

7. Kernel Counter Exports
   NVMe kernel driver maintainer (Keith) agreed to update kernel code to export additional counters such as:

   • Failovers,
   • Errors,
   • Retries
   • Few others, if required

   These counters can then be consumed and displayed by nvme-top.

[shroffni]

For reference, the LPC 2025 presentation slide deck is available here: https://lpc.events/event/19/contributions/2104/attachments/1739/4023/nvme-top.pdf