[All] Proposal: Simplify/Refactor of storage/filesystem APIs

[zatricky]

Summary

• Type: new-api
• Aim: Refactor of storage/filesystem APIs for better compatibility with long-term monitoring and modern filesystems

Description

I've been using and following development of glances, which relies on psutil and have noticed that the design of some of the storage-related APIs add unnecessary caveats, especially around modern filesystems (ZFS, btrfs, ReFS) and disk monitoring. This proposal is to add new APIs with the ultimate aim of obsoleting some of the existing APIs.

Worst case this proposal could act as a thought exercise. I'd appreciate any feedback, especially as it relates to any other plans that this might help or hinder.

I appreciate that this is not a simple ask. I do work with software engineering and have a few years of experience with Python - but I haven't touched C for about 20 years. I could work on it myself but I would likely need a bit of help with the quality.

Current API Issues

Main issues:

• The existing APIs impose arbitrary limitations in the relationships between block devices, filesystems, and mountpoints.
• The existing APIs use casual or legacy terminology that is reasonable language in day to day use - but adds ambiguity in the implementation.
• Modern multidisk filesystems like ZFS and btrfs are not properly represented. ZFS requires a special case just to appear in results at all, while btrfs multi-device filesystems silently lose their member device information with no clear way to account for them.
• no persistent storage identification

Specifics

1. disk_partitions() conflates partitions with filesystems

   • Named "disk_partitions" but actually returns mounted filesystems (getmntent() in Linux, GetVolumeInformation() in Windows)
   • device field can be partition (/dev/sda1), logical volume (/dev/mapper/vg-lv), network path (//server/share), or even drive letter (C:\)
   • No distinction between partitions and mountpoints (a filesystem can be mounted multiple times)
   • Windows: device and mountpoint are redundant (C:\ and C:\) - see references
   • References:
     • API: "Return mounted partitions" (not disk partitions)
     • Linux implementation: Uses glibc's getmntent() which reads mounted filesystems from /etc/mtab
     • Windows implementation: drive_letter passed as both device and mountpoint in the returned tuple

2. disk_io_counters(perdisk=True) mixes "disks" and partitions

   • Documentation claims "physical disk" but returns partitions too
   • Returns: sda (disk), sda1 (partition), dm-0 (LVM logical volume), zram0 (compressed RAM device)
   • Inconsistent what constitutes a "disk"
   • References:
     • API: Says "every physical disk" but includes partitions
     • Test case: Shows both nvme0n1 (disk) and nvme0n1p1 (partition) are returned
     • is_storage_device(): Checks for presence in /sys/block/; dm-* and zram* devices appear there and are therefore included

3. No persistent identifiers

   • Kernel names (sda, PhysicalDrive0) can change on reboot
   • No way to track same disk across OS migrations
   • Critical for S.M.A.R.T. monitoring and asset tracking
   • Reference:
     • Issue #1609: Feature request for persistent disk identification

4. disk_usage() tied to filesystem concept but named for disks

   • Takes a path/mountpoint (filesystem concept), not a disk/drive identifier
   • Name suggests drive-level operation but actually measures filesystem usage
   • Reference:
     • API: Takes "path" parameter (filesystem mountpoint)

Proposal

Key Principle: Separate Layers, Remove Conflation

The current API treats storage and filesystems as one layer. The proposal separates them into at least two layers:

Layer 1 - Block Devices:   [kernel_name: "sda"], [kernel_name: "sda1"], [kernel_name: "dm-0"], ...
                                        |
                                        | many-to-many (block devices can have multiple parents
                                        | and multiple children, which may be other block devices
                                        | or filesystems)
                                        |
Layer 2 - Filesystems:     ext4 on one block device, ZFS/btrfs/ReFS spanning multiple block devices, ...
                                        |
                                        | one-to-many (a filesystem has one or more mountpoints)
                                        |
Layer 3 - Mountpoints:     /, /home, C:\
                                        (just a list property in the Filesystem layer)

A physical disk is a BlockDevice where some property or function indicates it is a physical device. There is no separate class. Extending the classes to identify or support new types would become simple as long as it is possible to enumerate the relationships between the various objects.

---

Implementation Path

Step 1 - no breaking changes

psutil.block_devices()     # New
psutil.block_io_counters() # New; supplants the functionality provided by disk_io_counters()

psutil.disk_partitions()   # Unchanged
psutil.disk_io_counters()  # Unchanged
psutil.disk_usage()        # Unchanged

Step 2 - no breaking changes

psutil.filesystems()          # New; replaces disk_partitions()
psutil.filesystem_usage()     # New; replaces disk_usage()

psutil.disk_partitions()      # Unchanged
psutil.disk_usage()           # Unchanged

Step 3 - obsolete old functions/structures

psutil.disk_partitions()     # Add deprecation warnings
psutil.disk_io_counters()    # Add deprecation warnings
psutil.disk_usage()          # Add deprecation warnings

Step 3 would naturally only happen once the new APIs have full feature parity.

---

Step 1: Block Devices - psutil.block_devices()

New function and class:

psutil.block_devices() -> List[BlockDevice]

class BlockDevice:
    id: str
    ...

BlockDevice represents any block-layer device - physical storage hardware, partitions, LVM volumes, RAID arrays, etc. id is a normalised stable identifier resolved from the best available source according to the priority chain below. kernel_name is always transient and is only used as a last resort. Block devices do not have mount points. There is no block device named C:\ or /home.

Stable ID priority: wwn  ->  serial  ->  uuid  -> other (as long as the id is persistent) -> kernel_name

Relationships are expressed as lists of id strings rather than embedded objects, avoiding unnecessary I/O when only a single device's data is needed. To work with the full graph, call block_devices() once and index the results by id:

class BlockDevice:
    ...
    parents:  List[str]   # ids of parent BlockDevices
    children: List[str]   # ids of child devices (BlockDevices, Filesystems, etc)

Examples:

Physical:   parents=[typically empty]
Partition:  parents=[typically id of a physical block device]
LVM LV:     parents=[typically id of VG]
RAID:       parents=[typically ids of member devices]
bcache:     parents=[typically id of cache device, id of backing device]

The children lists will typically be other block devices, a filesystem id, or a swap id. A swap partition is not a filesystem but it does have its own identity. A block device with no children at all would be unformatted or otherwise in use outside any known layer.

As a side effect, the layered model also makes swap file -> physical device traversal straightforward: a swap file lives on a Filesystem, which in turn references its underlying BlockDevice(s). No special handling is needed - the path is already in the graph. The missing piece (out of scope for this proposal) would be a swap_locations() type function that exposes swap file paths as a starting point for that traversal, as sketched in #1681.

block_devices() should support filtering in two ways: a) with an identifier string ("sda", serial, uuid, etc) to look up a specific device, or b) with filter parameters. Example filters could be top_level=True which would return only devices with no parents (the natural starting point for traversing the device tree) and physical=True to apply additional heuristics that exclude virtual devices with no kernel-visible parents. The physical filter is necessarily best-effort - for example, paravirtual disks in a VM may be indistinguishable from physical hardware.

There will of course need to be other fields in the class, such as interface type (generic, ata, fc, usb, nvme), size, and kernel identifier (sda, nvme0n1p2, PhysicalDrive0). A dict field could provide type-specific information. For example physical drives could provide serial numbers and model information when available.

---

Step 2: Filesystems - psutil.filesystems()

New functions and class:

psutil.filesystems() -> List[Filesystem]
psutil.filesystem_usage(fs_or_mountpoint) -> FilesystemUsage

class Filesystem:
    id: str
    ...

Filesystem represents a formatted filesystem instance, independent of where (or whether) it is mounted. id follows the same normalised priority concept as BlockDevice.id - typically a filesystem UUID where available. On Windows, NTFS stores a 32-bit volume serial number in the superblock (persistent; survives migrations) rather than a full UUID, so the priority chain would use that rather than the Windows-assigned volume GUID (which is not stored in the filesystem itself and does not survive migration). Switching a filesystem to a different drive letter or having multiple mountpoints would not affect the persistent id.

A Filesystem can have a many-to-many relationship with BlockDevice - most filesystems live on a single block device, but volume managers like ZFS, btrfs, and ReFS can span multiple. As with BlockDevice, the relationship is expressed as id strings:

class Filesystem:
    ...
    block_devices: List[str]   # ids of underlying BlockDevices

A filesystem may be mounted at zero or more mountpoints (e.g. not mounted at all or via bind mounts):

class Filesystem:
    ...
    mountpoints: List[str]

filesystem_usage(fsid_or_mountpoint) accepts either a Filesystem object, a mountpoint path string, or a filesystem id string, and returns filesystem usage statistics. It obsoletes disk_usage().

There will of course need to be other fields, such as filesystem type (ext4, btrfs, ntfs, zfs, ReFS, etc.) and mount options.

---

Mountpoints

Mountpoints are simply exposed as List[str] in Filesystem objects. I don't see an obvious need for a separate function and class - but it wouldn't be particularly hard to implement one. All proposed "nesting" of parent and child objects are via List[str] anyway.

---

WWN - World Wide Name

A WWN (World Wide Name) is a globally unique 64-bit identifier assigned to a storage device by its manufacturer. It is defined by the IEEE NAA (Network Address Authority) standard and embedded in hardware - it does not change across reboots, OS reinstalls, or migrations between machines.

WWN support by interface type:

Interface	WWN availability
SATA / SAS	Mandatory per ATA-8 (~2008); very old drives may lack WWN
NVMe	Available as EUI-64 or NGUID
USB	Rarely available; fall back to serial
FC / SAN	Present but uses a different format from SATA/SAS NAA
Virtual disks	Not applicable; not physical hardware

On Linux, WWNs are exposed via sysfs (e.g. /sys/block/sda/device/wwid) with OS-specific prefixes (naa., eui., 0x) that need to be stripped for normalisation. On Windows, they are available via IOCTL_STORAGE_QUERY_PROPERTY. On macOS, IOKit exposes them for some devices; a media UUID is the preferred fallback.

When a WWN is not available (USB drives, some virtual disks), the serial number is the next best stable identifier.

---

Related Issues

• Extend disk_partitions() to get disk name / ID #1609 - feature request for persistent disk identification (from Glances' author)
• [Linux] new API to list swap_partitions() #1681 - swap partitions not visible in disk_partitions() because getmntent() only sees mounted filesystems; swap partitions would be naturally exposed as block devices with a swap id in its children list.
• [Linux] expose more disk I/O metrics #2749 - expose additional disk I/O metrics (discard, flush) from /proc/diskstats
• Allow for diskio alias by UUID nicolargo/glances#2750

A search for is:issue state:open partitions returns ~30 results, many of which are symptoms of the same underlying problems described in this issue.

[giampaolo]

I'm tempted to close this issue. There is zero effort on your side. You just pasted a huge blob of text generated by AI which is entirely up to me to decode, evaluate, figure out what's relevant, think through, etc. This isn't actionable.

[zatricky]

"Zero effort". If it was generated by AI then I would totally agree with you - but I actually did spend a lot of effort over the last week or so structuring it properly so it would be easy to understand and think through.

EDIT: Per the intro text, at worst I at least hope this would provide a thought exercise. I believe this would simplify many existing issues that currently have no clear fix/implementation path.

2ndEdit: It's late here, heading to bed. Perhaps because of how big it is, it makes sense you might say it "isn't actionable". If you're in favour of smaller related tasks, I don't mind closing this in favour of doing just one small meaningful portion at a time. Please let me know.

[thegreatzaxby]

I'm tempted to close this issue. There is zero effort on your side. You just pasted a huge blob of text generated by AI which is entirely up to me to decode, evaluate, figure out what's relevant, think through, etc. This isn't actionable.

I 1000% understand being weary of reading all the time especially at the end of the day. Love of the game brother. Ask good questions: "is any of this synthesized? A bit sleepy rn ngl, might get to it tomorrow" you gotta admit saying it's not actionable with no specifics demonstrating that noise not signal isn't the best move, yeah?

[thegreatzaxby]

I'm tempted to close this issue. There is zero effort on your side. You just pasted a huge blob of text generated by AI which is entirely up to me to decode, evaluate, figure out what's relevant, think through, etc. This isn't actionable.

The appropriate response is: how much of this is synthesized and what safeguards were established to ensure deterministic evaluation?

[thegreatzaxby]

"Zero effort". If it was generated by AI then I would totally agree with you - but I actually did spend a lot of effort over the last week or so structuring it properly so it would be easy to understand and think through.

EDIT: Per the intro text, at worst I at least hope this would provide a thought exercise. I believe this would simplify many existing issues that currently have no clear fix/implementation path.

2ndEdit: It's late here, heading to bed. Perhaps because of how big it is, it makes sense you might say it "isn't actionable". If you're in favour of smaller related tasks, I don't mind closing this in favour of doing just one small meaningful portion at a time. Please let me know.

Hey bud I'd have my feelings a bit hurt too. Thanks for being a passionate member of the community. But to be fair, you know we're all looking at every day. You invited the assumption by not mitigating it. ('AI only used for [grammar and formatting, some research- ] [background -] [where you left off/where OP can start])

Please keep open and collaborate. I have no dog in this fight, but y'all were both plenty reasonable. "Sorry for assuming, you know how it is"
"Guess I made it too pretty lol. What I am hoping can be addressed is x, which ties into y and could make z possible. I put t hours into it, where did I lose you"

Misunderstandings happen. See the humor in it. It's not a bug deal ;P

[zatricky]

@thegreatzaxby I use markdown as part of my daily work, especially around documentation. I follow strict linting and deliberate git principals for my personal and work code. "Guess I made it too pretty lol" 100% applies xD

Slow, deliberate, predictable change is valued in a tool/lib like this, so I wasn't expecting prompt feedback either way.

@giampaolo At some point in the next few weeks I'm going to do mockup functions of the initial part of this idea then we can see if you like or hate it. That way I'm not wasting anybody's time if it doesn't fit your vision.