[Linux] process_iter(attrs=...) spends ~75% of its time building Python objects, not doing I/O

[Dipet]

Summary

• OS: Linux (Ubuntu 22.04, x86_64, 128 cores, ~15 600 processes)
• Type: performance

Description

On a busy machine process_iter(attrs=[...]) is slow, and I wanted to find out where the time actually goes before proposing anything. The answer surprised me, so I am opening this before writing a real patch.

Setup: psutil master, Python 3.12, ~15 600 processes, 12 attributes (the set Glances asks for):
cpu_percent, cpu_times, memory_percent, name, status, num_threads, io_counters, memory_info, nice, pid, gids, cpu_num.

Where the time goes

In steady state psutil opens 4 files per process — stat, status, statm, io — and all four are genuinely needed for those attributes. There is no redundant read to remove. (On the first process_iter() call stat is read twice per process, because Process.__init__ calls create_time() outside the oneshot() window, but that is a one-off.)

So I measured the ceiling of "make I/O free": I preloaded every one of those files into a dict and monkeypatched open_binary/bcat to serve from memory, so the measured run does literally zero file I/O.

psutil as-is                       min=3036.6ms   1.00x
psutil with ALL file I/O removed   min=2269.6ms   1.34x

Even with I/O costing nothing, it is only 1.34x. Roughly three quarters of the time is the per-process Python machinery: creating a Process, as_dict(), the wrap_exceptions / memoize_when_activated decorator chain, and the namedtuple constructors.

That rules out the obvious fix (reading the files more cheaply, in C or otherwise) — it cannot pay off. The only way to a large win is to not build the per-process Python objects at all.

Proof of concept

I put a draft together to check that the ceiling is real:
https://github.com/Dipet/psutil/tree/perf/batch-process-scan-draft

It adds psutil/arch/linux/proc_scan.c, which reads stat/status/statm/io for every pid in one call with the GIL released, and a batch_scan() in _pslinux.py that process_iter() uses to prefill Process._prefetch. It falls back to the existing per-process path whenever the requested attrs are not fully covered, or for a process that appeared after the scan. Because the scan holds no GIL, it splits across a few threads (min(4, num_pids // 4000)).

process_iter(12 attrs), 15 623 procs, 4 interleaved pairs
  per-process path  median=3407.0ms
  batch path        median=1198.6ms
  speedup: 2.84x

Of the remaining 1.2s, 59% is the C scan and 41% is still Python. Two obvious leftovers: posix.listdir costs 0.19s because pids() walks /proc again after the scan already enumerated it, and 0.17s goes into namedtuple constructors.

I compared the resulting dicts against the per-process path for every process: the key sets are identical and the only differing values are ones that genuinely move between two samples (cpu_times off by one tick, cpu_num, kworkers renaming themselves). As a control, two consecutive process_iter() runs of unmodified psutil disagree with each other on those same fields at the same rate or higher.

The part I cannot decide myself

241 tests pass, but 8 in TestProcessIter fail, and they are not sloppiness on my side — they are a genuine collision:

test_attrs, test_emulate_access_denied, test_emulate_nsp,
test_prefetch_ad_value, test_prefetch_ad_value_is_not_none,
test_prefetch_derived_methods, test_prefetch_derived_username,
test_prefetch_memory_percent

They mock psutil._psplatform.Process.<method> to raise AccessDenied and assert that process_iter() substitutes ad_value. A batch path does not call those methods, so the mocks cannot take effect. Making them pass would mean routing through the platform methods again, which is exactly the cost being removed.

There is also a real gap underneath the test artifact: as_dict(ad_value=...) substitutes on AccessDenied, and the batch has no such layer. Today it returns None for an unreadable /proc/<pid>/io, which coincides with the default ad_value but is wrong for any other value. That part I can fix — the scan already distinguishes "could not read" from "read".

So before I invest in a mergeable patch: is a batch path that bypasses _psplatform.Process something you would consider at all, given it needs TestProcessIter to be reworked? If not, I would rather know now. If yes, I am happy to do the work — including whatever shape you prefer for the API (internal-only, opt-in, or transparent as in the draft).

Happy to provide the profiles or rerun anything on this machine.

[giampaolo]

I preloaded every one of those files into a dict and monkeypatched open_binary/bcat to serve from memory, so the measured run does literally zero file I/O.

What did you do exactly? Can you show the patch for this?

More in general: this aligns with #2463. There is space to speedup process_iter() by pre-collecting multiple info, similar to oneshot(), but system-wide instead of per-process.

Until now, I have assumed that the Python implementation on Linux was not a bottleneck, because reading and parsing /proc files in Python rather than C should not make a significant difference. In the end, it is mostly open(), read(), and close() syscalls, which map directly to their C counterparts in CPython. But I may be wrong.

More importantly, there should be only one /proc implementation, either in Python or in C. Maintaining two separate implementations is a no-go.

Even with I/O costing nothing, it is only 1.34x. Roughly three quarters of the time is the per-process Python machinery: creating a Process, as_dict(), the wrap_exceptions / memoize_when_activated decorator chain, and the namedtuple constructors.

That surprises me: wrap_exceptions / memoize_when_activated, etc. overhead should be absolutely negligible, unless we're talking about micro benchmarks here. What did you use for benchmarking (actual code)? It may also be worth it to use cProfile to check where most of the time is being spent.

More in general: I appreciate the research but I suggest not to make any PR about this yet. "system-wide oneshot" is something that should be clear how to solve architecturally first (aka how to wire up all the pieces into process_iter() and Process class in particular, so that it can work in "2 modes"). It's not an easy task and it's not Linux specific.

[Dipet]

Thanks for pushing back — you were right to, and the number I posted was wrong.

The 1.34x was a bug in my harness

To make I/O free I preloaded every /proc file into a dict and replaced open_binary/bcat with lookups. For files I could not read (/proc/<pid>/io of other users' processes) I stored the exception instance and re-raised it:

except OSError as err:
    store[path] = err
...
if isinstance(value, OSError):
    raise value        # <-- wrong

Re-raising the same instance appends a frame to its __traceback__ every single time, so on a box with 16 000 unreadable io files the raises get progressively more expensive. AccessDenied.__init__ showed up at 1.6 s of tottime for 16 002 calls, about 100 µs each, which is obviously nonsense for setting three attributes — that is what tipped me off.

Storing (type(err), err.args) and raising a fresh instance fixes it. Corrected numbers, unmodified master, 16 242 processes, Python 3.12, Linux:

process_iter(attrs=[12 attrs])
  as-is                  min=3366.9ms  median=3767.8ms
  all /proc I/O removed  min=1611.4ms  median=1838.0ms
  ceiling of making I/O free: 2.09x

So I/O is 52% of the time, not the ~25% I claimed. Your intuition was closer than my conclusion.

Where the time actually goes

Files opened per process in the steady state:

  stat         17015   1.05 per process
  status       16185   1.00 per process
  statm        16185   1.00 per process
  io           16185   1.00 per process
  TOTAL        65570   4.05 per process

(stat is 1.05 rather than 1.00 only because new processes appear mid-scan. On the first process_iter() call it is 2.0, because Process.__init__ calls create_time() outside the oneshot() window — a one-off, not worth chasing.)

cProfile, as-is:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    68922    0.928    0.000    0.932    0.000 {built-in method _io.open}
    33094    0.924    0.000    0.924    0.000 {method 'read' of '_io.BufferedReader' objects}
    16206    0.303    0.000    4.938    0.000 psutil/__init__.py:671(as_dict)
197713/178196    0.261    0.000    4.234    0.000 psutil/__init__.py:377(wrapper)
    16199    0.241    0.000    0.241    0.000 {built-in method posix.sysconf}
278937/165426    0.227    0.000    3.341    0.000 psutil/_pslinux.py:1549(wrapper)
        1    0.198    0.198    0.198    0.198 {built-in method posix.listdir}
    52825    0.159    0.000    0.159    0.000 {method '__exit__' of '_io._IOBase' objects}
162108/145909    0.114    0.000    2.435    0.000 psutil/_common.py:253(wrapper)
    32412    0.112    0.000    0.263    0.000 psutil/__init__.py:609(oneshot)

cProfile, with all /proc I/O served from memory:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    16194    0.552    0.000    2.894    0.000 psutil/__init__.py:671(as_dict)
197024/177574    0.213    0.000    2.010    0.000 psutil/__init__.py:377(wrapper)
    16132    0.210    0.000    0.210    0.000 {built-in method posix.sysconf}
277868/164821    0.186    0.000    1.267    0.000 psutil/_pslinux.py:1549(wrapper)
        1    0.181    0.181    0.181    0.181 {built-in method posix.listdir}
    32388    0.093    0.000    0.220    0.000 psutil/__init__.py:609(oneshot)
161443/145311    0.084    0.000    0.586    0.000 psutil/_common.py:253(wrapper)
    16132    0.082    0.000    0.115    0.000 psutil/__init__.py:1377(memory_percent)
    32264    0.078    0.000    0.123    0.000 psutil/_pslinux.py:1789(cpu_times)
    16132    0.076    0.000    0.197    0.000 psutil/_pslinux.py:1827(memory_info)

On the decorators specifically: _use_prefetch, wrap_exceptions and memoize_when_activated together are about 0.48 s of the 1.61 s I/O-free run, so roughly 14% of the original 3.37 s. Not free, but you are right that they are not the story. Listing them alongside the real cause in my first post was sloppy of me.

The cost is dominated by the sheer number of open()s: 65 570 of them. Splitting that further, on this box 15 000 files cost 366 ms through raw os.open/os.read/os.close and 482 ms through builtin open() — so the io module wrapper is about a quarter of it and the syscalls themselves are the rest.

What that changes

It means my C batch scanner was aimed at the wrong thing, and your "only one /proc implementation" rule is the right call regardless. Most of the 2.09x is reachable by reading fewer files per process, which is exactly the system-wide oneshot() idea in #2463 and needs no C at all. I withdraw the draft as a merge candidate and will not open a PR for it.

One small thing visible in the profile above and unrelated to all this: posix.sysconf at 16 199 calls, 0.24 s, is cpu_count() being called once per process from Process.cpu_percent(). It is a separate one-liner, #2955.

The script

Self-contained, run it against an unmodified checkout with python3 repro.py.

repro.py

"""Where does process_iter(attrs=...) spend its time on a machine with many processes?

Run it against an unmodified psutil checkout:

    python3 psutil_repro.py

It measures three things:

  1. process_iter(attrs=...) as-is.
  2. The same run with every /proc file served from a dict preloaded beforehand, so the
     measured loop performs no syscalls and allocates no file objects at all. The gap
     between 1 and 2 is everything psutil could possibly gain by making I/O cheaper.
  3. cProfile of both, so the remaining cost can be attributed.

It also counts how many files are opened per process.
"""

import cProfile
import io
import os
import pstats
import statistics
import time

import psutil
import psutil._common as _common
import psutil._pslinux as pl

# The attribute set Glances asks for; a realistic "give me everything for a process table".
ATTRS = [
    'cpu_percent',
    'cpu_times',
    'memory_percent',
    'name',
    'status',
    'num_threads',
    'io_counters',
    'memory_info',
    'nice',
    'pid',
    'gids',
    'cpu_num',
]

PROC_FILES = ('stat', 'status', 'statm', 'io')


def run():
    return [p.info for p in psutil.process_iter(attrs=ATTRS, ad_value=None)]


def timeit(fn, n=3):
    fn()  # warm the pid map, so we measure the steady state and not first-call setup
    times = []
    for _ in range(n):
        started = time.perf_counter()
        result = fn()
        times.append(time.perf_counter() - started)
    return min(times), statistics.median(times), result


def profile(fn, label, lines=10):
    profiler = cProfile.Profile()
    profiler.enable()
    fn()
    profiler.disable()
    stream = io.StringIO()
    pstats.Stats(profiler, stream=stream).sort_stats('tottime').print_stats(lines)
    print(f"\n----- cProfile: {label} -----")
    print('\n'.join(stream.getvalue().splitlines()[4:]))


def count_opens():
    """How many files does psutil open per process, in the steady state?"""
    counts = {}
    real = _common.open_binary

    def counting(fname, *args, **kwargs):
        name = str(fname).rsplit('/', 1)[-1]
        counts[name] = counts.get(name, 0) + 1
        return real(fname, *args, **kwargs)

    _common.open_binary = counting
    pl.open_binary = counting
    _common.bcat.__globals__['open_binary'] = counting
    try:
        run()  # first pass creates the Process objects
        counts.clear()
        procs = run()  # steady state
    finally:
        _common.open_binary = real
        pl.open_binary = real
        _common.bcat.__globals__['open_binary'] = real
    total = sum(counts.values())
    print(f"\n----- files opened per process, steady state ({len(procs)} processes) -----")
    for name, count in sorted(counts.items(), key=lambda kv: -kv[1]):
        print(f"  {name:10s} {count:7d}   {count / len(procs):.2f} per process")
    print(f"  {'TOTAL':10s} {total:7d}   {total / len(procs):.2f} per process")


class Preloaded:
    """Stands in for an open file. Built once per path, so serving it costs no allocation."""

    __slots__ = ('data', 'lines')

    def __init__(self, data):
        self.data = data
        self.lines = data.splitlines(True)

    def __enter__(self):
        return self

    def __exit__(self, *args):
        return False

    def read(self, size=-1):
        return self.data

    def readline(self):
        return self.lines[0] if self.lines else b''

    def __iter__(self):
        return iter(self.lines)


def preload():
    """Read every file psutil will ask for, and wrap each in a ready-made fake handle."""
    store = {}
    for entry in os.listdir('/proc'):
        if not entry.isdigit():
            continue
        for name in PROC_FILES:
            path = f"/proc/{entry}/{name}"
            try:
                with open(path, 'rb') as handle:
                    data = handle.read()
            except OSError as err:
                # Store the class and args, never the instance: re-raising one instance
                # appends a frame to its __traceback__ every time, so the raises get
                # progressively more expensive and poison the measurement.
                store[path] = (type(err), err.args)
            else:
                store[path] = data
    return store


def install_preloaded(store):
    """Serve every /proc read from memory: no syscall, no file object, parsing untouched."""
    handles = {k: (v if isinstance(v, tuple) else Preloaded(v)) for k, v in store.items()}
    real_open_binary = _common.open_binary
    real_bcat = _common.bcat

    def fake_open_binary(fname, *args, **kwargs):
        value = handles.get(str(fname))
        if value is None:
            return real_open_binary(fname, *args, **kwargs)  # appeared after the snapshot
        if isinstance(value, tuple):
            raise value[0](*value[1])
        return value

    def fake_bcat(fname, fallback=_common._DEFAULT):
        value = store.get(str(fname))
        if value is None:
            with open(fname, 'rb') as handle:
                return handle.read()
        if isinstance(value, tuple):
            if fallback is not _common._DEFAULT:
                return fallback
            raise value[0](*value[1])
        return value

    _common.open_binary = fake_open_binary
    pl.open_binary = fake_open_binary
    _common.bcat = fake_bcat
    pl.bcat = fake_bcat


def main():
    print(f"psutil {psutil.__version__}  python {'.'.join(map(str, os.sys.version_info[:3]))}")
    print(f"{len(psutil.pids())} processes, {os.cpu_count()} logical CPUs")

    count_opens()

    base_min, base_med, procs = timeit(run)
    print(f"\n----- process_iter(attrs=[{len(ATTRS)} attrs]) -----")
    print(f"  as-is                  min={base_min * 1000:8.1f}ms  median={base_med * 1000:8.1f}ms  n={len(procs)}")
    profile(run, "as-is")

    install_preloaded(preload())
    free_min, free_med, procs = timeit(run)
    print(f"\n  all /proc I/O removed  min={free_min * 1000:8.1f}ms  median={free_med * 1000:8.1f}ms  n={len(procs)}")
    print(f"\n  ceiling of making I/O free: {base_min / free_min:.2f}x")
    profile(run, "with /proc I/O removed")


if __name__ == '__main__':
    main()

Happy to run any variation of this on the same machine if it would help settle the architecture question.

[giampaolo]

I see no difference in commenting here vs. talking with claude code in my terminal. Issue trackers should be for humans not copy/pasting stuff from the agent as-is. Real discussion. Real effort. Not this. Seriously I can simply launch claude on a "cProfile quest" when I feel like it.
No offense, but closing.