[Performance] ModelPatcher creates a new namedtuple class for every weight backup

[HelloClyde]

Custom Node Testing

• I have tried disabling custom nodes and the issue persists (see how to disable custom nodes if you need help)

Expected Behavior

ModelPatcher.backup entries should reuse one tuple type. For N backed-up weights:

len(self.backup) == N
len({type(entry) for entry in self.backup.values()}) == 1

Creating and clearing backup entries should not generate a new class and its associated cyclic objects for every weight.

Actual Behavior

comfy/model_patcher.py calls collections.namedtuple() inside two per-key hot paths. Because collections.namedtuple() creates a new class on every call, ComfyUI creates one distinct Dimension type per backed-up weight:

self.backup[key] = collections.namedtuple(
    "Dimension", ["weight", "inplace_update"]
)(weight.to(device=self.offload_device, copy=inplace_update), inplace_update)

and:

self.backup[key] = collections.namedtuple(
    "Dimension", ["weight", "inplace_update"]
)(weight, False)

Therefore:

len({type(entry) for entry in self.backup.values()}) == len(self.backup)

Each generated class brings its own methods, closure cells, descriptors, staticmethod, classmethod, and two _tuplegetter objects. Repeated model patch/load/unload cycles generate large amounts of class-shaped cyclic garbage.

In one 178.1-second production observation window, the GC-tracked heap grew with the exact signature of approximately 925 newly generated two-field namedtuple classes:

total tracked    +65,105
type                 +931
staticmethod         +925
classmethod          +925
_tuplegetter        +1,850
function            +5,568
cell                +8,365
tuple              +23,049
frozenset          +17,550

Gen 2 collections:          +13
Objects collected by Gen 2: +310,277
Average per Gen 2:          ~23,867

In an isolated reproduction using the actual upstream assignment, 2,000 backup entries created 2,000 distinct classes. Clearing the backup made Gen 2 collect approximately 58,000–62,000 class-related cyclic objects, depending on the Python version. The shared-type implementation created one class and no per-entry class-related cyclic garbage.

Steps to Reproduce

This reproduces the core allocation behavior on a clean ComfyUI checkout without loading custom nodes or requiring a workflow/model file:

1. Check out current ComfyUI master and activate its Python environment.

2. Save the following as /tmp/reproduce_model_patcher_namedtuple_gc.py:

   import gc
   import torch
   
   from comfy.model_patcher import ModelPatcher
   
   
   model = torch.nn.Module()
   keys = []
   for index in range(1000):
       key = f"weight_{index}"
       keys.append(key)
       model.register_parameter(
           key,
           torch.nn.Parameter(torch.ones(1)),
       )
   
   patcher = ModelPatcher(
       model,
       load_device=torch.device("cpu"),
       offload_device=torch.device("cpu"),
   )
   
   # Exclude the interpreter/import baseline from the cleanup measurement.
   gc.collect()
   gc.freeze()
   gc.disable()
   
   for key in keys:
       patcher.patch_weight_to_device(key, force_cast=True)
   
   print("backup entries:", len(patcher.backup))
   print(
       "unique backup types:",
       len({type(entry) for entry in patcher.backup.values()}),
   )
   
   patcher.backup.clear()
   print("gen2 collected after clear:", gc.collect(2))

3. Run it from the ComfyUI repository root:

   python /tmp/reproduce_model_patcher_namedtuple_gc.py

4. Current upstream behavior is approximately:

   backup entries: 1000
   unique backup types: 1000
   gen2 collected after clear: ~29,000-31,000
   

5. With the proposed shared module-level type:

   backup entries: 1000
   unique backup types: 1
   gen2 collected after clear: 0 class-related objects
   

The issue is in ComfyUI core and the clean reproduction does not import custom nodes. The production workload did contain custom nodes, but they are not needed to reproduce the class-per-backup behavior.

Debug Logs

These production logs motivated the investigation. They demonstrate the symptom but, as explained under Other, subsequent testing found an independent execution-cache factor; the absolute pause duration should not be attributed solely to the inline namedtuple calls.

Pre-fix examples:

2026-08-11 09:40:58.853 | WARNING | req_id:- | hooks.gc_trace:_gc_callback:73 - slow gc_trace | gen:2 cost:15379.6ms collected:26200 uncollectable:0 counts:(1, 0, 0)
2026-08-11 15:03:54.158 | WARNING | req_id:- | hooks.gc_trace:_gc_callback:73 - slow gc_trace | gen:2 cost:17517.8ms collected:31913 uncollectable:0 counts:(2, 0, 0)

Summary of 3,495 consecutive pre-fix Gen 2 collections over 24 hours:

time range: 2026-08-10 11:03:08.072 -> 2026-08-11 11:02:54.467
samples:    3495
min:        6498.3 ms
mean:       7351.1 ms
median:     7344.7 ms
p90:        7954.3 ms
p95:        8033.6 ms
p99:        8183.6 ms
max:        8455.0 ms
max collected: 47808
uncollectable total: 0

The hourly median increased from 6,636.4 ms to 8,042.6 ms during that 24-hour log.

Post-fix examples from a newly started worker:

2026-08-12 10:34:42.815 | WARNING | req_id:- | hooks.gc_trace:_gc_callback:73 - slow gc_trace | gen:2 cost:485.3ms collected:948 uncollectable:0 counts:(1, 0, 0)
2026-08-12 10:40:14.053 | WARNING | req_id:- | hooks.gc_trace:_gc_callback:73 - slow gc_trace | gen:2 cost:493.5ms collected:996 uncollectable:0 counts:(1, 0, 0)

Summary of 100 consecutive post-fix collections:

samples: 100
min:     437.3 ms
median:  480.1 ms
p95:     513.0 ms
max:     520.1 ms
last:    517.9 ms
max collected: 1313

This production before/after comparison is observational rather than a controlled benchmark because deploying the fix also restarted the worker and reset its execution cache. The controlled evidence for this issue is the class count and isolated GC-object count in Steps to Reproduce and Other.

The GC timer uses gc.callbacks and measures wall time between the start and stop phases with time.perf_counter(). The collected and uncollectable values come directly from the callback's info argument.

Other

Upstream revision used for reproduction

The two inline allocations were verified on upstream commit 27bca654eb9a70237d93f56a6ea336ab55f8925d on 2026-08-12:

• ComfyUI/comfy/model_patcher.py

  Line 907 in 27bca65

  self.backup[key] = collections.namedtuple('Dimension', ['weight', 'inplace_update'])(weight.to(device=self.offload_device, copy=inplace_update), inplace_update)

• ComfyUI/comfy/model_patcher.py

  Line 1976 in 27bca65

  self.backup[key] = collections.namedtuple('Dimension', ['weight', 'inplace_update'])(weight, False)

The issue was originally validated against commit 62b3c94bd45154f6486c7abf1b9efcacee96ea69 and reproduced on Python 3.9.6, 3.11.15, and 3.12.13.

Investigation update: independent execution-cache growth

Further investigation found a separate source of full-GC growth in the production service. The default RAM-pressure execution cache retained input-signature keys composed primarily of tuples and frozensets until memory pressure caused eviction.

On the same worker running the shared-type fix, the execution cache was allowed to grow and was then cleared through ComfyUI's /free endpoint:

Metric	Before clearing execution cache	After clearing execution cache
GC-tracked objects	1,839,066	152,673
tuple	876,414	35,960
frozenset	873,963	28,355
Gen 2 GC duration	approximately 624 ms	44.9 ms

The removed tuples and frozensets accounted for 1,686,062 of the 1,686,393-object decrease. Meanwhile, the tracked type count remained stable at 887 across repeated post-fix workflows, and _tuplegetter showed no growth. This confirms that the per-backup class creation stopped, while also showing that execution-cache state independently affects absolute full-GC duration.

Consequently, the earlier production before/after timing cannot be used to claim that the inline namedtuple construction alone caused the 15–17 second pauses. This report is intentionally limited to the independently reproducible class-per-backup allocation and its associated cyclic-GC churn.

Proposed fix

Create the type once at module scope while preserving the existing tuple behavior, typename, and .weight / .inplace_update attributes:

ModelPatcherBackup = collections.namedtuple(
    "Dimension",
    ["weight", "inplace_update"],
)

Then use it at both call sites:

self.backup[key] = ModelPatcherBackup(
    weight.to(
        device=self.offload_device,
        copy=inplace_update,
    ),
    inplace_update,
)

and:

self.backup[key] = ModelPatcherBackup(weight, False)

A regression test creates two backup entries through patch_weight_to_device() and asserts that both are instances of ModelPatcherBackup and share exactly one type.

Additional diagnostics

An isolated benchmark on the affected production Python 3.11.15 build produced the following results:

Backup entries	Implementation	Unique types	Live Gen 2 GC	Cleanup Gen 2 GC	Cleanup collected
925	Current source	925	3.03 ms	5.60 ms	26,825
925	Shared type	1	0.04 ms	0.00 ms	0
2,000	Current source	2,000	7.53 ms	17.01 ms	58,000
2,000	Shared type	1	0.08 ms	0.00 ms	0
10,000	Current source	10,000	54.80 ms	103.69 ms	290,000
10,000	Shared type	1	0.36 ms	0.00 ms	0

The production log's common collected-object counts closely match the benchmark's approximately 29 cyclic objects per generated class. For example, 925 generated types produced 26,825 collectible objects.

The production deployment was followed by a large reduction in Gen 2 duration, but that comparison also included a process restart and execution-cache reset. It therefore cannot isolate the timing improvement caused by this change. The controlled benchmark establishes the narrower claim: the current implementation creates one class and approximately 29 cyclic objects per backup entry, whereas the shared-type implementation eliminates that per-entry class allocation and associated cyclic garbage.

A /proc sampler and a py-spy --gil --native profile found the observed slow collections CPU-bound in CPython's gc_collect_main, without cgroup throttling, memory/IO pressure, or Python-level model finalizers.

Environment:

Python: 3.11.15
PyTorch: 2.8.0+cu128
OS: Linux container
Pre-fix worker uptime at initial observation: approximately 6 days
cpu.cfs_quota_us: -1
nr_throttled: 0
throttled_time: 0
swap: 0
oom_kill: 0

[coderabbitai]

🔗 Related PRs

#14413 - model_patcher: skip synthetic quant keys in get_key_patches [open]
#15456 - Free other models' memory before retrying handles_tiling VAE decode [open]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[HelloClyde]

Investigation update

Further investigation found an additional, independent source of Gen 2
GC growth: the default RAM-pressure execution cache retains input-signature
keys composed primarily of tuples and frozensets until memory pressure causes
eviction.

On the same patched process, clearing the execution cache reduced:

• tracked objects from 1,839,066 to 152,673
• tuples from 876,414 to 35,960
• frozensets from 873,963 to 28,355
• Gen 2 GC duration from approximately 624 ms to 44.9 ms

Therefore, the original production before/after comparison was affected by
both the ModelPatcher fix and execution-cache/process state. The 15–17 second
pauses should not be attributed solely to the inline namedtuple construction.

The core issue reported here remains independently reproducible: every call
to collections.namedtuple() creates a new class. With 925 backup entries, the
current implementation creates 925 distinct types and approximately 26,825
class-related cyclic objects, while a shared module-level type creates one
type and no per-entry class-related cyclic garbage.