cuda_ipc: imported remote mappings are never released until importer process exit (no per-peer eviction) — pins exporter VMM/fabric memory domain-wide

[404mario]

Environment (loaded binary → source)

• UCX 1.21.0 (in-process ucp_get_version_string() on the loaded
  libucp; the process maps the bundled libuct_cuda.so, not a system UCX).
• Loaded DSO SHA-256 (full values below), bundled via NIXL 1.3.2 wheel:
  • libucs … fa900fe7c8d685ee70407ab681c655c25404b3e4e01145602df2422ac6cd59e3
  • libucp … 2c200186a59932afc88a8059527df5aae7c524860781b62c9b50719af4296475
  • libuct … 0f13475685a1b7b1422e31eec7f89310baba98da5f3e300cd78c2d2f34310cdf
  • ucx/libuct_cuda … 8dbeb2dedd5b1efb0c60335fca259acc21cee2ae7e165d922201285e6ccaafc3
• Source referenced against tag v1.21.0
  (tag → commit b6a9d47fccce849c28111f05a7fa8f1c930ff17d),
  src/uct/cuda/cuda_ipc/cuda_ipc_cache.c (file SHA-256 prefix e9594315…).
• Hardware: multi-node NVLink (MNNVL, fabric handles via IMEX);
  CU_MEM_HANDLE_TYPE_FABRIC VMM allocations transferred cross-node over the
  cuda_ipc transport. Transfer bytes did not appear on any NIC counter during a
  cross-node copy (consistent with an on-fabric NVLink path, i.e. VMM cuda_ipc
  dispatch — not directly confirmed at the handle-type/dispatch level).

Caveat on provenance: the above proves runtime binary identity + a matching
source tag; it does not prove the bundled .so is bit-identical to an
unmodified tag build. Anyone reproducing on a from-source v1.21.0 build is
welcome to confirm.

Symptom

An importer that has mapped a peer's CU_MEM_HANDLE_TYPE_FABRIC (VMM) memory
over the cuda_ipc transport keeps that mapping alive for its entire process
lifetime. Releasing every handle the transport/consumer layer exposes does
not free it; the exporter-side allocation therefore stays resident on the
exporter GPU for as long as any importer process is alive — even after the
importer has finished all transfers and torn down its endpoints/agent.

At fleet scale this is a real operational hazard: tearing down a subset of a
fabric memory domain leaves exporter-GPU memory (in our case ~200 GiB/GPU)
pinned and unreclaimable until the surviving importer processes on other nodes
exit.

Minimal reproduction (2 process, in-process, no framework)

Two small Python processes (exporter allocates a CU_MEM_HANDLE_TYPE_FABRIC
VMM buffer, registers it; importer maps it via one cuda_ipc READ, then walks a
graded teardown ladder while staying alive). Per-step exporter-GPU HBM sampled
with nvidia-smi, real rc per step. Scripts + raw logs:
lifecycle_repro*.py, lifecycle_{E,v5}.sh, lifecycle_*/controller.log.

Graded destruction ladder inside a LIVING importer (exporter killed first;
exporter-GPU HBM, serving baseline 237.7 GiB, +8–9 GiB test buffer):

step (importer stays alive)	exporter HBM
release transfer handle	pinned (~245.9 GiB)
remove remote agent	pinned
deregister own memory	pinned
destroy entire agent object + gc	pinned
importer process exit	freed (→ 237.7 GiB)

Also measured with UCX_CUDA_IPC_CACHE=n: still pinned until process exit
(env recognition positive-controlled — a bogus UCX_* var warns "unused",
this one does not, so the toggle is parsed). A plausible reading, consistent
with the source but not proven by this experiment alone, is that the
consumer-layer teardown never reaches the unmap path so the cache toggle has
nothing to act on.

Where it comes from (source reading, v1.21.0 cuda_ipc_cache.c)

• Imported mappings live in a module-global cache
  uct_cuda_ipc_remote_cache (hash + recursive spinlock), keyed by the remote
  {pid, pid_ns, cu_device} triple.
• The import path (uct_cuda_ipc_map_memhandle → ..._open_memhandle_vmm:
  cuMemImportFromShareableHandle + cuMemAddressReserve + cuMemMap +
  cuMemSetAccess) inserts into that cache with refcount semantics.
• Hash entries are destroyed only in UCS_STATIC_CLEANUP (library
  unload / process exit). Endpoint/worker/context/md teardown never removes a
  peer's entry. uct_cuda_ipc_cache_purge is static; the LRU capacity path
  is an insert-pressure valve, not a peer-close path.
• There is no public API to evict one peer's cached mappings. So a consumer
  that has finished with a peer (peer gone, or last endpoint closed) has no way
  to release the imported mappings short of exiting.

Confidence: the symptom (release-only-at-process-exit) is directly measured
and reproducible; the "module-global cache, static-cleanup-only" mechanism is
from source reading of the matching tag and is consistent with the
measurements, but I have not instrumented the running binary at that line
(this libucs build has ucs_trace compiled out). Independent confirmation
on a from-source build would be welcome.

Proposed direction (for maintainer feedback — not a submitted patch)

A per-peer eviction entry point, callable by a transport/consumer that knows a
peer is done, e.g.:

ucs_status_t uct_cuda_ipc_remote_cache_evict(pid_t pid, ucs_sys_ns_t pid_ns,
                                             int cu_device);

Design notes we've worked through (full doc: UCX_PURGE_API_DESIGN.md):

• In-place region eviction, cache object stays in the hash — avoids both a
  use-after-free of the cache object and the unmap-relookup-by-key hazard (the
  existing unmap path re-looks-up the cache by triple key and asserts the
  region is present, so detach-and-replace of the whole cache object is unsafe).
• Drain-before-evict precondition (caller closes admission for the peer and
  waits for in-flight region refcounts to reach 0), so eviction is a clean
  two-phase scan-then-remove: phase 1 under the cache lock verifies all regions
  are idle (else UCS_ERR_BUSY, no side effect); phase 2 removes from pgtable
  and closes handles after dropping the lock (no CUDA call under any lock).
• Idempotent, per-peer isolation (only that key's pgtable touched), triple
  key so container PID-ns / PID reuse is disambiguated.

Open questions for maintainers:

1. Is a per-peer eviction API the right shape, or is there an existing
   mechanism I've missed (beyond LRU capacity limits and static cleanup)?
2. Ownership/lifetime: does UCX expect the transport md/ep teardown to drive
   this, or a new explicit call from the consumer (NIXL, in our case)?
3. The separate uct_cuda_ipc_rem_mpool_cache (fabric mempool path) has the
   same static-cleanup-only lifetime — should a fix cover both?

I'm happy to prepare a PR with CPU-race unit tests (TSAN/ASan) and a
compute-sanitizer'd 2-process test if the direction is acceptable. Attaching
the minimal repro, per-step HBM logs, source-anchored analysis, and the API
design doc.

[404mario]

Attaching a self-contained 2-process reproduction (GitHub comments don't take file uploads, so inlining — copy/paste runs).

Dependencies / caveat: the consumer here is NIXL 1.3.2 (which uses the UCX cuda_ipc transport underneath) and a small CU_MEM_HANDLE_TYPE_FABRIC VMM allocator (vllm_fabric_kv.alloc_kv_tensor). So this exercises UCX cuda_ipc through NIXL rather than pure uct calls. The point it isolates: the imported mapping survives every teardown NIXL exposes (release transfer handle → remove remote agent → deregister → destroy the whole agent object + gc) and is reclaimed only at importer process exit. This is consistent with (and motivates checking) a UCX-side per-peer cache with no eviction entry point reachable from the consumer layer — it does not by itself prove that location. A maintainer with a pure-uct harness could check the same at the transport layer.

How to run (2 nodes sharing an MNNVL/IMEX fabric domain, GPU0 each; NODE_1 = exporter host):

• exporter: python lifecycle_repro.py exporter --gb 2 --port 18600
• importer (other node): python lifecycle_repro.py importer --peer <exporter-ip> --port 18600 --mode ladder
• watch exporter-GPU nvidia-smi memory across the ladder steps; it stays pinned until the importer process exits.

lifecycle_repro5.py (strict-fabric + nonzero-pattern + end-to-end checksum variant)

"""P0 repro v5 — strict fabric + nonzero pattern + end-to-end checksum + honest
terminal states. Exits NONZERO on any failure (ERR state, checksum mismatch,
fabric fallback under strict). Negative-control flags let the harness prove the
checker actually rejects failures."""
import argparse, hashlib, os, pickle, socket, struct, sys, time
import torch

def log(*a): print(time.strftime("%H:%M:%S"), *a, flush=True)
def die(msg, code=3): log("FAIL:", msg); sys.exit(code)

p = argparse.ArgumentParser()
p.add_argument("role", choices=["exporter", "importer"])
p.add_argument("--gb", type=float, default=2.0)
p.add_argument("--port", type=int, default=18600)
p.add_argument("--peer", default="")
p.add_argument("--dev", type=int, default=0)
p.add_argument("--corrupt", action="store_true", help="NEG: importer flips a byte before hashing -> checksum mismatch")
p.add_argument("--exit-code", type=int, default=0, help="NEG: importer force-exits with this code post-READ")
p.add_argument("--bad-size", action="store_true", help="NEG: importer requests wrong xfer size -> ERR")
args = p.parse_args()

torch.cuda.set_device(args.dev)
n = int(args.gb * (1 << 30))
STRICT = os.environ.get("VLLM_FABRIC_STRICT") == "1" or os.path.exists("$CLUSTER_ROOT/FABRIC_STRICT")

def pattern(nb):
    return (torch.arange(nb, dtype=torch.int64, device=f"cuda:{args.dev}") % 251).to(torch.int8)
def sha(t): return hashlib.sha256(t.cpu().numpy().tobytes()).hexdigest()

from vllm_fabric_kv import alloc_kv_tensor, fabric_enabled
if STRICT and not fabric_enabled():
    die("STRICT set but USE_FABRIC sentinel absent", 4)
buf = alloc_kv_tensor(n, torch.device(f"cuda:{args.dev}"))
# assert the allocation actually went fabric (marker check), not silent zeros fallback
mk = "/tmp/fabric_kv_marker.log"
if STRICT:
    tail = open(mk).read().splitlines()[-8:] if os.path.exists(mk) else []
    if any("FABRIC-FALLBACK" in l for l in tail): die("FABRIC-FALLBACK under strict", 4)
    if not any("FABRIC-OK" in l for l in tail): die("no FABRIC-OK marker under strict", 4)
log(f"[{args.role}] fabric buf {args.gb}GiB ptr=0x{buf.data_ptr():x} strict={STRICT}")

from nixl._api import nixl_agent, nixl_agent_config
agent = nixl_agent(f"lc5-{args.role}-{os.getpid()}", nixl_agent_config(backends=["UCX"]))
reg = agent.get_reg_descs([(buf.data_ptr(), n, args.dev, "")], "VRAM")
agent.register_memory(reg)
xfer = agent.get_xfer_descs([(buf.data_ptr(), n, args.dev)], "VRAM")

def rx(c,k):
    d=b""
    while len(d)<k:
        x=c.recv(k-len(d))
        if not x: raise RuntimeError("eof")
        d+=x
    return d
def sendo(c,o): b=pickle.dumps(o); c.sendall(struct.pack("!I",len(b))+b)
def recvo(c): return pickle.loads(rx(c,struct.unpack("!I",rx(c,4))[0]))

if args.role == "exporter":
    buf.copy_(pattern(n)); torch.cuda.synchronize()
    expect = sha(buf)
    srv=socket.socket(); srv.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
    srv.bind(("0.0.0.0",args.port)); srv.listen(1)
    log(f"[exporter] READY sha={expect[:16]}")
    c,_=srv.accept()
    sendo(c,{"meta":agent.get_agent_metadata(),"descs":agent.get_serialized_descs(xfer),
             "expect":expect,"n":n})
    log("[exporter] metadata+expect sent; holding")
    while True: time.sleep(5)
else:
    c=socket.socket(); c.connect((args.peer,args.port))
    rem=recvo(c)
    rname=agent.add_remote_agent(rem["meta"])
    if isinstance(rname,bytes): rname=rname.decode()
    rdescs=agent.deserialize_descs(rem["descs"])
    if args.bad_size:  # NEG: local xfer desc smaller than remote -> invalid param / ERR
        xfer=agent.get_xfer_descs([(buf.data_ptr(), n//2, args.dev)], "VRAM")
    h=agent.initialize_xfer("READ",xfer,rdescs,rname,b"lc5")
    _t0=time.time()
    agent.transfer(h)
    st=None
    while True:
        st=agent.check_xfer_state(h)
        if st in ("DONE","ERR"): break
        time.sleep(0.005)
    _dt=time.time()-_t0
    log(f"[importer] READ-DONE state={st} bytes={n} ms={_dt*1000:.1f} MBps={n/_dt/1e6:.1f}")
    if st != "DONE": die(f"xfer state={st}", 5)   # ERR is NOT a success terminal
    got=sha(buf)
    if args.corrupt:
        b=bytearray(buf[:1].cpu().numpy().tobytes()); b[0]^=0xFF  # flip -> mismatch
        got=hashlib.sha256(bytes(b)+b"corrupt").hexdigest()
    match = got == rem["expect"]
    log(f"[importer] CHECKSUM {'MATCH' if match else 'MISMATCH'} got={got[:16]} expect={rem['expect'][:16]}")
    if not match: die("checksum mismatch", 6)
    if args.exit_code: die(f"forced exit-code {args.exit_code}", args.exit_code)
    agent.release_xfer_handle(h); agent.remove_remote_agent(rname); agent.deregister_memory(reg)
    log("[importer] CLEAN-EXIT-OK"); sys.exit(0)

controller.log — negative-control harness (GREEN passes; 4 injected failures each correctly propagate non-zero)

=== lifecycle_v5 start 2026-08-11T08:36:25Z ===
GREEN importer_rc=0
NEG1_err importer_rc=1
NEG1 expected-fail OK
/tmp/lifecycle_v5.sh: line 29: 1255911 Killed                  setsid env $ENVP $V/bin/python /tmp/lifecycle_repro5.py exporter --gb 2 --port 18600 > $OUT/$name.exporter.log 2>&1
NEG2_noready importer_rc=1
NEG2 expected-fail OK
NEG3_corrupt importer_rc=6
NEG3 expected-fail OK
NEG4_rc importer_rc=7
NEG4 expected-fail OK
DIFF cudamalloc importer_rc=4 (fabric GREEN rc was 0)
=== lifecycle_v5 done STATUS=PASS fails=[] ===

Correction on the NEG* labels (matching the raw rc's shown): the four negative controls demonstrate that the harness propagates non-zero rather than only ever printing green, but to be precise about what each actually exercised — NEG1 (--bad-size, rc=1): a NIXL_ERR_INVALID_PARAM raised at initialize_xfer (a pre-transfer param rejection, not a check_xfer_state()==ERR terminal). NEG2 (rc=1): the exporter was killed before serving, so the importer fails to connect (a connection error, not a clean controller-level missing-READY assertion). NEG3 (--corrupt, rc=6): end-to-end checksum mismatch correctly caught. NEG4 (--exit-code, rc=7): forced non-zero child rc. The DIFF cudamalloc importer_rc=4 line in the log is not a cudaMalloc bandwidth differential either — rc=4 is a strict-mode guard rejecting the non-fabric allocation before any transfer ran. A genuine check_xfer_state()==ERR terminal, a controller-level READY-timeout gate, and an actual cudaMalloc-vs-fabric bandwidth comparison were only added in a later harness iteration not shown in this log. The load-bearing evidence for this issue is the pinning ladder above, not the NEG coverage.

[404mario]

Follow-up: two maintainer decisions that gate a PR for this issue

I have a drafted per-peer eviction change for the cuda_ipc remote-cache
lifetime problem above, plus a CPU stub-model test (contract semantics, TSAN/
ASan-clean) and a design writeup for the consumer (NIXL) side. Before I open a
PR I'd like a maintainer steer on two decisions, because each determines the
function signature and where the drain/UCS_ERR_BUSY contract lives. I don't
want to guess the layering and send a PR shaped the wrong way.

The draft primitive (internal shape) is:

ucs_status_t uct_cuda_ipc_remote_cache_evict(pid_t pid, ucs_sys_ns_t pid_ns,
                                             CUdevice cu_device);

with drain-before-evict semantics: caller closes admission + drains the peer;
UCS_ERR_BUSY (any live refcount) is a zero-side-effect no-op; a clean scan
purges the peer's regions in place; the cache object itself is never removed
from the module-global hash (so a concurrent unmap re-lookup by triple always
resolves to the same object); a CUDA-close failure is reported as a partial
teardown rather than a false UCS_OK.

Decision 1 — API ownership / reachability

In v1.21.0, cuda_ipc_cache.h is in noinst_HEADERS and the implementation is
in the separate libuct_cuda module; a UCP consumer (e.g. NIXL) links
-lucp -lucs -luct, not -luct_cuda, and does not hold the cuda_ipc MD/iface.
So a bare new symbol in that internal header is neither a stable public API nor
guaranteed link/load-reachable by the consumer. Which boundary do you prefer?

• (a) Transport-teardown self-driven — no new public symbol; eviction is
  driven internally from an existing teardown path. My concern: the
  module-global cache is shared across endpoints and (in v1.21.0)
  uct_cuda_ipc_ep_t carries only remote_pid (the key also needs pid_ns +
  CUdevice), and there is no last-owner refcount — so a single ep/iface/md
  close doesn't obviously mean "this peer's imports are done." Is there a
  reliable last-owner lifecycle point I've missed, or would you accept adding a
  per-peer last-owner refcount so one can be defined?
• (b) A UCP-reachable explicit cleanup hook that UCP routes internally down
  to cuda_ipc — so the consumer calls installed UCP/UCT surface it already
  links, without linking libuct_cuda. (I could not find an existing public
  uct_md_control/uct_iface_control entry to ride in v1.21.0; if there is
  one, pointing me at it is all I need.)
• (c) A formally installed CUDA-specific public API — promote the symbol to
  an installed header with an explicit export/link story. Heaviest ABI burden;
  listed for completeness.

If the answer is "(b), extend this op" or "(a), hook this event / yes add a
last-owner count," that's enough for me to shape the whole PR.

Decision 2 — close-failure resource policy

Once a region is detached and freed, a failed cuMemUnmap/cuIpcCloseMemHandle
leaves an untracked driver mapping — there's no handle left to retry.
Returning the error (instead of a false UCS_OK) is necessary but not closure.
Which policy do you want?

• (i) Fatal — treat as unrecoverable (abort / poison the domain).
• (ii) Quarantine + retry list — on close failure, do not free the
  region; move it to a UCX-internal list retaining handle/addr/len so the close
  can be retried. (Note: this must be UCX-side — a consumer that only sees
  the error status has nothing left to retry.)
• (iii) Defer to process teardown — accept leak-until-exit (today's status
  quo for the whole cache) and only surface the error for observability.

My tentative lean is (ii), but it adds per-cache state + a retry driver and
interacts with Decision 1 (who owns the retry), so I'd rather you choose.

I'm ready to build the PR to whichever answers you give. And if there's an
existing peer-close eviction mechanism I've overlooked (beyond LRU capacity
limits and UCS_STATIC_CLEANUP), I'd much rather use that than add anything new.