`cudf-polars` streaming runtime hangs on OOM

[nirandaperera]

I was running cudf-polars pdsh 1k runs on GH200 node with @rjzamora 's PR.

I noticed that when a query fails with an OOM error, the python process hangs, and I have to manually kill the process. It's not cleaning up properly.

[wence-]

This is likely because we are not propagating exceptions by shutting down all the channels in the nodes that turn out the fail. Can you check where everyone is in a debugger when you get an OOM and things fail?

[nirandaperera]

My current setup doesn't have debug symbols. So it might not give a lot of info. I'll setup a debug env and get back to you

[wence-]

I doubt you will see anything useful unfortunately (at least on the C++ side), but maybe.

[TomAugspurger]

Here's a run on an H100 (PDX cluster) with export CUDF_POLARS_LOG_TRACES=1 and these flags:

python \
>     "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/benchmarks/pdsh.py" \
>     --executor="streaming" \
>     --suffix="" \
>     --blocksize=2_000_000_000 \
>     --spill-device="0.5" \
>     --shuffle=rapidsmpf \
>     --runtime="rapidsmpf" \
>     --stream-policy="pool" \
>     --n-workers 1 \
>     --no-print-results \
>     --no-summarize \
>     --path="/data/tpch-rs/scale-1000" \
>     --explain \
>     --explain-logical \
>     --rmm-async \
>     --iterations 2 \
>     4

Logical Plan

Query 4 - Logical plan

SORT ('o_orderpriority',) ('o_orderpriority', 'order_count') row_count='unknown'
  SELECT ('o_orderpriority', 'order_count') row_count='unknown'
    GROUPBY ('o_orderpriority',) ('o_orderpriority', '_______________0') row_count='unknown'
      JOIN Semi ('o_orderkey',) ('l_orderkey',) ('o_orderkey', 'o_orderpriority', 'o_orderdate') row_count='unknown'
        SCAN PARQUET ('o_orderkey', 'o_orderpriority', 'o_orderdate') row_count='unknown'
        SCAN PARQUET ('l_orderkey', 'l_commitdate', 'l_receiptdate') row_count='unknown'

Physical Plan

Query 4 - Physical plan

SORT ('o_orderpriority',) ('o_orderpriority', 'order_count') [1]
  REPARTITION ('o_orderpriority', 'order_count') [1]
    SELECT ('o_orderpriority', 'order_count') [1]
      SELECT ('o_orderpriority', '_______________0') [1]
        GROUPBY ('o_orderpriority',) ('o_orderpriority', '_______________0') [1]
          REPARTITION ('o_orderpriority', '_______________0') [1]
            GROUPBY ('o_orderpriority',) ('o_orderpriority', '_______________0') [28]
              JOIN Semi ('o_orderkey',) ('l_orderkey',) ('o_orderkey', 'o_orderpriority', 'o_orderdate') [28]
                SHUFFLE ('o_orderkey', 'o_orderpriority', 'o_orderdate') [28]
                  SCAN PARQUET ('o_orderkey', 'o_orderpriority', 'o_orderdate') [8]
                SHUFFLE ('l_orderkey', 'l_commitdate', 'l_receiptdate') [28]
                  SCAN PARQUET ('l_orderkey', 'l_commitdate', 'l_receiptdate') [28]

IR Node Traces

The last successful node exected was a Scan (something is up with the units here. And for some reason we're missing RMM statistics. I'm looking into it).

2025-10-31 09:50:18 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (7337878, 3), 'size': 208397591}] nvml_current_bytes_input=553648128 nvml_current_bytes_output=11620319232 overhead_duration=9940685 start=1615225915382247 stop=1615226661806376 total_bytes_input=0 total_bytes_output=208397591 type=Scan
2025-10-31 09:50:18 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136566732, 3), 'size': 3277601568}] nvml_current_bytes_input=553648128 nvml_current_bytes_output=11857297408 overhead_duration=8951754 start=1615225914235155 stop=1615226947442179 total_bytes_input=0 total_bytes_output=3277601568 type=Scan
2025-10-31 09:50:19 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (7339879, 3), 'size': 208456729}] nvml_current_bytes_input=11620319232 nvml_current_bytes_output=17662214144 overhead_duration=435856 start=1615226668702056 stop=1615227515949622 total_bytes_input=0 total_bytes_output=208456729 type=Scan
2025-10-31 09:50:19 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136559027, 3), 'size': 3277416648}] nvml_current_bytes_input=12461277184 nvml_current_bytes_output=18903728128 overhead_duration=662727 start=1615226959535406 stop=1615227873736837 total_bytes_input=0 total_bytes_output=3277416648 type=Scan
2025-10-31 09:50:20 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (7343006, 3), 'size': 208531178}] nvml_current_bytes_input=17662214144 nvml_current_bytes_output=22158508032 overhead_duration=488893 start=1615227519188418 stop=1615228392167633 total_bytes_input=0 total_bytes_output=208531178 type=Scan
2025-10-31 09:50:20 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136556778, 3), 'size': 3277362672}] nvml_current_bytes_input=18903728128 nvml_current_bytes_output=22762487808 overhead_duration=554006 start=1615227883077422 stop=1615228751510607 total_bytes_input=0 total_bytes_output=3277362672 type=Scan
2025-10-31 09:50:21 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (7343836, 3), 'size': 208556739}] nvml_current_bytes_input=22158508032 nvml_current_bytes_output=26285703168 overhead_duration=492912 start=1615228395571532 stop=1615229271998854 total_bytes_input=0 total_bytes_output=208556739 type=Scan
2025-10-31 09:50:21 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136548067, 3), 'size': 3277153608}] nvml_current_bytes_input=22762487808 nvml_current_bytes_output=26319257600 overhead_duration=620459 start=1615228760991393 stop=1615229629825397 total_bytes_input=0 total_bytes_output=3277153608 type=Scan
2025-10-31 09:50:21 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (7337526, 3), 'size': 208402007}] nvml_current_bytes_input=26285703168 nvml_current_bytes_output=29842472960 overhead_duration=508368 start=1615229275406778 stop=1615230160538964 total_bytes_input=0 total_bytes_output=208402007 type=Scan
2025-10-31 09:50:22 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136569686, 3), 'size': 3277672464}] nvml_current_bytes_input=26319257600 nvml_current_bytes_output=29842472960 overhead_duration=857104 start=1615229639629389 stop=1615230524205544 total_bytes_input=0 total_bytes_output=3277672464 type=Scan
2025-10-31 09:50:22 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (7343032, 3), 'size': 208543597}] nvml_current_bytes_input=29842472960 nvml_current_bytes_output=33097252864 overhead_duration=524639 start=1615230163994473 stop=1615231031245154 total_bytes_input=0 total_bytes_output=208543597 type=Scan
2025-10-31 09:50:23 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136562577, 3), 'size': 3277501848}] nvml_current_bytes_input=29842472960 nvml_current_bytes_output=33533460480 overhead_duration=571962 start=1615230534232578 stop=1615231383606441 total_bytes_input=0 total_bytes_output=3277501848 type=Scan
2025-10-31 09:50:23 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (7339498, 3), 'size': 208435639}] nvml_current_bytes_input=33097252864 nvml_current_bytes_output=36888903680 overhead_duration=560926 start=1615231034741185 stop=1615231919710483 total_bytes_input=0 total_bytes_output=208435639 type=Scan
2025-10-31 09:50:24 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136568915, 3), 'size': 3277653960}] nvml_current_bytes_input=33533460480 nvml_current_bytes_output=37325111296 overhead_duration=738482 start=1615231393316642 stop=1615232284602037 total_bytes_input=0 total_bytes_output=3277653960 type=Scan
2025-10-31 09:50:24 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (5968933, 3), 'size': 169513443}] nvml_current_bytes_input=36888903680 nvml_current_bytes_output=38130417664 overhead_duration=569640 start=1615231923156062 stop=1615232695776740 total_bytes_input=0 total_bytes_output=169513443 type=Scan
2025-10-31 09:50:24 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136568311, 3), 'size': 3277639464}] nvml_current_bytes_input=37325111296 nvml_current_bytes_output=40345010176 overhead_duration=581917 start=1615232294358450 stop=1615233082607306 total_bytes_input=0 total_bytes_output=3277639464 type=Scan
2025-10-31 09:50:25 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136566407, 3), 'size': 3277593768}] nvml_current_bytes_input=40345010176 nvml_current_bytes_output=43868225536 overhead_duration=683925 start=1615233091875714 stop=1615233525661634 total_bytes_input=0 total_bytes_output=3277593768 type=Scan
2025-10-31 09:50:25 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136563664, 3), 'size': 3277527936}] nvml_current_bytes_input=43868225536 nvml_current_bytes_output=47928311808 overhead_duration=719398 start=1615233534737910 stop=1615233963008126 total_bytes_input=0 total_bytes_output=3277527936 type=Scan
2025-10-31 09:50:26 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136557180, 3), 'size': 3277372320}] nvml_current_bytes_input=47928311808 nvml_current_bytes_output=51921289216 overhead_duration=721026 start=1615233971934025 stop=1615234394001768 total_bytes_input=0 total_bytes_output=3277372320 type=Scan
2025-10-31 09:50:26 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136545811, 3), 'size': 3277099464}] nvml_current_bytes_input=51921289216 nvml_current_bytes_output=55444504576 overhead_duration=743144 start=1615234403167296 stop=1615234836425875 total_bytes_input=0 total_bytes_output=3277099464 type=Scan
2025-10-31 09:50:27 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136562800, 3), 'size': 3277507200}] nvml_current_bytes_input=55444504576 nvml_current_bytes_output=58900611072 overhead_duration=777668 start=1615234845648511 stop=1615235271765698 total_bytes_input=0 total_bytes_output=3277507200 type=Scan
2025-10-31 09:50:27 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136569414, 3), 'size': 3277665936}] nvml_current_bytes_input=58900611072 nvml_current_bytes_output=64000884736 overhead_duration=795294 start=1615235280973170 stop=1615235704195021 total_bytes_input=0 total_bytes_output=3277665936 type=Scan
2025-10-31 09:50:27 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136562134, 3), 'size': 3277491216}] nvml_current_bytes_input=64000884736 nvml_current_bytes_output=66282586112 overhead_duration=817186 start=1615235713576234 stop=1615236162335672 total_bytes_input=0 total_bytes_output=3277491216 type=Scan
2025-10-31 09:50:28 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136559875, 3), 'size': 3277437000}] nvml_current_bytes_input=66282586112 nvml_current_bytes_output=69805801472 overhead_duration=820107 start=1615236171926035 stop=1615236603029173 total_bytes_input=0 total_bytes_output=3277437000 type=Scan
2025-10-31 09:50:28 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136559886, 3), 'size': 3277437264}] nvml_current_bytes_input=69805801472 nvml_current_bytes_output=73396125696 overhead_duration=819659 start=1615236612493231 stop=1615237044217907 total_bytes_input=0 total_bytes_output=3277437264 type=Scan
2025-10-31 09:50:29 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136546450, 3), 'size': 3277114800}] nvml_current_bytes_input=73396125696 nvml_current_bytes_output=76852232192 overhead_duration=854703 start=1615237053804715 stop=1615237486241006 total_bytes_input=0 total_bytes_output=3277114800 type=Scan
2025-10-31 09:50:29 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136549998, 3), 'size': 3277199952}] nvml_current_bytes_input=76852232192 nvml_current_bytes_output=81449189376 overhead_duration=855618 start=1615237495821645 stop=1615237946500807 total_bytes_input=0 total_bytes_output=3277199952 type=Scan
2025-10-31 09:50:30 [info     ] Execute IR                     count_frames_input=0 count_frames_output=1 frames_input=[] frames_output=[{'shape': (136564824, 3), 'size': 3277555776}] nvml_current_bytes_input=81449189376 nvml_current_bytes_output=84401979392 overhead_duration=884554 start=1615237956226995 stop=1615238394815670 total_bytes_input=0 total_bytes_output=3277555776 type=Scan

Traceback

We do get a Python traceback before the hang:

Exception in callback Future.set_result(None)
handle: <Handle Future.set_result(None)>
Traceback (most recent call last):
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "rapidsmpf/streaming/core/node.pyx", line 255, in runner
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 684, in _wrap_awaitable
    return await awaitable
           ^^^^^^^^^^^^^^^
  File "rapidsmpf/streaming/core/node.pyx", line 174, in run
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/rapidsmpf/io.py", line 339, in scan_node
    await asyncio.gather(*tasks)
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/rapidsmpf/io.py", line 211, in read_chunk
    df = await asyncio.to_thread(
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/dsl/tracing.py", line 185, in wrapper
    result = func(cls, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/dsl/ir.py", line 785, in do_evaluate
    chunk = reader.read_chunk()
            ^^^^^^^^^^^^^^^^^^^
  File "pylibcudf/io/parquet.pyx", line 366, in pylibcudf.io.parquet.ChunkedParquetReader.read_chunk
  File "pylibcudf/io/parquet.pyx", line 385, in pylibcudf.io.parquet.ChunkedParquetReader.read_chunk
MemoryError: std::bad_alloc: out_of_memory: CUDA error (failed to allocate 1728253408 bytes) at: /tmp/pip-build-env-nd5tpyj0/normal/lib/python3.12/site-packages/librmm/include/rmm/mr/device/cuda_async_view_memory_resource.hpp:87: cudaErrorMemoryAllocation out of memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
asyncio.exceptions.InvalidStateError: invalid state
Exception in callback Future.set_result(None)
handle: <Handle Future.set_result(None)>
Traceback (most recent call last):
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "rapidsmpf/streaming/core/node.pyx", line 255, in runner
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 684, in _wrap_awaitable
    return await awaitable
           ^^^^^^^^^^^^^^^
  File "rapidsmpf/streaming/core/node.pyx", line 174, in run
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/rapidsmpf/io.py", line 339, in scan_node
    await asyncio.gather(*tasks)
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/rapidsmpf/io.py", line 211, in read_chunk
    df = await asyncio.to_thread(
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/dsl/tracing.py", line 185, in wrapper
    result = func(cls, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/dsl/ir.py", line 785, in do_evaluate
    chunk = reader.read_chunk()
            ^^^^^^^^^^^^^^^^^^^
  File "pylibcudf/io/parquet.pyx", line 366, in pylibcudf.io.parquet.ChunkedParquetReader.read_chunk
  File "pylibcudf/io/parquet.pyx", line 385, in pylibcudf.io.parquet.ChunkedParquetReader.read_chunk
MemoryError: std::bad_alloc: out_of_memory: CUDA error (failed to allocate 1728253408 bytes) at: /tmp/pip-build-env-nd5tpyj0/normal/lib/python3.12/site-packages/librmm/include/rmm/mr/device/cuda_async_view_memory_resource.hpp:87: cudaErrorMemoryAllocation out of memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
asyncio.exceptions.InvalidStateError: invalid state
Exception in callback Future.set_result(None)
handle: <Handle Future.set_result(None)>
Traceback (most recent call last):
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "rapidsmpf/streaming/core/node.pyx", line 255, in runner
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 684, in _wrap_awaitable
    return await awaitable
           ^^^^^^^^^^^^^^^
  File "rapidsmpf/streaming/core/node.pyx", line 174, in run
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/rapidsmpf/io.py", line 339, in scan_node
    await asyncio.gather(*tasks)
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/rapidsmpf/io.py", line 211, in read_chunk
    df = await asyncio.to_thread(
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/dsl/tracing.py", line 185, in wrapper
    result = func(cls, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/dsl/ir.py", line 785, in do_evaluate
    chunk = reader.read_chunk()
            ^^^^^^^^^^^^^^^^^^^
  File "pylibcudf/io/parquet.pyx", line 366, in pylibcudf.io.parquet.ChunkedParquetReader.read_chunk
  File "pylibcudf/io/parquet.pyx", line 385, in pylibcudf.io.parquet.ChunkedParquetReader.read_chunk
MemoryError: std::bad_alloc: out_of_memory: CUDA error (failed to allocate 1728253408 bytes) at: /tmp/pip-build-env-nd5tpyj0/normal/lib/python3.12/site-packages/librmm/include/rmm/mr/device/cuda_async_view_memory_resource.hpp:87: cudaErrorMemoryAllocation out of memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
asyncio.exceptions.InvalidStateError: invalid state
Exception in callback recv.<locals>.f()
handle: <Handle recv.<locals>.f()>
Traceback (most recent call last):
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "rapidsmpf/streaming/core/node.pyx", line 255, in runner
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 684, in _wrap_awaitable
    return await awaitable
           ^^^^^^^^^^^^^^^
  File "rapidsmpf/streaming/core/node.pyx", line 174, in run
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/rapidsmpf/io.py", line 339, in scan_node
    await asyncio.gather(*tasks)
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/rapidsmpf/io.py", line 211, in read_chunk
    df = await asyncio.to_thread(
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/dsl/tracing.py", line 185, in wrapper
    result = func(cls, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/dsl/ir.py", line 785, in do_evaluate
    chunk = reader.read_chunk()
            ^^^^^^^^^^^^^^^^^^^
  File "pylibcudf/io/parquet.pyx", line 366, in pylibcudf.io.parquet.ChunkedParquetReader.read_chunk
  File "pylibcudf/io/parquet.pyx", line 385, in pylibcudf.io.parquet.ChunkedParquetReader.read_chunk
MemoryError: std::bad_alloc: out_of_memory: CUDA error (failed to allocate 1728253408 bytes) at: /tmp/pip-build-env-nd5tpyj0/normal/lib/python3.12/site-packages/librmm/include/rmm/mr/device/cuda_async_view_memory_resource.hpp:87: cudaErrorMemoryAllocation out of memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "rapidsmpf/streaming/core/channel.pyx", line 316, in rapidsmpf.streaming.core.channel.Channel.recv.f
asyncio.exceptions.InvalidStateError: invalid state
Exception in callback Future.set_result(None)
handle: <Handle Future.set_result(None)>
Traceback (most recent call last):
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "rapidsmpf/streaming/core/node.pyx", line 255, in runner
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 684, in _wrap_awaitable
    return await awaitable
           ^^^^^^^^^^^^^^^
  File "rapidsmpf/streaming/core/node.pyx", line 174, in run
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/rapidsmpf/io.py", line 339, in scan_node
    await asyncio.gather(*tasks)
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/rapidsmpf/io.py", line 211, in read_chunk
    df = await asyncio.to_thread(
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/dsl/tracing.py", line 185, in wrapper
    result = func(cls, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/dsl/ir.py", line 785, in do_evaluate
    chunk = reader.read_chunk()
            ^^^^^^^^^^^^^^^^^^^
  File "pylibcudf/io/parquet.pyx", line 366, in pylibcudf.io.parquet.ChunkedParquetReader.read_chunk
  File "pylibcudf/io/parquet.pyx", line 385, in pylibcudf.io.parquet.ChunkedParquetReader.read_chunk
MemoryError: std::bad_alloc: out_of_memory: CUDA error (failed to allocate 1728253408 bytes) at: /tmp/pip-build-env-nd5tpyj0/normal/lib/python3.12/site-packages/librmm/include/rmm/mr/device/cuda_async_view_memory_resource.hpp:87: cudaErrorMemoryAllocation out of memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
asyncio.exceptions.InvalidStateError: invalid state

At this point the process hangs and needs to be terminated sigkill. Running with -X faulthandler doesn't give any additional tracebacks.
I haven't been able to diagnose the program with external tools like py-spy.

[wence-]

That looks like an exception from a python node is failing to shut down a channel. Since it's in the producer can you just make a test that throws when reading and see if you can go from there?

[rjzamora]

That looks like an exception from a python node is failing to shut down a channel. Since it's in the producer can you just make a test that throws when reading and see if you can go from there?

Yeah, raising an error in scan_node definitely causes a hang right now. There is also a hang if you just immediately call shutdown on the output channel in scan_node.

[TomAugspurger]

With this patch:

diff --git a/python/cudf_polars/cudf_polars/dsl/ir.py b/python/cudf_polars/cudf_polars/dsl/ir.py
index d707f02828..bc3a130bbe 100644
--- a/python/cudf_polars/cudf_polars/dsl/ir.py
+++ b/python/cudf_polars/cudf_polars/dsl/ir.py
@@ -642,6 +642,8 @@ class Scan(IR):
         context: IRExecutionContext,
     ) -> DataFrame:
         """Evaluate and return a dataframe."""
+        print("scan", paths)
+        raise RuntimeError("test hang")
         stream = context.get_cuda_stream()
         if typ == "csv":

and this program (working on getting a fully reproducible example; for now, this uses a tpch dataset):

import faulthandler, signal
faulthandler.register(signal.SIGUSR1)

import polars as pl
import datetime
import os

print("kill -USR1", os.getpid())

root = "/datasets/toaugspurger/tpch/scale-10"

lineitem = pl.scan_parquet(f"{root}/lineitem.parquet")
orders = pl.scan_parquet(f"{root}/orders.parquet")

var1 = datetime.date(1993, 7, 1)
var2 = datetime.date(1993, 10, 1)

left = orders.filter(pl.col("o_orderdate").is_between(var1, var2, closed="left"))
right = lineitem.filter(pl.col("l_commitdate").lt(pl.col("l_receiptdate")))
q = left.join(right, left_on="o_orderkey", right_on="l_orderkey", how="semi")

engine = pl.GPUEngine(
    raise_on_fail=True,
    executor="streaming",
    executor_options={
        "runtime": "rapidsmpf",
        "cluster": "single",
        "max_rows_per_partition": 100_000_000,
        "target_partition_size": 100_000_000,
    }
)

q.collect(engine=engine)

Which outputs this before hanging

kill -USR1 3525889
scan ['/datasets/toaugspurger/tpch/scale-10/orders.parquet']
scan ['/datasets/toaugspurger/tpch/scale-10/lineitem.parquet']

Sending that signal gives these backtraces:

Thread 0x00007fd7f0ffc700 (most recent call first):
  File "/home/nfs/toaugspurger/.local/share/uv/python/cpython-3.13.3-linux-x86_64-gnu/lib/python3.13/concurrent/futures/thread.py", line 90 in _worker
  File "/home/nfs/toaugspurger/.local/share/uv/python/cpython-3.13.3-linux-x86_64-gnu/lib/python3.13/threading.py", line 992 in run
  File "/home/nfs/toaugspurger/.local/share/uv/python/cpython-3.13.3-linux-x86_64-gnu/lib/python3.13/threading.py", line 1041 in _bootstrap_inner
  File "/home/nfs/toaugspurger/.local/share/uv/python/cpython-3.13.3-linux-x86_64-gnu/lib/python3.13/threading.py", line 1012 in _bootstrap

Current thread 0x00007fd9aaa52740 (most recent call first):
  File "/home/nfs/toaugspurger/gh/rapidsai/cudf/python/cudf_polars/cudf_polars/experimental/rapidsmpf/core.py", line 132 in evaluate_logical_plan
  File "/home/nfs/toaugspurger/gh/rapidsai/cudf/python/cudf_polars/cudf_polars/experimental/parallel.py", line 247 in evaluate_rapidsmpf
  File "/home/nfs/toaugspurger/gh/rapidsai/cudf/python/cudf_polars/cudf_polars/experimental/parallel.py", line 276 in evaluate_streaming
  File "/home/nfs/toaugspurger/gh/rapidsai/cudf/python/cudf_polars/cudf_polars/callback.py", line 249 in _callback
  File "/raid/toaugspurger/envs/gh/rapidsai/cudf/lib/python3.13/site-packages/polars/_utils/scan.py", line 27 in _execute_from_rust
  File "/raid/toaugspurger/envs/gh/rapidsai/cudf/lib/python3.13/site-packages/polars/lazyframe/frame.py", line 2407 in collect
  File "/raid/toaugspurger/envs/gh/rapidsai/cudf/lib/python3.13/site-packages/polars/lazyframe/opt_flags.py", line 330 in wrapper
  File "/raid/toaugspurger/envs/gh/rapidsai/cudf/lib/python3.13/site-packages/polars/_utils/deprecation.py", line 97 in wrapper
  File "/home/nfs/toaugspurger/gh/rapidsai/cudf/test_hang.py", line 38 in <module>

Which doesn't tell us a ton I thikn. The main (Current) thread is at https://github.com/rjzamora/cudf/blob/961898df403fbd53751ee5daccd7d494be44626e/python/cudf_polars/cudf_polars/experimental/rapidsmpf/core.py#L132 calling run_streaming_pipeline.

I'm not sure what that other thread (0x00007fd7f0ffc700) is doing. Presumably it's from that py_executor thread pool, or potentially the thread pool attached to the dedicated event loop. I'll try reproducing parts of run_streaming_pipeline in pure python if possible.

[wence-]

Can you just write a network where the producer is a py node that raises and the consumer is a pull_from_channel. That should presumably also hang.

[TomAugspurger]

Yep, I'm working on that now. The simplest possible version of that errored, but didn't hang:

@define_py_node()
async def producer(context: Context, ch_out: Channel) -> None:
    async with shutdown_on_error(context, ch_out):
        raise Exception("test")


@define_py_node()
async def consumer(context: Context, ch_in: Channel) -> None:
    async with shutdown_on_error(context, ch_in):
        while True:
            a = await ch_in.recv(context)
            print("Got a", a)

I'll keep investigating.

[TomAugspurger]

I can't explain why yet, but while reading some asyncio source code I noticed that there are a few spots where they catch BaseException instead of Exception. Changing https://github.com/rjzamora/cudf/blob/55eb6eddaca3d2815daaf4796a121133ef76f627/python/cudf_polars/cudf_polars/experimental/rapidsmpf/utils.py#L47 to catch BaseException seems to avoid the hang.

My guess right now is that we aren't handling task cancellation correctly somewhere (asyncio.CancelledError subclasses BaseException, not Exception), but I haven't found out where yet.

[TomAugspurger]

Just speculation, but the call to Future.set_result at

rapidsmpf/python/rapidsmpf/rapidsmpf/streaming/core/channel.pyx

Line 233 in 5e8fdad

loop.call_soon_threadsafe(ret.set_result, None)

in Channel.shutdown may not handle cancellation correctly. Calling fut.set_result on a cancelled future raises an InvalidStateError:

In [14]: fut = loop.create_future()

In [15]: fut.cancel()
Out[15]: True

In [16]: fut.set_result(1)
---------------------------------------------------------------------------
InvalidStateError                         Traceback (most recent call last)
Cell In[16], line 1
----> 1 fut.set_result(1)

InvalidStateError: invalid state

More things I can't yet explain: with the original code (catching just Exception instead of BaseException, and Scan.do_evaluate patched to raise a RuntimeError) I see different behavior for an inner join vs. semi:

• how="inner": InvalidStateError printed in the traceback, no hang.
• how="semi": No InvalidStateError in the traceback, but hangs.

[TomAugspurger]

With that change to catch BaseException, we avoid the hang but hit a segfault when running Q4:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/benchmarks/utils.py", line 933, in run_polars
    result = execute_query(q_id, i, q, run_config, args, engine)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/benchmarks/utils.py", line 585, in execute_query
    return q.collect(engine=engine)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/polars/_utils/deprecation.py", line 97, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/polars/lazyframe/opt_flags.py", line 330, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/polars/lazyframe/frame.py", line 2407, in collect
    return wrap_df(ldf.collect(engine, callback))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/polars/_utils/scan.py", line 27, in _execute_from_rust
    return function(with_columns, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/callback.py", line 249, in _callback
    return evaluate_streaming(ir, config_options).to_polars()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/parallel.py", line 276, in evaluate_streaming
    return evaluate_rapidsmpf(ir, config_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/parallel.py", line 247, in evaluate_rapidsmpf
    return evaluate_logical_plan(ir, config_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/rapidsmpf/core.py", line 132, in evaluate_logical_plan
    run_streaming_pipeline(nodes=nodes, py_executor=executor)
  File "rapidsmpf/streaming/core/node.pyx", line 268, in rapidsmpf.streaming.core.node.run_streaming_pipeline
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "rapidsmpf/streaming/core/node.pyx", line 255, in runner
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/tasks.py", line 684, in _wrap_awaitable
    return await awaitable
           ^^^^^^^^^^^^^^^
  File "rapidsmpf/streaming/core/node.pyx", line 174, in run
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/rapidsmpf/io.py", line 339, in scan_node
    await asyncio.gather(*tasks)
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/experimental/rapidsmpf/io.py", line 211, in read_chunk
    df = await asyncio.to_thread(
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/dsl/tracing.py", line 185, in wrapper
    result = func(cls, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/cudf_polars/dsl/ir.py", line 785, in do_evaluate
    chunk = reader.read_chunk()
            ^^^^^^^^^^^^^^^^^^^
  File "pylibcudf/io/parquet.pyx", line 366, in pylibcudf.io.parquet.ChunkedParquetReader.read_chunk
  File "pylibcudf/io/parquet.pyx", line 385, in pylibcudf.io.parquet.ChunkedParquetReader.read_chunk
MemoryError: std::bad_alloc: out_of_memory: CUDA error (failed to allocate 1728253408 bytes) at: /tmp/pip-build-env-nd5tpyj0/normal/lib/python3.12/site-packages/librmm/include/rmm/mr/device/cuda_async_view_memory_resource.hpp:87: cudaErrorMemoryAllocation out of memory

[gpu-h100-0440:4024077:0:4024531] Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil))
==== backtrace (tid:4024531) ====
 0  /app/.venv/lib/python3.12/site-packages/libucx/lib/libucs.so(ucs_handle_error+0x294) [0x15547edb3b14]
 1  /app/.venv/lib/python3.12/site-packages/libucx/lib/libucs.so(+0x34cca) [0x15547edb3cca]
 2  /app/.venv/lib/python3.12/site-packages/libucx/lib/libucs.so(+0x34f7e) [0x15547edb3f7e]
 3  /usr/lib64/libpthread.so.0(+0x12990) [0x155555116990]
 4  /usr/lib64/libcuda.so.1(+0x30c404) [0x154f73691404]
 5  /usr/lib64/libcuda.so.1(+0x3427b8) [0x154f736c77b8]
 6  /usr/lib64/libcuda.so.1(+0x44214e) [0x154f737c714e]
 7  /usr/lib64/libcuda.so.1(+0x3f9603) [0x154f7377e603]
 8  /usr/lib64/libcuda.so.1(cuMemFreeAsync+0x20) [0x154f737646f0]
 9  /app/.venv/lib/python3.12/site-packages/rmm/pylibrmm/memory_resource/_memory_resource.cpython-312-x86_64-linux-gnu.so(+0x7bea0) [0x15550d291ea0]
10  /app/.venv/lib/python3.12/site-packages/rmm/pylibrmm/memory_resource/_memory_resource.cpython-312-x86_64-linux-gnu.so(+0xe6c5b) [0x15550d2fcc5b]
11  /app/.venv/lib/python3.12/site-packages/rmm/pylibrmm/memory_resource/_memory_resource.cpython-312-x86_64-linux-gnu.so(_ZN3rmm2mr26cuda_async_memory_resource13do_deallocateEPvmNS_16cuda_stream_viewE+0x27) [0x15550d26a387]
12  /app/.venv/lib/python3.12/site-packages/rmm/pylibrmm/memory_resource/_memory_resource.cpython-312-x86_64-linux-gnu.so(+0x53aad) [0x15550d269aad]
13  /app/.venv/lib/python3.12/site-packages/rmm/pylibrmm/memory_resource/_memory_resource.cpython-312-x86_64-linux-gnu.so(_ZN3rmm2mr27statistics_resource_adaptorINS0_22device_memory_resourceEE13do_deallocateEPvmNS_16cuda_stream_viewE+0x47) [0x15550d26aab7]
14  /app/.venv/lib/python3.12/site-packages/pylibcudf/binaryop.cpython-312-x86_64-linux-gnu.so(_ZN4cuda2mr3__424_Resource_vtable_builder14_Dealloc_asyncIN3rmm2mr22device_memory_resourceEEEvPvS7_mmNS_3__410stream_refE+0x4d) [0x15550d36a18d]
15  /app/.venv/lib/python3.12/site-packages/librapidsmpf/lib64/librapidsmpf.so(_ZN9rapidsmpf18RmmResourceAdaptor13do_deallocateEPvmN3rmm16cuda_stream_viewE+0xcb) [0x15547debf99b]
16  /app/.venv/lib/python3.12/site-packages/rmm/pylibrmm/memory_resource/_memory_resource.cpython-312-x86_64-linux-gnu.so(+0x53aad) [0x15550d269aad]
17  /app/.venv/lib/python3.12/site-packages/librmm/lib64/librmm.so(_ZN3rmm13device_buffer16deallocate_asyncEv+0x60) [0x155547bb2710]
18  /app/.venv/lib/python3.12/site-packages/librmm/lib64/librmm.so(_ZN3rmm13device_bufferD1Ev+0x21) [0x155547bb27d1]
19  /app/.venv/lib/python3.12/site-packages/libcudf/lib64/libcudf.so(_ZN4cudf2io7parquet6detail11reader_implD2Ev+0x65) [0x15550ea0a565]
20  /app/.venv/lib/python3.12/site-packages/libcudf/lib64/libcudf.so(_ZN4cudf2io7parquet6detail6readerD1Ev+0x20) [0x15550eada590]
21  /app/.venv/lib/python3.12/site-packages/libcudf/lib64/libcudf.so(_ZN4cudf2io7parquet6detail14chunked_readerD0Ev+0x9) [0x15550eada5f9]
22  /app/.venv/lib/python3.12/site-packages/pylibcudf/io/parquet.cpython-312-x86_64-linux-gnu.so(+0x1c004) [0x1554809b6004]
23  python() [0x1829723]
24  python() [0x1a14335]
25  python() [0x188c653]
26  python(_PyEval_EvalFrameDefault+0xfd3a) [0x182277a]
27  python() [0x1852170]
28  python() [0x18027ab]
29  python(_PyEval_EvalFrameDefault+0x322f) [0x1815c6f]
30  python() [0x18129d0]
31  /app/.venv/lib/python3.12/site-packages/rapidsmpf/streaming/core/node.cpython-312-x86_64-linux-gnu.so(+0x15d1d) [0x15547c069d1d]
32  /app/.venv/lib/python3.12/site-packages/rapidsmpf/streaming/core/node.cpython-312-x86_64-linux-gnu.so(+0x16e7b) [0x15547c06ae7b]
33  /app/.venv/lib/python3.12/site-packages/rapidsmpf/streaming/core/channel.cpython-312-x86_64-linux-gnu.so(+0xea09) [0x15547c087a09]
34  /app/.venv/lib/python3.12/site-packages/rapidsmpf/streaming/core/channel.cpython-312-x86_64-linux-gnu.so(+0x12886) [0x15547c08b886]
35  python(_PyEval_EvalFrameDefault+0x6633) [0x1819073]
36  python() [0x185340e]
37  python() [0x4a2ac2]
38  python() [0x4a28ac]
39  python(_PyObject_MakeTpCall+0x77) [0x185adf7]
40  python() [0x485050]
41  python() [0x1806a27]
42  python(_PyEval_EvalFrameDefault+0xed1d) [0x182175d]
43  python() [0x1811f1d]
44  python() [0x19514d8]
45  python() [0x1951421]
46  /usr/lib64/libpthread.so.0(+0x81ca) [0x15555510c1ca]
47  /usr/lib64/libc.so.6(clone+0x43) [0x1555543d48d3]
=================================
Segmentation fault (core dumped)

[nirandaperera]

@TomAugspurger Segfault seems to be related to rmm device buffer deallocation.
https://github.com/rapidsai/rmm/blob/main/cpp/src/device_buffer.cpp#L93-L99
It seems to be not checking if the data() is nullptr.
allocate_async is setting the size before allocating. So, I'm assuming, this is coming from the destructor of the rmm buffer which failed on cuda async malloc.
I think its a rmm bug.

[wence-]

That might be an incorrectly stream-ordered deallocation.

Edit: ah no, I see what you mean. Yes device_buffer::deallocate_async probably needs to run in the inverse order of allocate_async