How to get SF3K to pass on g7e.24large?
- Drop driver count and exchange batch sizes
we can make all the queries pass by tuning driver count and exchange batch sizes
fast mode
task.max-drivers-per-task=5
cudf.batch_size_min_threshold=50000000
cudf.partitioned_output_batch_rows=50000000
safe mode
task.max-drivers-per-task=1
cudf.batch_size_min_threshold=20000000
cudf.partitioned_output_batch_rows=20000000
- enable
prefetch_managed_async
This was segfaulting for the last few months, but all you need to do is set export UCX_CUDA_COPY_DMABUF=no to progress further.
see rapidsai/ucxx#749
prefetch_managed_pool does not work well. it doesn't page fault but the pool growth is very expensive
prefetch_managed_async works as expected, but really needs single driver execution. multi driver doesn't page fault, but it does excess migrations, evictions and prefetching. Single driver works well and can compete with CPU TCO.
managed memory can run all the conditions without OOMing
(reference is 16 nodes of m9gd.4xlarge)
- enable adaptive back pressure
see facebookincubator/velox#18848
this conflicts with facebookincubator/velox#18746 so was difficult to resolve and build. Adaptive exchange resulted in worse performance than async MR baseline when undersubscribed and a similar amount of OOMs, at least on sf3k on g7e.24xlarge.
How to get SF3K to pass on
g7e.24large?we can make all the queries pass by tuning driver count and exchange batch sizes
fast mode
safe mode
prefetch_managed_asyncThis was segfaulting for the last few months, but all you need to do is set
export UCX_CUDA_COPY_DMABUF=noto progress further.see rapidsai/ucxx#749
prefetch_managed_pooldoes not work well. it doesn't page fault but the pool growth is very expensiveprefetch_managed_asyncworks as expected, but really needs single driver execution. multi driver doesn't page fault, but it does excess migrations, evictions and prefetching. Single driver works well and can compete with CPU TCO.managed memory can run all the conditions without OOMing
(reference is 16 nodes of
m9gd.4xlarge)see facebookincubator/velox#18848
this conflicts with facebookincubator/velox#18746 so was difficult to resolve and build. Adaptive exchange resulted in worse performance than async MR baseline when undersubscribed and a similar amount of OOMs, at least on sf3k on g7e.24xlarge.