Locally import pyarrow.dataset and fsspec for import cudf performance

[mroeschke]

Description

These are some low hanging fruit imports that we can make lazy to marginally improve import performance xref #627

Pyinstrument of import cudf before

3.021 <module>  tmp.py:1
└─ 3.020 <module>  cudf/__init__.py:1
   ├─ 1.081 _setup_numba  cudf/utils/_numba.py:100
   │  ├─ 0.861 safe_get_versions  cudf/utils/_ptxcompiler.py:78
   │  │  └─ 0.861 get_versions  cudf/utils/_ptxcompiler.py:54
   │  │     └─ 0.861 run  subprocess.py:506
   │  │           [3 frames hidden]  subprocess, selectors
   │  │              0.857 poll.poll  <built-in>
   │  └─ 0.220 _get_cuda_build_version  cudf/utils/_numba.py:17
   │     └─ 0.220 <module>  cudf/_lib/__init__.py:1
   │        ├─ 0.111 <module>  pylibcudf/__init__.py:1
   │        │  └─ 0.040 <module>  pyarrow/__init__.py:1
   │        └─ 0.084 <module>  rmm/__init__.py:1
   │              [3 frames hidden]  rmm, enum
   ├─ 0.609 <module>  cupy/__init__.py:1
   │     [40 frames hidden]  cupy, importlib, email, cupyx, scipy,...
   ├─ 0.329 <module>  cudf/utils/_numba.py:1
   │  └─ 0.325 <module>  numba/__init__.py:1
   │        [12 frames hidden]  numba, numpy
   ├─ 0.299 <module>  cudf/api/__init__.py:1
   │  └─ 0.296 <module>  cudf/api/extensions/__init__.py:1
   │     └─ 0.295 <module>  pandas/__init__.py:1
   │           [13 frames hidden]  pandas, pyarrow
   ├─ 0.287 validate_setup  cudf/utils/gpu_utils.py:4
   │  └─ 0.286 getDeviceCount  rmm/_cuda/gpu.py:92
   ├─ 0.135 <module>  cudf/core/dataframe.py:1
   │  └─ 0.111 <module>  cudf/core/groupby/__init__.py:1
   │     └─ 0.110 <module>  cudf/core/groupby/groupby.py:1
   │        └─ 0.104 <module>  cudf/core/udf/__init__.py:1
   │           └─ 0.098 <module>  cudf/core/udf/groupby_lowering.py:1
   ├─ 0.116 <module>  cudf/core/algorithms.py:1
   │  ├─ 0.081 <module>  cudf/core/index.py:1
   │  │  └─ 0.062 <module>  cudf/core/_base_index.py:1
   │  │     └─ 0.061 <module>  cudf/utils/ioutils.py:1
   │  │        └─ 0.060 <module>  fsspec/__init__.py:1
   │  │              [3 frames hidden]  fsspec, importlib
   │  └─ 0.032 <module>  cudf/core/column/__init__.py:1
   ├─ 0.112 <module>  numba_cuda/numba/cuda/__init__.py:1
   │     [9 frames hidden]  numba_cuda, pynvjitlink, <built-in>
   └─ 0.036 <module>  cudf/io/__init__.py:1

Pyinstrument of import cudf after

2.984 <module>  tmp.py:1
└─ 2.984 <module>  cudf/__init__.py:1
   ├─ 1.096 _setup_numba  cudf/utils/_numba.py:100
   │  ├─ 0.872 safe_get_versions  cudf/utils/_ptxcompiler.py:78
   │  │  └─ 0.872 get_versions  cudf/utils/_ptxcompiler.py:54
   │  │     └─ 0.872 run  subprocess.py:506
   │  │           [3 frames hidden]  subprocess, selectors
   │  │              0.870 poll.poll  <built-in>
   │  └─ 0.224 _get_cuda_build_version  cudf/utils/_numba.py:17
   │     └─ 0.224 <module>  cudf/_lib/__init__.py:1
   │        ├─ 0.113 <module>  pylibcudf/__init__.py:1
   │        │  └─ 0.045 <module>  pyarrow/__init__.py:1
   │        └─ 0.085 <module>  rmm/__init__.py:1
   │              [3 frames hidden]  rmm, enum
   ├─ 0.625 <module>  cupy/__init__.py:1
   │     [44 frames hidden]  cupy, importlib, email, cupyx, scipy,...
   ├─ 0.335 <module>  cudf/utils/_numba.py:1
   │  └─ 0.331 <module>  numba/__init__.py:1
   │        [15 frames hidden]  numba, numpy, llvmlite
   ├─ 0.297 <module>  cudf/api/__init__.py:1
   │  └─ 0.293 <module>  cudf/api/extensions/__init__.py:1
   │     └─ 0.292 <module>  pandas/__init__.py:1
   │           [12 frames hidden]  pandas, pyarrow
   ├─ 0.288 validate_setup  cudf/utils/gpu_utils.py:4
   │  └─ 0.287 getDeviceCount  rmm/_cuda/gpu.py:92
   ├─ 0.138 <module>  cudf/core/dataframe.py:1
   │  └─ 0.115 <module>  cudf/core/groupby/__init__.py:1
   │     └─ 0.113 <module>  cudf/core/groupby/groupby.py:1
   │        └─ 0.107 <module>  cudf/core/udf/__init__.py:1
   │           └─ 0.103 <module>  cudf/core/udf/groupby_lowering.py:1
   │              └─ 0.077 <module>  cudf/core/udf/groupby_typing.py:1
   │                 └─ 0.064 <module>  cudf/core/udf/utils.py:1
   │                    └─ 0.063 <module>  cudf/core/udf/masked_typing.py:1
   │                       └─ 0.059 MaskedScalarScalarOp.decorator  numba/core/typing/templates.py:1289
   │                             [5 frames hidden]  numba
   ├─ 0.112 <module>  numba_cuda/numba/cuda/__init__.py:1
   │     [9 frames hidden]  numba_cuda, pynvjitlink, <built-in>
   └─ 0.059 <module>  cudf/core/algorithms.py:1
      └─ 0.032 <module>  cudf/c

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[mroeschke]

@bdice just wanted to get your second blessing here since this changed after you approved this PR. I opened #18741 to track this in a follow up.

[vyasr]

Note that @bdice is mostly out this week so if you want to get him to take a look you will either need to wait a while or ping him directly and see if he has a few moments. If nothing is too controversial you could also merge and do a follow-up next week when he's back if he has additional requests.

[bdice]

Approving as-is. I don't have time to dig deep into the root cause for this but the comment is helpful for now.

[bdice]

Approving as-is. I don't have time to dig deep into the root cause for this but the comment is helpful for now.

[mroeschke]

/merge