Commit 56de570
Add nccl_version to the source-checkout torch_info fallback (#8383)
Running the unit tests from a source checkout — one that was not `pip
install`ed, so
`git_version_info_installed.py` does not exist — fails before any test
body runs:
```
FAILED unit/runtime/zero/test_zero_tensor_fragment.py::TestTensorFragmentSet::test_zero_fragments[none-1-local-dtype0]
- KeyError: 'nccl_version'
```
`setup.py:292` writes five keys:
```python
torch_info = {
"version": torch_version,
"bf16_support": bf16_support,
"cuda_version": cuda_version,
"nccl_version": nccl_version,
"hip_version": hip_version,
}
```
and the fallback in `deepspeed/git_version_info.py:22` lists three:
```python
torch_info = {'version': "0.0", "cuda_version": "0.0", "hip_version": "0.0"}
```
`tests/unit/util.py` reads `nccl_version` on the way into
`bf16_required_version_check`:
```python
if torch_info['nccl_version'] == '0.0':
# Use runtime NCCL version if available
```
The `"0.0"` is a sentinel — the reader's own comment says so, and
`version`, `cuda_version` and
`hip_version` all carry it for exactly that reason. `nccl_version` was
left out, so instead of
taking the runtime-detection branch the lookup raises.
## The change
Add the missing key with the same sentinel the other three use, so the
reader takes the fallback
it was written to take.
Not adding `bf16_support`, the other key `setup.py` writes and this dict
omits: nothing reads it
off `torch_info` today (`get_accelerator().is_bf16_supported()` is used
instead), and there is no
established sentinel for a bool the way `"0.0"` is for a version. Happy
to add it if you would
rather the two dicts match key-for-key.
## Verification
Reproduced on both a CPU checkout and 2×H100:
```
# upstream/master, source checkout
>>> from unit.util import bf16_required_version_check; bf16_required_version_check()
KeyError: 'nccl_version'
# this branch
>>> bf16_required_version_check()
False # no CUDA on that box, which is the correct answer
```
On 2 GPUs, `unit/runtime/zero/test_zero_tensor_fragment.py`:
```
master 19 failures, all KeyError: 'nccl_version'
this branch 0 occurrences of nccl_version anywhere in the log
```
The failures that remain in my environment are all `Unable to JIT load
the {cpu_adam,fused_adam}
op due to ninja not being installed` — a missing build tool on that box,
unrelated to this
change and present identically before and after.
```
yapf==0.40.0 --diff (no diff)
```
---------
Signed-off-by: alanhuangyoo <alanhuangyoo@gmail.com>
Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>1 parent 80f19f3 commit 56de570
1 file changed
Lines changed: 3 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
23 | 25 | | |
24 | 26 | | |
25 | 27 | | |
| |||
0 commit comments