Commit 3a47d1d
activation_checkpointing: default num_layers to None so configure() assert fires (#8041)
## Summary
Calling
`deepspeed.checkpointing.configure(contiguous_checkpointing=True,
partition_activations=True)` *without* setting `num_checkpoints` (or
otherwise providing a layer count) is supposed to fail fast with the
assert at
`deepspeed/runtime/activation_checkpointing/checkpointing.py:1108`:
```python
if CONTIGUOUS_CHECKPOINTING:
assert num_layers is not None, "Must specify the number of layers with contiguous memory checkpointing"
```
That assert never fires because `_configure_defaults()` (called inside
`configure()` at line 1079) initialized:
```python
num_layers = False
```
`False is not None` is `True`, so the assert silently passes. The user
instead hits a much later cryptic `IndexError` from `range(num_layers)`
(lines 399 / 406) or a 0-element allocation from `numel * num_layers`
(lines 457 / 461).
The module-level default at line 46 is already `num_layers = None`, and
every other path that sets it (`set_num_layers`,
`config.number_checkpoints`) assigns an integer — only
`_configure_defaults` used `False`, which looks like a copy-paste from
the surrounding `PARTITION_ACTIVATIONS = False` etc. block.
## Fix
```diff
- num_layers = False
+ num_layers = None
```
One-character change. No callers compare `num_layers` to `False`
(downstream uses are `range(num_layers)` and `numel * num_layers`, both
requiring an int), so the only path this changes is the broken one:
users now get the documented "Must specify the number of layers" assert
instead of a downstream `IndexError`.
## Test
Adds
`test_configure_with_contiguous_checkpointing_requires_num_checkpoints`
to the existing
`tests/unit/runtime/activation_checkpointing/test_activation_checkpointing.py`
(consolidating per `AGENTS.md`, not a new file). It calls
`configure(contiguous_checkpointing=True, partition_activations=True)`
and asserts the expected `AssertionError` matches `"number of layers"`.
On `main` the assert silently passes and the test fails; with this
change it passes.
## CI/lint
- `pre-commit run --files <changed files>`: all hooks pass (yapf,
flake8, `check-torchdist`, `check-license`, codespell,
`check-torchcuda`).
- DCO `Signed-off-by` on the commit.
- No competing PR (`gh pr list --search "num_layers checkpointing OR
_configure_defaults OR contiguous_checkpointing assert"`).
---------
Signed-off-by: Kymi808 <zeng.kyle13@gmail.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
Co-authored-by: Olatunji Ruwase <tunji.ruwase@snowflake.com>1 parent 4421665 commit 3a47d1d
2 files changed
Lines changed: 43 additions & 1 deletion
File tree
- deepspeed/runtime/activation_checkpointing
- tests/unit/runtime/activation_checkpointing
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1019 | 1019 | | |
1020 | 1020 | | |
1021 | 1021 | | |
1022 | | - | |
| 1022 | + | |
1023 | 1023 | | |
1024 | 1024 | | |
1025 | 1025 | | |
| |||
Lines changed: 42 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
309 | 309 | | |
310 | 310 | | |
311 | 311 | | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
0 commit comments