[BUG] v0.10.0 Graceful shutdown does not generate a model

[rossbalch]

Thanks for taking the time to write a bug report! Use the following prompts to help you describe what's going on. The more info you provide, the easier it'll be to understand how to fix it for you without having to come back and ask you questions.

Describe the bug
When using ctrl+c to end a training run using Anaconda PowerShell on Win11 24h2, "attempting graceful shutdown" is displayed but the CLI never indicates that a model was generated, nor indeed is a model file present in the output folder.

To Reproduce
I can confirm the bug in both the NAM Gui initiated training, and the nam-full initiated training.
I have ensured I have not accidentally pressed ctrl+c multiple times.
This issue has happened every time I have attempted an early termination of training using ctrl+c

[sdatkinson]

Hmm. I've seen this on Colab. I wasn't able to reproduce it just now on my desktop with my "day to day" nam environment, but I just made a new one and I think it did what this Issue is talking about.

I'll look more into this, but could you share your pip list and conda list to get me started? I'll compare notes later.

[rossbalch]

Package                 Version
----------------------- -----------
absl-py                 2.1.0
aiohappyeyeballs        2.4.3
aiohttp                 3.10.10
aiosignal               1.3.1
anaconda-anon-usage     0.4.4
annotated-types         0.7.0
archspec                0.2.3
attrs                   24.2.0
auraloss                0.3.0
boltons                 23.0.0
Brotli                  1.0.9
certifi                 2024.8.30
cffi                    1.17.1
charset-normalizer      3.3.2
colorama                0.4.6
coloredlogs             15.0.1
conda                   24.9.2
conda-content-trust     0.2.0
conda-libmamba-solver   24.9.0
conda-package-handling  2.3.0
conda_package_streaming 0.10.0
contourpy               1.3.0
cryptography            43.0.0
cycler                  0.12.1
distro                  1.9.0
filelock                3.13.1
flatbuffers             24.3.25
fonttools               4.54.1
frozendict              2.4.2
frozenlist              1.5.0
fsspec                  2024.10.0
grpcio                  1.67.0
huggingface-hub         0.26.1
humanfriendly           10.0
idna                    3.7
Jinja2                  3.1.4
jsonpatch               1.33
jsonpointer             2.1
kiwisolver              1.4.7
libmambapy              1.5.8
lightning-utilities     0.11.8
Markdown                3.7
MarkupSafe              2.1.3
matplotlib              3.9.2
menuinst                2.1.2
mpmath                  1.3.0
multidict               6.1.0
networkx                3.3
neural-amp-modeler      0.10.0
numpy                   1.26.4
onnx                    1.17.0
onnxruntime             1.19.2
packaging               24.1
pillow                  11.0.0
pip                     24.2
platformdirs            3.10.0
pluggy                  1.0.0
propcache               0.2.0
protobuf                5.28.3
pycosat                 0.6.6
pycparser               2.21
pydantic                2.9.2
pydantic_core           2.23.4
pyparsing               3.2.0
pyreadline3             3.5.4
PySocks                 1.7.1
python-dateutil         2.9.0.post0
pytorch-lightning       2.4.0
PyYAML                  6.0.2
regex                   2024.9.11
requests                2.32.3
ruamel.yaml             0.18.6
ruamel.yaml.clib        0.2.8
safetensors             0.4.5
scipy                   1.14.1
setuptools              75.1.0
six                     1.16.0
sounddevice             0.5.1
sympy                   1.13.1
tensorboard             2.18.0
tensorboard-data-server 0.7.2
tokenizers              0.20.1
torch                   2.5.0
torchmetrics            1.5.1
tqdm                    4.66.5
transformers            4.46.0
truststore              0.8.0
typing_extensions       4.11.0
urllib3                 2.2.3
wavio                   0.0.9
Werkzeug                3.0.6
wheel                   0.44.0
win-inet-pton           1.1.0
yarl                    1.16.0
zstandard               0.23.0```

```conda list
# packages in environment at H:\conda:
#
# Name                    Version                   Build  Channel
absl-py                   2.1.0                    pypi_0    pypi
aiohappyeyeballs          2.4.3                    pypi_0    pypi
aiohttp                   3.10.10                  pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
anaconda-anon-usage       0.4.4           py312hfc23b7f_100
anaconda_powershell_prompt 1.1.0                haa95532_0
anaconda_prompt           1.1.0                haa95532_0
annotated-types           0.7.0                    pypi_0    pypi
archspec                  0.2.3              pyhd3eb1b0_0
attrs                     24.2.0                   pypi_0    pypi
auraloss                  0.3.0                    pypi_0    pypi
blas                      1.0                         mkl
boltons                   23.0.0          py312haa95532_0
brotli-python             1.0.9           py312hd77b12b_8
bzip2                     1.0.8                h2bbff1b_6
ca-certificates           2024.9.24            haa95532_0
certifi                   2024.8.30       py312haa95532_0
cffi                      1.17.1          py312h827c3e9_0
charset-normalizer        3.3.2              pyhd3eb1b0_0
colorama                  0.4.6           py312haa95532_0
coloredlogs               15.0.1                   pypi_0    pypi
conda                     24.9.2          py312haa95532_0
conda-content-trust       0.2.0           py312haa95532_1
conda-libmamba-solver     24.9.0             pyhd3eb1b0_0
conda-package-handling    2.3.0           py312haa95532_0
conda-package-streaming   0.10.0          py312haa95532_0
contourpy                 1.3.0                    pypi_0    pypi
cryptography              43.0.0          py312h89fc84f_0
cuda-cccl                 12.6.77                       0    nvidia
cuda-cccl_win-64          12.6.77                       0    nvidia
cuda-cudart               12.1.105                      0    nvidia
cuda-cudart-dev           12.1.105                      0    nvidia
cuda-cupti                12.1.105                      0    nvidia
cuda-libraries            12.1.0                        0    nvidia
cuda-libraries-dev        12.1.0                        0    nvidia
cuda-nvrtc                12.1.105                      0    nvidia
cuda-nvrtc-dev            12.1.105                      0    nvidia
cuda-nvtx                 12.1.105                      0    nvidia
cuda-opencl               12.6.77                       0    nvidia
cuda-opencl-dev           12.6.77                       0    nvidia
cuda-profiler-api         12.6.77                       0    nvidia
cuda-runtime              12.1.0                        0    nvidia
cuda-version              12.6                          3    nvidia
cycler                    0.12.1                   pypi_0    pypi
distro                    1.9.0           py312haa95532_0
expat                     2.6.3                h5da7b33_0
filelock                  3.13.1          py312haa95532_0
flatbuffers               24.3.25                  pypi_0    pypi
fmt                       9.1.0                h6d14046_1
fonttools                 4.54.1                   pypi_0    pypi
frozendict                2.4.2           py312haa95532_0
frozenlist                1.5.0                    pypi_0    pypi
fsspec                    2024.10.0                pypi_0    pypi
grpcio                    1.67.0                   pypi_0    pypi
huggingface-hub           0.26.1                   pypi_0    pypi
humanfriendly             10.0                     pypi_0    pypi
idna                      3.7             py312haa95532_0
intel-openmp              2023.1.0         h59b6b97_46320
jinja2                    3.1.4           py312haa95532_0
jsonpatch                 1.33            py312haa95532_1
jsonpointer               2.1                pyhd3eb1b0_0
kiwisolver                1.4.7                    pypi_0    pypi
libarchive                3.7.4                h9243413_0
libcublas                 12.1.0.26                     0    nvidia
libcublas-dev             12.1.0.26                     0    nvidia
libcufft                  11.0.2.4                      0    nvidia
libcufft-dev              11.0.2.4                      0    nvidia
libcurand                 10.3.7.77                     0    nvidia
libcurand-dev             10.3.7.77                     0    nvidia
libcurl                   8.9.1                h0416ee5_0
libcusolver               11.4.4.55                     0    nvidia
libcusolver-dev           11.4.4.55                     0    nvidia
libcusparse               12.0.2.55                     0    nvidia
libcusparse-dev           12.0.2.55                     0    nvidia
libffi                    3.4.4                hd77b12b_1
libiconv                  1.16                 h2bbff1b_3
libmamba                  1.5.8                h99b1521_3
libmambapy                1.5.8           py312h77c03ed_3
libnpp                    12.0.2.50                     0    nvidia
libnpp-dev                12.0.2.50                     0    nvidia
libnvjitlink              12.1.105                      0    nvidia
libnvjitlink-dev          12.1.105                      0    nvidia
libnvjpeg                 12.1.1.14                     0    nvidia
libnvjpeg-dev             12.1.1.14                     0    nvidia
libsolv                   0.7.24               h23ce68f_1
libssh2                   1.11.0               h291bd65_0
libuv                     1.48.0               h827c3e9_0
libxml2                   2.13.1               h24da03e_2
lightning-utilities       0.11.8                   pypi_0    pypi
lz4-c                     1.9.4                h2bbff1b_1
markdown                  3.7                      pypi_0    pypi
markupsafe                2.1.3           py312h2bbff1b_0
matplotlib                3.9.2                    pypi_0    pypi
menuinst                  2.1.2           py312h5da7b33_0
mkl                       2023.1.0         h6b88ed4_46358
mpmath                    1.3.0           py312haa95532_0
multidict                 6.1.0                    pypi_0    pypi
networkx                  3.3             py312haa95532_0
neural-amp-modeler        0.10.0                   pypi_0    pypi
numpy                     1.26.4                   pypi_0    pypi
onnx                      1.17.0                   pypi_0    pypi
onnxruntime               1.19.2                   pypi_0    pypi
openssl                   3.0.15               h827c3e9_0
packaging                 24.1            py312haa95532_0
pcre2                     10.42                h0ff8eda_1
pillow                    11.0.0                   pypi_0    pypi
pip                       24.2            py312haa95532_0
platformdirs              3.10.0          py312haa95532_0
pluggy                    1.0.0           py312haa95532_1
propcache                 0.2.0                    pypi_0    pypi
protobuf                  5.28.3                   pypi_0    pypi
pybind11-abi              5                    hd3eb1b0_0
pycosat                   0.6.6           py312h2bbff1b_1
pycparser                 2.21               pyhd3eb1b0_0
pydantic                  2.9.2                    pypi_0    pypi
pydantic-core             2.23.4                   pypi_0    pypi
pyparsing                 3.2.0                    pypi_0    pypi
pyreadline3               3.5.4                    pypi_0    pypi
pysocks                   1.7.1           py312haa95532_0
python                    3.12.7               h14ffc60_0
python-dateutil           2.9.0.post0              pypi_0    pypi
pytorch                   2.5.0           py3.12_cuda12.1_cudnn9_0    pytorch
pytorch-cuda              12.1                 hde6ce7c_6    pytorch
pytorch-lightning         2.4.0                    pypi_0    pypi
pytorch-mutex             1.0                        cuda    pytorch
pyyaml                    6.0.2           py312h827c3e9_0
regex                     2024.9.11                pypi_0    pypi
reproc                    14.2.4               hd77b12b_2
reproc-cpp                14.2.4               hd77b12b_2
requests                  2.32.3          py312haa95532_0
ruamel.yaml               0.18.6          py312h827c3e9_0
ruamel.yaml.clib          0.2.8           py312h827c3e9_0
safetensors               0.4.5                    pypi_0    pypi
scipy                     1.14.1                   pypi_0    pypi
setuptools                75.1.0          py312haa95532_0
six                       1.16.0                   pypi_0    pypi
sounddevice               0.5.1                    pypi_0    pypi
sqlite                    3.45.3               h2bbff1b_0
sympy                     1.13.1                   pypi_0    pypi
tbb                       2021.8.0             h59b6b97_0
tensorboard               2.18.0                   pypi_0    pypi
tensorboard-data-server   0.7.2                    pypi_0    pypi
tk                        8.6.14               h0416ee5_0
tokenizers                0.20.1                   pypi_0    pypi
torchmetrics              1.5.1                    pypi_0    pypi
tqdm                      4.66.5          py312hfc267ef_0
transformers              4.46.0                   pypi_0    pypi
truststore                0.8.0           py312haa95532_0
typing_extensions         4.11.0          py312haa95532_0
tzdata                    2024b                h04d1e81_0
urllib3                   2.2.3           py312haa95532_0
vc                        14.40                h2eaa2aa_1
vs2015_runtime            14.40.33807          h98bb1dd_1
wavio                     0.0.9                    pypi_0    pypi
werkzeug                  3.0.6                    pypi_0    pypi
wheel                     0.44.0          py312haa95532_0
win_inet_pton             1.1.0           py312haa95532_0
xz                        5.4.6                h8cc25b3_1
yaml                      0.2.5                he774522_0
yaml-cpp                  0.8.0                hd77b12b_1
yarl                      1.16.0                   pypi_0    pypi
zlib                      1.2.13               h8cc25b3_1
zstandard                 0.23.0          py312h4fc1ca9_0
zstd                      1.5.6                h8880b57_0```

[sdatkinson]

Graceful shutdown appears to attempt to exit(1) at the end of its except block in version 2.5.0.post0.

From trainer/call.py:

def _call_and_handle_interrupt(trainer: "pl.Trainer", trainer_fn: Callable, *args: Any, **kwargs: Any) -> Any:
    r"""Error handling, intended to be used only for main trainer function entry points (fit, validate, test, predict)
    as all errors should funnel through them.

    Args:
        trainer_fn: one of (fit, validate, test, predict)
        *args: positional arguments to be passed to the `trainer_fn`
        **kwargs: keyword arguments to be passed to `trainer_fn`

    """
    try:
        if trainer.strategy.launcher is not None:
            return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
        return trainer_fn(*args, **kwargs)

    except _TunerExitException:
        _call_teardown_hook(trainer)
        trainer._teardown()
        trainer.state.status = TrainerStatus.FINISHED
        trainer.state.stage = None

    except KeyboardInterrupt as exception:
        rank_zero_info("\nDetected KeyboardInterrupt, attempting graceful shutdown ...")
        # user could press Ctrl+C many times, disable KeyboardInterrupt for shutdown
        signal.signal(signal.SIGINT, signal.SIG_IGN)
        _interrupt(trainer, exception)
        trainer._teardown()
        launcher = trainer.strategy.launcher
        if isinstance(launcher, _SubprocessScriptLauncher):
            launcher.kill(_get_sigkill_signal())
        exit(1)

    except BaseException as exception:
        _interrupt(trainer, exception)
        trainer._teardown()
        # teardown might access the stage so we reset it after
        trainer.state.stage = None
        raise

I wonder if this changed.