Description
Commit 0090622 added an atexit handler that unconditionally calls SimulationApp.close():
def _atexit_close(app=self._app):
with contextlib.suppress(Exception):
app.close()
When an uncaught exception occurs after AppLauncher initialization, this handler runs during interpreter shutdown. Kit fast shutdown exits with the default SimulationApp.close(exit_code=0), replacing the pending nonzero Python exit status.
As a result, launchers such as torchrun, CI systems, and workflow schedulers report failed training processes as successful. We observed an unhandled exception and traceback from every torchrun worker, followed by torchrun and the workflow task exiting with code 0.
Expected behavior
Uncaught exceptions preserve a nonzero process exit code.
Actual behavior
The traceback is printed, but the process exits with code 0.
Regression
The behavior was introduced by commit 0090622 and is present in v3.0.0-beta2. v2.3.2 and v3.0.0-beta predate the change.
The current SimulationApp.close(exit_code=...) documentation explicitly notes that a nonzero value is required to prevent fast shutdown from replacing the status with 0.
Description
Commit
0090622added anatexithandler that unconditionally callsSimulationApp.close():When an uncaught exception occurs after
AppLauncherinitialization, this handler runs during interpreter shutdown. Kit fast shutdown exits with the defaultSimulationApp.close(exit_code=0), replacing the pending nonzero Python exit status.As a result, launchers such as
torchrun, CI systems, and workflow schedulers report failed training processes as successful. We observed an unhandled exception and traceback from everytorchrunworker, followed bytorchrunand the workflow task exiting with code 0.Expected behavior
Uncaught exceptions preserve a nonzero process exit code.
Actual behavior
The traceback is printed, but the process exits with code 0.
Regression
The behavior was introduced by commit
0090622and is present inv3.0.0-beta2.v2.3.2andv3.0.0-betapredate the change.The current
SimulationApp.close(exit_code=...)documentation explicitly notes that a nonzero value is required to prevent fast shutdown from replacing the status with 0.