Skip to content

AppLauncher atexit handler masks uncaught exceptions as exit code 0 #6573

Description

@nblauch

Description

Commit 0090622 added an atexit handler that unconditionally calls SimulationApp.close():

def _atexit_close(app=self._app):
    with contextlib.suppress(Exception):
        app.close()

When an uncaught exception occurs after AppLauncher initialization, this handler runs during interpreter shutdown. Kit fast shutdown exits with the default SimulationApp.close(exit_code=0), replacing the pending nonzero Python exit status.

As a result, launchers such as torchrun, CI systems, and workflow schedulers report failed training processes as successful. We observed an unhandled exception and traceback from every torchrun worker, followed by torchrun and the workflow task exiting with code 0.

Expected behavior

Uncaught exceptions preserve a nonzero process exit code.

Actual behavior

The traceback is printed, but the process exits with code 0.

Regression

The behavior was introduced by commit 0090622 and is present in v3.0.0-beta2. v2.3.2 and v3.0.0-beta predate the change.

The current SimulationApp.close(exit_code=...) documentation explicitly notes that a nonzero value is required to prevent fast shutdown from replacing the status with 0.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions