Skip to content

worker-utils: SIGTERM shutdown is reported as a crash #1318

Description

@kristinapathak

What happens

service.Run() panics when the process is stopped with SIGTERM.

The nvkit servers run group models a shutdown signal as a terminal error and
returns fmt.Errorf("received signal %s", sig)
(src/libraries/go/lib/pkg/nvkit/servers/grpc.go). service.Run() then
classifies that error:

if err != nil && err.Error() != "received signal interrupt" {
    utils.ExitReason(err)
    zap.S().Panic(err)
}

Only the SIGINT wording is excused. SIGTERM stringifies as terminated, so the
message is received signal terminated, which does not match and falls through
to zap.S().Panic.

Why it matters

SIGTERM is how Kubernetes asks a container to stop, and SIGINT essentially never
arrives there. The check excuses the signal that does not happen in production
and panics on the one that always does. Every graceful shutdown is recorded as a
crash, and utils.ExitReason writes the panic into the pod termination log, so
normal terminations look like failures.

Related exposure

The server goroutine in src/compute-plane-services/worker-utils/worker/worker.go
makes the same decision by a different route:

err := w.server.Run()
if err != nil && w.shutdownCtx.Err() == nil {

The run group always returns a non-nil error on SIGTERM, so the only thing
preventing a panic is the shutdown context having already been cancelled. That
depends on interrupt ordering inside the run group rather than on the error
itself.

Suggested fix

Recognize a shutdown-signal error explicitly, derived from the signal names
rather than a single hard-coded string, and use it at both call sites.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions