Skip to content

Running CRIU in the snapshot-agent container throws an error #3090

Description

@AL-Cui

I'm testing Dynamo's snapshot-agent and using CRIU to checkpoint and restore the vLLM inference engine, but it keeps showing the following error. Can you help me out?

controller/controller.go:236	Checkpoint target detected, triggering checkpoint	{"pod": "default/checkpoint-job-00ccafce5ef260a9-1-8r8bx", "checkpoint_id": "00ccafce5ef260a9"}
2026-07-13T08:38:25.208822415Z	INFO	controller	executor/checkpoint.go:55	=== Starting checkpoint operation ===	{"pod": "default/checkpoint-job-00ccafce5ef260a9-1-8r8bx", "checkpoint_id": "00ccafce5ef260a9"}
2026-07-13T08:38:28.246543252Z	ERROR	controller	criu/dump.go:113	CRIU dump failed	{"pod": "default/checkpoint-job-00ccafce5ef260a9-1-8r8bx", "checkpoint_id": "00ccafce5ef260a9", "duration": 0.307958197, "checkpoint_dir": "/checkpoints/00ccafce5ef260a9/versions/tmp/6d0ac58e-ef9b-4059-8419-943a21fd92db", "dump_log_path": "/checkpoints/00ccafce5ef260a9/versions/tmp/6d0ac58e-ef9b-4059-8419-943a21fd92db/dump.log", "error": "operation failed (msg:Error (criu/util.c:1625): Unable to create the binfmt_misc file system: Operation not permitted\n err:0)\ncriu swrk failed: exit status 1"}
github.com/ai-dynamo/dynamo/deploy/snapshot/internal/criu.ExecuteDump
	/workspace/internal/criu/dump.go:113
github.com/ai-dynamo/dynamo/deploy/snapshot/internal/executor.captureCheckpoint
	/workspace/internal/executor/checkpoint.go:264
github.com/ai-dynamo/dynamo/deploy/snapshot/internal/executor.Checkpoint
	/workspace/internal/executor/checkpoint.go:89
github.com/ai-dynamo/dynamo/deploy/snapshot/internal/controller.(*NodeController).runCheckpoint
	/workspace/internal/controller/controller.go:475
github.com/ai-dynamo/dynamo/deploy/snapshot/internal/controller.(*NodeController).reconcileCheckpointPod.func1
	/workspace/internal/controller/controller.go:240

I've already mounted the proc directory in the snapshot-agent container

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions