container "pytorch" is waiting to start: PodInitializing

[gogogwwb]

When the master is finished running, the worker is still initializing.

worker log：
Error from server (BadRequest): container "pytorch" in pod"xxx-jxosi-worker-0" is waiting to start: PodInitializing

What is the reason for this?

[gaocegege]

Could you please run kubectl describe and post the result here?

[gaocegege]

Can you show more about it? Especially the events section.

[gaocegege]

Seems that the init container is pending. Can you show its log?

[gogogwwb]

init-pytorch log：

[gogogwwb]

master svc ：

[gaocegege]

Can you try kubectl debug to run an ephemeral container, then run ping xxx-master-0?

[gogogwwb]

ping master-0 in an ephemeral container :

[gogogwwb]

kubectl get ep -A，then i found ep appeared, but disappeared again after a while

[gaocegege]

It's weird.

[gogogwwb]

I put the program to sleep for a while and found that the worker can run. Is there any restriction on the creation order of service and pod in pytorchjob?

[gogogwwb]

It should be that the master executes too fast, so that the ep of the service finally becomes none, and the worker cannot obtain the IP address of the master.

[gaocegege]

It should be that the master executes too fast, so that the ep of the service finally becomes none, and the worker cannot obtain the IP address of the master.

Interesting. /cc @johnugeorge

[johnugeorge]

But in that case, master should not start the job until workers are up. Are you using distributed setup itself in the code?

[gogogwwb]

I did not use distributed steps in the code. After master running, it becomes completed state, "kubectl get ep -ntest", and found that ep is none