分配了虚拟gpu但pod提示不足(Virtual GPU is allocated but pod prompts insufficient)

[ZNFY]

我按照教程成功安装了Hami，并且都处于running状态下
I have successfully installed Hami according to the tutorial and it is all running

但当我使用教程中的示例，出现了问题
But when I used the example in the tutorial, I got a problem

我使用了这个示例
I used this example

并且我已经提前查看了拥有gpu节点的情况，我分配了五个虚拟gpu
And I have checked the situation of having gpu nodes in advance, I have allocated five virtual gpus

但是奇怪的，当我使用示例，会提示我资源不足
But the strange thing is that when I use the example, it prompts me that there are insufficient resources

我查看了Hami的日志，除了显示成功调度，没有什么错误提示
I checked Hami's log and found no error messages except successful scheduling.
我应该如何解决？我希望能得到答复
How should I solve this? I hope to get an answer

Thank you.

[ZNFY]

我重启了gpu节点，现在变成这样了

[ZNFY]

新创建的pod还是提示
Allocate failed due to rpc error: code = Unknown desc = no binding pod found on node node4, which is unexpected

[ZNFY]

我找到了api的日志
Trace[417025689]: [10.003858256s] [10.003858256s] END
W0306 13:50:36.308908 1 dispatcher.go:181] Failed calling webhook, failing open vgpu.hami.io: failed calling webhook "vgpu.hami.io": failed to call webhook: Post "https://hami-scheduler.kube-system.svc:443/webhook?timeout=10s": context deadline exceeded
E0306 13:50:36.308975 1 dispatcher.go:185] failed calling webhook "vgpu.hami.io": failed to call webhook: Post "https://hami-scheduler.kube-system.svc:443/webhook?timeout=10s": context deadline exceeded
I0306 13:50:36.314888 1 trace.go:219] Trace[1027626634]: "Create" accept:application/vnd.kubernetes.protobuf, /,audit-id:fe2799f7-0f03-45cc-b279-1e5dec42337e,client:127.0.0.1,protocol:HTTP/2.0,resource:pods,scope:resource,url:/api/v1/namespaces/project1/pods,user-agent:kube-controller-manager/v1.26.4 (linux/amd64) kubernetes/f89670c/system:serviceaccount:kube-system:replicaset-controller,verb:POST (06-Mar-2025 13:50:26.307) (total time: 10007ms):
Trace[1027626634]: ["Call mutating webhook" configuration:hami-webhook,webhook:vgpu.hami.io,resource:/v1, Resource=pods,subresource:,operation:CREATE,UID:d14a1c1e-2237-428d-b203-c5b21db477e3 10006ms (13:50:26.308)]
Trace[1027626634]: [10.007066185s] [10.007066185s] END
W0306 13:56:54.310224 1 watcher.go:221] watch chan error: etcdserver: mvcc: required revision has been compacted
W0306 13:57:53.701686 1 dispatcher.go:181] Failed calling webhook, failing open vgpu.hami.io: failed calling webhook "vgpu.hami.io": failed to call webhook: Post "https://hami-scheduler.kube-system.svc:443/webhook?timeout=10s": dial tcp 10.233.239.112:443: i/o timeout
E0306 13:57:53.701715 1 dispatcher.go:185] failed calling webhook "vgpu.hami.io": failed to call webhook: Post "https://hami-scheduler.kube-system.svc:443/webhook?timeout=10s": dial tcp 10.233.239.112:443: i/o timeout
I0306 13:57:53.704325 1 trace.go:219] Trace[150870325]: "Create" accept:application/json, text/plain, /,audit-id:7f236d4f-e76e-466e-a923-c76855a7c685,client:10.234.168.1,protocol:HTTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/default/pods,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36,verb:POST (06-Mar-2025 13:57:43.698) (total time: 10005ms):
Trace[150870325]: ["Call mutating webhook" configuration:hami-webhook,webhook:vgpu.hami.io,resource:/v1, Resource=pods,subresource:,operation:CREATE,UID:743c032a-e7da-4401-8132-95ffc182bd95 10003ms (13:57:43.700)]
Trace[150870325]: [10.005615843s] [10.005615843s] END
W0306 13:58:06.546286 1 dispatcher.go:181] Failed calling webhook, failing open vgpu.hami.io: failed calling webhook "vgpu.hami.io": failed to call webhook: Post "https://hami-scheduler.kube-system.svc:443/webhook?timeout=10s": context deadline exceeded
E0306 13:58:06.546366 1 dispatcher.go:185] failed calling webhook "vgpu.hami.io": failed to call webhook: Post "https://hami-scheduler.kube-system.svc:443/webhook?timeout=10s": context deadline exceeded
I0306 13:58:06.550077 1 trace.go:219] Trace[391410148]: "Create" accept:application/json, text/plain, /,audit-id:454e3303-0c15-4704-bf91-bafe0e3b5040,client:10.234.168.1,protocol:HTTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/default/pods,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36,verb:POST (06-Mar-2025 13:57:56.545) (total time: 10004ms):
Trace[391410148]: ["Call mutating webhook" configuration:hami-webhook,webhook:vgpu.hami.io,resource:/v1, Resource=pods,subresource:,operation:CREATE,UID:9e0017d8-fcf3-4037-a18b-f2e5157a81c4 10004ms (13:57:56.545)]
Trace[391410148]: [10.004871455s] [10.004871455s] END

[lixd]

看日志是请求 HAMi webhook 失败了，然后应该是没有切换到 hami-scheduler 使用 default scheduler 做了调度，然后调用 DevicePlugin 分配 vGPU 的时候缺少部分信息，冲突了。 先看下 HAMi webhook 能否正常访问，然后可以尝试直接pod 中指定 schedulerName: hami-scheduler