已经分配的pod注解中hami.io/vgpu-devices-to-allocate不被清除

[zhegemingzimeibanquan]

What happened:
已经分配的pod注解中hami.io/vgpu-devices-to-allocate不被清除
而且注解中的显卡uuid和实际上pod内部nvidia-smi -L查询出来的uuid不相同

pod的yaml：
`apiVersion: v1
kind: Pod
metadata:
annotations:
aip/deployment-history-id: 0998e190-1d93-40af-855c-8b4befd091fd
hami.io/bind-phase: success
hami.io/bind-time: "1744011458"
hami.io/vgpu-devices-allocated: GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:;
hami.io/vgpu-devices-to-allocate: GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:;
hami.io/vgpu-node: dashuju-gpu-04
hami.io/vgpu-time: "1744017459"
creationTimestamp: "2025-04-07T07:37:38Z"
generateName: a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-
labels:
aip/deployment-id: a2e16143-9fe9-4504-9673-ec17c6cad005
hami.io/vgpu-node: dashuju-gpu-04
juicefs-uniqueid: ""
pod-template-hash: 654f4566f7
managedFields:

• apiVersion: v1
  fieldsType: FieldsV1
  fieldsV1:
  f:metadata:
  f:annotations:
  .: {}
  f:aip/deployment-history-id: {}
  f:generateName: {}
  f:labels:
  .: {}
  f:aip/deployment-id: {}
  f:pod-template-hash: {}
  f:ownerReferences:
  .: {}
  k:{"uid":"43393d13-53c8-4504-8852-74b4f44d6743"}:
  .: {}
  f:apiVersion: {}
  f:blockOwnerDeletion: {}
  f:controller: {}
  f:kind: {}
  f:name: {}
  f:uid: {}
  f:spec:
  f:affinity:
  .: {}
  f:podAntiAffinity:
  .: {}
  f:preferredDuringSchedulingIgnoredDuringExecution: {}
  f:containers:
  k:{"name":"aip-c8ddeda3528b461ea5cd715d9e424e9d"}:
  .: {}
  f:command: {}
  f:image: {}
  f:imagePullPolicy: {}
  f:name: {}
  f:ports:
  .: {}
  k:{"containerPort":29010,"protocol":"TCP"}:
  .: {}
  f:containerPort: {}
  f:protocol: {}
  f:resources:
  .: {}
  f:limits:
  .: {}
  f:cpu: {}
  f:memory: {}
  f:nvidia.com/gpu: {}
  f:nvidia.com/gpumem: {}
  f:requests:
  .: {}
  f:cpu: {}
  f:memory: {}
  f:nvidia.com/gpu: {}
  f:nvidia.com/gpumem: {}
  f:terminationMessagePath: {}
  f:terminationMessagePolicy: {}
  f:volumeMounts:
  .: {}
  k:{"mountPath":"/home"}:
  .: {}
  f:mountPath: {}
  f:name: {}
  f:subPath: {}
  f:dnsPolicy: {}
  f:enableServiceLinks: {}
  f:imagePullSecrets:
  .: {}
  k:{"name":"1-docker-registry"}:
  .: {}
  f:name: {}
  f:nodeSelector:
  .: {}
  f:bdps.group/8a748260909c8bb00190b0685483059d: {}
  f:restartPolicy: {}
  f:schedulerName: {}
  f:securityContext: {}
  f:terminationGracePeriodSeconds: {}
  f:tolerations: {}
  f:volumes:
  .: {}
  k:{"name":"aip-ability-pvc-2"}:
  .: {}
  f:name: {}
  f:persistentVolumeClaim:
  .: {}
  f:claimName: {}
  k:{"name":"aip-portal-sensitive"}:
  .: {}
  f:name: {}
  f:persistentVolumeClaim:
  .: {}
  f:claimName: {}
  manager: kube-controller-manager
  operation: Update
  time: "2025-04-07T07:37:38Z"
• apiVersion: v1
  fieldsType: FieldsV1
  fieldsV1:
  f:metadata:
  f:annotations:
  f:hami.io/bind-phase: {}
  manager: nvidia-device-plugin
  operation: Update
  time: "2025-04-07T07:37:38Z"
• apiVersion: v1
  fieldsType: FieldsV1
  fieldsV1:
  f:metadata:
  f:annotations:
  f:hami.io/bind-time: {}
  f:hami.io/vgpu-devices-allocated: {}
  f:hami.io/vgpu-devices-to-allocate: {}
  f:hami.io/vgpu-node: {}
  f:hami.io/vgpu-time: {}
  f:labels:
  f:hami.io/vgpu-node: {}
  manager: scheduler
  operation: Update
  time: "2025-04-07T07:37:40Z"
• apiVersion: v1
  fieldsType: FieldsV1
  fieldsV1:
  f:metadata:
  f:labels:
  f:juicefs-uniqueid: {}
  manager: juicefs-csi-driver
  operation: Update
  time: "2025-04-07T07:37:49Z"
• apiVersion: v1
  fieldsType: FieldsV1
  fieldsV1:
  f:status:
  f:conditions:
  k:{"type":"ContainersReady"}:
  .: {}
  f:lastProbeTime: {}
  f:lastTransitionTime: {}
  f:status: {}
  f:type: {}
  k:{"type":"Initialized"}:
  .: {}
  f:lastProbeTime: {}
  f:lastTransitionTime: {}
  f:status: {}
  f:type: {}
  k:{"type":"Ready"}:
  .: {}
  f:lastProbeTime: {}
  f:lastTransitionTime: {}
  f:status: {}
  f:type: {}
  f:containerStatuses: {}
  f:hostIP: {}
  f:phase: {}
  f:podIP: {}
  f:podIPs:
  .: {}
  k:{"ip":"10.244.7.67"}:
  .: {}
  f:ip: {}
  f:startTime: {}
  manager: kubelet
  operation: Update
  time: "2025-04-07T09:17:33Z"
• apiVersion: v1
  fieldsType: FieldsV1
  fieldsV1:
  f:status:
  f:conditions:
  .: {}
  k:{"type":"PodScheduled"}:
  .: {}
  f:lastProbeTime: {}
  f:lastTransitionTime: {}
  f:message: {}
  f:reason: {}
  f:status: {}
  f:type: {}
  manager: kube-scheduler
  operation: Update
  time: "2025-04-07T09:17:39Z"
  name: a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x
  namespace: "1"
  ownerReferences:
• apiVersion: apps/v1
  blockOwnerDeletion: true
  controller: true
  kind: ReplicaSet
  name: a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7
  uid: 43393d13-53c8-4504-8852-74b4f44d6743
  resourceVersion: "187365186"
  uid: e4e7a715-b15b-4fbe-a616-e6cb97f37c5c
  spec:
  affinity:
  podAntiAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
  • podAffinityTerm:
    labelSelector:
    matchExpressions:
    - key: aip/deployment-id
    operator: In
    values:
    - a2e16143-9fe9-4504-9673-ec17c6cad005
    topologyKey: kubernetes.io/hostname
    weight: 100
    containers:
• command:
  • bash
  • -c
  • cd /home/nlp_hw_business_hall_113M_service;python3 nlp_hw_business_hall_service.py
    image: harbor.harbor:30001/3d044511-45fd-439c-9e71-4482c1dde03c/hotword:v1
    imagePullPolicy: Always
    name: aip-c8ddeda3528b461ea5cd715d9e424e9d
    ports:
  • containerPort: 29010
    protocol: TCP
    resources:
    limits:
    cpu: "1"
    memory: 4000Mi
    nvidia.com/gpu: "1"
    nvidia.com/gpumem: 2k
    requests:
    cpu: "1"
    memory: 4000Mi
    nvidia.com/gpu: "1"
    nvidia.com/gpumem: 2k
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
  • mountPath: /home
    name: aip-ability-pvc-2
    subPath: bf4da92c-601f-4d6e-8829-561cbc12fa58/b2cbfa6d-5600-4159-b3c9-e9b592d80c73/a2e16143-9fe9-4504-9673-ec17c6cad005/faed84ad-7c7d-430d-8d17-f75cf9032313
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name: default-token-qxmhp
    readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    imagePullSecrets:
• name: 1-docker-registry
  nodeName: dashuju-gpu-04
  nodeSelector:
  bdps.group/8a748260909c8bb00190b0685483059d: 8a748260909c8bb00190b0685483059d
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: hami-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
• effect: NoSchedule
  key: gpu
  operator: Exists
• effect: NoExecute
  key: node.kubernetes.io/not-ready
  operator: Exists
  tolerationSeconds: 300
• effect: NoExecute
  key: node.kubernetes.io/unreachable
  operator: Exists
  tolerationSeconds: 300
  volumes:
• name: aip-ability-pvc-2
  persistentVolumeClaim:
  claimName: aip-ability-pvc-2
• name: aip-portal-sensitive
  persistentVolumeClaim:
  claimName: aip-portal-sensitive
• name: default-token-qxmhp
  secret:
  defaultMode: 420
  secretName: default-token-qxmhp
  status:
  conditions:
• lastProbeTime: null
  lastTransitionTime: "2025-04-07T07:37:38Z"
  status: "True"
  type: Initialized
• lastProbeTime: null
  lastTransitionTime: "2025-04-07T07:37:54Z"
  status: "True"
  type: Ready
• lastProbeTime: null
  lastTransitionTime: "2025-04-07T07:37:54Z"
  status: "True"
  type: ContainersReady
• lastProbeTime: null
  lastTransitionTime: "2025-04-07T09:17:39Z"
  message: pod e4e7a715-b15b-4fbe-a616-e6cb97f37c5c is in the cache, so can't be
  assumed
  reason: SchedulerError
  status: "False"
  type: PodScheduled
  containerStatuses:
• containerID: containerd://375bdf9a498ceb86cd237e6f9b3f4fc9e7aa131501eef1c64ae1bcafd72b552d
  image: harbor.harbor:30001/3d044511-45fd-439c-9e71-4482c1dde03c/hotword:v1
  imageID: harbor.harbor:30001/3d044511-45fd-439c-9e71-4482c1dde03c/hotword@sha256:6127dde70e22d37514738bc8991eaf594f3e220b873c1f5ad401fabe4495d0f8
  lastState: {}
  name: aip-c8ddeda3528b461ea5cd715d9e424e9d
  ready: true
  restartCount: 0
  started: true
  state:
  running:
  startedAt: "2025-04-07T07:37:54Z"
  hostIP: 192.168.129.15
  phase: Running
  podIP: 10.244.7.67
  podIPs:
• ip: 10.244.7.67
  qosClass: Guaranteed
  startTime: "2025-04-07T07:37:38Z"
  `
  What you expected to happen:
  正常调度
  How to reproduce it (as minimally and precisely as possible):

device-plugin-hami-device-plugin-2tjvc.log
kube-scheduler.log
vgpu-scheduler-extender.log

Environment:

• HAMi version:2.5.0
• k8s 1.20.14

[ouyangluwei163]

apiVersion: v1
kind: Pod
metadata:
  annotations:
    aip/deployment-history-id: 0998e190-1d93-40af-855c-8b4befd091fd
    hami.io/bind-phase: success
    hami.io/bind-time: "1744011458"  #2025-04-07 15:37:38   dp中拿到的注解1744011458 2025-04-07 15:37:38 
    hami.io/vgpu-devices-allocated: GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:;
    hami.io/vgpu-devices-to-allocate: GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:;
    hami.io/vgpu-node: dashuju-gpu-04
    hami.io/vgpu-time: "1744017459"   #2025-04-07 17:17:39  dp中拿到的注解1744011458 2025-04-07 15:37:38
  creationTimestamp: "2025-04-07T07:37:38Z"
  generateName: a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-

根据pod提供的信息来看，在2025-04-07 17:17:3917:17:39的时候触发过重新调度的情况。但是只走了Filter的逻辑，没有走到Bind的逻辑

I0407 07:56:10.618520       1 scheduler.go:462] "Starting schedule filter process" pod="a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" uuid="e4e7a715-b15b-4fbe-a616-e6cb97f37c5c" namespace="1"
I0407 07:56:10.618571       1 pod.go:29] "Processing resource requirements" pod="1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" containerCount=1
I0407 07:56:10.618590       1 pod.go:36] "Processing container resources" pod="1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" containerIndex=0 containerName="aip-c8ddeda3528b461ea5cd715d9e424e9d"
I0407 07:56:10.618604       1 device.go:122] Start to count metax devices for container aip-c8ddeda3528b461ea5cd715d9e424e9d
I0407 07:56:10.618622       1 device.go:253] Start to count mlu devices for container aip-c8ddeda3528b461ea5cd715d9e424e9d
I0407 07:56:10.618633       1 device.go:258] idx= nvidia.com/gpu val= {{1 0} {<nil>} 1 DecimalSI} {{0 0} {<nil>}  }
I0407 07:56:10.618654       1 device.go:258] idx= nvidia.com/gpumem val= {{2 3} {<nil>} 2k DecimalSI} {{0 0} {<nil>}  }
I0407 07:56:10.618665       1 device.go:258] idx= cpu val= {{1 0} {<nil>} 1 DecimalSI} {{0 0} {<nil>}  }
I0407 07:56:10.618681       1 device.go:258] idx= memory val= {{4194304000 0} {<nil>}  BinarySI} {{0 0} {<nil>}  }
I0407 07:56:10.618694       1 device.go:187] Start to count dcu devices for container aip-c8ddeda3528b461ea5cd715d9e424e9d
I0407 07:56:10.618703       1 device.go:182] Start to count iluvatar devices for container aip-c8ddeda3528b461ea5cd715d9e424e9d
I0407 07:56:10.618712       1 device.go:212] Start to count mthreads devices for container aip-c8ddeda3528b461ea5cd715d9e424e9d
I0407 07:56:10.618743       1 pod.go:51] "Resource requirements collected" pod="1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" requests=[{"NVIDIA":{"Nums":1,"Type":"NVIDIA","Memreq":2000,"MemPercentagereq":101,"Coresreq":0}}]
I0407 07:56:10.618767       1 pods.go:93] "Pod deleted" pod="1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" namespace="1" name="a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" nodeID="dashuju-gpu-04"
I0407 07:56:10.624004       1 score.go:69] "Allocating device for container request" pod="1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" card request={"Nums":1,"Type":"NVIDIA","Memreq":2000,"MemPercentagereq":101,"Coresreq":0}
I0407 07:56:10.624023       1 score.go:73] "scoring pod" pod="1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" Memreq=2000 MemPercentagereq=101 Coresreq=0 Nums=1 device index=7 device="GPU-983d1f82-a74c-2843-105a-f117adfa9bf9"
I0407 07:56:10.624041       1 score.go:40] "Type check" device="NVIDIA-NVIDIA H800 PCIe" req="NVIDIA"
I0407 07:56:10.624073       1 score.go:61] checkUUID result is true for NVIDIA type
I0407 07:56:10.624095       1 score.go:73] "scoring pod" pod="1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" Memreq=2000 MemPercentagereq=101 Coresreq=0 Nums=1 device index=6 device="GPU-02a85cb6-60f4-30d9-7f26-354c3accb7cb"
I0407 07:56:10.624110       1 score.go:40] "Type check" device="NVIDIA-NVIDIA H800 PCIe" req="NVIDIA"
I0407 07:56:10.624122       1 score.go:61] checkUUID result is true for NVIDIA type
I0407 07:56:10.624140       1 score.go:73] "scoring pod" pod="1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" Memreq=2000 MemPercentagereq=101 Coresreq=0 Nums=1 device index=5 device="GPU-718c4646-9de9-0850-a87d-9001a93bd69c"
I0407 07:56:10.624152       1 score.go:40] "Type check" device="NVIDIA-NVIDIA H800 PCIe" req="NVIDIA"
I0407 07:56:10.624162       1 score.go:61] checkUUID result is true for NVIDIA type
I0407 07:56:10.624184       1 score.go:128] "first fitted" pod="1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" device="GPU-718c4646-9de9-0850-a87d-9001a93bd69c"
I0407 07:56:10.624215       1 score.go:139] "device allocate success" pod="1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" allocate device={"NVIDIA":[{"Idx":1,"UUID":"GPU-718c4646-9de9-0850-a87d-9001a93bd69c","Type":"NVIDIA","Usedmem":2000,"Usedcores":0}]}
I0407 07:56:10.624232       1 score.go:186] After AddResourceUsage: &{GPU-718c4646-9de9-0850-a87d-9001a93bd69c 1 9 10 81140 81559 100 0 hami-core [] {0 []} 0 NVIDIA-NVIDIA H800 PCIe true}
I0407 07:56:10.624260       1 scheduler.go:505] nodeScores_len= 1
I0407 07:56:10.624286       1 scheduler.go:508] "Scheduling pod to node" podNamespace="1" podName="a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" nodeID="dashuju-gpu-04" devices={"NVIDIA":[[{"Idx":1,"UUID":"GPU-718c4646-9de9-0850-a87d-9001a93bd69c","Type":"NVIDIA","Usedmem":2000,"Usedcores":0}]]}
I0407 07:56:10.624307       1 util.go:234] Encoded container Devices: GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:
I0407 07:56:10.624328       1 util.go:257] Encoded pod single devices GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:;
I0407 07:56:10.624342       1 device.go:370] pod add notation key [hami.io/vgpu-devices-to-allocate], values is [GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:;]
I0407 07:56:10.624353       1 device.go:371] pod add notation key [hami.io/vgpu-devices-allocated], values is [GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:;]
I0407 07:56:10.624365       1 device.go:155] annoinput map[hami.io/vgpu-devices-allocated:GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:; hami.io/vgpu-devices-to-allocate:GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:; hami.io/vgpu-node:dashuju-gpu-04 hami.io/vgpu-time:1744012570]
I0407 07:56:10.624404       1 pods.go:69] "Pod added" pod="1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" namespace="1" name="a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" nodeID="dashuju-gpu-04" devices={"NVIDIA":[[{"Idx":1,"UUID":"GPU-718c4646-9de9-0850-a87d-9001a93bd69c","Type":"NVIDIA","Usedmem":2000,"Usedcores":0}]]}
I0407 07:56:10.624434       1 util.go:376] patch pod 1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x annotation content is {"metadata":{"annotations":{"hami.io/vgpu-devices-allocated":"GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:;","hami.io/vgpu-devices-to-allocate":"GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:;","hami.io/vgpu-node":"dashuju-gpu-04","hami.io/vgpu-time":"1744012570"},"labels":{"hami.io/vgpu-node":"dashuju-gpu-04"}}}
I0407 07:56:10.633414       1 event.go:307] "Event occurred" object="1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" fieldPath="" kind="Pod" apiVersion="v1" type="Normal" reason="FilteringSucceed" message="Successfully filtered to following nodes: [dashuju-gpu-04] for 1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x "
I0407 07:56:10.636015       1 scheduler.go:99] "Pod added" pod="a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" namespace="1"
I0407 07:56:10.636105       1 util.go:303] checklist is [map[DCU:hami.io/dcu-devices-allocated Iluvatar:hami.io/iluvatar-vgpu-devices-allocated MLU:hami.io/cambricon-mlu-devices-allocated Metax:hami.io/metax-gpu-devices-allocated Mthreads:hami.io/mthreads-vgpu-devices-allocated NVIDIA:hami.io/vgpu-devices-allocated]], annos is [map[aip/deployment-history-id:0998e190-1d93-40af-855c-8b4befd091fd hami.io/bind-phase:success hami.io/bind-time:1744011458 hami.io/vgpu-devices-allocated:GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:; hami.io/vgpu-devices-to-allocate:GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:; hami.io/vgpu-node:dashuju-gpu-04 hami.io/vgpu-time:1744012570]]
I0407 07:56:10.636164       1 util.go:278] Start to decode container device GPU-718c4646-9de9-0850-a87d-9001a93bd69c,NVIDIA,2000,0:
I0407 07:56:10.636182       1 util.go:298] Finished decoding container devices. Total devices: 1
I0407 07:56:10.636222       1 util.go:325] "Decoded pod annos" poddevices={"NVIDIA":[[{"Idx":0,"UUID":"GPU-718c4646-9de9-0850-a87d-9001a93bd69c","Type":"NVIDIA","Usedmem":2000,"Usedcores":0}]]}
I0407 07:56:10.636268       1 pods.go:78] "Pod devices updated" pod="1/a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" namespace="1" name="a2e16143-9fe9-4504-9673-ec17c6cad005-654f4566f7-2lk5x" devices={"NVIDIA":[[{"Idx":0,"UUID":"GPU-718c4646-9de9-0850-a87d-9001a93bd69c","Type":"NVIDIA","Usedmem":2000,"Usedcores":0}]]}

[zhegemingzimeibanquan]

这个问题只在k8s1.20版本出现，在1.28版本中未出现

[zhegemingzimeibanquan]

出现这种情况的pod会显示这个事件

[shoted]

+1