gpu-manager
gpu-manager copied to clipboard
tkestack
[need help] Failed to create a deployment requesting 0.x GPU cores
hello guys,
when I create a deployment with one GPU card, everything goes well
but, if with GPU that less than 1, it fails
bellow are the logs and node info, please kindly review them, and supply any suggestion or solution, thanks.
deployment log
2022-01-27T13:41:21.696682567+08:00 F0127 05:41:21.696330 7 client.go:78] fail to get response from manager, error rpc error: code = Unknown desc = can't find kubepods-burstable-podf96b0b34_8379_4581_9c87_2e98637f2fc2.slice from docker
2022-01-27T13:41:21.698605243+08:00 /tmp/cuda-control/src/register.c:87 rpc client exit with 255
node info
% kubectl describe node
Name: 192.168.41.104
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=192.168.41.104
kubernetes.io/os=linux
node-role.kubernetes.io/master=
nvidia-device-enable=enable
platform.tkestack.io/machine-ip=192.168.41.104
Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"62:f1:6f:67:5d:d5"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 192.168.41.104
kubeadm.alpha.kubernetes.io/cri-socket: /var/run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 25 Jan 2022 10:23:41 +0800
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: 192.168.41.104
AcquireTime: <unset>
RenewTime: Thu, 27 Jan 2022 15:06:29 +0800
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 27 Jan 2022 15:04:50 +0800 Tue, 25 Jan 2022 10:23:37 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 27 Jan 2022 15:04:50 +0800 Tue, 25 Jan 2022 10:23:37 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 27 Jan 2022 15:04:50 +0800 Tue, 25 Jan 2022 10:23:37 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 27 Jan 2022 15:04:50 +0800 Wed, 26 Jan 2022 18:16:45 +0800 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.168.41.104
Hostname: 192.168.41.104
Capacity:
cpu: 48
ephemeral-storage: 307050000Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 32660268Ki
pods: 256
tencent.com/vcuda-core: 100
tencent.com/vcuda-memory: 62
Allocatable:
cpu: 47800m
ephemeral-storage: 282977279532
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 31533868Ki
pods: 256
tencent.com/vcuda-core: 100
tencent.com/vcuda-memory: 62
System Info:
Machine ID: 9ffa73bff8a9485ea512c442e554d291
System UUID: 00000000-0000-0000-0000-AC1F6B450EAE
Boot ID: 28318fa8-fe7d-4b63-a1eb-3665b240af1a
Kernel Version: 3.10.0-1160.53.1.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.5.4
Kubelet Version: v1.21.4-tke.1
Kube-Proxy Version: v1.21.4-tke.1
PodCIDR: 172.16.0.0/24
PodCIDRs: 172.16.0.0/24
Non-terminated Pods: (47 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
default empty9-8547ff7fd-2dmkf 250m (0%) 500m (1%) 256Mi (0%) 1Gi (3%) 86m
kube-system coredns-6c654f5cc6-58bz2 100m (0%) 0 (0%) 70Mi (0%) 170Mi (0%) 2d4h
kube-system coredns-6c654f5cc6-wsg7s 100m (0%) 0 (0%) 70Mi (0%) 170Mi (0%) 2d4h
kube-system etcd-192.168.41.104 100m (0%) 0 (0%) 100Mi (0%) 0 (0%) 2d4h
kube-system flannel-pmfcv 100m (0%) 100m (0%) 50Mi (0%) 256Mi (0%) 2d4h
kube-system galaxy-daemonset-tc6t6 100m (0%) 0 (0%) 200Mi (0%) 0 (0%) 2d4h
kube-system gpu-manager-daemonset-89827 0 (0%) 0 (0%) 0 (0%) 0 (0%) 4h46m
kube-system gpu-quota-admission-76cfff49b6-gs8b5 1 (2%) 2 (4%) 1Gi (3%) 2Gi (6%) 2d4h
kube-system kube-apiserver-192.168.41.104 250m (0%) 0 (0%) 0 (0%) 0 (0%) 2d4h
kube-system kube-controller-manager-192.168.41.104 200m (0%) 0 (0%) 0 (0%) 0 (0%) 2d4h
kube-system kube-proxy-26g7w 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2d4h
kube-system kube-scheduler-192.168.41.104 100m (0%) 0 (0%) 0 (0%) 0 (0%) 2d4h
kube-system metrics-server-v0.3.6-794ccd69c8-8sqxb 87m (0%) 182m (0%) 154Mi (0%) 404Mi (1%) 2d4h
tke influxdb-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2d4h
tke tke-application-api-6db6c5f8dd-l98gc 50m (0%) 500m (1%) 128Mi (0%) 1Gi (3%) 2d4h
tke tke-application-api-6db6c5f8dd-tbf2x 50m (0%) 500m (1%) 128Mi (0%) 1Gi (3%) 2d4h
tke tke-application-controller-6c769cc9cc-8v4zc 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-application-controller-6c769cc9cc-lbf2s 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-auth-api-6fcfdb454d-kv7bb 250m (0%) 500m (1%) 256Mi (0%) 512Mi (1%) 2d4h
tke tke-auth-api-6fcfdb454d-srxtq 250m (0%) 500m (1%) 256Mi (0%) 512Mi (1%) 2d4h
tke tke-auth-controller-5fbbdd9dd9-npjrd 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-auth-controller-5fbbdd9dd9-qj5fx 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-business-api-77ff89ff7-kwcnw 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-business-api-77ff89ff7-s94ml 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-business-controller-86879f65df-5wf94 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-business-controller-86879f65df-rz2kv 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-gateway-4gcpl 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-logagent-api-6d9b6fc4-7mg2r 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-logagent-api-6d9b6fc4-fbsqk 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-logagent-controller-c6c8bcbf6-gc6z6 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-logagent-controller-c6c8bcbf6-wc7bf 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-monitor-api-54bc7f44f6-7pl4s 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-monitor-api-54bc7f44f6-xxcl5 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-monitor-controller-bb56765d-5vl2x 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-monitor-controller-bb56765d-78p6v 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-notify-api-df68cb499-hzc8w 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-notify-api-df68cb499-qtnwz 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-notify-controller-85cf89b5bb-ngkqf 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-notify-controller-85cf89b5bb-p9bbx 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-platform-api-b4d45f4bc-4z5ln 100m (0%) 150m (0%) 256Mi (0%) 512Mi (1%) 2d4h
tke tke-platform-api-b4d45f4bc-j5dbq 100m (0%) 150m (0%) 256Mi (0%) 512Mi (1%) 2d4h
tke tke-platform-controller-546fc4fc97-phzgj 50m (0%) 100m (0%) 128Mi (0%) 1Gi (3%) 2d4h
tke tke-platform-controller-546fc4fc97-sbp9l 50m (0%) 100m (0%) 128Mi (0%) 1Gi (3%) 2d4h
tke tke-registry-api-55b6f778f7-j849x 50m (0%) 500m (1%) 128Mi (0%) 1Gi (3%) 2d4h
tke tke-registry-api-55b6f778f7-mkhwn 50m (0%) 500m (1%) 128Mi (0%) 1Gi (3%) 2d4h
tke tke-registry-controller-5bbb898bc5-hk6rw 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
tke tke-registry-controller-5bbb898bc5-r8p26 50m (0%) 100m (0%) 128Mi (0%) 256Mi (0%) 2d4h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 4537m (9%) 8582m (17%)
memory 6660Mi (21%) 18152Mi (58%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
tencent.com/vcuda-core 20 20
tencent.com/vcuda-memory 3 3
Events: <none>
GPU-Manager log
2022-01-27T15:14:57.338651164+08:00 copy /usr/local/host/lib/libnvidia-ml.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:14:57.353867616+08:00 copy /usr/local/host/lib/libnvidia-ml.so.1 to /usr/local/nvidia/lib
2022-01-27T15:14:57.365707593+08:00 copy /usr/local/host/lib/libnvidia-ml.so to /usr/local/nvidia/lib
2022-01-27T15:14:57.377306514+08:00 copy /usr/local/host/lib64/libnvidia-ml.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:14:57.390168998+08:00 copy /usr/local/host/lib64/libnvidia-ml.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:14:57.403748079+08:00 copy /usr/local/host/lib64/libnvidia-ml.so to /usr/local/nvidia/lib64
2022-01-27T15:14:57.752285035+08:00 copy /usr/local/host/lib/libcuda.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:14:57.789709907+08:00 copy /usr/local/host/lib/libcuda.so.1 to /usr/local/nvidia/lib
2022-01-27T15:14:57.801813172+08:00 copy /usr/local/host/lib/libcuda.so to /usr/local/nvidia/lib
2022-01-27T15:14:57.814628166+08:00 copy /usr/local/host/lib64/libcuda.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:14:57.855645410+08:00 copy /usr/local/host/lib64/libcuda.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:14:57.868238721+08:00 copy /usr/local/host/lib64/libcuda.so to /usr/local/nvidia/lib64
2022-01-27T15:14:58.221402375+08:00 copy /usr/local/host/lib/libnvidia-ptxjitcompiler.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:14:58.246832945+08:00 copy /usr/local/host/lib/libnvidia-ptxjitcompiler.so.1 to /usr/local/nvidia/lib
2022-01-27T15:14:58.260925497+08:00 copy /usr/local/host/lib/libnvidia-ptxjitcompiler.so to /usr/local/nvidia/lib
2022-01-27T15:14:58.272734588+08:00 copy /usr/local/host/lib64/libnvidia-ptxjitcompiler.so to /usr/local/nvidia/lib64
2022-01-27T15:14:58.284625993+08:00 copy /usr/local/host/lib64/libnvidia-ptxjitcompiler.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:14:58.306610183+08:00 copy /usr/local/host/lib64/libnvidia-ptxjitcompiler.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:14:59.021397422+08:00 copy /usr/local/host/lib/libnvidia-opencl.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:14:59.053452117+08:00 copy /usr/local/host/lib/libnvidia-opencl.so.1 to /usr/local/nvidia/lib
2022-01-27T15:14:59.064877049+08:00 copy /usr/local/host/lib64/libnvidia-opencl.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:14:59.093746142+08:00 copy /usr/local/host/lib64/libnvidia-opencl.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:14:59.436163522+08:00 copy /usr/local/host/lib/libnvidia-compiler.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:14:59.506278506+08:00 copy /usr/local/host/lib64/libnvidia-compiler.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:14:59.887614862+08:00 copy /usr/local/host/lib/vdpau/libvdpau_nvidia.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:14:59.900631544+08:00 copy /usr/local/host/lib/vdpau/libvdpau_nvidia.so.1 to /usr/local/nvidia/lib
2022-01-27T15:14:59.914070936+08:00 copy /usr/local/host/lib/libvdpau_nvidia.so to /usr/local/nvidia/lib
2022-01-27T15:14:59.925889044+08:00 copy /usr/local/host/lib64/vdpau/libvdpau_nvidia.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:14:59.937774449+08:00 copy /usr/local/host/lib64/vdpau/libvdpau_nvidia.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:14:59.949215912+08:00 copy /usr/local/host/lib64/libvdpau_nvidia.so to /usr/local/nvidia/lib64
2022-01-27T15:15:00.276571393+08:00 copy /usr/local/host/lib/libnvidia-encode.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:00.289498988+08:00 copy /usr/local/host/lib/libnvidia-encode.so.1 to /usr/local/nvidia/lib
2022-01-27T15:15:00.302716100+08:00 copy /usr/local/host/lib/libnvidia-encode.so to /usr/local/nvidia/lib
2022-01-27T15:15:00.314625777+08:00 copy /usr/local/host/lib64/libnvidia-encode.so to /usr/local/nvidia/lib64
2022-01-27T15:15:00.326298946+08:00 copy /usr/local/host/lib64/libnvidia-encode.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:00.337656884+08:00 copy /usr/local/host/lib64/libnvidia-encode.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:00.630485163+08:00 copy /usr/local/host/lib/libnvcuvid.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:00.646156009+08:00 copy /usr/local/host/lib/libnvcuvid.so.1 to /usr/local/nvidia/lib
2022-01-27T15:15:00.658620776+08:00 copy /usr/local/host/lib/libnvcuvid.so to /usr/local/nvidia/lib
2022-01-27T15:15:00.669256945+08:00 copy /usr/local/host/lib64/libnvcuvid.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:00.685615089+08:00 copy /usr/local/host/lib64/libnvcuvid.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:00.696418920+08:00 copy /usr/local/host/lib64/libnvcuvid.so to /usr/local/nvidia/lib64
2022-01-27T15:15:00.986894243+08:00 copy /usr/local/host/lib/libnvidia-fbc.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:00.998607169+08:00 copy /usr/local/host/lib/libnvidia-fbc.so.1 to /usr/local/nvidia/lib
2022-01-27T15:15:01.010264318+08:00 copy /usr/local/host/lib/libnvidia-fbc.so to /usr/local/nvidia/lib
2022-01-27T15:15:01.022598489+08:00 copy /usr/local/host/lib64/libnvidia-fbc.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:01.033919311+08:00 copy /usr/local/host/lib64/libnvidia-fbc.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:01.044758481+08:00 copy /usr/local/host/lib64/libnvidia-fbc.so to /usr/local/nvidia/lib64
2022-01-27T15:15:01.397140614+08:00 copy /usr/local/host/lib/libnvidia-ifr.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:01.410247941+08:00 copy /usr/local/host/lib/libnvidia-ifr.so.1 to /usr/local/nvidia/lib
2022-01-27T15:15:01.421835077+08:00 copy /usr/local/host/lib/libnvidia-ifr.so to /usr/local/nvidia/lib
2022-01-27T15:15:01.435419169+08:00 copy /usr/local/host/lib64/libnvidia-ifr.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:01.447542578+08:00 copy /usr/local/host/lib64/libnvidia-ifr.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:01.459584691+08:00 copy /usr/local/host/lib64/libnvidia-ifr.so to /usr/local/nvidia/lib64
2022-01-27T15:15:01.814857723+08:00 copy /usr/local/host/lib64/libGL.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:01.827774087+08:00 copy /usr/local/host/lib64/libGL.so to /usr/local/nvidia/lib64
2022-01-27T15:15:01.841417973+08:00 copy /usr/local/host/lib64/libGL.so.1.7.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:02.195583222+08:00 copy /usr/local/host/lib64/libGLX.so.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:02.207088332+08:00 copy /usr/local/host/lib64/libGLX.so to /usr/local/nvidia/lib64
2022-01-27T15:15:02.221320790+08:00 copy /usr/local/host/lib64/libGLX.so.0.0.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:02.559862624+08:00 copy /usr/local/host/lib64/libOpenGL.so.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:02.572714936+08:00 copy /usr/local/host/lib64/libOpenGL.so.0.0.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:02.584301511+08:00 copy /usr/local/host/lib64/libOpenGL.so to /usr/local/nvidia/lib64
2022-01-27T15:15:02.941268059+08:00 copy /usr/local/host/lib64/libGLESv1_CM.so to /usr/local/nvidia/lib64
2022-01-27T15:15:02.955333940+08:00 copy /usr/local/host/lib64/libGLESv1_CM.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:02.967220940+08:00 copy /usr/local/host/lib64/libGLESv1_CM.so.1.2.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:03.302953742+08:00 copy /usr/local/host/lib64/libGLESv2.so to /usr/local/nvidia/lib64
2022-01-27T15:15:03.314786648+08:00 copy /usr/local/host/lib64/libGLESv2.so.2 to /usr/local/nvidia/lib64
2022-01-27T15:15:03.326296602+08:00 copy /usr/local/host/lib64/libGLESv2.so.2.1.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:03.666788341+08:00 copy /usr/local/host/lib64/libEGL.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:03.679512295+08:00 copy /usr/local/host/lib64/libEGL.so to /usr/local/nvidia/lib64
2022-01-27T15:15:03.691450703+08:00 copy /usr/local/host/lib64/libEGL.so.1.1.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:04.021903077+08:00 copy /usr/local/host/lib64/libGLdispatch.so.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:04.032325862+08:00 copy /usr/local/host/lib64/libGLdispatch.so.0.0.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:04.043387116+08:00 copy /usr/local/host/lib64/libGLdispatch.so to /usr/local/nvidia/lib64
2022-01-27T15:15:04.377466614+08:00 copy /usr/local/host/lib/libGLX_nvidia.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:04.389392925+08:00 copy /usr/local/host/lib/libGLX_nvidia.so.0 to /usr/local/nvidia/lib
2022-01-27T15:15:04.399746140+08:00 copy /usr/local/host/lib64/libGLX_nvidia.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:04.409042528+08:00 copy /usr/local/host/lib64/libGLX_nvidia.so.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:04.775731657+08:00 copy /usr/local/host/lib/libEGL_nvidia.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:04.786955708+08:00 copy /usr/local/host/lib/libEGL_nvidia.so.0 to /usr/local/nvidia/lib
2022-01-27T15:15:04.798537412+08:00 copy /usr/local/host/lib64/libEGL_nvidia.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:04.809180290+08:00 copy /usr/local/host/lib64/libEGL_nvidia.so.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:05.103482513+08:00 copy /usr/local/host/lib/libGLESv2_nvidia.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:05.117117954+08:00 copy /usr/local/host/lib/libGLESv2_nvidia.so.2 to /usr/local/nvidia/lib
2022-01-27T15:15:05.129766669+08:00 copy /usr/local/host/lib64/libGLESv2_nvidia.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:05.141248921+08:00 copy /usr/local/host/lib64/libGLESv2_nvidia.so.2 to /usr/local/nvidia/lib64
2022-01-27T15:15:05.495297811+08:00 copy /usr/local/host/lib/libGLESv1_CM_nvidia.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:05.510392919+08:00 copy /usr/local/host/lib/libGLESv1_CM_nvidia.so.1 to /usr/local/nvidia/lib
2022-01-27T15:15:05.522212773+08:00 copy /usr/local/host/lib64/libGLESv1_CM_nvidia.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:05.534953503+08:00 copy /usr/local/host/lib64/libGLESv1_CM_nvidia.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:05.841311187+08:00 copy /usr/local/host/lib/libnvidia-eglcore.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:05.881322726+08:00 copy /usr/local/host/lib64/libnvidia-eglcore.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:06.252357737+08:00 copy /usr/local/host/lib64/libnvidia-egl-wayland.so.1.1.7 to /usr/local/nvidia/lib64
2022-01-27T15:15:06.262656714+08:00 copy /usr/local/host/lib64/libnvidia-egl-wayland.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:06.628317661+08:00 copy /usr/local/host/lib/libnvidia-glcore.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:06.663536096+08:00 copy /usr/local/host/lib64/libnvidia-glcore.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:07.054514561+08:00 copy /usr/local/host/lib/libnvidia-tls.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:07.068586386+08:00 copy /usr/local/host/lib64/libnvidia-tls.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:07.429548212+08:00 copy /usr/local/host/lib/libnvidia-glsi.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:07.443867613+08:00 copy /usr/local/host/lib64/libnvidia-glsi.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:07.788040611+08:00 copy /usr/local/host/lib/libnvidia-opticalflow.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:07.800287508+08:00 copy /usr/local/host/lib/libnvidia-opticalflow.so.1 to /usr/local/nvidia/lib
2022-01-27T15:15:07.811975123+08:00 copy /usr/local/host/lib/libnvidia-opticalflow.so to /usr/local/nvidia/lib
2022-01-27T15:15:07.823496372+08:00 copy /usr/local/host/lib64/libnvidia-opticalflow.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:07.835343610+08:00 copy /usr/local/host/lib64/libnvidia-opticalflow.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:07.846720954+08:00 copy /usr/local/host/lib64/libnvidia-opticalflow.so to /usr/local/nvidia/lib64
2022-01-27T15:15:08.172378279+08:00 copy /usr/local/host/bin/nvidia-cuda-mps-control to /usr/local/nvidia/bin/
2022-01-27T15:15:08.513841922+08:00 copy /usr/local/host/bin/nvidia-cuda-mps-server to /usr/local/nvidia/bin/
2022-01-27T15:15:08.856701556+08:00 copy /usr/local/host/bin/nvidia-debugdump to /usr/local/nvidia/bin/
2022-01-27T15:15:09.201883209+08:00 copy /usr/local/host/bin/nvidia-persistenced to /usr/local/nvidia/bin/
2022-01-27T15:15:09.544941255+08:00 copy /usr/local/host/bin/nvidia-smi to /usr/local/nvidia/bin/
2022-01-27T15:15:09.578972619+08:00 rebuild ldcache
2022-01-27T15:15:09.589852232+08:00 launch gpu manager
2022-01-27T15:15:09.605431237+08:00 I0127 07:15:09.605194 44050 app.go:92] Wait for internal server ready
2022-01-27T15:15:09.607011470+08:00 I0127 07:15:09.606825 44050 volume.go:133] Find binaries: [/usr/local/nvidia/bin/nvidia-cuda-mps-control /usr/local/nvidia/bin/nvidia-cuda-mps-server /usr/local/nvidia/bin/nvidia-debugdump /usr/local/nvidia/bin/nvidia-persistenced /usr/local/nvidia/bin/nvidia-smi /usr/bin/gpu-client]
2022-01-27T15:15:09.609057613+08:00 I0127 07:15:09.608892 44050 volume.go:138] Find 32bit libraries: [/usr/local/nvidia/lib/libvdpau_nvidia.so.470.82.01 /usr/local/nvidia/lib/libnvidia-tls.so.470.82.01 /usr/local/nvidia/lib/libnvidia-ptxjitcompiler.so.470.82.01 /usr/local/nvidia/lib/libnvidia-opticalflow.so.470.82.01 /usr/local/nvidia/lib/libnvidia-opencl.so.470.82.01 /usr/local/nvidia/lib/libnvidia-ml.so.470.82.01 /usr/local/nvidia/lib/libnvidia-ifr.so.470.82.01 /usr/local/nvidia/lib/libnvidia-glsi.so.470.82.01 /usr/local/nvidia/lib/libnvidia-glcore.so.470.82.01 /usr/local/nvidia/lib/libnvidia-fbc.so.470.82.01 /usr/local/nvidia/lib/libnvidia-encode.so.470.82.01 /usr/local/nvidia/lib/libnvidia-eglcore.so.470.82.01 /usr/local/nvidia/lib/libnvidia-compiler.so.470.82.01 /usr/local/nvidia/lib/libnvcuvid.so.470.82.01 /usr/local/nvidia/lib/libcuda.so.470.82.01 /usr/local/nvidia/lib/libGLX_nvidia.so.470.82.01 /usr/local/nvidia/lib/libGLESv2_nvidia.so.470.82.01 /usr/local/nvidia/lib/libGLESv1_CM_nvidia.so.470.82.01 /usr/local/nvidia/lib/libEGL_nvidia.so.470.82.01]
2022-01-27T15:15:09.609081308+08:00 I0127 07:15:09.608935 44050 volume.go:139] Find 64bit libraries: [/usr/local/nvidia/lib64/libvdpau_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-tls.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-ptxjitcompiler.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-opticalflow.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-opencl.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-ml.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-ifr.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-glsi.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-glcore.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-fbc.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-encode.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-eglcore.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-egl-wayland.so.1.1.7 /usr/local/nvidia/lib64/libnvidia-compiler.so.470.82.01 /usr/local/nvidia/lib64/libnvcuvid.so.470.82.01 /usr/local/nvidia/lib64/libcuda.so.470.82.01 /usr/lib64/libcuda-control.so /usr/local/nvidia/lib64/libOpenGL.so.0.0.0 /usr/local/nvidia/lib64/libGLdispatch.so.0.0.0 /usr/local/nvidia/lib64/libGLX_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libGLX.so.0.0.0 /usr/local/nvidia/lib64/libGLESv2_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libGLESv2.so.2.1.0 /usr/local/nvidia/lib64/libGLESv1_CM_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libGLESv1_CM.so.1.2.0 /usr/local/nvidia/lib64/libGL.so.1.7.0 /usr/local/nvidia/lib64/libEGL_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libEGL.so.1.1.0]
2022-01-27T15:15:09.610118040+08:00 I0127 07:15:09.609986 44050 volume.go:133] Find binaries: [/usr/local/nvidia/bin/nvidia-cuda-mps-control /usr/local/nvidia/bin/nvidia-cuda-mps-server /usr/local/nvidia/bin/nvidia-debugdump /usr/local/nvidia/bin/nvidia-persistenced /usr/local/nvidia/bin/nvidia-smi]
2022-01-27T15:15:09.614388448+08:00 I0127 07:15:09.614219 44050 volume.go:138] Find 32bit libraries: [/usr/local/nvidia/lib/libvdpau_nvidia.so.470.82.01 /usr/local/nvidia/lib/libnvidia-tls.so.470.82.01 /usr/local/nvidia/lib/libnvidia-ptxjitcompiler.so.470.82.01 /usr/local/nvidia/lib/libnvidia-opticalflow.so.470.82.01 /usr/local/nvidia/lib/libnvidia-opencl.so.470.82.01 /usr/local/nvidia/lib/libnvidia-ml.so.470.82.01 /usr/local/nvidia/lib/libnvidia-ifr.so.470.82.01 /usr/local/nvidia/lib/libnvidia-glsi.so.470.82.01 /usr/local/nvidia/lib/libnvidia-glcore.so.470.82.01 /usr/local/nvidia/lib/libnvidia-fbc.so.470.82.01 /usr/local/nvidia/lib/libnvidia-encode.so.470.82.01 /usr/local/nvidia/lib/libnvidia-eglcore.so.470.82.01 /usr/local/nvidia/lib/libnvidia-compiler.so.470.82.01 /usr/local/nvidia/lib/libnvcuvid.so.470.82.01 /usr/local/nvidia/lib/libcuda.so.470.82.01 /usr/local/nvidia/lib/libGLX_nvidia.so.470.82.01 /usr/local/nvidia/lib/libGLESv2_nvidia.so.470.82.01 /usr/local/nvidia/lib/libGLESv1_CM_nvidia.so.470.82.01 /usr/local/nvidia/lib/libEGL_nvidia.so.470.82.01]
2022-01-27T15:15:09.614407646+08:00 I0127 07:15:09.614264 44050 volume.go:139] Find 64bit libraries: [/usr/local/nvidia/lib64/libvdpau_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-tls.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-ptxjitcompiler.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-opticalflow.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-opencl.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-ml.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-ifr.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-glsi.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-glcore.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-fbc.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-encode.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-eglcore.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-egl-wayland.so.1.1.7 /usr/local/nvidia/lib64/libnvidia-compiler.so.470.82.01 /usr/local/nvidia/lib64/libnvcuvid.so.470.82.01 /usr/local/nvidia/lib64/libcuda.so.470.82.01 /usr/local/nvidia/lib64/libOpenGL.so.0.0.0 /usr/local/nvidia/lib64/libGLdispatch.so.0.0.0 /usr/local/nvidia/lib64/libGLX_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libGLX.so.0.0.0 /usr/local/nvidia/lib64/libGLESv2_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libGLESv2.so.2.1.0 /usr/local/nvidia/lib64/libGLESv1_CM_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libGLESv1_CM.so.1.2.0 /usr/local/nvidia/lib64/libGL.so.1.7.0 /usr/local/nvidia/lib64/libEGL_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libEGL.so.1.1.0]
2022-01-27T15:15:09.614414781+08:00 I0127 07:15:09.614309 44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-cuda-mps-control to /etc/gpu-manager/vdriver/nvidia/bin
2022-01-27T15:15:09.614958244+08:00 I0127 07:15:09.614857 44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-cuda-mps-server to /etc/gpu-manager/vdriver/nvidia/bin
2022-01-27T15:15:09.615319757+08:00 I0127 07:15:09.615222 44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-debugdump to /etc/gpu-manager/vdriver/nvidia/bin
2022-01-27T15:15:09.615929873+08:00 I0127 07:15:09.615841 44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-persistenced to /etc/gpu-manager/vdriver/nvidia/bin
2022-01-27T15:15:09.616484687+08:00 I0127 07:15:09.616400 44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-smi to /etc/gpu-manager/vdriver/nvidia/bin
2022-01-27T15:15:09.617562557+08:00 I0127 07:15:09.617472 44050 volume.go:176] Mirror /usr/bin/gpu-client to /etc/gpu-manager/vdriver/nvidia/bin
2022-01-27T15:15:09.638130223+08:00 I0127 07:15:09.637990 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libvdpau_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.639448607+08:00 I0127 07:15:09.639358 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-tls.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.639847885+08:00 I0127 07:15:09.639762 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-ptxjitcompiler.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.654420601+08:00 I0127 07:15:09.654257 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-opticalflow.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.654894720+08:00 I0127 07:15:09.654786 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-opencl.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.674533377+08:00 I0127 07:15:09.674368 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-ml.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.677278132+08:00 I0127 07:15:09.677145 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-ifr.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.677920586+08:00 I0127 07:15:09.677824 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-glsi.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.678857755+08:00 I0127 07:15:09.678786 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-glcore.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.711395371+08:00 I0127 07:15:09.711228 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-fbc.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.711971243+08:00 I0127 07:15:09.711874 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-encode.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.712518377+08:00 I0127 07:15:09.712389 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-eglcore.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.742380053+08:00 I0127 07:15:09.742212 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-compiler.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.805781712+08:00 I0127 07:15:09.805637 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvcuvid.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.811704362+08:00 I0127 07:15:09.811559 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libcuda.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.836848789+08:00 I0127 07:15:09.836743 44050 volume.go:185] Driver version: 470.0
2022-01-27T15:15:09.836873936+08:00 I0127 07:15:09.836761 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libGLX_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.838548410+08:00 I0127 07:15:09.838447 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libGLESv2_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.839075531+08:00 I0127 07:15:09.838947 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libGLESv1_CM_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.839516821+08:00 I0127 07:15:09.839393 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libEGL_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.842164547+08:00 I0127 07:15:09.841988 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libvdpau_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.843602999+08:00 I0127 07:15:09.843474 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-tls.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.843980882+08:00 I0127 07:15:09.843864 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-ptxjitcompiler.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.859241489+08:00 I0127 07:15:09.859094 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-opticalflow.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.859691405+08:00 I0127 07:15:09.859536 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-opencl.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.882554855+08:00 I0127 07:15:09.882382 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-ml.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.885252141+08:00 I0127 07:15:09.885112 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-ifr.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.885872805+08:00 I0127 07:15:09.885773 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-glsi.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.886845510+08:00 I0127 07:15:09.886723 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-glcore.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.920564818+08:00 I0127 07:15:09.920398 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-fbc.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.921103454+08:00 I0127 07:15:09.921002 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-encode.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.921610440+08:00 I0127 07:15:09.921474 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-eglcore.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.955770883+08:00 I0127 07:15:09.955584 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-egl-wayland.so.1.1.7 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.956240461+08:00 I0127 07:15:09.956096 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-compiler.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.029161286+08:00 I0127 07:15:10.029032 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvcuvid.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.037381521+08:00 I0127 07:15:10.037237 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libcuda.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.072454944+08:00 I0127 07:15:10.072296 44050 volume.go:185] Driver version: 470.0
2022-01-27T15:15:10.072479484+08:00 I0127 07:15:10.072316 44050 volume.go:176] Mirror /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.073931016+08:00 I0127 07:15:10.073817 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libOpenGL.so.0.0.0 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.074624543+08:00 I0127 07:15:10.074515 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLdispatch.so.0.0.0 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.075807915+08:00 I0127 07:15:10.075715 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLX_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.078407236+08:00 I0127 07:15:10.078327 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLX.so.0.0.0 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.080281753+08:00 I0127 07:15:10.080191 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv2_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.081492821+08:00 I0127 07:15:10.081363 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv2.so.2.1.0 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.082613821+08:00 I0127 07:15:10.082482 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv1_CM_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.083597334+08:00 I0127 07:15:10.083469 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv1_CM.so.1.2.0 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.084542206+08:00 I0127 07:15:10.084415 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGL.so.1.7.0 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.086745915+08:00 I0127 07:15:10.086607 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libEGL_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.090284804+08:00 I0127 07:15:10.090168 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libEGL.so.1.1.0 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.091062468+08:00 I0127 07:15:10.090970 44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-cuda-mps-control to /etc/gpu-manager/vdriver/origin/bin
2022-01-27T15:15:10.091526374+08:00 I0127 07:15:10.091471 44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-cuda-mps-server to /etc/gpu-manager/vdriver/origin/bin
2022-01-27T15:15:10.092071065+08:00 I0127 07:15:10.091941 44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-debugdump to /etc/gpu-manager/vdriver/origin/bin
2022-01-27T15:15:10.092743201+08:00 I0127 07:15:10.092611 44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-persistenced to /etc/gpu-manager/vdriver/origin/bin
2022-01-27T15:15:10.093329998+08:00 I0127 07:15:10.093244 44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-smi to /etc/gpu-manager/vdriver/origin/bin
2022-01-27T15:15:10.094585093+08:00 I0127 07:15:10.094501 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libvdpau_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.096084665+08:00 I0127 07:15:10.095959 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-tls.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.096466696+08:00 I0127 07:15:10.096347 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-ptxjitcompiler.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.113695447+08:00 I0127 07:15:10.113559 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-opticalflow.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.114315495+08:00 I0127 07:15:10.114248 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-opencl.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.137646435+08:00 I0127 07:15:10.137540 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-ml.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.140070290+08:00 I0127 07:15:10.139897 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-ifr.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.140649545+08:00 I0127 07:15:10.140560 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-glsi.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.141657828+08:00 I0127 07:15:10.141530 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-glcore.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.177298095+08:00 I0127 07:15:10.177150 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-fbc.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.178047202+08:00 I0127 07:15:10.177906 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-encode.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.178855251+08:00 I0127 07:15:10.178744 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-eglcore.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.227882827+08:00 I0127 07:15:10.227722 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-compiler.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.311913030+08:00 I0127 07:15:10.311770 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvcuvid.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.318614913+08:00 I0127 07:15:10.318483 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libcuda.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.347139455+08:00 I0127 07:15:10.347009 44050 volume.go:185] Driver version: 470.0
2022-01-27T15:15:10.347165855+08:00 I0127 07:15:10.347028 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libGLX_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.348952866+08:00 I0127 07:15:10.348889 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libGLESv2_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.349464258+08:00 I0127 07:15:10.349390 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libGLESv1_CM_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.349878024+08:00 I0127 07:15:10.349825 44050 volume.go:176] Mirror /usr/local/nvidia/lib/libEGL_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.351781633+08:00 I0127 07:15:10.351714 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libvdpau_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.353081536+08:00 I0127 07:15:10.352996 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-tls.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.353428512+08:00 I0127 07:15:10.353306 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-ptxjitcompiler.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.367311128+08:00 I0127 07:15:10.367158 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-opticalflow.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.367729244+08:00 I0127 07:15:10.367618 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-opencl.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.391425529+08:00 I0127 07:15:10.391265 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-ml.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.394071588+08:00 I0127 07:15:10.393962 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-ifr.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.394730318+08:00 I0127 07:15:10.394617 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-glsi.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.395761756+08:00 I0127 07:15:10.395689 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-glcore.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.436364636+08:00 I0127 07:15:10.436229 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-fbc.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.437002370+08:00 I0127 07:15:10.436909 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-encode.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.437472487+08:00 I0127 07:15:10.437411 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-eglcore.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.474168277+08:00 I0127 07:15:10.474005 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-egl-wayland.so.1.1.7 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.474726464+08:00 I0127 07:15:10.474608 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-compiler.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.540799094+08:00 I0127 07:15:10.540689 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvcuvid.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.550966010+08:00 I0127 07:15:10.550850 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libcuda.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.585864559+08:00 I0127 07:15:10.585756 44050 volume.go:185] Driver version: 470.0
2022-01-27T15:15:10.585897631+08:00 I0127 07:15:10.585778 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libOpenGL.so.0.0.0 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.586628457+08:00 I0127 07:15:10.586528 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLdispatch.so.0.0.0 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.587898431+08:00 I0127 07:15:10.587816 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLX_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.590044638+08:00 I0127 07:15:10.589945 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLX.so.0.0.0 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.590593784+08:00 I0127 07:15:10.590468 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv2_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.591173574+08:00 I0127 07:15:10.591037 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv2.so.2.1.0 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.591647319+08:00 I0127 07:15:10.591579 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv1_CM_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.592241121+08:00 I0127 07:15:10.592122 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv1_CM.so.1.2.0 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.592757185+08:00 I0127 07:15:10.592620 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGL.so.1.7.0 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.593922948+08:00 I0127 07:15:10.593834 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libEGL_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.595955171+08:00 I0127 07:15:10.595886 44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libEGL.so.1.1.0 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.597452250+08:00 I0127 07:15:10.597370 44050 volume.go:205] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib/libcuda.so.1
2022-01-27T15:15:10.598282680+08:00 I0127 07:15:10.598225 44050 volume.go:216] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib/libcuda.so
2022-01-27T15:15:10.599117899+08:00 I0127 07:15:10.599054 44050 volume.go:205] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib64/libcuda.so.1
2022-01-27T15:15:10.599971725+08:00 I0127 07:15:10.599883 44050 volume.go:216] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib64/libcuda.so
2022-01-27T15:15:10.600847591+08:00 I0127 07:15:10.600761 44050 volume.go:205] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib/libnvidia-ml.so.1
2022-01-27T15:15:10.601695896+08:00 I0127 07:15:10.601619 44050 volume.go:216] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib/libnvidia-ml.so
2022-01-27T15:15:10.602574446+08:00 I0127 07:15:10.602472 44050 volume.go:205] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib64/libnvidia-ml.so.1
2022-01-27T15:15:10.603436041+08:00 I0127 07:15:10.603325 44050 volume.go:216] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib64/libnvidia-ml.so
2022-01-27T15:15:10.603451301+08:00 I0127 07:15:10.603340 44050 volume.go:152] Volume manager is running
2022-01-27T15:15:10.603456583+08:00 E0127 07:15:10.603381 44050 server.go:133] Unable to set Type=notify in systemd service file?
2022-01-27T15:15:10.603470085+08:00 W0127 07:15:10.603401 44050 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2022-01-27T15:15:10.605219002+08:00 I0127 07:15:10.605101 44050 logs.go:79] parsed scheme: ""
2022-01-27T15:15:10.605233835+08:00 I0127 07:15:10.605114 44050 logs.go:79] scheme "" not registered, fallback to default scheme
2022-01-27T15:15:10.605238595+08:00 I0127 07:15:10.605123 44050 logs.go:79] ccResolverWrapper: sending update to cc: {[{/run/containerd/containerd.sock 0 <nil>}] <nil>}
2022-01-27T15:15:10.605242981+08:00 I0127 07:15:10.605147 44050 logs.go:79] ClientConn switching balancer to "pick_first"
2022-01-27T15:15:10.605276764+08:00 I0127 07:15:10.605239 44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc000123460, CONNECTING
2022-01-27T15:15:10.605378798+08:00 I0127 07:15:10.605320 44050 app.go:92] Wait for internal server ready
2022-01-27T15:15:10.605933302+08:00 I0127 07:15:10.605861 44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc000123460, READY
2022-01-27T15:15:10.608222773+08:00 I0127 07:15:10.608170 44050 runtime.go:69] Container runtime is containerd
2022-01-27T15:15:10.608237690+08:00 I0127 07:15:10.608202 44050 server.go:157] Container runtime manager is running
2022-01-27T15:15:10.608469937+08:00 I0127 07:15:10.608407 44050 reflector.go:150] Starting reflector *v1.Pod (1m0s) from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105
2022-01-27T15:15:10.608490996+08:00 I0127 07:15:10.608432 44050 reflector.go:185] Listing and watching *v1.Pod from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105
2022-01-27T15:15:11.605702531+08:00 I0127 07:15:11.605496 44050 app.go:92] Wait for internal server ready
2022-01-27T15:15:11.608649989+08:00 I0127 07:15:11.608505 44050 watchdog.go:64] Pod cache is running
2022-01-27T15:15:11.608694722+08:00 I0127 07:15:11.608533 44050 server.go:160] Watchdog is running
2022-01-27T15:15:11.608704756+08:00 I0127 07:15:11.608559 44050 label.go:102] Labeler for hostname 192.168.41.104
2022-01-27T15:15:11.660964248+08:00 I0127 07:15:11.660820 44050 label.go:153] Auto label is running
2022-01-27T15:15:11.660994136+08:00 I0127 07:15:11.660850 44050 server.go:167] Load container response data
2022-01-27T15:15:11.661034245+08:00 I0127 07:15:11.660957 44050 util.go:131] Try NUMA checkpoint data format
2022-01-27T15:15:11.661629051+08:00 I0127 07:15:11.661449 44050 manager.go:69] Insert f96b0b34-8379-4581-9c87-2e98637f2fc2/empty9 allocResp
2022-01-27T15:15:11.661655118+08:00 I0127 07:15:11.661521 44050 manager.go:203] Start vDevice watcher
2022-01-27T15:15:11.661702695+08:00 I0127 07:15:11.661586 44050 util.go:206] Pod tke-platform-api-b4d45f4bc-4z5ln in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661711925+08:00 I0127 07:15:11.661599 44050 util.go:206] Pod tke-monitor-controller-bb56765d-78p6v in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661719272+08:00 I0127 07:15:11.661606 44050 util.go:206] Pod tke-application-api-6db6c5f8dd-tbf2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661726285+08:00 I0127 07:15:11.661614 44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-gc6z6 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661733532+08:00 I0127 07:15:11.661622 44050 util.go:206] Pod gpu-manager-daemonset-vv7zz in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.661741012+08:00 I0127 07:15:11.661630 44050 util.go:206] Pod tke-auth-api-6fcfdb454d-kv7bb in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661747859+08:00 I0127 07:15:11.661637 44050 util.go:206] Pod influxdb-0 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661801675+08:00 I0127 07:15:11.661645 44050 util.go:206] Pod tke-platform-controller-546fc4fc97-phzgj in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661835995+08:00 I0127 07:15:11.661659 44050 util.go:206] Pod tke-platform-api-b4d45f4bc-j5dbq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661843678+08:00 I0127 07:15:11.661681 44050 util.go:206] Pod tke-registry-api-55b6f778f7-mkhwn in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661850598+08:00 I0127 07:15:11.661688 44050 util.go:206] Pod coredns-6c654f5cc6-58bz2 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.661858152+08:00 I0127 07:15:11.661694 44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-hk6rw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661864480+08:00 I0127 07:15:11.661702 44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-xxcl5 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661871729+08:00 I0127 07:15:11.661710 44050 util.go:206] Pod galaxy-daemonset-tc6t6 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.661877872+08:00 I0127 07:15:11.661723 44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-npjrd in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661883938+08:00 I0127 07:15:11.661730 44050 util.go:206] Pod tke-gateway-4gcpl in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661890038+08:00 I0127 07:15:11.661742 44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-ngkqf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661896452+08:00 I0127 07:15:11.661752 44050 util.go:206] Pod tke-business-api-77ff89ff7-s94ml in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661925284+08:00 I0127 07:15:11.661762 44050 util.go:206] Pod tke-application-api-6db6c5f8dd-l98gc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661954530+08:00 I0127 07:15:11.661776 44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-p9bbx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661966864+08:00 I0127 07:15:11.661787 44050 util.go:206] Pod kube-proxy-26g7w in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.661977941+08:00 I0127 07:15:11.661798 44050 util.go:206] Pod tke-business-api-77ff89ff7-kwcnw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661988461+08:00 I0127 07:15:11.661810 44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-fbsqk in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661998930+08:00 I0127 07:15:11.661817 44050 util.go:206] Pod tke-business-controller-86879f65df-rz2kv in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662010047+08:00 I0127 07:15:11.661830 44050 util.go:206] Pod tke-notify-api-df68cb499-qtnwz in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662039020+08:00 I0127 07:15:11.661837 44050 util.go:206] Pod tke-registry-api-55b6f778f7-j849x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662075124+08:00 I0127 07:15:11.661847 44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-r8p26 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662090697+08:00 I0127 07:15:11.661862 44050 util.go:206] Pod gpu-quota-admission-76cfff49b6-gs8b5 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662098611+08:00 I0127 07:15:11.661874 44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-qj5fx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662105957+08:00 I0127 07:15:11.661890 44050 util.go:206] Pod kube-controller-manager-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662138381+08:00 I0127 07:15:11.661903 44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-7pl4s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662146684+08:00 I0127 07:15:11.661918 44050 util.go:206] Pod kube-apiserver-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662153574+08:00 I0127 07:15:11.661929 44050 util.go:206] Pod flannel-pmfcv in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662160511+08:00 I0127 07:15:11.661942 44050 util.go:206] Pod tke-application-controller-6c769cc9cc-8v4zc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662167361+08:00 I0127 07:15:11.661953 44050 util.go:206] Pod coredns-6c654f5cc6-wsg7s in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662174497+08:00 I0127 07:15:11.661965 44050 util.go:206] Pod tke-monitor-controller-bb56765d-5vl2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662181251+08:00 I0127 07:15:11.661980 44050 util.go:206] Pod kube-scheduler-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662188054+08:00 I0127 07:15:11.661992 44050 util.go:206] Pod etcd-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662195121+08:00 I0127 07:15:11.662007 44050 util.go:206] Pod tke-platform-controller-546fc4fc97-sbp9l in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662201808+08:00 I0127 07:15:11.662037 44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-7mg2r in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662208548+08:00 I0127 07:15:11.662052 44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-wc7bf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662226041+08:00 I0127 07:15:11.662066 44050 util.go:206] Pod tke-application-controller-6c769cc9cc-lbf2s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662233368+08:00 I0127 07:15:11.662082 44050 util.go:206] Pod metrics-server-v0.3.6-794ccd69c8-8sqxb in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662239985+08:00 I0127 07:15:11.662093 44050 util.go:206] Pod tke-notify-api-df68cb499-hzc8w in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662246948+08:00 I0127 07:15:11.662107 44050 util.go:206] Pod tke-auth-api-6fcfdb454d-srxtq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662253681+08:00 I0127 07:15:11.662115 44050 util.go:206] Pod tke-business-controller-86879f65df-5wf94 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662635853+08:00 I0127 07:15:11.662572 44050 manager.go:232] Recover vDevice server for /etc/gpu-manager/vm/f96b0b34-8379-4581-9c87-2e98637f2fc2
2022-01-27T15:15:11.662804880+08:00 I0127 07:15:11.662745 44050 manager.go:245] Recover vDevice server for /etc/gpu-manager/vm/f96b0b34-8379-4581-9c87-2e98637f2fc2
2022-01-27T15:15:11.662839423+08:00 I0127 07:15:11.662796 44050 manager.go:199] Virtual manager is running
2022-01-27T15:15:11.662859680+08:00 I0127 07:15:11.662811 44050 manager.go:270] Starting garbage directory collector
2022-01-27T15:15:11.662909969+08:00 I0127 07:15:11.662863 44050 util.go:206] Pod tke-monitor-controller-bb56765d-5vl2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662918503+08:00 I0127 07:15:11.662881 44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-qj5fx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662924866+08:00 I0127 07:15:11.662892 44050 util.go:206] Pod kube-controller-manager-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662937095+08:00 I0127 07:15:11.662901 44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-7pl4s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662956341+08:00 I0127 07:15:11.662894 44050 manager.go:354] Starting process vm events
2022-01-27T15:15:11.662969852+08:00 I0127 07:15:11.662908 44050 util.go:206] Pod kube-apiserver-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662976389+08:00 I0127 07:15:11.662934 44050 util.go:206] Pod flannel-pmfcv in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662982318+08:00 I0127 07:15:11.662949 44050 util.go:206] Pod tke-application-controller-6c769cc9cc-8v4zc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662988055+08:00 I0127 07:15:11.662956 44050 util.go:206] Pod coredns-6c654f5cc6-wsg7s in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663010438+08:00 I0127 07:15:11.662964 44050 util.go:206] Pod kube-scheduler-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663016629+08:00 I0127 07:15:11.662972 44050 util.go:206] Pod etcd-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663035716+08:00 I0127 07:15:11.662980 44050 util.go:206] Pod tke-business-controller-86879f65df-5wf94 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663071806+08:00 I0127 07:15:11.662987 44050 util.go:206] Pod tke-platform-controller-546fc4fc97-sbp9l in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663081499+08:00 I0127 07:15:11.662994 44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-7mg2r in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663115482+08:00 I0127 07:15:11.663001 44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-wc7bf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663154719+08:00 I0127 07:15:11.663012 44050 util.go:206] Pod tke-application-controller-6c769cc9cc-lbf2s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663164349+08:00 I0127 07:15:11.663022 44050 util.go:206] Pod metrics-server-v0.3.6-794ccd69c8-8sqxb in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663172519+08:00 I0127 07:15:11.663029 44050 util.go:206] Pod tke-notify-api-df68cb499-hzc8w in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663180189+08:00 I0127 07:15:11.663036 44050 util.go:206] Pod tke-auth-api-6fcfdb454d-srxtq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663187556+08:00 I0127 07:15:11.663043 44050 util.go:206] Pod tke-platform-api-b4d45f4bc-4z5ln in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663194640+08:00 I0127 07:15:11.663051 44050 util.go:206] Pod tke-monitor-controller-bb56765d-78p6v in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663202110+08:00 I0127 07:15:11.663058 44050 util.go:206] Pod tke-application-api-6db6c5f8dd-tbf2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663209243+08:00 I0127 07:15:11.663066 44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-gc6z6 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663215916+08:00 I0127 07:15:11.663078 44050 util.go:206] Pod gpu-manager-daemonset-vv7zz in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663222833+08:00 I0127 07:15:11.663085 44050 util.go:206] Pod tke-auth-api-6fcfdb454d-kv7bb in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663243787+08:00 I0127 07:15:11.663101 44050 util.go:206] Pod galaxy-daemonset-tc6t6 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663264773+08:00 I0127 07:15:11.663110 44050 util.go:206] Pod influxdb-0 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663284400+08:00 I0127 07:15:11.663119 44050 util.go:206] Pod tke-platform-controller-546fc4fc97-phzgj in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663291598+08:00 I0127 07:15:11.663126 44050 util.go:206] Pod tke-platform-api-b4d45f4bc-j5dbq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663319659+08:00 I0127 07:15:11.663135 44050 util.go:206] Pod tke-registry-api-55b6f778f7-mkhwn in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663332158+08:00 I0127 07:15:11.663147 44050 util.go:206] Pod coredns-6c654f5cc6-58bz2 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663342038+08:00 I0127 07:15:11.663154 44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-hk6rw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663352473+08:00 I0127 07:15:11.663161 44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-xxcl5 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663360622+08:00 I0127 07:15:11.663174 44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-npjrd in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663367115+08:00 I0127 07:15:11.663181 44050 util.go:206] Pod tke-gateway-4gcpl in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663373616+08:00 I0127 07:15:11.663190 44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-ngkqf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663380016+08:00 I0127 07:15:11.663199 44050 util.go:206] Pod tke-business-api-77ff89ff7-s94ml in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663386281+08:00 I0127 07:15:11.663206 44050 util.go:206] Pod tke-application-api-6db6c5f8dd-l98gc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663393105+08:00 I0127 07:15:11.663214 44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-p9bbx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663399539+08:00 I0127 07:15:11.663222 44050 util.go:206] Pod kube-proxy-26g7w in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663417302+08:00 I0127 07:15:11.663229 44050 util.go:206] Pod tke-business-api-77ff89ff7-kwcnw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663424294+08:00 I0127 07:15:11.663236 44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-fbsqk in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663430748+08:00 I0127 07:15:11.663247 44050 util.go:206] Pod tke-business-controller-86879f65df-rz2kv in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663437001+08:00 I0127 07:15:11.663256 44050 util.go:206] Pod tke-notify-api-df68cb499-qtnwz in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663443414+08:00 I0127 07:15:11.663263 44050 util.go:206] Pod tke-registry-api-55b6f778f7-j849x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663449854+08:00 I0127 07:15:11.663270 44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-r8p26 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663456228+08:00 I0127 07:15:11.663277 44050 util.go:206] Pod gpu-quota-admission-76cfff49b6-gs8b5 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.668414267+08:00 I0127 07:15:11.668300 44050 tree.go:182] Detect 1 gpu cards
2022-01-27T15:15:11.674361124+08:00 E0127 07:15:11.674269 44050 tree.go:333] No topology level found at 0
2022-01-27T15:15:11.674375390+08:00 I0127 07:15:11.674291 44050 tree.go:336] Only one card topology
2022-01-27T15:15:11.676188533+08:00 I0127 07:15:11.676107 44050 tree.go:119] Update device information
2022-01-27T15:15:11.683856601+08:00 I0127 07:15:11.683735 44050 tree.go:135] node 0, pid: [], memory: 0, utilization: 0, pendingReset: false
2022-01-27T15:15:11.684440017+08:00 I0127 07:15:11.684324 44050 allocator.go:275] Load extra config from /etc/gpu-manager/extra-config.json
2022-01-27T15:15:11.684782237+08:00 I0127 07:15:11.684713 44050 allocator.go:196] Nvidia GPU "/dev/nvidia0" is in use by container: "empty9"
2022-01-27T15:15:11.684795359+08:00 I0127 07:15:11.684733 44050 allocator.go:197] Uid: f96b0b34-8379-4581-9c87-2e98637f2fc2, Name: empty9, util: 20, memory: 805306368
2022-01-27T15:15:11.684819400+08:00 I0127 07:15:11.684789 44050 tree.go:487] Occupy /dev/nvidia0 with 20 805306368, mask 1
2022-01-27T15:15:11.684826375+08:00 I0127 07:15:11.684802 44050 tree.go:514] Occupy /dev/nvidia0 parent 1
2022-01-27T15:15:11.684840315+08:00 I0127 07:15:11.684814 44050 tree.go:497] /dev/nvidia0 cores 100->80
2022-01-27T15:15:11.684857017+08:00 I0127 07:15:11.684825 44050 tree.go:503] /dev/nvidia0 memory 16911040512->16105734144
2022-01-27T15:15:11.684903638+08:00 I0127 07:15:11.684855 44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-xxcl5 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.684913151+08:00 I0127 07:15:11.684864 44050 util.go:206] Pod galaxy-daemonset-tc6t6 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.684919133+08:00 I0127 07:15:11.684870 44050 util.go:206] Pod influxdb-0 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.684924824+08:00 I0127 07:15:11.684877 44050 util.go:206] Pod tke-platform-controller-546fc4fc97-phzgj in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.684945061+08:00 I0127 07:15:11.684884 44050 util.go:206] Pod tke-platform-api-b4d45f4bc-j5dbq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.684968501+08:00 I0127 07:15:11.684896 44050 util.go:206] Pod tke-registry-api-55b6f778f7-mkhwn in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.684975959+08:00 I0127 07:15:11.684903 44050 util.go:206] Pod coredns-6c654f5cc6-58bz2 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.684982507+08:00 I0127 07:15:11.684911 44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-hk6rw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.684997910+08:00 I0127 07:15:11.684918 44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-npjrd in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685004852+08:00 I0127 07:15:11.684929 44050 util.go:206] Pod tke-gateway-4gcpl in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685010984+08:00 I0127 07:15:11.684939 44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-ngkqf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685017044+08:00 I0127 07:15:11.684946 44050 util.go:206] Pod tke-business-api-77ff89ff7-s94ml in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685023050+08:00 I0127 07:15:11.684952 44050 util.go:206] Pod tke-application-api-6db6c5f8dd-l98gc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685029064+08:00 I0127 07:15:11.684959 44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-p9bbx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685038433+08:00 I0127 07:15:11.684965 44050 util.go:206] Pod kube-proxy-26g7w in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685044350+08:00 I0127 07:15:11.684972 44050 util.go:206] Pod gpu-quota-admission-76cfff49b6-gs8b5 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685066677+08:00 I0127 07:15:11.684989 44050 util.go:206] Pod tke-business-api-77ff89ff7-kwcnw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685092317+08:00 I0127 07:15:11.684996 44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-fbsqk in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685132083+08:00 I0127 07:15:11.685003 44050 util.go:206] Pod tke-business-controller-86879f65df-rz2kv in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685140989+08:00 I0127 07:15:11.685014 44050 util.go:206] Pod tke-notify-api-df68cb499-qtnwz in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685147726+08:00 I0127 07:15:11.685020 44050 util.go:206] Pod tke-registry-api-55b6f778f7-j849x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685154238+08:00 I0127 07:15:11.685026 44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-r8p26 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685160552+08:00 I0127 07:15:11.685035 44050 util.go:206] Pod coredns-6c654f5cc6-wsg7s in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685166933+08:00 I0127 07:15:11.685042 44050 util.go:206] Pod tke-monitor-controller-bb56765d-5vl2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685185902+08:00 I0127 07:15:11.685048 44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-qj5fx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685199160+08:00 I0127 07:15:11.685054 44050 util.go:206] Pod kube-controller-manager-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685205633+08:00 I0127 07:15:11.685061 44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-7pl4s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685211613+08:00 I0127 07:15:11.685067 44050 util.go:206] Pod kube-apiserver-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685217763+08:00 I0127 07:15:11.685074 44050 util.go:206] Pod flannel-pmfcv in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685223763+08:00 I0127 07:15:11.685081 44050 util.go:206] Pod tke-application-controller-6c769cc9cc-8v4zc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685229745+08:00 I0127 07:15:11.685087 44050 util.go:206] Pod kube-scheduler-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685235545+08:00 I0127 07:15:11.685093 44050 util.go:206] Pod etcd-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685241385+08:00 I0127 07:15:11.685099 44050 util.go:206] Pod tke-auth-api-6fcfdb454d-srxtq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685247156+08:00 I0127 07:15:11.685106 44050 util.go:206] Pod tke-business-controller-86879f65df-5wf94 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685252916+08:00 I0127 07:15:11.685113 44050 util.go:206] Pod tke-platform-controller-546fc4fc97-sbp9l in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685283654+08:00 I0127 07:15:11.685120 44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-7mg2r in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685296579+08:00 I0127 07:15:11.685126 44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-wc7bf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685303025+08:00 I0127 07:15:11.685133 44050 util.go:206] Pod tke-application-controller-6c769cc9cc-lbf2s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685309428+08:00 I0127 07:15:11.685140 44050 util.go:206] Pod metrics-server-v0.3.6-794ccd69c8-8sqxb in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685316079+08:00 I0127 07:15:11.685147 44050 util.go:206] Pod tke-notify-api-df68cb499-hzc8w in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685327993+08:00 I0127 07:15:11.685153 44050 util.go:206] Pod tke-platform-api-b4d45f4bc-4z5ln in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685335556+08:00 I0127 07:15:11.685160 44050 util.go:206] Pod tke-monitor-controller-bb56765d-78p6v in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685353964+08:00 I0127 07:15:11.685167 44050 util.go:206] Pod tke-application-api-6db6c5f8dd-tbf2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685360751+08:00 I0127 07:15:11.685178 44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-gc6z6 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685367548+08:00 I0127 07:15:11.685184 44050 util.go:206] Pod gpu-manager-daemonset-vv7zz in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685373876+08:00 I0127 07:15:11.685191 44050 util.go:206] Pod tke-auth-api-6fcfdb454d-kv7bb in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685380527+08:00 I0127 07:15:11.685217 44050 allocator.go:631] Pods to be removed: []
2022-01-27T15:15:11.845831500+08:00 I0127 07:15:11.845712 44050 allocator.go:216] Checking allocation of pods on this node
2022-01-27T15:15:11.935165188+08:00 I0127 07:15:11.935015 44050 util.go:216] Determine if the container empty9 needs GPU resource
2022-01-27T15:15:11.935189368+08:00 I0127 07:15:11.935053 44050 util.go:206] Pod coredns-6c654f5cc6-58bz2 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935196871+08:00 I0127 07:15:11.935063 44050 util.go:206] Pod coredns-6c654f5cc6-wsg7s in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935203689+08:00 I0127 07:15:11.935070 44050 util.go:206] Pod etcd-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935209691+08:00 I0127 07:15:11.935076 44050 util.go:206] Pod flannel-pmfcv in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935216094+08:00 I0127 07:15:11.935082 44050 util.go:206] Pod galaxy-daemonset-tc6t6 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935225360+08:00 I0127 07:15:11.935089 44050 util.go:206] Pod gpu-manager-daemonset-vv7zz in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935231614+08:00 I0127 07:15:11.935095 44050 util.go:206] Pod gpu-quota-admission-76cfff49b6-gs8b5 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935237706+08:00 I0127 07:15:11.935102 44050 util.go:206] Pod kube-apiserver-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935243666+08:00 I0127 07:15:11.935108 44050 util.go:206] Pod kube-controller-manager-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935249723+08:00 I0127 07:15:11.935115 44050 util.go:206] Pod kube-proxy-26g7w in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935255963+08:00 I0127 07:15:11.935121 44050 util.go:206] Pod kube-scheduler-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935261969+08:00 I0127 07:15:11.935128 44050 util.go:206] Pod metrics-server-v0.3.6-794ccd69c8-8sqxb in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935268138+08:00 I0127 07:15:11.935135 44050 util.go:206] Pod influxdb-0 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935288864+08:00 I0127 07:15:11.935141 44050 util.go:206] Pod tke-application-api-6db6c5f8dd-l98gc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935295526+08:00 I0127 07:15:11.935147 44050 util.go:206] Pod tke-application-api-6db6c5f8dd-tbf2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935311675+08:00 I0127 07:15:11.935154 44050 util.go:206] Pod tke-application-controller-6c769cc9cc-8v4zc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935318455+08:00 I0127 07:15:11.935160 44050 util.go:206] Pod tke-application-controller-6c769cc9cc-lbf2s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935324315+08:00 I0127 07:15:11.935167 44050 util.go:206] Pod tke-auth-api-6fcfdb454d-kv7bb in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935330158+08:00 I0127 07:15:11.935174 44050 util.go:206] Pod tke-auth-api-6fcfdb454d-srxtq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935335984+08:00 I0127 07:15:11.935182 44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-npjrd in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935341884+08:00 I0127 07:15:11.935189 44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-qj5fx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935347967+08:00 I0127 07:15:11.935196 44050 util.go:206] Pod tke-business-api-77ff89ff7-kwcnw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935353692+08:00 I0127 07:15:11.935204 44050 util.go:206] Pod tke-business-api-77ff89ff7-s94ml in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935359744+08:00 I0127 07:15:11.935211 44050 util.go:206] Pod tke-business-controller-86879f65df-5wf94 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935365547+08:00 I0127 07:15:11.935219 44050 util.go:206] Pod tke-business-controller-86879f65df-rz2kv in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935372301+08:00 I0127 07:15:11.935225 44050 util.go:206] Pod tke-gateway-4gcpl in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935396787+08:00 I0127 07:15:11.935232 44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-7mg2r in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935403179+08:00 I0127 07:15:11.935238 44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-fbsqk in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935408870+08:00 I0127 07:15:11.935245 44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-gc6z6 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935414884+08:00 I0127 07:15:11.935253 44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-wc7bf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935420732+08:00 I0127 07:15:11.935260 44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-7pl4s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935426492+08:00 I0127 07:15:11.935267 44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-xxcl5 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935432262+08:00 I0127 07:15:11.935274 44050 util.go:206] Pod tke-monitor-controller-bb56765d-5vl2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935448064+08:00 I0127 07:15:11.935280 44050 util.go:206] Pod tke-monitor-controller-bb56765d-78p6v in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935454479+08:00 I0127 07:15:11.935287 44050 util.go:206] Pod tke-notify-api-df68cb499-hzc8w in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935460475+08:00 I0127 07:15:11.935293 44050 util.go:206] Pod tke-notify-api-df68cb499-qtnwz in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935466224+08:00 I0127 07:15:11.935300 44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-ngkqf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935471947+08:00 I0127 07:15:11.935306 44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-p9bbx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935481562+08:00 I0127 07:15:11.935313 44050 util.go:206] Pod tke-platform-api-b4d45f4bc-4z5ln in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935488010+08:00 I0127 07:15:11.935319 44050 util.go:206] Pod tke-platform-api-b4d45f4bc-j5dbq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935493987+08:00 I0127 07:15:11.935326 44050 util.go:206] Pod tke-platform-controller-546fc4fc97-phzgj in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935499802+08:00 I0127 07:15:11.935333 44050 util.go:206] Pod tke-platform-controller-546fc4fc97-sbp9l in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935505787+08:00 I0127 07:15:11.935340 44050 util.go:206] Pod tke-registry-api-55b6f778f7-j849x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935511527+08:00 I0127 07:15:11.935347 44050 util.go:206] Pod tke-registry-api-55b6f778f7-mkhwn in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935517202+08:00 I0127 07:15:11.935354 44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-hk6rw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935522927+08:00 I0127 07:15:11.935360 44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-r8p26 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935537453+08:00 I0127 07:15:11.935384 44050 server.go:191] Starting the GRPC server, driver nvidia, queryPort 5678
2022-01-27T15:15:11.935592763+08:00 I0127 07:15:11.935513 44050 server.go:245] Server tencent.com/vcuda-core is running
2022-01-27T15:15:11.935631370+08:00 I0127 07:15:11.935529 44050 server.go:245] Server tencent.com/vcuda-memory is running
2022-01-27T15:15:11.935654220+08:00 I0127 07:15:11.935619 44050 server.go:259] Server is ready at /var/run/gpu-manager.sock
2022-01-27T15:15:11.935736299+08:00 I0127 07:15:11.935634 44050 logs.go:79] parsed scheme: ""
2022-01-27T15:15:11.935766376+08:00 I0127 07:15:11.935652 44050 logs.go:79] scheme "" not registered, fallback to default scheme
2022-01-27T15:15:11.935796624+08:00 I0127 07:15:11.935685 44050 logs.go:79] ccResolverWrapper: sending update to cc: {[{/var/run/gpu-manager.sock 0 <nil>}] <nil>}
2022-01-27T15:15:11.935808699+08:00 I0127 07:15:11.935721 44050 logs.go:79] ClientConn switching balancer to "pick_first"
2022-01-27T15:15:11.935891561+08:00 I0127 07:15:11.935829 44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc0006dc160, CONNECTING
2022-01-27T15:15:11.935918288+08:00 I0127 07:15:11.935864 44050 vcore.go:82] Server tencent.com/vcuda-core is ready at /var/lib/kubelet/device-plugins/vcore.sock
2022-01-27T15:15:11.935980500+08:00 I0127 07:15:11.935910 44050 vmemory.go:81] Server tencent.com/vcuda-memory is ready at /var/lib/kubelet/device-plugins/vmemory.sock
2022-01-27T15:15:11.936195686+08:00 I0127 07:15:11.936133 44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc0006dc160, READY
2022-01-27T15:15:12.605930673+08:00 I0127 07:15:12.605740 44050 logs.go:79] parsed scheme: ""
2022-01-27T15:15:12.605962556+08:00 I0127 07:15:12.605775 44050 logs.go:79] scheme "" not registered, fallback to default scheme
2022-01-27T15:15:12.605973233+08:00 I0127 07:15:12.605791 44050 logs.go:79] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/device-plugins/vcore.sock 0 <nil>}] <nil>}
2022-01-27T15:15:12.605981086+08:00 I0127 07:15:12.605821 44050 logs.go:79] ClientConn switching balancer to "pick_first"
2022-01-27T15:15:12.606003403+08:00 I0127 07:15:12.605937 44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc0000dd290, CONNECTING
2022-01-27T15:15:12.606505102+08:00 I0127 07:15:12.606346 44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc0000dd290, READY
2022-01-27T15:15:12.606542916+08:00 I0127 07:15:12.606383 44050 server.go:92] Server /var/lib/kubelet/device-plugins/vcore.sock is ready, readyServers: 1
2022-01-27T15:15:12.606552794+08:00 I0127 07:15:12.606401 44050 logs.go:79] parsed scheme: ""
2022-01-27T15:15:12.606559828+08:00 I0127 07:15:12.606410 44050 logs.go:79] scheme "" not registered, fallback to default scheme
2022-01-27T15:15:12.606566773+08:00 I0127 07:15:12.606422 44050 logs.go:79] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/device-plugins/vmemory.sock 0 <nil>}] <nil>}
2022-01-27T15:15:12.606573514+08:00 I0127 07:15:12.606440 44050 logs.go:79] ClientConn switching balancer to "pick_first"
2022-01-27T15:15:12.606590551+08:00 I0127 07:15:12.606443 44050 logs.go:79] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-01-27T15:15:12.606598263+08:00 I0127 07:15:12.606503 44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc0000dd4d0, CONNECTING
2022-01-27T15:15:12.606686133+08:00 I0127 07:15:12.606588 44050 logs.go:79] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-01-27T15:15:12.607031082+08:00 I0127 07:15:12.606933 44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc0000dd4d0, READY
2022-01-27T15:15:12.607050519+08:00 I0127 07:15:12.606966 44050 server.go:92] Server /var/lib/kubelet/device-plugins/vmemory.sock is ready, readyServers: 2
2022-01-27T15:15:12.607063302+08:00 I0127 07:15:12.606995 44050 logs.go:79] parsed scheme: ""
2022-01-27T15:15:12.607075405+08:00 I0127 07:15:12.607003 44050 logs.go:79] scheme "" not registered, fallback to default scheme
2022-01-27T15:15:12.607108937+08:00 I0127 07:15:12.607013 44050 logs.go:79] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/device-plugins/kubelet.sock 0 <nil>}] <nil>}
2022-01-27T15:15:12.607134556+08:00 I0127 07:15:12.607028 44050 logs.go:79] ClientConn switching balancer to "pick_first"
2022-01-27T15:15:12.607154116+08:00 I0127 07:15:12.607059 44050 logs.go:79] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-01-27T15:15:12.607193453+08:00 I0127 07:15:12.607114 44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc00098a1b0, CONNECTING
2022-01-27T15:15:12.607207313+08:00 I0127 07:15:12.607135 44050 logs.go:79] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-01-27T15:15:12.607641528+08:00 I0127 07:15:12.607531 44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc00098a1b0, READY
2022-01-27T15:15:12.607692309+08:00 I0127 07:15:12.607566 44050 server.go:343] Register to kubelet with endpoint vcore.sock
2022-01-27T15:15:12.608429472+08:00 I0127 07:15:12.608330 44050 server.go:343] Register to kubelet with endpoint vmemory.sock
2022-01-27T15:15:12.609061739+08:00 I0127 07:15:12.608933 44050 vcore.go:94] ListAndWatch request for vcore
2022-01-27T15:15:12.609099686+08:00 I0127 07:15:12.609011 44050 logs.go:79] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-01-27T15:15:12.609675039+08:00 I0127 07:15:12.609515 44050 vmemory.go:98] ListAndWatch request for vmemory
2022-01-27T15:15:41.935937280+08:00 I0127 07:15:41.935722 44050 allocator.go:216] Checking allocation of pods on this node
2022-01-27T15:15:41.992878184+08:00 I0127 07:15:41.992695 44050 util.go:216] Determine if the container empty9 needs GPU resource
2022-01-27T15:15:41.992912047+08:00 I0127 07:15:41.992742 44050 util.go:206] Pod coredns-6c654f5cc6-58bz2 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992921351+08:00 I0127 07:15:41.992751 44050 util.go:206] Pod coredns-6c654f5cc6-wsg7s in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992938301+08:00 I0127 07:15:41.992758 44050 util.go:206] Pod etcd-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992947464+08:00 I0127 07:15:41.992764 44050 util.go:206] Pod flannel-pmfcv in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992954514+08:00 I0127 07:15:41.992771 44050 util.go:206] Pod galaxy-daemonset-tc6t6 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992961351+08:00 I0127 07:15:41.992777 44050 util.go:206] Pod gpu-manager-daemonset-vv7zz in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992968861+08:00 I0127 07:15:41.992783 44050 util.go:206] Pod gpu-quota-admission-76cfff49b6-gs8b5 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992976224+08:00 I0127 07:15:41.992790 44050 util.go:206] Pod kube-apiserver-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992983728+08:00 I0127 07:15:41.992795 44050 util.go:206] Pod kube-controller-manager-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992990651+08:00 I0127 07:15:41.992802 44050 util.go:206] Pod kube-proxy-26g7w in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992997654+08:00 I0127 07:15:41.992808 44050 util.go:206] Pod kube-scheduler-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.993004478+08:00 I0127 07:15:41.992817 44050 util.go:206] Pod metrics-server-v0.3.6-794ccd69c8-8sqxb in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.993013738+08:00 I0127 07:15:41.992823 44050 util.go:206] Pod influxdb-0 in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993021311+08:00 I0127 07:15:41.992830 44050 util.go:206] Pod tke-application-api-6db6c5f8dd-l98gc in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993028378+08:00 I0127 07:15:41.992837 44050 util.go:206] Pod tke-application-api-6db6c5f8dd-tbf2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993035424+08:00 I0127 07:15:41.992844 44050 util.go:206] Pod tke-application-controller-6c769cc9cc-8v4zc in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993070969+08:00 I0127 07:15:41.992851 44050 util.go:206] Pod tke-application-controller-6c769cc9cc-lbf2s in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993108709+08:00 I0127 07:15:41.992857 44050 util.go:206] Pod tke-auth-api-6fcfdb454d-kv7bb in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993116041+08:00 I0127 07:15:41.992863 44050 util.go:206] Pod tke-auth-api-6fcfdb454d-srxtq in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993122619+08:00 I0127 07:15:41.992869 44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-npjrd in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993128795+08:00 I0127 07:15:41.992875 44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-qj5fx in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993135224+08:00 I0127 07:15:41.992881 44050 util.go:206] Pod tke-business-api-77ff89ff7-kwcnw in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993141647+08:00 I0127 07:15:41.992887 44050 util.go:206] Pod tke-business-api-77ff89ff7-s94ml in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993147719+08:00 I0127 07:15:41.992901 44050 util.go:206] Pod tke-business-controller-86879f65df-5wf94 in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993153532+08:00 I0127 07:15:41.992907 44050 util.go:206] Pod tke-business-controller-86879f65df-rz2kv in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993170450+08:00 I0127 07:15:41.992914 44050 util.go:206] Pod tke-gateway-4gcpl in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993177115+08:00 I0127 07:15:41.992919 44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-7mg2r in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993183119+08:00 I0127 07:15:41.992926 44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-fbsqk in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993205924+08:00 I0127 07:15:41.992934 44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-gc6z6 in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993212827+08:00 I0127 07:15:41.992940 44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-wc7bf in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993218964+08:00 I0127 07:15:41.992950 44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-7pl4s in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993224793+08:00 I0127 07:15:41.992956 44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-xxcl5 in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993230635+08:00 I0127 07:15:41.992962 44050 util.go:206] Pod tke-monitor-controller-bb56765d-5vl2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993236493+08:00 I0127 07:15:41.992968 44050 util.go:206] Pod tke-monitor-controller-bb56765d-78p6v in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993242225+08:00 I0127 07:15:41.992974 44050 util.go:206] Pod tke-notify-api-df68cb499-hzc8w in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993248247+08:00 I0127 07:15:41.992979 44050 util.go:206] Pod tke-notify-api-df68cb499-qtnwz in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993253927+08:00 I0127 07:15:41.992985 44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-ngkqf in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993259602+08:00 I0127 07:15:41.992991 44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-p9bbx in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993265333+08:00 I0127 07:15:41.992997 44050 util.go:206] Pod tke-platform-api-b4d45f4bc-4z5ln in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993271027+08:00 I0127 07:15:41.993003 44050 util.go:206] Pod tke-platform-api-b4d45f4bc-j5dbq in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993276750+08:00 I0127 07:15:41.993010 44050 util.go:206] Pod tke-platform-controller-546fc4fc97-phzgj in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993282407+08:00 I0127 07:15:41.993017 44050 util.go:206] Pod tke-platform-controller-546fc4fc97-sbp9l in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993288039+08:00 I0127 07:15:41.993023 44050 util.go:206] Pod tke-registry-api-55b6f778f7-j849x in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993293773+08:00 I0127 07:15:41.993029 44050 util.go:206] Pod tke-registry-api-55b6f778f7-mkhwn in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993299445+08:00 I0127 07:15:41.993035 44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-hk6rw in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993305108+08:00 I0127 07:15:41.993041 44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-r8p26 in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663647683+08:00 I0127 07:16:11.663480 44050 util.go:206] Pod tke-auth-api-6fcfdb454d-kv7bb in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663713963+08:00 I0127 07:16:11.663517 44050 util.go:206] Pod influxdb-0 in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663734892+08:00 I0127 07:16:11.663526 44050 util.go:206] Pod tke-platform-controller-546fc4fc97-phzgj in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663742486+08:00 I0127 07:16:11.663533 44050 util.go:206] Pod tke-platform-api-b4d45f4bc-j5dbq in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663748972+08:00 I0127 07:16:11.663542 44050 util.go:206] Pod tke-registry-api-55b6f778f7-mkhwn in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663755620+08:00 I0127 07:16:11.663550 44050 util.go:206] Pod coredns-6c654f5cc6-58bz2 in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.663762140+08:00 I0127 07:16:11.663558 44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-hk6rw in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663768231+08:00 I0127 07:16:11.663565 44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-xxcl5 in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663774708+08:00 I0127 07:16:11.663572 44050 util.go:206] Pod galaxy-daemonset-tc6t6 in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.663780721+08:00 I0127 07:16:11.663579 44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-npjrd in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663801148+08:00 I0127 07:16:11.663586 44050 util.go:206] Pod tke-gateway-4gcpl in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663829214+08:00 I0127 07:16:11.663598 44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-ngkqf in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663864451+08:00 I0127 07:16:11.663612 44050 util.go:206] Pod tke-business-api-77ff89ff7-s94ml in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663872879+08:00 I0127 07:16:11.663620 44050 util.go:206] Pod tke-application-api-6db6c5f8dd-l98gc in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663879396+08:00 I0127 07:16:11.663627 44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-p9bbx in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663886742+08:00 I0127 07:16:11.663634 44050 util.go:206] Pod kube-proxy-26g7w in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.663893465+08:00 I0127 07:16:11.663641 44050 util.go:206] Pod tke-business-api-77ff89ff7-kwcnw in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663900131+08:00 I0127 07:16:11.663649 44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-fbsqk in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663906393+08:00 I0127 07:16:11.663656 44050 util.go:206] Pod tke-business-controller-86879f65df-rz2kv in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663912540+08:00 I0127 07:16:11.663679 44050 util.go:206] Pod tke-notify-api-df68cb499-qtnwz in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663918922+08:00 I0127 07:16:11.663687 44050 util.go:206] Pod tke-registry-api-55b6f778f7-j849x in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663925209+08:00 I0127 07:16:11.663695 44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-r8p26 in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663931196+08:00 I0127 07:16:11.663703 44050 util.go:206] Pod gpu-quota-admission-76cfff49b6-gs8b5 in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.663937282+08:00 I0127 07:16:11.663712 44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-qj5fx in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663943414+08:00 I0127 07:16:11.663719 44050 util.go:206] Pod kube-controller-manager-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.664006182+08:00 I0127 07:16:11.663727 44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-7pl4s in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664027932+08:00 I0127 07:16:11.663735 44050 util.go:206] Pod kube-apiserver-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.664038942+08:00 I0127 07:16:11.663742 44050 util.go:206] Pod flannel-pmfcv in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.664050282+08:00 I0127 07:16:11.663750 44050 util.go:206] Pod tke-application-controller-6c769cc9cc-8v4zc in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664059222+08:00 I0127 07:16:11.663758 44050 util.go:206] Pod coredns-6c654f5cc6-wsg7s in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.664066719+08:00 I0127 07:16:11.663765 44050 util.go:206] Pod tke-monitor-controller-bb56765d-5vl2x in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664073649+08:00 I0127 07:16:11.663771 44050 util.go:206] Pod kube-scheduler-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.664080656+08:00 I0127 07:16:11.663778 44050 util.go:206] Pod etcd-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.664087666+08:00 I0127 07:16:11.663785 44050 util.go:206] Pod tke-platform-controller-546fc4fc97-sbp9l in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664094399+08:00 I0127 07:16:11.663798 44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-7mg2r in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664104456+08:00 I0127 07:16:11.663806 44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-wc7bf in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664127939+08:00 I0127 07:16:11.663812 44050 util.go:206] Pod tke-application-controller-6c769cc9cc-lbf2s in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664140716+08:00 I0127 07:16:11.663820 44050 util.go:206] Pod metrics-server-v0.3.6-794ccd69c8-8sqxb in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.664150160+08:00 I0127 07:16:11.663827 44050 util.go:206] Pod tke-notify-api-df68cb499-hzc8w in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664157103+08:00 I0127 07:16:11.663835 44050 util.go:206] Pod tke-auth-api-6fcfdb454d-srxtq in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664164113+08:00 I0127 07:16:11.663842 44050 util.go:206] Pod tke-business-controller-86879f65df-5wf94 in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664171170+08:00 I0127 07:16:11.663849 44050 util.go:206] Pod tke-platform-api-b4d45f4bc-4z5ln in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664178167+08:00 I0127 07:16:11.663857 44050 util.go:206] Pod tke-monitor-controller-bb56765d-78p6v in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664185153+08:00 I0127 07:16:11.663864 44050 util.go:206] Pod tke-application-api-6db6c5f8dd-tbf2x in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664192213+08:00 I0127 07:16:11.663871 44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-gc6z6 in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664198790+08:00 I0127 07:16:11.663878 44050 util.go:206] Pod gpu-manager-daemonset-vv7zz in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.935739239+08:00 I0127 07:16:11.935632 44050 allocator.go:216] Checking allocation of pods on this node
looks like https://github.com/tkestack/vcuda-controller/blob/master/src/loader.c#L943 cant parse containerd's cgroup
nvidia hook (nvidia-container-runtime) disabled, right?
i can prepare MR to fix vanilla containerd
@paragor I can create gpu pod by gpu-manager, but when i execute the command nvidia-smi in the gpu pod, i get the following errors: F0627 09:09:43.017690 18 client.go:78] fail to get response from manager, error rpc error: code = Unknown desc = can't find containerd-918d8cef52c37c77df775137619b3ccffcf626cdd0d57d71a82d6113a6df5dcd from docker /tmp/cuda-control/src/register.c:87 rpc client exit with 255
Do you know how to solve this problem?
@guofengtd My cgroup driver is systemd, runtime is containerd and the version of kubernetes is 1.23
@seanchen022 please, try this fix https://github.com/tkestack/gpu-manager/compare/master...paragor:master
i will have gpu cluster only after 4 days for tests :(
@seanchen022 i had the same problem like u days ago.
when i change the images/codes of gpu-manager newer, it solved.
maybe the branch master, commit: f0669de works.
@paragor by the way, have u tried it ur way?
