Nathan J. Williams comments

Results 4 comments of


                                            Nathan J. Williams

GPU resources are not recovered even XID error is resolved

I agree that it seems like Xid 94 is essentially an application error and should not disable the device. But as a workaround you can tell it to ignore this...

DCGM_FI_DEV_GPU_UTIL abnormal point

I'm seeing the same thing, momentary spikes in the DCGM_FI_DEV_GPU_UTIL metric to about 800 million - (808793649 in one recent capture) for intervals somewhere between 15 and 45 seconds (I...

DCGM_FI_DEV_GPU_UTIL abnormal point

I ran `dcgmi dmon -e 203 -d 5000` on all of my fleet hosts for the past day, and never saw numbers out of the range of 0-100. During that...

Multiple GPUs are visible in a container despite setting limits in Kubernetes manifest

This looks a lot like https://github.com/NVIDIA/gpu-operator/issues/421 - the image you're using is has `NVIDIA_VISIBLE_DEVICES=all` set*, and the plugin is respecting that. You may want to follow the directions in that...