GPU used by a user

[scatenag]

I rephrase this.

It would be nice to have 'per user' GPU monitoring.

Something similar to what we can get with users_cpu_user_percentage_average I would like to have users_gpu_user_percentage_average.

I have seen something similar in gpustat.

Could it be possible to get some useful info?

If I'm not wrong both (this and gpustat) use the same library pynvml...

I asked the same here...

--it was ---

Would it be possible to create an alarm that triggers when a user uses more than one GPU?

That is: if on a GPU there are running processes of more than a user?

Thanks

[wookayin]

One can add a chart entry for GPU memory utilization per GPU, which consists of many lines (one per user). This should be straightforward to implement.

[scatenag]

For the moment I implemented per user GPU memory utilization modifying nvidia-smi collector https://github.com/netdata/netdata/pull/9372

Implementing by pynvml should be more efficient (but, for the moment, I am not python-ninja enough to try).

Do you think it would be possible to monitor per user GPU utilization (not memory)?

[coraxx]

Correct me if I'm wrong, but the only per user metric that is extractable, is how much memory is used. I am not able to find a metric for percent GPU usage per user.

Please let me know if I am just blind ^^

[scatenag]

You are right! I haven't found any way to get percent of GPU usage per user.

With nvidia-smi plugin I use percent of GPU memory usage (as approximation of) for user GPU usage that, for user accounting, is still better than no user info ..