serving Memory usage of TensorFlow models

We want to monitor memory usage of TensorFlow serving runtime on a per model basis. Currently we can get the memory used by Tensorflow completely but we dont have a way to get this information on a per model basis. We would like to have a solution that can work for both CPU and GPU memory

No alternatives are available that we know of

This is a similar issue to this one - https://github.com/tensorflow/serving/issues/2156 where I wanted to follow up on whether there are any solutions for CPUs

Aug 16 '23 05:08 ndeepesh

@singhniraj08 Any pointers on above?

Aug 21 '23 00:08 ndeepesh

Hey @singhniraj08 Can you point us to any workarounds in the meantime?

Aug 22 '23 02:08 ndeepesh

@ndeepesh, Can you try using Memory profiling tool if that helps. Since the models are running online on c++ model servers, tracking memory usage becomes difficult in that case.

I am bumping up this issue internally for solution and we will update this thread. Thank you!

Aug 22 '23 09:08 singhniraj08

Thanks @singhniraj08. Without a good estimate this is causing regressions in our hosts where we are going OOM easily with no way to tell which model is the issue

Aug 23 '23 01:08 ndeepesh

@singhniraj08 Will the memory profile tool gets populated for profiling on CPUs? I havent see this getting populated on cpu profiles

Aug 25 '23 17:08 ndeepesh

@ndeepesh, Memory profiling tools monitors the memory usage of your device during the profiling interval. If you are looking for CPU usage while running TF Serving, Profiling inference requests with Tensorboard may help you achieve that. Thanks.

Aug 28 '23 08:08 singhniraj08

@singhniraj08 This only gives us what tensorflow op is taking how much time right? Not the amount of memory it occupies on CPU.

Sep 02 '23 17:09 ndeepesh