livelossplot icon indicating copy to clipboard operation
livelossplot copied to clipboard

WARNING : tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time

Open essanhaji opened this issue 5 years ago • 2 comments

tensorflow:Callback

Hi, Thank you very much for this amazing tool it helps a lot. Just a have a simple warning that will make this tool great if it is fixed. This warning occurs when I use PlotLossesKeras callback.

WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.1298s vs `on_train_batch_end` time: 0.2397s). Check your callbacks.

Environment

  • livelossplot==0.5.3
  • tensorflow==2.4.0
  • Keras==2.4.3
  • Windows 10
  • Jupyter Lab
  • Python version 3.6.8

Thank you

essanhaji avatar Jan 05 '21 11:01 essanhaji

Do you use Livelossplot without any other callbacks? We don't have on_train_batch_end method in our callback (only 'on_epoch_end').

Bartolo1024 avatar Feb 03 '21 16:02 Bartolo1024

Multiprocessing supported by TensorFlow itself is not supported on Windows. This is a suggestion to improve it by using the multiprocessing provided by Python directly. https://docs.python.org/en/3.7/library/multiprocessing.html

And, to see if multiprocessing is running, you need to look at what each process is doing, not the overall cpu stats.

In Linux, you can check by simply creating and disappearing multiple python processes with the same command as top, but I don't know Windows well, so if you google it, I think it might not come up.

Noah-Junseo avatar May 20 '22 04:05 Noah-Junseo