Tracing is expensive and the excessive number of tracings could be due to ...

[GiuseppeDiGuglielmo]

I have just noticed this warning with the latest code in the main branch combined with Tensorflow 2.4.1, Keras 2.4.3, Python 3.8.1

WARNING:tensorflow:11 out of the last 11 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7fe08a0295e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.

I am not sure if it was previously reported.

[jmduarte]

This was introduced by my PR: #310, where I create a different intermediate model for each layer in order to provide the appropriate intermediate outputs.

In particular, the issue is that I recreate the intermediate model for every inference, which is definitely excessive. So we should come up with a way to create the intermediate models once (say when profiling is first activated/started), and then reuse those models.

This would be a good issue for a student to fix I think.