How to make profile data returned to local host where tensorboard is running

[nrobeR]

Hi, I'm trying to use Tensorboard to profile TF running on a remote host with the commend below: python3 profiler/install_and_run.py --envdir=profile_env --logdir=${datadir} --version=2.5

And grab profile on Tensorboard profile UI with remote host.

But it seems like the profile data is stored in remote host instead of local host where Tensorboard is running. So I have to scp the file from remote to local such that Tensorboard can read it. Is there a way to have the profile data returned back through the Tensorboard gRPC profile call and store the profile data locally? Thanks!

[trisolaran]

if you are profiling using tensorboard, the design is to save it in tensorboard's logdir parameter, more specifically, under {logdir}/plugin/profile directory.

[nrobeR]

Just to clarify, I am using tensorboard to grab profile from a remote server. But the profile data is saved in the remote server's {logdir}/plugin/profile instead of on the local host where tensorboard is running. Did I miss something in the config? I didn't find any documentation abou this. thanks!

[trisolaran]

you are right, we seems to have that issues right now. but the fix will not appear when new tf and profiler plugin release. I will file an internal bug to track it.

[nrobeR]

Thanks for confirming. Is there an ETA for the fix?

[trisolaran]

typical setup between TF and TB will share access to the log_dir (network backed file system). so normally this is not an issue for our TF/TB users so far. we will make sure this is handled in next release of TF (i.e. give an option to store it locally). are you using TF nightly?

[nrobeR]

Thanks! No, we are not using TF nightly. But we can patch your change. Is there a pull request already?