Remove assets copied to tmp right after uploading

[ivan94fi]

Is your feature request related to a problem? Please describe.

Now that copy_to_tmp is ignored and always set to true (see #504 and #505 for example), our workstation hard drive constantly fills up with temporary artifacts. They should be cleaned at the end of experiments, but we have experiments lasting weeks, and we cannot afford to keep those artifacts forever (they are mostly images logged with log_image).

Describe the solution you'd like

I would love the logging mechanism to be able to remove temporary artifacts once they are uploaded: this may be done with some callbacks, which get fired when the upload is successful to remove the file in /tmp.

Describe alternatives you've considered

The alternative is spawning a process that removes files in tmp older than some age, but this has two main problems:

• first of all, files saved by comet are spawned in /tmp with no structure, that is some directories are created with random names and that's all. I cannot be really sure to delete the right files and not some other process tmp files. Maybe using a top level directory in /tmp, for example /tmp/comet_ml_tmp_data or /tmp/comet, would help a bit with this.
• then obviously it might happen that I delete some asset that is still to be uploaded.

Additional context

N/A

[dsblank]

Comet should clean up after itself. I will report this bug to the Engineering team.

[dsblank]

This is reported internally as CM-10358.

[dsblank]

Engineering reports that the files are cleaned up at the end of the experiment as some files are not uploaded until the end (not immediately after logging them).