Running `omniperf profile` after `omniperf profile --roof-only` erases all roofline pdfs

[gsitaram]

Describe the bug If we profiled the application with omniperf profile and then obtained roofline chart PDFs using omniperf profile --kernel-names --roof-only, then we have both the profiling data and the roofline PDFs. If we reversed the order of these two operations, we lose the PDFs and legend. Could this be fixed that the user does not have to remember this ordering detail?

Development Environment:

• Linux Distribution: SLES
• Omniperf Version: 1.0.10
• GPU: MI250X
• Custer (if applicable): Frontier

To Reproduce Steps to reproduce the behavior:

1. `omniperf profile -n new --roof-only --kernel-names --device 0 -- ./exe ^--- this will create the roofline PDFs and legend
2. `omniperf profile -n new --device 0 -- ./exe ^--- this will collect all other profiling data, but erase the PDFs collected in the previous step.

If you did the steps in the opposite order, we will have all profiling data intact.

Expected behavior If we profile and collect different things in different commands but save to the same workload, nothing should be erased by subsequent profile steps unless the subsequent step collects the same info and has to overwrite the data collected. I can see how the second command above would have assumed that all roofline data collected earlier has to be overwritten.. but could we have the PDFs regenerated in this case?

[ppanchad-amd]

Hi @gsitaram. Internal ticket has been created to investigate your issue. Thanks!

[zichguan-amd]

Hi @gsitaram, the default behaviour is to clear the workload directory if it exists during setup, see this line. If you run multiple workloads with the same name, the directory will be cleared in-between runs. This does not happen with the --roof-only flag so you should run the roofline after you have done the profiling.

[gsitaram]

It is not a good idea to ask users to remember an order of issuing commands. I can think of some possibilities. Please comment on them.

• Instead of clearing an existing workload completely, warn the user or abort informing the user about the existence of that workload. Maybe we can add an option to force overwrite an output workload.
• If the tool is deciding to clear an existing workload assuming that the user may have used the same name by mistake, then is it possible to identify this particular situation that I reported? If the user did a --roof-only first, then can that be identified when running the second command? Clearly, --roof-only is the only way to generate those roofline PDFs and legends. The command in step 2 does not generate those. So, why erase them if they exist?
  • If this situation is identifiable, could a warning can be presented to the user that those files are now going to be cleared and that they have to regenerate those? Or could those roofline plots could be left alone? Or could a command with the right set of options be recommended so that they can generate both the profiling data and the roofline plots?