kineto icon indicating copy to clipboard operation
kineto copied to clipboard

Add Logging for Empty Traces

Open sraikund16 opened this issue 1 year ago • 4 comments

Summary: Recently, we have had users seen empty traces when the system is idle leading to confusion as to whether it was caused by a bug in kineto formatting or not. This diff adds further logging to ensure that the user is aware that kineto finds no valid trace events and is expecting an empty trace.

Differential Revision: D60311331

sraikund16 avatar Jul 26 '24 21:07 sraikund16

This pull request was exported from Phabricator. Differential Revision: D60311331

facebook-github-bot avatar Jul 26 '24 21:07 facebook-github-bot

This pull request was exported from Phabricator. Differential Revision: D60311331

facebook-github-bot avatar Jul 29 '24 16:07 facebook-github-bot

This pull request was exported from Phabricator. Differential Revision: D60311331

facebook-github-bot avatar Jul 29 '24 16:07 facebook-github-bot

This pull request was exported from Phabricator. Differential Revision: D60311331

facebook-github-bot avatar Jul 29 '24 16:07 facebook-github-bot

Should we also add the functions into roctracer's handleactivity functions?

@aaronenyeshi Sure, can we do that as a followup diff instead?

sraikund16 avatar Jul 30 '24 17:07 sraikund16

Should we also add the functions into roctracer's handleactivity functions?

@aaronenyeshi Sure, can we do that as a followup diff instead?

yes that sounds good

aaronenyeshi avatar Jul 31 '24 15:07 aaronenyeshi

This pull request was exported from Phabricator. Differential Revision: D60311331

facebook-github-bot avatar Aug 01 '24 07:08 facebook-github-bot

This pull request was exported from Phabricator. Differential Revision: D60311331

facebook-github-bot avatar Aug 01 '24 07:08 facebook-github-bot

This pull request was exported from Phabricator. Differential Revision: D60311331

facebook-github-bot avatar Aug 01 '24 07:08 facebook-github-bot

Should we also add the functions into roctracer's handleactivity functions?

@aaronenyeshi Sure, can we do that as a followup diff instead?

yes that sounds good

Looking at it now, I think this already covers the roctracer impl. handleGpuActivity with ItraceActivity as an arg is shared between CUPTI and Rocm, and the Rocm handleRuntimeActivity is also covered in this diff. We will see for sure when the next diff tests on all architectures though

sraikund16 avatar Aug 01 '24 17:08 sraikund16

This pull request has been merged in pytorch/kineto@da2f2682cabaf95d601fa2a9b7e0979f84fe7667.

facebook-github-bot avatar Aug 01 '24 19:08 facebook-github-bot