Performance Profiling

[jeromeku]

@stas00

Wondering if you have any tips & tricks for working with performance profiling tools such as nsys? Or recommendations for systematically optimizing model architecture and single / multi-node training workflows?

[stas00]

Wondering if you have any tips & tricks for working with performance profiling tools such as nsys?

I don't have experience with nsys.

Or recommendations for systematically optimizing model architecture

Neural Architecture Search (NAS) https://en.wikipedia.org/wiki/Neural_architecture_search? e.g. see https://developer.nvidia.com/blog/advancing-the-accuracy-efficiency-frontier-with-llama-3-1-nemotron-51b/ though I have no direct experience with it.

and single / multi-node training workflows?

This part is too vague for me to understand what you're asking about? Can you be more specific?

[stas00]

Closing due to inactivity. Please feel free to re-open if needed.