Alex Seceleanu comments

Repositories
Issues
Comments

Results 3 comments of


                                            Alex Seceleanu

PyTorch Profiler produces large trace files (~1GB) causing TensorBoard to crash

I tried reducing the block size in the profiler context, but with no luck. I get 1.9G torch_trace.json.

NCCL:Broadcast collectives are missing from the converted trace but present in the trace_link

Tested the same steps with the latest version of Chakra, installed from repository, 16 Oct, the behavior is the same.

NCCL:Broadcast collectives are missing from the converted trace but present in the trace_link

While looking into this issue I observed that the nccl:broadcast operation is a CPU operation and therefore is does not pass this check from pytorch_converter.py: ``` def get_protobuf_node_type_from_json_node( self, json_node_map:...