Kuo Zhang comments

Repositories
Issues
Comments

Results 3 comments of


                                            Kuo Zhang

[BUG] grad_norm and loss is nan when deepspeed==0.13.5 but ok with deepspeed==0.10.2

> > So, is this bug solved or not? > > Are you able to test with the latest version from source to confirm? Hi, is there any fix of...

By default, in megatron, condition fais on ibgda_get_state()->num_rc_per_pe == num_channels || ibgda_get_state()->num_rc_per_pe >= num_sms

the root cause is: ![Image](https://github.com/user-attachments/assets/94eaf93f-0d92-4c59-a2b8-e584cc702c85)

By default, in megatron, condition fais on ibgda_get_state()->num_rc_per_pe == num_channels || ibgda_get_state()->num_rc_per_pe >= num_sms

``` EP_DEVICE_ASSERT(ibgda_get_state()->num_rc_per_pe == num_channels || ibgda_get_state()->num_rc_per_pe >= num_sms); ``` I printed the values, by default , num_rc_per_pe = 12, num_channels = 10, num_sms = 20, Is this by design ?...