Shufang Xie

That may not be the reason because norm is computed in FP32. [code](https://github.com/facebookresearch/fairseq/blob/0272196aa803ecc94a8bffa6132a8c64fd7de286/fairseq/utils.py#L368). ```python total_norm = torch.norm(grads[0], p=2, dtype=torch.float32) ```

Hello, was wondering if this is still being worked. And is there any chance to also add `non_blocking` and `pin_memory`, which according to the PyTorch official [tutorial](https://pytorch.org/tutorials/intermediate/pinmem_nonblock.html) is also helpful...

Willing to have a try. Is there any standard training speed benchmark scripts? If not I can run the recipes and manually measure the speed.

Do you have plan to support multi-class classification tasks? The logic should be very similar to binary classification. It will be great if the `num_classes` param is configurable in the...

Same issue. Is there any plan to support ID authentication?