Pretraining codes

[YeDeming]

Hi Siqi,

Thanks for releasing the great code.

I cannot find the pretraining code in this repository about the implement of the in-batch negative examples? Could you point out it?

And it seems there is no 'masked_lm_distill_crd' in the criterions?

Thanks!

Best, Deming

[jinxinglu]

HI, Deming, did you find the implement of the negative examples in the pre-training?

thanks