Yichun Yin comments

Results 11 comments of


                                            Yichun Yin

TinyBert fine tune SQuAD

Hi, we have no plans to release the distillation code for SQuAD dataset. For SQuAD fine-tune code, you can refer to https://github.com/huawei-noah/Pretrained-Language-Model/blob/master/AutoTinyBERT/superbert_run_en_classifier.py.

Wikipedia with proper data format

Hi, you should preprocess the wikidata yourself.

Glove data

Hi, GloVe data is at https://nlp.stanford.edu/projects/glove/ and we use the version of glove.840B.300d.

Unrecognized model in huawei-noah/TinyBERT_4L_zh

Hi, I think the reason is that the Transformers does not include the TinyBERT model. TinyBERT has a similar architecture toBERT, so you can use it in Transformers like BERT.

Including task loss

Hi, we have tried this loss, but we have not seen improvement.

General_TinyBERT(4layer-312dim)/General_TinyBERT(6layer-768dim) 模型下载链接失效？

请查看 https://huggingface.co/huawei-noah

TinyBERT learning w/o GD

hi，问题1：是的；问题2：设置和两步蒸馏是一样的；问题3：权重都是1，我们没有调节权重。

the TEACHER model used for the distillation of specific tasks

Hi we have no plans to release the fine-tuned model. You can use https://github.com/huawei-noah/Pretrained-Language-Model/blob/master/AutoTinyBERT/superbert_run_en_classifier.py for GLUE and SQuAD finetune.

tinybert的tf代码有提供吗

您好！暂时不提供tf版本~

中文Tinybert的训练

你好！中文没有涉及词向量替换的问题，因为tokenization之后基本上是单个字。