Reproduction Issue of BiT on STS-B

Open NicoNico6 opened this issue 2 years ago • 2 comments

Great work and thanks a lot for opening up the great work!

While reusing the code released, I found some issues below:

I can not reproduce the W1A1 version BiT accuracy on the STS-B dataset as reported in the paper (68.7% vs 71.1%).

I have basically followed the setting in the code and paper, can you share some suggestions for the issue? https://github.com/facebookresearch/bit/blob/37d2bd73111dda9787424bc9e3a48edb7f18cc88/utils_glue.py#L689

Mar 27 '23 07:03 NicoNico6

I can not reproduce most W1A2 experiments by simply tuning abits from 1 to 2.

More details of the multi-stage distillation are needed to close the performance gap.

Apr 02 '23 08:04 NicoNico6

I can not reproduce most W1A2 experiments by simply tuning abits from 1 to 2.

More details of the multi-stage distillation are needed to close the performance gap.

You can try to set the seed to 42 and adjust the temperature to 4. from 1.

Dec 19 '23 02:12 we1sher