generated results are terrible with bloom

[raihan0824]

I use this repo to finetune bloomz-7b1-mt with alpaca data (50k conversation) and the results are terrible. It takes 8 hours to train with the same arguments as in how you finetune the llama. What could be the reason of this?

[guoday]

Firstly, alpaca data is not intended for conversation but rather for instructional purposes. You should consider leveraging other data specifically designed for conversation. It appears that Bloomz-7b1-mt is a model fine-tuned on other tasks, so it's hard to align using instruction data.

[raihan0824]

no, I did convert the data to the same format as in this repo. And when I use bloomz-7b1-mton alpaca repo, it works just fine, however, I want to make it conversational so I use this repo. I think the problem is the training hyperparameter because I find this repo's hyperparameter is different from this repo. What do you think?

[raihan0824]

and for some reason, the adapter_model.bin file size is so small (<1MB) even though I trained for 8 hours. it's so weird

[guoday]

I see. Please adjust batch size or epochs here, because you only update 50k/64= 781 steps. However, we update about 3000 steps using all data.