[BUG] Training using init-model gives normal lcurve and bad model

[zjgemi]

Bug summary

In the DPGen workflow, training in iter-0 seems all right. The model trained in iter-1 (with init-model) has a large RMSE ~100meV, while the lcurve shows a better accuracy For the worst system, the RMSE increase by a factor >30 after training of iter-1. This phenomenon does not appear when using finetune (instead of init-model) in iter-1.

DeePMD-kit Version

stable-0411

Backend and its version

Pytorch

How did you download the software?

docker

Input Files, Running Commands, Error Log, etc.

iter1input.zip

Steps to Reproduce

bash aefcb166ade9f2faf80a15e8a6f0d0cb70a6d33a.sub

Further Information, Files, and Links

No response

[Chengqian-Zhang]

I followed the steps of

1. fine-tuning based on the multitask pre-trained model
2. init-model based on the finetuning model obtained in step1

and found that I am able to reproduce the bug on the stable_0411 branch, but everything works well on the latest devel branch, so you can test if it's still an issue on the latest devel branch.