Guoheng Sun
Guoheng Sun
同问，有什么解决方法吗？ lol
@PeiqinSun Hello, I have encountered the same issue. The loss dramatically decreases after each epoch and then gradually increases. I have also observed this phenomenon in a [paper](https://arxiv.org/abs/2304.14454). After conducting...
> Same issue here.Have you solved it? I did not solve this problem. Recently, I have some new discoveries: when I use the default code of FastChat, the loss curve...
@LuJunru Hi, does this mean you have successfully finetuned a 33-B-parameter model using zero stage3 + offload optimizer & param on A100 40G * 8 + 600G CPU RAM? I...
@LuJunru Many Thanks! Would you mind sharing your Deepspeed script, please? I have tried other scripts from this issue and Deepspeed's official default script, but I am hoping to rule...
@LuJunru I understand your situation. Thanks again.
在生成阶段可以设置一下repetition_penalty，如： generation_config = GenerationConfig( temperature=temperature, top_p=top_p, top_k=top_k, num_beams=num_beams, repetition_penalty=1.2, **kwargs, ) generate_params = { "input_ids": input_ids, "generation_config": generation_config, "return_dict_in_generate": True, "output_scores": True, "max_new_tokens": max_new_tokens, }
> Repetition Penalty 在web界面上可以调整吧 默认设置是2 。 repetition_penalty：控制生成的文本中重复标记的惩罚力度。 你为什么还调整小了。。 我用的是alpaca-lora的代码，alpaca-lora默认是没设置repetition_penalty的