CHNRyan

Fail to use zero_init to construct llama2 with deepspeed zero3 and bnb!

1

### System Info ``` bitsandbytes==0.43.1 sentencepiece==0.1.97 huggingface_hub==0.23.2 accelerate==0.30.1 tokenizers==0.19.1 transformers==4.41.1 trl==0.8.6 peft==0.11.1 datasets==2.14.6 ``` ### Reproduction When I run the code, parameters first be fully loaded to each GPU and...

[BUG] is_zero_init_model is always False when I'm using zero_init!

5

**Describe the bug** When I'm fine tuning llama2 with deepspeed zero3, I set "zero3_init_flag: true" in my accelerate config. The "is_deepspeed_zero3_enabled()" in transformers/integrations/deepspeed.py is also judged to True. But the...