wenzhi fang

I encountered the same issue. I found the following two approaches works for me First, set ds_config = { "zero_optimization": { "stage": 3, "stage3_gather_16bit_weights_on_model_save": True, "offload_param": {"device": "none"}, "offload_optimizer": {"device":...