vllm_enable_sleep undefined in train_ppo args

[jzhang-x]

Environment OS: [Ubuntu 22.04.5 LTS] Python Version: [3.10.14] Package Version: [openrlhf 0.6.0.post3]

Reproduction Steps

1. Ran command: from Readme example deepspeed --module openrlhf.cli.train_ppo \ --pretrain OpenRLHF/Llama-3-8b-sft-mixture \ --reward_pretrain OpenRLHF/Llama-3-8b-rm-mixture \ --save_path ./checkpoint/llama-3-8b-rlhf \ --save_steps -1 \ --logging_steps 1 \ --eval_steps -1 \ --micro_train_batch_size 2 \ --train_batch_size 128 \ --micro_rollout_batch_size 4 \ --rollout_batch_size 1024 \ --max_epochs 1 \ --prompt_max_len 1024 \ --generate_max_len 1024 \ --zero_stage 2 \ --bf16 \ --actor_learning_rate 5e-7 \ --critic_learning_rate 9e-6 \ --init_kl_coef 0.01 \ --prompt_data OpenRLHF/prompt-collection-v0.1 \ --input_key context_messages \ --apply_chat_template \ --max_samples 100000 \ --normalize_reward \ --adam_offload \ --flash_attn \ --gradient_checkpointing \ --use_wandb {wandb_token}
2. Observed error [rank0]: Traceback (most recent call last): [rank0]: File "/cpfs/user/x/miniforge3/envs/openrlhf/lib/python3.10/runpy.py", line 196, in _run_module_as_main [rank0]: return _run_code(code, main_globals, None, [rank0]: File "/cpfs/user/x/miniforge3/envs/openrlhf/lib/python3.10/runpy.py", line 86, in _run_code [rank0]: exec(code, run_globals) [rank0]: File "/cpfs/user/x/miniforge3/envs/openrlhf/lib/python3.10/site-packages/openrlhf/cli/train_ppo.py", line 465, in <module> [rank0]: train(args) [rank0]: File "/cpfs/user/x/miniforge3/envs/openrlhf/lib/python3.10/site-packages/openrlhf/cli/train_ppo.py", line 272, in train [rank0]: trainer.fit(args, prompts_dataloader, pretrain_dataloader, consumed_samples, num_update_steps_per_episodes) [rank0]: File "/cpfs/user/x/miniforge3/envs/openrlhf/lib/python3.10/site-packages/openrlhf/trainer/ppo_trainer.py", line 234, in fit [rank0]: self.experience_maker.make_experience_list(rand_prompts, labels, **self.generate_kwargs) [rank0]: File "/cpfs/user/x/miniforge3/envs/openrlhf/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context [rank0]: return func(*args, **kwargs) [rank0]: File "/cpfs/user/x/miniforge3/envs/openrlhf/lib/python3.10/site-packages/openrlhf/trainer/ppo_utils/experience_maker.py", line 206, in make_experience_list [rank0]: if self.strategy.args.vllm_enable_sleep: [rank0]: AttributeError: 'Namespace' object has no attribute 'vllm_enable_sleep'
3. Code Analysis vllm_enable_sleep undefined in train_ppo args parser parser = argparse.ArgumentParser() parser.add_argument("--save_path", type=str, default="./ckpt") parser.add_argument("--save_steps", type=int, default=-1) parser.add_argument("--save_hf_ckpt", action="store_true", default=False) parser.add_argument("--disable_ds_ckpt", action="store_true", default=False) parser.add_argument("--logging_steps", type=int, default=1) parser.add_argument("--eval_steps", type=int, default=-1) parser.add_argument("--ckpt_path", type=str, default="./ckpt/checkpoints_ppo") parser.add_argument("--max_ckpt_num", type=int, default=3) parser.add_argument("--max_ckpt_mem", type=int, default=1e8) parser.add_argument("--load_checkpoint", action="store_true", default=False) ...

[hijkzzz]

please use train_ppo_ray.py

[wenxiao7]

i just commented the few lines related to this, and it works

[lebronjamesking]

please use train_ppo_ray.py

Basically I cannot run ray with vllm. I have 1 node with 4 A100 cards.