Chen Wu

code ```python from transformers import pipeline import transformers import deepspeed import torch import os from transformers.models.t5.modeling_t5 import T5Block import sys import torch.distributed as dist local_rank = int(os.getenv('LOCAL_RANK', '0')) world_size =...

> > May you try `python3 -m sglang.bench_serving --backend sglang --num-prompts 1024` instead? > > Hi, sorry for the delay, nodes were busy yesterday so just got a chance to...

> any one tried vllm 0.3.3 + torch 2.1.1+cu118 with nccl==2.19.3? . By default vllm 0.3.3 + torch 2.1.1+cu118 installs nccl==2.18.3 that is giving the all_reduce error with multiple nodes...