ZelinTan

icon company University of Science and Technology of China icon location Hefei,China icon location University of Science and Technology of China(BA&Master[in-process])

Results 14 comments of ZelinTan

how to install from source

Hi!I also run into this weird problem.Have you solved it?

how to install from source

Hi, I have just solved the problem, try using pip install --upgrade pip to upgrade pip(>= 21.3).Then try again run pip install -e . I recognize that you are interested...

Evaluation.py failing on KeyError: 'test/0'

thanks Henry, your reply is really helpful

clang-9: error: linker command failed with exit code 1 (use -v to see invocation)

besides, I sincerely recommend that you try the same operation I mentioned above on FPGA compute node.When I did those operations on A10,compilation also falied.

🐛 [Bug] error: backend='torch_tensorrt' raised: TypeError: pybind11::init(): factory function returned nullptr

@geraldstanje I tried the resnet example in https://pytorch.org/TensorRT/tutorials/_rendered_examples/dynamo/torch_compile_resnet_example.html with : `| NVIDIA-SMI 470.103.01 Driver Version: 470.103.01 CUDA Version: 11.8 |` The GPU is Nvidia-A100 80G and run nvcc --version: ```...

[RFC]: Disaggregated prefilling and KV cache transfer roadmap

> We (Alibaba Cloud) are actively developing a disaggregated prefilling feature for vLLM to tackle latency issues and minimize interference during prefilling and decoding.  Leveraging fully asynchronous I/O, it ensures ...

[Feature] Automatically truncate when the maximum tokens are exceeded instead of throwing an error

@richardodliu could you please give us an example so that we can locate the problem more efficiently?

[Feature] Automatically truncate when the maximum tokens are exceeded instead of throwing an error

will take a look on it soon

Question for EP parameter in command for MoE model

Meanwhile, I found that when I want ep(dp)=4, but tp set to 1, there is only one GPU running but not 4 GPUS running in my machine... don't know why

Question for EP parameter in command for MoE model

Thanks for your reply! I also want to ask if it is possible to turn off DP while enabling EP? Because in my opinion, EP and TP working together can...