shangshng comments

Repositories
Issues
Comments

Results 3 comments of


                                            shangshng

Parallel multi-GPU training

> There are actually quite a few non-trivial changes required. I will make a pull request soon with the changes. Right now just validating my changes on a 2 GPU...

[FeatureRequest] Gather sparse logprobs

> @Marks101, the logits processor is supported on `ModelRunnerCppExecutor`: https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/runtime/model_runner_cpp.py#L48 Could you try that please? Hello, it looks like logits processor is disabled here.... ![image](https://github.com/NVIDIA/TensorRT-LLM/assets/22979596/5532c0ed-12fe-42a7-9985-0c7f23e1e0aa)

[Usage]: LLM.beam_search is much slower in vLLM 0.7.3 compared to 0.5.4

Same here. And setting n>1 in sample parameter will also get really low throughput.