Xinchi Huang comments

Repositories
Issues
Comments

Results 4 comments of


                                            Xinchi Huang

llama trt-engine streaming mode does not work

I tried to use AsyncLLMEngine but got an error: ![image](https://github.com/NVIDIA/TensorRT-LLM/assets/37529028/94e7246f-016a-4421-b405-bfa841ecb3e7)

llama trt-engine streaming mode does not work

Thanks for your reply! I ran streaming mode in this way: ``` import gc import random import time from asyncio import run from pathlib import Path import torch import tensorrt_llm...

llama trt-engine streaming mode does not work

get signal 6 here: ![image](https://github.com/NVIDIA/TensorRT-LLM/assets/37529028/2afd007c-073c-4866-a285-1a9f27b24bb7)

llama trt-engine streaming mode does not work

Thanks for your reply, I will have a try!