ninono12345

Hi @zerollzeng I apologize, a batch of 8 in regular pytorch takes about 160ms, I used time.time() to measure, but I placed it in the wrong place perhaps... now for...

@zerollzeng I understand that this might be a very difficult task, but perhaps you can share your experience at which direction should I be looking at? Should I look into...

@zerollzeng so now I am trying to run inference on tensorrt each batch in parallel. So I am using a static single batch engine, that has inference time of 16ms....

hi @noahc1510 . I have a ubuntu 23.10 running on RTX 4090, I've followed your instructions, after pip install -r requirements.txt I downloaded the engine and tokenizer files. Now I'm...

yeah... I just found out that I had to change llama_index.bridge to llama_index.legacy.bridge , same for some others

Hello again, thank you for your reply, @gs-olive, and I apologize, for replying so late. Thank you for pointing me to the windows_CI branch, I haven't noticed it then. Now...

Hi again @gs-olive, I've successfully installed it like you said, but now when i try to import torch_tensorrt, I get this error: C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\_Device.py:19: UserWarning: Unable to import torchscript frontend core...

Thank you @gs-olive for your fast reply. Now I'm really confused. I installed torch_tensorrt, now when I try to import torch_tensorrt I get this error and the program stops: `C:\Users\Tomas\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensorrt\_Device.py:19:...

@zerollzeng thank you for your answer, I just wanted to write that I've done a mistake in my code, now it successfully converted in tensorrt 8.6 just changed the dataset...

@zerollzeng Now I'm getting this, a lot of layers are not converted to int8, how can I fix that?: [02/12/2024-00:29:48] [TRT] [W] Missing scale and zero-point for tensor (Unnamed Layer*...