TensorRT icon indicating copy to clipboard operation
TensorRT copied to clipboard
verified publisher icon NVIDIA

flux model engine_from_bytes(bytes_from_path(self.engine_path)) OutOfMemory

Open algorithmconquer opened this issue 1 year ago • 2 comments

from polygraphy.backend.trt import engine_from_bytes; when run engine_from_bytes(bytes_from_path(self.engine_path)) OutOfMemory on L40 with 1gpu with flux-dev,how to solve

algorithmconquer avatar Oct 17 '24 10:10 algorithmconquer

Try to use trtexec, trt version >=8.6

lix19937 avatar Oct 18 '24 14:10 lix19937

Related issue: https://github.com/NVIDIA/TensorRT/issues/4205

yuanyao-nv avatar Oct 18 '24 22:10 yuanyao-nv

@algorithmconquer the flux demo should now run on L40S as we have added memory optimizations in release/10.6. Can you please try again and update here?

asfiyab-nvidia avatar Nov 05 '24 22:11 asfiyab-nvidia