pytorch
❓ [Question] How to compile model when input is a list of tensors
❓ Question
I am trying to follow the tutorial here and am stuck at compiling the model with tensor-rt. The model i am using takes a list of tensors as inputs and hence i could not get the following compile code to work as i cannot get the shape of a list:
trt_model = torch_tensorrt.compile(self.model,
inputs= [torch_tensorrt.Input(inputs.shape)],
enabled_precisions= { torch.half} # Run with FP32
)
Inputs have the following tensors:
ic| i.shape: torch.Size([1, 3, 256, 256]) ic| i.shape: torch.Size([1, 98, 3]) ic| i.shape: torch.Size([1, 3, 3])
What you have already tried
I have tried using (3,) but i am getting the following errror:
File "/home/default/anaconda3/envs/driverstate_ttrt/lib/python3.10/site-packages/torch/jit/_recursive.py", line 397, in create_methods_and_properties_from_stubs
concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
RuntimeError:
forward(__torch__.spiga.models.cnn.layers.___torch_mangle_24.Residual self, Tensor x) -> Tensor:
Keyword argument core unknown.
:
File "/home/default/driver-state-detection/Fabian/headpose/SPIGA/spiga/models/cnn/hourglass.py", line 45
low1 = self.low1(pool1)
if self.n > 1:
low2, core = self.low2(low1, core=core)
~~~~~~~~~ <--- HERE
else:
low2 = self.low2(low1)
Environment
Build information about Torch-TensorRT can be found by turning on debug messages
- PyTorch Version (e.g., 1.0): 2.0.1+cu118
- OS (e.g., Linux): WSL2 on Windows11
- How you installed PyTorch (
conda,pip,libtorch, source): pip - Python version: 3.10.12
- GPU models and configuration: 2070Super
- Any other relevant information: torch-tensorrt 1.4.0
Additional context
Basically asking what should i used as inputs shape if it is a list of tensors. Should i instead look to this?
In torchtrt 1.4.0 where Torchscript is the default, the best way is to use the input_signature API which is an alternative to inputs but allows you to group inputs like you would an input to your original forward function
@narendasan
I am following the input_signature guide from here. However, i am still getting an error.
# i have tried both
input_signature = (
torch_tensorrt.Input(shape=[1, 3, 256, 256], dtype=torch.float),
torch_tensorrt.Input(shape=[1, 98, 3], dtype=torch.float),
torch_tensorrt.Input(shape=[1, 3, 3], dtype=torch.float)
)
# and
input_signature = (
[torch_tensorrt.Input(shape=[1, 3, 256, 256], dtype=torch.half),
torch_tensorrt.Input(shape=[1, 98, 3], dtype=torch.half),
torch_tensorrt.Input(shape=[1, 3, 3], dtype=torch.half)]
)
trt_model = torch_tensorrt.compile(self.model,
input_signature=input_signature,
enabled_precisions= {torch.half} # Run with FP32
)
Error:
File "/home/default/anaconda3/envs/driverstate_ttrt/lib/python3.10/site-packages/torch/jit/_recursive.py", line 397, in create_methods_and_properties_from_stubs
concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
RuntimeError:
forward(__torch__.spiga.models.cnn.layers.___torch_mangle_24.Residual self, Tensor x) -> Tensor:
Keyword argument core unknown.
:
File "/home/default/driver-state-detection/Fabian/headpose/SPIGA/spiga/models/cnn/hourglass.py", line 45
low1 = self.low1(pool1)
if self.n > 1:
low2, core = self.low2(low1, core=core)
~~~~~~~~~ <--- HERE
else:
low2 = self.low2(low1)
how my list of tensors look like:
ic| inputs: [tensor([[[[0.6078, 0.6078, 0.6078, ..., 0.3294, 0.3490, 0.3843],
[0.6118, 0.6118, 0.6118, ..., 0.4863, 0.4667, 0.5608],
...
[0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000],
[0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000]]]],
device='cuda:0'),
tensor([[[-0.8532, 0.7105, -0.3933],
[-0.8659, 0.6979, -0.2815],
...
[-0.3289, 0.2912, -0.3630],
[-0.3146, -0.2947, -0.3540]]], device='cuda:0'),
tensor([[[96., 0., 32.],
[ 0., 96., 32.],
[ 0., 0., 1.]]], device='cuda:0')]
Can you try
input_signature = (
[torch_tensorrt.Input(shape=[1, 3, 256, 256], dtype=torch.half),
torch_tensorrt.Input(shape=[1, 98, 3], dtype=torch.half),
torch_tensorrt.Input(shape=[1, 3, 3], dtype=torch.half)],
)
note the comma at the end since its a 1 argument tuple as the function signature
@narendasan Thank you for the suggestion but i am still getting the same error
File "/home/default/anaconda3/envs/driverstate_ttrt/lib/python3.10/site-packages/torch/jit/_recursive.py", line 397, in create_methods_and_properties_from_stubs
concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
RuntimeError:
forward(__torch__.spiga.models.cnn.layers.___torch_mangle_24.Residual self, Tensor x) -> Tensor:
Keyword argument core unknown.
:
File "/home/default/driver-state-detection/Fabian/headpose/SPIGA/spiga/models/cnn/hourglass.py", line 45
low1 = self.low1(pool1)
if self.n > 1:
low2, core = self.low2(low1, core=core)
~~~~~~~~~ <--- HERE
else:
low2 = self.low2(low1)
I reinstalled torch-tensorrt so i am using the torch version in their dependency.
pytorch-triton 2.1.0+6e4932cda8 pypi_0 pypi torch 2.0.1 pypi_0 pypi torch-tensorrt 1.4.0 pypi_0 pypi torchaudio 2.0.2 pypi_0 pypi torchvision 0.15.2 pypi_0 pypi
Hi @HeChengHui which particular dataloader are you using for the model, from which you are getting the above shapes?
@apbose i printed out the shape of the inputs from here.
def net_forward(self, inputs):
for i in inputs:
print(i.shape)
outputs = self.model(inputs)
return outputs
I didnt use any particular dataloader. app_CH.zip I have attached my testing code above. To be added to the main directory.