🐛 [Bug] Encountered bug when using Torch-TensorRT

[ninono12345]

Bug Description

When compiling a pytorch model with torch-tensorrt, using different compiling methods I get different errors:

using this code:

head_jit = torch.jit.trace(om.head, idd) trt_head2 = torch_tensorrt.compile( head_jit, ir="ts", inputs=idd, debug=True) torch.jit.save(trt_head2, "trt_head_ts.ts")

I get this error:

WARNING: [Torch-TensorRT] - For input target_labels, found user specified input dtype as Float. The compiler is going to use the user setting Float WARNING: [Torch-TensorRT] - For input train_ltrb, found user specified input dtype as Float. The compiler is going to use the user setting Float Traceback (most recent call last): File "/pytracking-master-dir/band4.py", line 47, in trt_head2 = torch_tensorrt.compile( File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/_compile.py", line 185, in compile compiled_ts_module: torch.jit.ScriptModule = torchscript_compile( File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/ts/_compiler.py", line 151, in compile compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec)) RuntimeError: [Error thrown at core/partitioning/shape_analysis.cpp:311] Unable to process subgraph input type of at::kLong/at::kDouble, try to compile model with truncate_long_and_double enabled

using this code:

trt_head2 = torch_tensorrt.compile( om.head, ir="ts", inputs=idd, debug=True) torch.jit.save(trt_head2, "trt_head_ts.ts")

I get this error:

Traceback (most recent call last): File "/pytracking-master-dir/band4.py", line 47, in trt_head2 = torch_tensorrt.compile( File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/_compile.py", line 183, in compile ts_mod = torch.jit.script(module) File "/usr/local/lib/python3.10/dist-packages/torch/jit/_script.py", line 1324, in script return torch.jit._recursive.create_script_module( File "/usr/local/lib/python3.10/dist-packages/torch/jit/_recursive.py", line 559, in create_script_module return create_script_module_impl(nn_module, concrete_type, stubs_fn) File "/usr/local/lib/python3.10/dist-packages/torch/jit/_recursive.py", line 632, in create_script_module_impl script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn) File "/usr/local/lib/python3.10/dist-packages/torch/jit/_script.py", line 639, in _construct init_fn(script_module) File "/usr/local/lib/python3.10/dist-packages/torch/jit/_recursive.py", line 608, in init_fn scripted = create_script_module_impl( File "/usr/local/lib/python3.10/dist-packages/torch/jit/_recursive.py", line 632, in create_script_module_impl script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn) File "/usr/local/lib/python3.10/dist-packages/torch/jit/_script.py", line 639, in _construct init_fn(script_module) File "/usr/local/lib/python3.10/dist-packages/torch/jit/_recursive.py", line 608, in init_fn scripted = create_script_module_impl( File "/usr/local/lib/python3.10/dist-packages/torch/jit/_recursive.py", line 636, in create_script_module_impl create_methods_and_properties_from_stubs( File "/usr/local/lib/python3.10/dist-packages/torch/jit/_recursive.py", line 469, in create_methods_and_properties_from_stubs concrete_type._create_methods_and_properties( RuntimeError: Unsupported value kind: Tensor

using this code:

head_jit = torch.jit.trace(om.head, idd) trt_head2 = torch_tensorrt.compile( head_jit, ir="ts", inputs=idd, debug=True, truncate_long_and_double=True) torch.jit.save(trt_head2, "trt_head_ts.ts")

I get this error:

WARNING: [Torch-TensorRT] - For input target_labels, found user specified input dtype as Float. The compiler is going to use the user setting Float WARNING: [Torch-TensorRT] - For input train_ltrb, found user specified input dtype as Float. The compiler is going to use the user setting Float WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util WARNING: [Torch-TensorRT] - Sum converter disregards dtype WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32 WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32 WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util WARNING: [Torch-TensorRT] - Sum converter disregards dtype WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32 WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32 WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util WARNING: [Torch-TensorRT] - Unable to process input type of at::kLong, truncate type to at::kInt in scalar_to_tensor_util WARNING: [Torch-TensorRT TorchScript Conversion Context] - Unused Input: input_1 WARNING: [Torch-TensorRT TorchScript Conversion Context] - [RemoveDeadLayers] Input Tensor input_1 is unused or used only at compile-time, but is not being removed. ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [elementWiseLayer.cpp::validate::61] Error Code 4: Internal Error ((Unnamed Layer* 13) [ElementWise]: operation SUM requires inputs with activation or Int32 type.) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [elementWiseLayer.cpp::validate::61] Error Code 4: Internal Error ((Unnamed Layer* 21) [ElementWise]: operation SUM requires inputs with activation or Int32 type.) WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32 ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 14) [LoopOutput]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 14) [LoopOutput]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 14) [LoopOutput]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 14) [LoopOutput]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 14) [LoopOutput]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 27) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 27) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 27) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 27) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 29) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 29) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 29) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 29) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 31) [Slice]: output shape can not be computed) WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32 ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 31) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%32 : Tensor = aten::sub(%y_embed.1, %33, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:55:0: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%39 : Tensor = aten::add(%38, %40, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:55:0: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%32 : Tensor = aten::sub(%y_embed.1, %33, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:55:0: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%39 : Tensor = aten::add(%38, %40, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:55:0: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%32 : Tensor = aten::sub(%y_embed.1, %33, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:55:0: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%39 : Tensor = aten::add(%38, %40, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:55:0: output shape can not be computed) WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32 ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 22) [LoopOutput]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 22) [LoopOutput]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 22) [LoopOutput]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 22) [LoopOutput]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 22) [LoopOutput]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 40) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 40) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 40) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 40) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 42) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 42) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 42) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 42) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 44) [Slice]: output shape can not be computed) WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32 ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error ((Unnamed Layer* 44) [Slice]: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%42 : Tensor = aten::sub(%x_embed.1, %33, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%46 : Tensor = aten::add(%45, %40, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%42 : Tensor = aten::sub(%x_embed.1, %33, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%46 : Tensor = aten::add(%45, %40, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%42 : Tensor = aten::sub(%x_embed.1, %33, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%46 : Tensor = aten::add(%45, %40, %12), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%x_embed.3 : Tensor = aten::div(%42, %46), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed) ERROR: [Torch-TensorRT TorchScript Conversion Context] - 4: [graphShapeAnalyzer.cpp::needTypeAndDimensions::2212] Error Code 4: Internal Error (%x_embed.3 : Tensor = aten::div(%42, %46), scope: __module.filter_predictor.pos_encoding # /pytracking-master-dir/ltr/models/transformer/position_encoding.py:56:0: output shape can not be computed) Traceback (most recent call last): File "/pytracking-master-dir/band4.py", line 47, in trt_head2 = torch_tensorrt.compile( File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/_compile.py", line 185, in compile compiled_ts_module: torch.jit.ScriptModule = torchscript_compile( File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/ts/_compiler.py", line 151, in compile compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec)) RuntimeError: [Error thrown at core/util/trt_util.cpp:165] pos >= (-d.nbDims - 1) && pos <= d.nbDims ASSERT FAILED at core/util/trt_util.cpp:165, consider filing a bug: https://www.github.com/NVIDIA/Torch-TensorRT/issues ERROR: Index to unsqueeze is out of bounds. Expected value in range [0, -1], but got -1

If anybody has any experience, please let me know

I you need more info or the full model, please tell me

THANK YOU