[Bug] InternalError: Check failed: (offset + needed_size <= this->buffer.size) is false: storage allocation failure

[coffezhou]

Expected behavior

TVM should run the model correctly.

Actual behavior

For the following model,

it can be executed by onnxruntime, the results are as follows:

ONNXRuntime:
 [array([[[[False, False, False, False],
         [False, False, False, False],
         [False, False, False, False],
         [False, False, False, False]]]])]

However, when compiling and running the model using TVM, TVM crashes:

  File "/home/carla/Documents/tvm/python/tvm/runtime/vm.py", line 295, in invoke_stateful
    self._invoke_stateful(func_name)
  File "tvm/ffi/cython/./function.pxi", line 228, in tvm.ffi.core.Function.__call__
tvm.error.InternalError: Check failed: (offset + needed_size <= this->buffer.size) is false: storage allocation failure, attempted to allocate 18446744073709551553 at offset 0 in region that is 0bytes

Environment

OS: Ubuntu 20.04 TVM: 0.22.dev0 (c6969d723) onnxruntime: 1.21.0

Steps to reproduce

This bug can be reproduced by the following code with the model in the attachment. As shown in the code, the model can be executed by onnxruntime. However, TVM crashes when calling the invoke_stateful function.

import sys

import numpy as np
import onnx
import onnxruntime

import tvm
from tvm import relax
from tvm.relax.frontend.onnx import from_onnx

import pickle

            
def main():
    onnx_model = onnx.load("111.onnx")
    
    with open("inputs.pkl", "rb") as fp:
        inputs = pickle.load(fp)
    print(inputs)
    try:
        ort_session = onnxruntime.InferenceSession(
            onnx_model.SerializeToString(), providers=["CPUExecutionProvider"]
        )
        ort_output = ort_session.run([], inputs)
    except Exception as e:
        print(e)
        sys.exit(1)
        
    print("ONNXRuntime:\n", ort_output)   

    # Convert the onnx model into relax through the onnx importer.
    tvm_model = from_onnx(onnx_model, keep_params_in_input=True)
    # Convert operators for inference mode.
    tvm_model = relax.transform.DecomposeOpsForInference()(tvm_model)
    # Legalize any relax ops into tensorir.
    tvm_model = relax.transform.LegalizeOps()(tvm_model)

    # Separate model from parameters.
    tvm_model, params = relax.frontend.detach_params(tvm_model)
    
    # Prepare inputs.
    input_list = [
        inputs[key.name_hint] for key in tvm_model["main"].params if key.name_hint in inputs
    ]
    if params:
        input_list += params["main"]
        
    # Compile the relax graph into a VM then run.
    #----------------------cpu-----------------------
    with tvm.transform.PassContext(opt_level=0):
        target = tvm.target.Target("llvm", host="llvm")
        relax_pipeline = relax.pipeline.get_default_pipeline(target)
        
        ex = relax.build(tvm_model, target="llvm", relax_pipeline=relax_pipeline)
        vm = relax.VirtualMachine(ex, tvm.cpu())
    
        # Run model and check outputs.
        vm.set_input("main", *input_list)
        vm.invoke_stateful("main")
        tvm_cpu_output = vm.get_outputs("main")

    
if __name__ == "__main__":
    
    main()

testcase.zip

Triage

Please refer to the list of label tags here to find the relevant tags and add them below in a bullet format (example below).

• needs-triage

[ConvolutedDog]

The error occurs because the expand_shape parameter you passed to the main function is tvm.nd.array([1, 1, 4, 4]), but TVM's Exapnd node requires that the expand_shape passed in is the size of the target tensor. Therefore, tvm.nd.array([1, 3, 4, 4]) should be passed in here:

if params:
    input_list += params["main"]
    input_list[-1] = tvm.nd.array([1, 3, 4, 4])

[coffezhou]

The error occurs because the expand_shape parameter you passed to the main function is tvm.nd.array([1, 1, 4, 4]), but TVM's Exapnd node requires that the expand_shape passed in is the size of the target tensor. Therefore, tvm.nd.array([1, 3, 4, 4]) should be passed in here:

if params: input_list += params["main"] input_list[-1] = tvm.nd.array([1, 3, 4, 4])

Thanks! This issue could be resolved by adding the code "input_list[-1] = tvm.nd.array([1, 3, 4, 4])". However, according to the onnx specification of Expand, the Expand operator supports the broadcast rule. The Expand operator in TVM is not compatible with that of onnx. Maybe some documents of this or an assertion should be added in the code of the Expand operator in the onnx frontend of TVM.

[ConvolutedDog]

Yes this is a good suggestion, it needs to be better documented. Maybe there should be an operator documentation similar to ONNX or PyTorch.