opencv
Opencv loads models of dynamic input types
Actual error message (3.4.17/4.5.5+): DNN/ONNX(Shape): dynamic 'zero' shapes are not supported
- OpenCV => 4.5.1
- Operating System / Platform => Windows 64 Bit
- Compiler => MinGW 7.3
Can opencv load dynamic input models?
An error is reported when using opencv to load the onnx model of dynamic input type.
Detailed description
The following is the compressed package of the input type and model file.
After I converted a pytorch model to onnx, the input size was 1x3x640x640. This model can run at onnxronnuntime. Then I tried to use opencv's readNet(path) function. The model can also work normally.
When I convert this model to dynamic input, this model can also run in onnxronnuntime, but when I use opencv's readNet to load this dynamic input model, ,the program starts to report an error, the error is as follows
OpenCV: terminate handler is called! The last OpenCV error is: OpenCV(4.5.1) Error: Assertion failed (total(os[i]) > 0) in getLayerShapesRecursively, file F:\OpenCV\opencv451\sources\modules\dnn\src\dnn.cpp, line 3520
If opencv can load dynamic input models,do I need to set the input and output size after the model is loaded by opencv? If yes, is there any information on this part of the content?
Steps to reproduce
I first use python to convert the model to onnx. The conversion code of pytorch to onnx I used is as follows
input_names = ["input0"]
output_names = ["output0","output1","output2"]
x = torch.randn(1, 3, 640, 640, requires_grad=True)
dynamic_axes = {'input0': {2: "height", 3: 'width'}, # variable lenght axes
'output0': {1: '_size'},
'output1': {1: '_size'},
'output2': {1: '_size'}}
# Export the model
torch.onnx.export(net, # model being run
x, # model input (or a tuple for multiple inputs)
"super_resolution.onnx", # where to save the model (can be a file or file-like object)
export_params=True, # store the trained parameter weights inside the model file
opset_version=11, # the ONNX version to export the model to
do_constant_folding=True, # whether to execute constant folding for optimization
input_names=input_names, # the model's input names
output_names=output_names, # the model's output names
dynamic_axes=dynamic_axes)
Then use opencv to load, the key code is as follows
net = readNet(std::string(pth_file.toLocal8Bit()));
net.setPreferableBackend(DNN_BACKEND_OPENCV);
net.setPreferableTarget(DNN_TARGET_CPU);
cv::Mat blob;
blob = blobFromImage(img,1.0, cv::Size(inpHeight,inpWidth), cv::Scalar(104, 117, 123), true, false, CV_32F);
net.setInput(blob,"input0");
std::vector<cv::Mat> outs;
net.forward(outs, getOutputsNames(net));
std::vector<bbox_l> bbox;
bbox = post_processing(img, outs);
Issue submission checklist
- [x] I report the issue, it's not a question
- [x] I checked the problem with documentation, FAQ, open issues, answers.opencv.org, Stack Overflow, etc and have not found solution
- [x] I updated to latest OpenCV version and the issue is still there
- [x] There is reproducer code and related data files: videos, images, onnx, etc
@SunlifeV Thank you for the report!
Confirmed.
net.setInput(blob, "input0"); raises mentioned exception.
Assertion failed (total(os[i]) > 0) in getLayerShapesRecursively
This stops during handling of Resize node 409:
[ERROR:[email protected]] OPENCV/DNN: [Resize]:(409): getMemoryShapes() post validation failed. inputs=1 outputs=1/1 blobs=0 inplace=0
[ERROR:[email protected]] input[0] = [ 1 64 34 60 ]
[ERROR:[email protected]] output[0] = [ 1 64 0 0 ]
[ERROR:[email protected]] Exception message: OpenCV(3.4.17-pre) modules/dnn/src/dnn.cpp:3081: error: (-2:Unspecified error) in function 'void cv::dnn::experimental_dnn_34_v24::Net::Impl::getLayerShapesRecursively(int, cv::dnn::experimental_dnn_34_v24::Net::Impl::LayersShapesMap&)'
> (expected: 'total(os[i]) > 0'), where
> 'total(os[i])' is 0
> must be greater than
> '0' is 0
It doesn't support ONNX Resize completely. 4th input is expected to be a constant.
But it is not a constant due to dynamic input shapes of super_resolution.onnx.
#21322 added extra checks to stop in ONNX importer on unsupported case (fail fast).
[DEBUG:[email protected]] DNN/ONNX: processing node with 1 inputs and 1 outputs: [Shape]:(391)
[ERROR:[email protected]] DNN/ONNX(Shape): dynamic 'zero' shapes are not supported, input 387 [ 1 64 0 0 ]
[ERROR:[email protected]] DNN/ONNX: ERROR during processing node with 1 inputs and 1 outputs: [Shape]:(391)
[ INFO:[email protected]] Input[0] = '387'
[ INFO:[email protected]] Output[0] = '391'
unknown file: Failure
C++ exception with description "OpenCV(3.4.17-pre) modules/dnn/src/onnx/onnx_importer.cpp:662: error: (-2:Unspecified error) in function 'handleNode'
> Node [Shape]:(391) parse error: OpenCV(3.4.17-pre) modules/dnn/src/onnx/onnx_importer.cpp:2160: error: (-215:Assertion failed) !isDynamicShape in function 'parseShape'
> " thrown in the test body.
Same error in 4.5.5 when readNet. Looking forward to dynamic input feature, many other inference framework support this feature.
The problem is still present in v. 4.6.0 but only in the release build. With debug build I can use variable input size neural net and it works correctly. In my case I am loading the neural net from TF model by cv:dnn::readNetFromTensorflow so the problem is not onnx specific. You can reproduce the problem with this model: https://github.com/sambhav37/Mask-R-CNN/
@antoszy Can u tell me please how to make a debug build?
Which repo branch do u use? I built current 4.x branch, but it has version 4.6.0.3725898 instead of pip 4.6.0.66
Is it necessary to use command python setup.py bdist_wheel --build-type=Debug or pip wheel . --verbose is same?
I can confirm, this still happens in 4.6.0 on onnx model converted from YOLOv5 with variable input.
commit b0dc474160e389b9c9045da5db49d03ae17c6a6b (HEAD, tag: 4.6.0)
haven't tried latest sources, will try and report back.
I've built latest opencv version 4.6.0-587-gb16f76eede
I've exported YOLOv5 pt model to onnx with dynamic input. I've run this code (so I am passing two images in one batch). However, problem occurs even before net.forward(). It happens during cv2.dnn.readNet()
import cv2
INPUT_WIDTH = 640
INPUT_HEIGHT = 640
modelWeights = "yolov5s.onnx"
net = cv2.dnn.readNet(modelWeights)
images = []
images.append(cv2.imread("1.jpg"))
images.append(cv2.imread("2.jpg"))
blob = cv2.dnn.blobFromImages(images, 1/255, (INPUT_WIDTH, INPUT_HEIGHT), [0,0,0], 1, crop=False)
net.setInput(blob)
output_layers = net.getUnconnectedOutLayersNames()
outputs = net.forward(output_layers)
I am getting following error:
[ERROR:[email protected]] global D:\CODING\Developement\C++\Libs\OpenCV 4.6.current\modules\dnn\src\onnx\onnx_importer.cpp (1054) cv::dnn::dnn4_v20220524::ONNXImporter::handleNode DNN/ONNX: ERROR during processing node with 2 inputs and 1 outputs: [Concat]:(onnx_node!Concat_119) from domain='ai.onnx'
Traceback (most recent call last):
File "d:\CODING\Developement\Python\OpenCVCurrent\VariableShapeTest.py", line 7, in <module>
net = cv2.dnn.readNet(modelWeights)
cv2.error: OpenCV(4.6.0-dev) D:\CODING\Developement\C++\Libs\OpenCV 4.6.current\modules\dnn\src\onnx\onnx_importer.cpp:1073: error: (-2:Unspecified error) in function 'cv::dnn::dnn4_v20220524::ONNXImporter::handleNode'
> Node [[email protected]]:(onnx_node!Concat_119) parse error: OpenCV(4.6.0-dev) D:\CODING\Developement\C++\Libs\OpenCV 4.6.current\modules\dnn\src\layers\concat_layer.cpp:108: error: (-201:Incorrect size of input array) Inconsistent shape for ConcatLayer in function 'cv::dnn::ConcatLayerImpl::getMemoryShapes'
@alalek I don't know if this is a bug or just a lack of support for model with dynamic input ?
You can download the model from here:
same error
$ pip list
Package Version
--------------------- --------
numpy 1.25.0
opencv-contrib-python 4.7.0.72
opencv-python 4.7.0.72
pip 23.1.2
setuptools 68.0.0
wheel 0.40.0
$ python -V
Python 3.10.12
[ERROR:[email protected]] global net_impl.cpp:1196 getLayerShapesRecursively OPENCV/DNN: [Convolution]:(Conv2D): getMemoryShapes() post validation failed. inputs=1 outputs=1/1 blobs=1 inplace=0
[ERROR:[email protected]] global net_impl.cpp:1199 getLayerShapesRecursively input[0] = [ 1 3 2268 4032 ]
[ERROR:[email protected]] global net_impl.cpp:1203 getLayerShapesRecursively output[0] = [ 1 256 2268 4032 ]
[ERROR:[email protected]] global net_impl.cpp:1207 getLayerShapesRecursively blobs[0] = CV_32FC1 [ 256 3 3 3 ]
[ERROR:[email protected]] global net_impl.cpp:1209 getLayerShapesRecursively Exception message: OpenCV(4.7.0) /Users/runner/work/opencv-python/opencv-python/opencv/modules/dnn/src/net_impl.cpp:1188: error: (-2:Unspecified error) in function 'void cv::dnn::dnn4_v20221220::Net::Impl::getLayerShapesRecursively(int, cv::dnn::dnn4_v20221220::Net::Impl::LayersShapesMap &)'
> (expected: 'total(os[i]) > 0'), where
> 'total(os[i])' is -1953955840
> must be greater than
> '0' is 0
Traceback (most recent call last):
File "/Users/51pwn/MyWork/51pwn4ai/lib/isr.py", line 29, in <module>
upScalePic = sr.upsample(input)
cv2.error: OpenCV(4.7.0) /Users/runner/work/opencv-python/opencv-python/opencv/modules/dnn/src/net_impl.cpp:1188: error: (-2:Unspecified error) in function 'void cv::dnn::dnn4_v20221220::Net::Impl::getLayerShapesRecursively(int, cv::dnn::dnn4_v20221220::Net::Impl::LayersShapesMap &)'
> (expected: 'total(os[i]) > 0'), where
> 'total(os[i])' is -1953955840
> must be greater than
> '0' is 0
Opencv super resolution error: cv2.error: OpenCV(4.7.0) /io/opencv/modules/dnn/src/net_impl.cpp:1188
https://stackoverflow.com/questions/75311675/opencv-super-resolution-error-cv2-error-opencv4-7-0-io-opencv-modules-dnn-s
opencv4.8.0 do not work either
[ERROR:[email protected]] global onnx_importer.cpp:2588 cv::dnn::dnn4_v20230620::ONNXImporter::parseShape DNN/ONNX(Shape): dynamic 'zero' shapes are not supported, input 387 [ 1 64 0 0 ]
[ERROR:[email protected]] global onnx_importer.cpp:1064 cv::dnn::dnn4_v20230620::ONNXImporter::handleNode DNN/ONNX: ERROR during processing node with 1 inputs and 1 outputs: [Shape]:(onnx_node!Shape_60) from domain='ai.onnx'
OpenCV: terminate handler is called! The last OpenCV error is:
OpenCV(4.8.0) Error: Unspecified error (> Node [[email protected]]:(onnx_node!Shape_60) parse error: OpenCV(4.8.0) C:\GHA-OCV-1\_work\ci-gha-workflow\ci-gha-workflow\opencv\modules\dnn\src\onnx\onnx_importer.cpp:2589: error: (-215:Assertion failed) !isDynamicShape in function 'cv::dnn::dnn4_v20230620::ONNXImporter::parseShape'
> ) in cv::dnn::dnn4_v20230620::ONNXImporter::handleNode, file C:\GHA-OCV-1\_work\ci-gha-workflow\ci-gha-workflow\opencv\modules\dnn\src\onnx\onnx_importer.cpp, line 1083
The error thrown when loading
cv::dnn::readNet("super_resolution.onnx");
Build 16928806f998a3112d52a086090833f54b65a3f1 manually and test with the super_resolution.zip again, error message
[ERROR:[email protected]] global onnx_importer.cpp:2440 cv::dnn::dnn4_v20230620::ONNXImporter::parseShape DNN/ONNX(Shape): dynamic 'zero' shapes are not supported, input 504 [ 1 0 0 8 ]
[ERROR:[email protected]] global onnx_importer.cpp:1033 cv::dnn::dnn4_v20230620::ONNXImporter::handleNode DNN/ONNX: ERROR during processing node with 1 inputs and 1 outputs: [Shape]:(onnx_node!Shape_149) from domain='ai.onnx'
OpenCV: terminate handler is called! The last OpenCV error is:
OpenCV(4.8.0-dev) Error: Unspecified error (> Node [[email protected]]:(onnx_node!Shape_149) parse error: OpenCV(4.8.0-dev) D:\programming\3rdLibs\opencv\modules\dnn\src\onnx\onnx_importer.cpp:2441: error: (-215:Assertion failed) !isDynamicShape in function 'cv::dnn::dnn4_v20230620::ONNXImporter::parseShape'
> ) in cv::dnn::dnn4_v20230620::ONNXImporter::handleNode, file D:\programming\3rdLibs\opencv\modules\dnn\src\onnx\onnx_importer.cpp, line 1052
Hello @SunlifeV , we need to know input shapes ahead of inference so that memory can be allocated. In your model, height and width are all unknown, which is not supported by the current dnn engine.
If you export your model with a all-fixed input shape, you can load and infer your model with latest opencv dnn. You can also use onnxsim to overwrite the input shape:
onnxsim --overwrite-input-shape=1,3,640,640 super_resolution.onnx super_resolution.fixed.sim.onnx
Also checked with lasted dnn:
import numpy as np
import cv2 as cv
net = cv.dnn.readNet("./super_resolution.fixed.sim.onnx")
x = np.random.rand(1, 3, 640, 640).astype(np.float32)
net.setInput(x)
out = net.forward()
print(out.shape)
@stereomatchingkiss @hossein-moghaddam @hktalent @PasaOpasen @antoszy May be I solved your problem with https://github.com/LaurentBerger/dnnlegacy (put this module in opencv_contrib) and recompile opencv and try
int main(int argc, char** argv) { { std::map<std::string, int> val; val["width"] = 640; val["height"] = 480; cv::dnn::Net x = dnnlegacy::importOnnxWithFixedShape("super_resolution.onnx", val); Mat img = imread(samples::findFile("butterfly.jpg")); Image2BlobParams paramBlob; paramBlob.datalayout = DNN_LAYOUT_NCHW; paramBlob.ddepth = CV_32F; paramBlob.mean = Scalar(0, 0, 0, 0); paramBlob.scalefactor = Scalar(1, 1, 1, 1); paramBlob.size = Size(640, 480); paramBlob.swapRB = false; paramBlob.paddingmode = DNN_PMODE_NULL; Mat blobNHWC = blobFromImageWithParams(img, paramBlob); x.setInput(blobNHWC); Mat y = x.forward(); }
I did not check output blob. Can you check it?
