smartlab demo not working with OV 2022.1

[wallacezq]

I'm getting the following error when trying to run smartlab demo. The latest 2022.1 OV runtime is installed. Any ideas how i can fix this ?

Traceback (most recent call last): File "smartlab_demo.py", line 144, in main() File "smartlab_demo.py", line 105, in main frame_index=frame_counter) File "C:\test\open_model_zoo_2022\2022.1.0\open_model_zoo\demos\smartlab_demo\python\segmentor.py", line 87, in inference frame_index=frame_index) File "C:\test\open_model_zoo_2022\2022.1.0\open_model_zoo\demos\smartlab_demo\python\segmentor.py", line 136, in feature_embedding out_logits = infer_request.infer(input_dict)[self.i3d.outputs[0]] File "c:\Program Files (x86)\Intel\openvino_2022\python\python3.6\openvino\runtime\ie_api.py", line 110, in infer {} if inputs is None else normalize_inputs(inputs, get_input_types(self)) RuntimeError: Can't SetBlob with name: Placeholder, because model input (shape={1,16,224,224,3}) and blob (shape=(1.3.16.224.224)) are incompatible

Python command to launch the demo:

python smartlab_demo.py 
    -tv .\stream_1_top.mp4 
    -fv .\stream_1_high.mp4 
    -m_ta ".\intel\smartlab-object-detection-0001\FP32\smartlab-object-detection-0001.xml" 
    -m_tm ".\intel\smartlab-object-detection-0002\FP32\smartlab-object-detection-0002.xml" 
    -m_fa ".\intel\smartlab-object-detection-0003\FP32\smartlab-object-detection-0003.xml" 
    -m_fm ".\intel\smartlab-object-detection-0004\FP32\smartlab-object-detection-0004.xml" 
    -m_en ".\public\i3d-rgb-tf\FP32\i3d-rgb-tf.xml" 
    -m_de ".\intel\smartlab-sequence-modelling-0001/FP32/smartlab-sequence-modelling-0001.xml"

do note that openvino-dev 2022.1 is installed on the machine, and omz_downloader and omz_converter is used to download & convert the models.

[Wovchena]

@kminemur, can you look at it?

[kminemur]

@wallacezq

probably the error comes from reshape (line 57-58), since openvino API2.0, the model shape changed.

You may try

net.reshape({net.inputs[0]: PartialShape( [self.EmbedBatchSize, 3, self.EmbedWindowLength, self.ImgSizeHeight, self.ImgSizeWidth])})

[wallacezq]

@kminemur it doesn't work.

I get the following error: C:\test\open_model_zoo_2022\2022.1.0\open_model_zoo\demos\smartlab_demo\python>start.bat Traceback (most recent call last): File "smartlab_demo.py", line 144, in main() File "smartlab_demo.py", line 66, in main segmentor = SegmentorMstcn(core, args.device, args.m_encoder, args.m_decoder) File "C:\test\open_model_zoo_2022\2022.1.0\open_model_zoo\demos\smartlab_demo\python\segmentor.py", line 57, in init net.reshape({net.inputs[0]: PartialShape([self.EmbedBatchSize, 3, self.EmbedWindowLength, self.ImgSizeHeight, self.ImgSizeWidth])}) RuntimeError: Check 'input_shape[1].compatible(filters_shape[1])' failed at C:\j\workspace\private-ci\ie\build-windows-vs2019@3\b\repos\openvino\src\core\shape_inference\include\convolution_shape_inference.hpp:265: While validating node 'v1::Convolution RGB/inception_i3d/Conv3d_1a_7x7/conv_3d/convolution (Placeholder/mean/Fused_Add_[0]:f32{1,224,3,16,224}, RGB/inception_i3d/Conv3d_1a_7x7/batch_norm/batch_norm/mul_copy[0]:f32{64,3,7,7,7}) -> (f32{1,64,40,112,112})' with friendly_name 'RGB/inception_i3d/Conv3d_1a_7x7/conv_3d/convolution': Data batch channel count (224) does not match filter input channel count (3).

[kminemur]

the models come from which openvino version? 2022.1?

[wallacezq]

@kminemur downloaded with omz_downloader of the openvino-dev==2022.1.0

[kminemur]

I reproduced your issue. will take a look

[kminemur]

Hi i3d-rgb-tf model was changed, so need to fix some part of codes.

Workaround: please change segmentor.py as following

[L56-61] net = core.read_model(i3d_path) net.reshape({net.inputs[0]: PartialShape(
[self.EmbedBatchSize, self.EmbedWindowLength, self.ImgSizeHeight, self.ImgSizeWidth, 3])})

nodes = net.get_ops() net.add_outputs(nodes[11].output(0))

[L128-129] ~~input_data = np.asarray(input_data).transpose((0, 4, 1, 2, 3))~~ input_data = np.asarray(input_data) * 127.5 + 127.5

[L181-185] predictions = predictions[:, np.newaxis, :len(self.ActionTerms), :]

temporal_logits = predictions[:, :, :len(self.ActionTerms), :] # 11x16x2048 temporal_logits = softmax(temporal_logits[-1], 1) # 16x2048 temporal_logits = temporal_logits.transpose((0, 2, 1)).squeeze(axis=0)

[wallacezq]

Thanks for the workaround @kminemur . It only works for me if i added a minor change to the workaround proposed by you, else i would get an error that says TypeError: can't multiply sequence by non-int of type 'float'

@L128 
- input_data = np.asarray(input_data).transpose(0,4,1,2,3))
+ input_data = np.asarray(input_data)

[kminemur]

Thanks for the comment will create patch

[vladimir-dudnik]

patch created https://github.com/openvinotoolkit/open_model_zoo/pull/3430

[vladimir-dudnik]

@wallacezq patch #3430 merged to master branch, please verify