Issue with model converted with OpenVino 2022.1 and not 2021.4

meaton

Hi,
I am using "Depthai Core", release v2.17.4, to create a C++ application utilizing a DeepLabV3 algorithm running on the OAK-D Pro. Currently, I am having a problem using the BLOB generated with OpenVino 2022.1 and not OpenVino 2021.4. The DeepLabV3 model is created with Keras, and I used the steps from "depthai-ml-training/colab-notebooks/DeepLabV3plus_MNV2.ipynb" to convert the TF model to OpenVINO Intermediate Representation: .xml and .bin and then convert the XML/Bin to a BLOB. The BLOB created with OpenVino 2021.4 runs but does not generates the results I get in Keras. This issue may be related to the XML fix discussed in the Colab notebook (still looking into this). So, I moved to OpenVino 2022.1, and when I run the BLOB on the OAK-D Pro, I get this error "[18443010A157641200] [1.1.1.3] [9.855] [NeuralNetwork(2)] [warning] Input image (256x256) does not match NN (3x256)". The input image to the BLOB is a 256x256x3 (RGB) preview image from the Color Camera. It appears the input is getting scrambled some how.

Is there an example code (python script or Colab notebook) using the new OpenVino 2022.1 or has anybody come across a similar problem?

Here is an excerpt from my python script to calls OpenVino 2021.4.
subprocess.call("source /opt/intel/openvino_2021/bin/setupvars.sh", shell=True) subprocess.call("python /opt/intel/openvino_2021/deployment_tools/model_optimizer/mo_tf.py \ --input_model \"" + frozen_file + "\" \ --model_name " + model_base_name + " \ --data_type " + data_type + " \ --input_shape \"[1," + str(IMAGE_SIZE) + "," + str(IMAGE_SIZE) + ",3]\" \ --reverse_input_channel \ --output_dir \"" + export_vino_dir + "\"", shell=True)

Here is my python call to OpenVino 2022.1
subprocess.call("mo \ --saved_model_dir \"" + model_export_dir + "\" \ --model_name " + model_base_name + " \ --data_type " + data_type + " \ --input_shape [1," + str(IMAGE_SIZE) + "," + str(IMAGE_SIZE) + ",3] \ --reverse_input_channel \ --output_dir \"" + export_vino_dir + "\"", shell=True)

Here is the BlobConverter code. I only change the OpenVino version in the code
print("(Step 18): Convert OpenVino IR to blob") os.chdir(export_vino_dir) blob_path = blobconverter.from_openvino( xml=xml_file, bin=bin_file, data_type=data_type, shaves=6, version="2021.4")

Thanks for the help!
Mike E

stephansturges

meaton

If you use OpenVino 2021.4 with the fix (change layer 490 to 32 bit if memory serves) it will work with DeepLabV3 networks and you won't have to mess with this mess... it's entirely possible that Intel have messed something up in 2022.1 that's causing your issue, I would avoid spending too much time digging into this.

Matija

Hey @meaton ,

Best to try what @stephansturges suggested. I think there is a bug in 2022.1, which should be resolved in 2022.2 (not yet supported in DepthAI). But I think your issue is related to this bug fix:

Fixed scenario when both reverse_input_channel and changing the layout are applied in the same conversion

Best,
Matija

meaton

Thanks Matija !

Unfortunately, when I tried "DeepLabV3plus_MNV2.ipynb" with my custom dataset (2 classes instead of 21 classes), I was getting an error with the eval.py related to the class size change (I tried all types of fixes I found on the internet with no luck). So, I moved over to using Keras to train the DeepLabV3 model. However, the OpenVino XML file generated from the Keras model has a different structure than the XML file in "DeepLabV3plus_MNV2.ipynb". So, I am not sure if my OpenVino 2021.4 BLOB issue (no results) is related to the XML issue or a completely different problem.

Do you guys have a date of when OpenVino 2022.2 will be supported?

Thanks,
Mike E

Matija

The fix is there just for the version from the Colab, which when exporting mixes up some types. In case you're using a different model, then this does not relate to you. The issue is likely that the layout of the model in TF/Keras is NHWC rather than NCHW which is used on OAK.

Could you try using flag --layout NCHW:NHWC when calling model optimizer meaton ?

stephansturges

meaton

Unfortunately, when I tried "DeepLabV3plus_MNV2.ipynb" with my custom dataset (2 classes instead of 21 classes), I was getting an error with the eval.py related to the class size change (I tried all types of fixes I found on the internet with no luck)

Did you change the number of classes before generating tfrec files in data_generator.py? That should fix it 🙂

meaton

Matija

The flag --layout NCHW:NHWC gives me the following error

Layout name is invalid (:). Only english letters, digits and _ is allowed

However, it helped to point out that I need to handle the conversion from NHWC to NCHW. I plan to modify the model in the Keras script to handle the conversion.
Thanks!
Mike E

meaton

stephansturges

Yeah, I updated the number of classes in data_generator.py prior to generating tfrec files. At this point, I figure I will just focus on using Keras. Thanks for the suggestion..

Matija

meaton

Hm, it seems this was wrongly written in documentation. You can instead also try setting --source_layout nhwc and --target_layout nchw or vice versa.

meaton

Matija
Thanks for the tip! This new setting resolved the

[18443010A157641200] [1.1.1.3] [9.855] [NeuralNetwork(2)] [warning] Input image (256x256) does not match NN (3x256)

error when using OpenVino 2022.1. Below, is the updated python code I used in Keras. Unfortunately, my model still does not segment the object correctly (at all), whereas I get proper results with the Keras model. I plan to use the OpenVino Inference Engine to verify the converted model (IR format) still works.

subprocess.call("mo \ --saved_model_dir \"" + model_export_dir + "\" \ --model_name " + model_base_name + " \ --data_type " + data_type + " \ --input_shape [1," + str(IMAGE_SIZE) + "," + str(IMAGE_SIZE) + ",3] \ --reverse_input_channel \ --source_layout nhwc \ --target_layout nchw \ --output_dir \"" + export_vino_dir + "\"", shell=True)

Matija

meaton
Great to hear that. And yes, using OpenVINO Inference Engine is usually the best way to debug that. I noticed that you don't have any preprocessing flags when exporting the model.

Images on OAK are usually UINT8 BGR images. If your model in Keras accepts those directly, that is fine. But if your model in Keras takes normalized/standardized images, you might want to use --mean_values and --scale_values flags. If your model takes BGR images as input, then there is also no need for --reverse_input_channel.

Let me know how it goes 🙂

meaton

Matija
Thanks for the mean/scale value suggestion! That lead me down the right path and the results on the OAK-D are better. Now, I can focus on improving the training of the model.

Here is the call to convert the model
subprocess.call("mo \ --saved_model_dir \"" + model_export_dir + "\" \ --model_name " + model_base_name + " \ --data_type " + data_type + " \ --input_shape [1," + str(IMAGE_SIZE) + "," + str(IMAGE_SIZE) + ",3] \ --reverse_input_channel \ --source_layout nhwc \ --target_layout nchw \ --mean_values [104,117,123] \ --scale_values [255,255,255] \ --output_dir \"" + export_vino_dir + "\"", shell=True)

Here is the code for blob converter
blob_path = blobconverter.from_openvino( xml=xml_file, bin=bin_file, data_type=data_type, shaves=6, version="2022.1" )

Matija

meaton, glad to hear that! Often models use ImageNet mean and std for normalization. If this will be useful to someone else in the future, this is how flags look like after scaling to 0 - 255 range:
--mean_values [123.675,116.28,103.53] and --scale_values [58.395,57.12,57.375].

Have fun improving the models and using OAK-D 🙂

Best,
Matija

meaton

Matija

In Keras, I am using the ResNet50 model that was pre-trained on ImageNet. So, I used --mean_values [123.675,116.28,103.53] and --scale_values [58.395,57.12,57.375] to follow your advice and it improved the results on the OAK-D. Thanks again!

subprocess.call("mo \ --saved_model_dir \"" + model_export_dir + "\" \ --model_name " + model_base_name + " \ --data_type " + data_type + " \ --input_shape [1," + str(IMAGE_SIZE) + "," + str(IMAGE_SIZE) + ",3] \ --reverse_input_channel \ --source_layout nhwc \ --target_layout nchw \ --mean_values [123.675,116.28,103.53] \ --scale_values [58.395,57.12,57.375] \ --output_dir \"" + export_vino_dir + "\"", shell=True)

Matija

meaton Great to hear that! Happy to help 🙂

Nadi

I am still having the same issue with [NeuralNetwork(0)] [warning] Input image (512x512) does not match NN (3x512) using openvino_2022.2 optimizer (onnx->xml) and compiler(xml->blob).

any fix?

erik

Docs here on how to solve this.
Thanks, Erik