Faster RCNN Error: Input image (1024x600) does not match NN (3x1)

isagmz

I am trying to use Faster RCNN with ResNet50 backbone. I downloaded the model provided with

omz_downloader --name faster_rcnn_resnet50_coco

This is my code:

MODEL_NAME = 'faster_rcnn_resnet50_coco'
HEIGHT = 600
WIDTH = 1024
BATCH_SIZE = 1
CHANNELS = 3
SCALE_FACTOR = 1

# create pipeline
pipeline = depthai.Pipeline()

# create camera node
cam_rgb = pipeline.createColorCamera()
cam_rgb.setPreviewSize(WIDTH, HEIGHT) # input size to fasterRCNN, width=1024 and height=600
cam_rgb.setInterleaved(False) # need planar
cam_rgb.setColorOrder(depthai.ColorCameraProperties.ColorOrder.BGR) # Ensure BGR color order

# create nn node
detection_nn = pipeline.createNeuralNetwork()
detection_nn.setBlobPath(blobconverter.from_zoo(name=MODEL_NAME))

# create ouput nodes for camera and nn
xout_rgb = pipeline.createXLinkOut()
xout_rgb.setStreamName("rgb")

xout_nn = pipeline.createXLinkOut()
xout_nn.setStreamName("nn")

# link nodes
cam_rgb.preview.link(detection_nn.input) # cam output goes into nn
cam_rgb.preview.link(xout_rgb.input) # cam output goes to display
detection_nn.out.link(xout_nn.input) # nn output goes to display

# connect to device
with depthai.Device(pipeline) as device:
    # create queues for cam and nn
    q_rgb = device.getOutputQueue("rgb") # names set above
    q_nn = device.getOutputQueue("nn") # names set above

    # handle output from cam 
    frame = None # will hold output from camera
    nn_output = [] # image_id, label, confidence, x/y_min, x/y_max

    while True:
        in_rgb = q_rgb.tryGet()

        if in_rgb is not None:
            frame = in_rgb.getCvFrame()
        if frame is not None and frame.shape[0] > 0 and frame.shape[1] > 0:
            print(frame.shape)
            cv2.imshow("preview", frame)

        # break if press 'q'
        if cv2.waitKey(1) == ord('q'):
            break

I get this error:

My "frame" size is (600, 1024, 3). I am trying to follow this documentation but I am not sure how to edit the camera feed to fit into the input of the model.

Would appreciate any help, thanks.

jakaskerl

Hi @isagmz
The code looks ok, but the layout of the model is incorrectly specified (depthai expects a different layout) - docs.

Thanks,
Jaka

isagmz

Hi Jaka,

Thanks for the reply. I am trying to change the layout of the model.

I have a saved_model.pb file and I used the omz_converter to make the model IR format. When I use the .xml and .bin files on the blobConverter, I cannot set any flags for the layout because the flags are not recognized.

When I try to convert the file using:

mo --input_model /home/igomez/Documents/igomez/depthai_spot/public/faster_rcnn_resnet50_coco/faster_rcnn_resnet50_coco_2018_01_28/frozen_inference_graph.pb --mean_values=[123.675,116.28,103.53] --scale_values=[58.395,57.12,57.375] --reverse_input_channels --layout NCHW

It will fail with:

[ ERROR ] -------------------------------------------------

[ ERROR ] ----------------- INTERNAL ERROR ----------------

[ ERROR ] Unexpected exception happened.

[ ERROR ] Please contact Model Optimizer developers and forward the following information:

[ ERROR ] Check 'element_type.is_real()' failed at src/core/src/preprocess/preprocess_steps_impl.cpp:118:

Mean preprocessing can be applied to 'float' inputs. Consider using of 'convert_element_type' before scaling. Current type is: u8

[ ERROR ] Traceback (most recent call last):

File "/home/igomez/depthai/lib/python3.8/site-packages/openvino/tools/mo/convert_impl.py", line 892, in _convert

ov_model, legacy_path = driver(argv, {"conversion_parameters": non_default_params})

File "/home/igomez/depthai/lib/python3.8/site-packages/openvino/tools/mo/convert_impl.py", line 558, in driver

res_ngraph_function = moc_emit_ir(ngraph_function, argv)

File "/home/igomez/depthai/lib/python3.8/site-packages/openvino/tools/mo/moc_frontend/moc_emit_ir.py", line 14, in moc_emit_ir

apply_preprocessing(ov_function=ngraph_function, argv=argv)

File "/home/igomez/depthai/lib/python3.8/site-packages/openvino/tools/mo/back/preprocessing.py", line 435, in apply_preprocessing

ov_function = prep.build()

RuntimeError: Check 'element_type.is_real()' failed at src/core/src/preprocess/preprocess_steps_impl.cpp:118:

Mean preprocessing can be applied to 'float' inputs. Consider using of 'convert_element_type' before scaling. Current type is: u8

[ ERROR ] ---------------- END OF BUG REPORT --------------

[ ERROR ] -------------------------------------------------

I cannot find an existing ONNX file that will work with the blobConverter from the onnx github because:

Currently ONNX operator set version: 16 is unsupported. Falling back to: 15 FAILED: fasterrcnn_resnet50_fpn_v2_Opset16

I am wondering what I can do to get around this? I was hoping there would be a way I can convert my saved_model.pb file to .xml and .bin with the layout I need for the model.

Thanks,

Isabella

jakaskerl

isagmz I have a saved_model.pb file and I used the omz_converter to make the model IR format. When I use the .xml and .bin files on the blobConverter, I cannot set any flags for the layout because the flags are not recognized.

And if you go under tensorflow, upload your model and then set the optimizer params with the layout?

Alternatively, you can send me the model via support@luxonis.com and I can try to convert it.

Thanks,
Jaka