Hi,

I would like to use a keypoint detection model on my OAK-D to mark points on broccolis.

I already trained a Pytorch keypoint RCNN: torchvision.models.detection.keypointrcnn_resnet50_fpn.

I converted it to ONNX but it seems unconvertable for the online converter. It throws this error:
[ ERROR ] -------------------------------------------------
[ ERROR ] ----------------- INTERNAL ERROR ----------------
[ ERROR ] Unexpected exception happened.
[ ERROR ] Please contact Model Optimizer developers and forward the following information:
[ ERROR ] While validating ONNX node '<Node(ConstantOfShape): /rpn/anchor_generator/ConstantOfShape>':
Check 'broadcast_shape_rank.compatible(1)' failed at core/shape_inference/include/broadcast_shape_inference.hpp:171:
While validating node 'v3::Broadcast Broadcast_2893 (Constant_2892[0]:i64{}, /rpn/anchor_generator/Constant_12_output_0[0]:i64{}) -> (dynamic...)' with friendly_name 'Broadcast_2893':
Broadcast shape rank must be 1, but has 0

[ ERROR ] Traceback (most recent call last):
File "/app/venvs/venv2022_1/lib/python3.8/site-packages/openvino/tools/mo/main.py", line 533, in main
ret_code = driver(argv)
File "/app/venvs/venv2022_1/lib/python3.8/site-packages/openvino/tools/mo/main.py", line 489, in driver
graph, ngraph_function = prepare_ir(argv)
File "/app/venvs/venv2022_1/lib/python3.8/site-packages/openvino/tools/mo/main.py", line 394, in prepare_ir
ngraph_function = moc_pipeline(argv, moc_front_end)
File "/app/venvs/venv2022_1/lib/python3.8/site-packages/openvino/tools/mo/moc_frontend/pipeline.py", line 147, in moc_pipeline
ngraph_function = moc_front_end.convert(input_model)
RuntimeError: While validating ONNX node '<Node(ConstantOfShape): /rpn/anchor_generator/ConstantOfShape>':
Check 'broadcast_shape_rank.compatible(1)' failed at core/shape_inference/include/broadcast_shape_inference.hpp:171:
While validating node 'v3::Broadcast Broadcast_2893 (Constant_2892[0]:i64{}, /rpn/anchor_generator/Constant_12_output_0[0]:i64{}) -> (dynamic...)' with friendly_name 'Broadcast_2893':
Broadcast shape rank must be 1, but has 0

[ ERROR ] ---------------- END OF BUG REPORT --------------
[ ERROR ] -------------------------------------------------

Any ideas about this problem are welcome. I'm considering trying to convert this model or to try to train and convert another, better compatible, model. Any suggestions are welcome!

    15 days later

    Hi fredrik
    Sorry for late reply. I was unsure whether this was a luxonis tool related issue or not. Since your model has additional ouptut (keypoints), you cannot directly use it on depthai, unless you will be doing host side decoding of the results. What tool are you using to convert the .pt model? Seems like it's failing at model optimizer stage, which indicates the layers are unsuported.

    cc @Matija Any ideas what the issue might be?

    Thanks,
    Jaka

      6 days later

      It seems that the model has an operation that is not supported on our VPU.

      fredrik Is it possible to share the source code and the ONNX? You might have to prune the model and do some decoding on host.

        21 days later

        Hi @Jackmarks and @Matija,

        thanks for your response. would be interesting too check if model pruning makes it compatible with the VPU. I know the pose keypoint models work quite good on the VPU, so therefore I expect that it will be possible.

        I will share the source code and ONNX with you soon.