Deeplab C++

Tusk

Hello,

I am trying to write the same python app (gen2-deeplabv3_person) in a C++ application. When I get the vector by calling getLayerInt32, I want to generate the color cv::Mat. I can't figure out how to write the following python code into C++:

def decode_deeplabv3p(output_tensor):
class_colors = [[0,0,0], [0,255,0]]
class_colors = np.asarray(class_colors, dtype=np.uint8)

output = output_tensor.reshape(nn_shape,nn_shape)
output_colors = np.take(class_colors, output, axis=0)
return output_colors

Does anybody have an idea how this can be written in C++? I will appreciate any hints anybody can provide.

Thanks,

James

erik

Hi Tusk,
Unfortunately, I am not familiar with C++ / OpenCV, so I would suggest asking this question on OpenCV community (discord/stack overflow/etc.) as it's not specific to DepthAI/OAK cameras.
Thanks, Erik

Tusk

erik

Erick,

Thank you for your reply. My apologies if my question wasn't clear. I am asking about the 32 bit vector from the tensor layer, which is retrieved from the DataOutputQueue via the Depth AI SDK that I am using to perform segmentation with an oak-d lite camera. I believe my question is in the appropriate discussion forum, as it isn't about OpenCV itself, but about how to interpret what I am getting out of the layer and use it in a cv::Mat, just like it is used in the gen2-deeplabv3_person example in the Depth AI experiments git. It may not be you, but I believe there may be somebody in the forum that can help me.

Thanks,

James

Tusk

@Brandon Any ideas on what I am trying to do?

Brandon

Not sure. @Matija might know.

Matija

So, you are asking what is the output of the model (the Int32 layer that you get with DepthAI)?
The output layer (before reshaping) is a [256x256 vector,], which after being reshaped to 256x256 represents a binary mask, where 0=background and 1=person. In the deeplabv3_person experiment, the output is actually being reshaped to [256, 256, 1, 1], so that np.take() can be used while decoding the output (generating color mask). An alternative way of doing this is by first reshaping the array to [256, 256] only, scaling the binary mask to range [0-255], and then calling cv2.applyColorMap(). See decoding in deeplabv3_multiclass. Here the output layer is the same, with the difference that mask is not binary, but contains integers corresponding to each class. If you are using only a person classifier, this could be modified by simply multiplying all 1s corresponding to the person class by 255, and then applying the color map.

Is this what you are interested in @Tusk ?

Tusk

Matija thank you very much for the reply. I have a better understanding of the 32 bit vector. In python, it is very easy to perform reshaping of np arrays. Since I am doing this in cpp, my hope was to get some hints on how to do this with cv::Mat. Do you have any experience with this sort of array manipulation in cpp?

Thanks,

James

GergelySzabolcs

Tusk

Hello.
cv::Mat exposes reshape functionality, see API here: https://docs.opencv.org/4.x/d3/d63/classcv_1_1Mat.html#a4eb96e3251417fa88b78e2abd6cfd7d8

Matija

Hey, I personally don't unfortunately. I'll forward this internally and someone should get back to you.