Spatial Calculation

rsantan

Hello,

I am using the OAK-D LR camera to calculate the distance to external ROIs from my model. I am using 2 color cameras instead of mono to get stereo and the center for the detections.
After I output the bounding box in the depthframe, the bounding boxes do not match and are not around the object and they look correctly on the model side debug image.
The coordinates being send by the model match on the depth ai pipeline.
Before I start banging my head against the table to make them match I wanted to make sure that the setDepthAlign is not doing something on the background which is not reflected in the depthframe.

Are the bounding boxes supposed to match exactly? I am getting significant difference in the position on imshow for openCV

https://docs.luxonis.com/software/depthai/examples/spatial_tiny_yolo/
On this example the box is also different compared to the color image. Is the box shown in the depth map the one actually used to calculate distance?

jakaskerl

rsantan
You need to align depth to the same camera that is capturing images that are sent into the detection network. Only when depth and detection frames are aligned, can the spatial calculator correctly return the spatial coordinates of the object.

Thanks,
Jaka

rsantan

I am aligning the image, however the bounding box does not falls on top of the object in the depthframe.
You can see the result on the image and the distance is also off by 2 meters.

jakaskerl

rsantan
CAM_A is center I believe, so if you set one camera to left and the other one to right, then the camera sockets are B and C, and aligning to A will not work. Try aligning to left (CAM_B).

Thanks,
Jaka

rsantan

My color camera where the detection is being done is on CAM_A

That is why I was aligning to CAMA. I tried with CAMB and CAM_C as you suggested but the output is also not right.

jakaskerl

rsantan
I see. Can you perhaps send me the full code (maybe swap your model with a standard yolo), so I can test locally on LR?

I'd check the setOutputSize() if this is causing mismatch. Might be that the BB is not scaled correctly. I don't know what you are sending to the SLC config but seems like the scales are wrong.

Thanks,
Jaka

eolson20

Hey Jaka,

Working with rsantan here, we've verified that the normalized bounding boxes our model is sending are the same values right up until we send the ROI configuration. My thought is that because the color frame is just from the perspective of the left lens, when it gets to the depth frame (which is fused left and right) it gets scaled incorrectly.

We de-normalize the boxes and convert from x,y,w,h to x1,y1,x2,y2.

Do you have an email we can forward the code too?

eolson20

I don't know what I did… but I re-built our machine learning code, and now the boxes are aligning correctly… I'm baffled but I think we fixed it