Issues with depth align with color image. [OAK-D Short Range]

alfredogpt

Hello I am trying to depth align with the color image on my OAK-D Short range.

I was following the following forum which lead me to find the following page which is exactly what I need.

I usually run the sample code before I try and integrate it in my application.

When I first ran the code I got the following error

Traceback (most recent call last):
    lensPosition = calibData.getLensPosition(rgbCamSocket)
RuntimeError: There is no Camera data available corresponding to the the requested cameraID

Which makes sense because OAK-D SR does not have a CAM_A rgbCamSocket = dai.CameraBoardSocket.CAM_A #line 53, but it does have a CAM_B and CAM_C. When I tried switching this socket to either CAM_B or CAM_C I get the following error.

RuntimeError: Camera(0) - Specified camera board socket already used

I also tried playing around with using dai.node.ColorCamera instead of dai.node.Camera but I end up getting the same error message.

As far as I understand the error, somewhere in the code I am trying to use the camera socket more than once. But I am unsure where that is happening and why as I did a tiny modification.

If im able to find a solution before someone answers ill post it here. In the meantime any help is appreciated.

EDIT_ONE: I think I have an idea of what is going on wrong. When I set camRgb to be either socket B or C is when the issue is happening because I believe at

left = pipeline.create(dai.node.MonoCamera)
right = pipeline.create(dai.node.MonoCamera)

We are setting socket B to be left and Socket C to be right. Which when we set socket B or C to camRgb it does not like it. Ill have to do some more modifications to see if my theory is correct.

EDIT_TWO: I think I got it to work (not sure if it works 100% like I want it to) it runs the demo and does not crash. ill have to wait till next week before I can try and plug it into my application to see if everything is fine. Git gist

erik

Hi @alfredogpt ,
You should likely use depthai-viewer instead, as it has support for newer models (while depthai_demo does not).

You are correct on creating camera node twice in the code - first with MonoCamera (incorrect, should be ColorCamera/Camera). Depth from the OAK-D-SR is aligned to rectified right frame.

Thanks, Erik

alfredogpt

erik

Right, but if im not mistaken rectified right frame does not carry any rgb information ? I need to aligned frame to contain rgb information for my specific use case. More specifically this is for a point cloud application.

In the gist I linked in my original post I am having trouble piping the rgb (which I think is aligned with the depth) and depth to a point cloud. The point cloud depth points look super incorrect.

I believe im not aligning the color and depth correctly. As I can pipe a rgb (unaligned with depth) and depth to a point cloud and the resulting point cloud is correct.

erik

@alfredogpt I assume you are using RGB depth along (example here: https://docs.luxonis.com/projects/api/en/latest/samples/StereoDepth/rgb_depth_aligned/#rgb-depth-alignment) to eg. right camera? then depth + right color stream would be aligned.

alfredogpt

erik

Yes correct I am using that example. I have it linked in my original post. I made some progress on displaying the point cloud after I performed "alignment" but its still not aligned correctly. I am trying to depth align with the right color camera.

I added a small feature where you can freeze a pointcloud and click on the rgb image and then calculate XYZ points (calculated using HostSpatialsCalc) and feed those into the point cloud. The calculated XYZ points that are fed into the point cloud are blue to make it easy to distingish from my scene.

I drew a red line on a table and pointed the camera onto that as a reference. When I click on the start and end of the red line in the point cloud it appears as if the clicked XYZ points (calculated using HostSpatialsCalc) are shifted to the right (compared to where I clicked in the RGB image) . I can calculate where these XYZ points are in the real world in reference to a object I have in the table and I can visually see that the points are in fact shifted to the right slightly. (compared to where I clicked in the RGB image)

I then did some further debugging and decided to draw a circle on the RGB window and rectRight OpenCV window using the same clicked image XY cords . When I click on the RGB window i draw the same XY cords on the rect right image but on the rectRight image they appear more to the right.

If I make sure the circle lines up with the start and end in the rectRight frame the point cloud clicked points align perfectly with the red line in the point cloud. And also these same clicked points (the ones that align with the start and end rectRight frame red line) end up in the correct real world coordinates in reference to an object I have on the table.

The XYZ values are super correct whenever circle is aligned with my target in the RightRect frame but I need it to be aligned with the RGB image.

This makes me believe the depth is still aligned with the RectRight frame as having my circle over my target in the RectRight frame leads to the correct values in contracts with the RGB image which gives me incorrect values when that circle is over my target.

I believe im performing the depth alignment incorrectly maybe ? I really hope im explaining my self correctly. Sorry if there is bad english as it is not my first language.

EDIT NUMBER ONE: I noticed in the link (https://docs.luxonis.com/projects/api/en/latest/samples/StereoDepth/rgb_depth_aligned/#rgb-depth-alignment) it mentions some steps to do if you have high resolution. in my case the width is 1280 and height is 720. Not sure if that is considered high resolution but I will look into that more tomorrow.

I run my code like this python3 main.py

Here is my code

jakaskerl

Hi @alfredogpt
You are undistorting the right stream and then linking it to stereo node, where rectification is performed.
It's currently not possible to do this unless you manually create rectification/undistortion mesh and use it on a color camera.
Might create and example for it if I have some spare time.

Thanks
Jaka

alfredogpt

jakaskerl

An example would be nice thank you !

In the mean time given that information I will see if I can figure it out.

Thanks again