Hi, We are using the OAK-D Pro Wide(with OV972). And we would like to check the RGB Depth alignment like below link. RGB Depth alignment — DepthAI documentation | Luxonis But based on our test it seems not available its RGB-Depth Alignment function anymore with the OAK-D Pro Wide(with OV9782). Is our test result is right? and are there any way to make this RGB-Depth Alighment with the OAK-D Pro Wide(with OV972)? **** In the picture below, you can see that the angle of view of the depth image is narrower than that of the RGB image. There is a person at the edge of the RGB image, but in the Depth image the person is out of the field of view and cannot be seen. [URL=https://i.imgur.com/xj2V8or.png][IMG]https://i.imgur.com/xj2V8or.png[/IMG][/URL] [URL=https://i.imgur.com/aowc1S3.jpg][IMG]https://i.imgur.com/aowc1S3.jpg[/IMG][/URL] Best regards, Ryan.

Hi @"RyanLee"#p4891 , Could you please [provide full MRE,](https://docs.luxonis.com/en/latest/pages/support/) together with minimal script you are using? Thanks ,Erik

Hi Based on our test you can check the RGB-Depth angle of view difference by just running the demo code provided by Luxonis like below. How do you think? [URL=https://i.imgur.com/c3TEnQE.png][IMG]https://i.imgur.com/c3TEnQEh.png[/IMG][/URL] [URL=https://i.imgur.com/E95z622.png][IMG]https://i.imgur.com/E95z622h.png[/IMG][/URL] Best regards, Ryan.

Hi @"erik"#p4931 That means that the RGB Depth alignment is not available for OAK-D Pro wide OV9782 anymore? **** https://docs.luxonis.com/projects/api/en/latest/samples/StereoDepth/rgb_depth_aligned/ Best regards, Ryan.

Hi @"RyanLee"#p4937 , It should be possible to rgb-detph align the model, but you would need [to undistort the rgb](https://github.com/luxonis/depthai-python/pull/661), as calibration uses undistorted rgb when calculating intrinsics. Thanks, Erik

HI @"erik"#p4945 We tried the code [undistort the rgb image](https://github.com/luxonis/depthai-python/blob/develop/examples/ColorCamera/rgb_undistort.py). But we got some error result like below.[URL=https://i.imgur.com/7u6DMs5.png][IMG]https://i.imgur.com/7u6DMs5.png[/IMG][/URL] Please check it out and let us know how to handle this issue. Best regards, Ryan.

Hi @"RyanLee"#p5141 , I have developed rgb-depth alignemnt demo that also does rgb undistortion; https://gist.github.com/Erol444/a9e50f5d8cef689dc86c544c15ca3175 I tried it with the OAK-D W and it works as expected. Thanks, Erik

Hi @"erik"#p5162 This issue is cleared. So it is working now. And there was one of our mistake. https://gist.github.com/Erol444/a9e50f5d8cef689dc86c544c15ca3175#file-rgb-undistortion-depth-alignment-py Actually, this issue is cleared with above solution you shared. But we did some mistake (Bug) by ourselves. Thank you for your patience always. Best regards, Ryan.

OAK-D Pro wide OV9782 RGB-Depth Alignment

RyanLee

Hi,

We are using the OAK-D Pro Wide(with OV972). And we would like to check the RGB Depth alignment like below link.

RGB Depth alignment — DepthAI documentation | Luxonis

But based on our test it seems not available its RGB-Depth Alignment function anymore with the OAK-D Pro Wide(with OV9782). Is our test result is right? and are there any way to make this RGB-Depth Alighment with the OAK-D Pro Wide(with OV972)?

<picture explanation>
In the picture below, you can see that the angle of view of the depth image is narrower than that of the RGB image.

There is a person at the edge of the RGB image, but in the Depth image the person is out of the field of view and cannot be seen.

Best regards,
Ryan.

erik

Hi RyanLee ,
Could you please provide full MRE, together with minimal script you are using?
Thanks ,Erik

RyanLee

Based on our test you can check the RGB-Depth angle of view difference by just running the demo code provided by Luxonis like below. How do you think?

Best regards,
Ryan.

erik

Hi RyanLee ,
The OV9782 color image has much wider FOV than the depth (as mono cameras need to be rectified for the depth) - so that's expected.
Thanks, Erik

RyanLee

Hi erik

That means that the RGB Depth alignment is not available for OAK-D Pro wide OV9782 anymore?
<RGB Depth alignment>
https://docs.luxonis.com/projects/api/en/latest/samples/StereoDepth/rgb_depth_aligned/

Best regards,
Ryan.

erik

Hi RyanLee ,
It should be possible to rgb-detph align the model, but you would need to undistort the rgb, as calibration uses undistorted rgb when calculating intrinsics.
Thanks, Erik

RyanLee

HI erik

We tried the code undistort the rgb image. But we got some error result like below.

Please check it out and let us know how to handle this issue.

Best regards,
Ryan.

erik

Hi RyanLee ,
I have developed rgb-depth alignemnt demo that also does rgb undistortion;
I tried it with the OAK-D W and it works as expected.
Thanks, Erik

RyanLee

HI erik

I checked the test code you shared us above.
It has been confirmed that the alignment between RGB/Depth has been significantly improved compared to before.

However, the alignment between the two images still seems insufficient at the lens periphery for use in Spatial Detection.

Is it possible to further improve the alignment between the two images more precisely?

Best regards,
Ryan.

erik

Hi RyanLee ,
Have you modified the script so it uses 800P color camera (which is max of OV9782), and potentially disabled isp scaling?
So the first few lines of the script would be:

import cv2
import depthai as dai
import numpy as np

camRes = dai.ColorCameraProperties.SensorResolution.THE_800_P
camSocket = dai.CameraBoardSocket.RGB
ispScale = (1,1)

RyanLee

Hi erik

This issue is cleared. So it is working now. And there was one of our mistake.

Actually, this issue is cleared with above solution you shared. But we did some mistake (Bug) by ourselves. Thank you for your patience always.

Best regards,
Ryan.

LeonelSimo

Hi erik,

I am also using the wide camera (OV9782), and I have tried your script, but I have a delay of 3 seconds. How to fix it?

repena

I also experience the same depth-rgb missmatch.

Hello,

I'm unsure if it's appropriate to reply to this post, When I run the example code above I'm face the same issue as RyanLee regarding the depth-RGB alignment mismatch.

In my case I'm using an OAK-D Pro Wide with the IMX378 sensor, which has a 12MP resolution as opposed to 800P resolution on the OV972. Could this resolution difference be the cause of the misalignment between the depth and RGB images?

If not, could you please guide me on the camera resolution settings that would enable pixel-wise alignment between the depth and RGB images?

Thank you for your assistance.

erik

Hi repena ,
Does this example now work as expected? https://docs.luxonis.com/projects/api/en/latest/samples/StereoDepth/rgb_depth_aligned/#rgb-depth-alignment

repena

Hello erik,

No, when I run this example it does not work as expected.

As you can see in this picture the whiteboard depth mask is unaligned with the rgb image, this can be clearly seen in the right hand-side of the picture.

Here is another example:

erik

Hi repena ,
Could you try this script?

repena

erik I tried this code but this is the output:

TestCA

erik

I have tried and added Image parameter like lumianace and few more computer vision algorthims for smoothing :

Btw how to reach to your team ?

erik

Hi repena , it seems like you don't have the newest calibration - are you using an older device? If that's the case, you can recalibrate the device.

repena

Hello Erik,

I am using an OAKD PRO Wide camera, as stated by the documentation there is no need to calibrate this camera since it is a new model.

Do you think I should still try to calibrate it?

jakaskerl

Hi repena
Yes. Shouldn't take to long and it could fix the issue you are having.

Thanks,
Jaka

erik

Hi LeonelSimo , please see docs here: https://docs.luxonis.com/projects/api/en/latest/tutorials/low-latency/

jakaskerl

LeonelSimo
You can pass the --help to the calibrate script to see possible arguments. The capture delay can be set using: -cd <n seconds>.

Thanks,
Jaka