Hi,

We are using the OAK-D Pro Wide(with OV972). And we would like to check the RGB Depth alignment like below link.

RGB Depth alignment — DepthAI documentation | Luxonis

But based on our test it seems not available its RGB-Depth Alignment function anymore with the OAK-D Pro Wide(with OV9782). Is our test result is right? and are there any way to make this RGB-Depth Alighment with the OAK-D Pro Wide(with OV972)?

<picture explanation>
In the picture below, you can see that the angle of view of the depth image is narrower than that of the RGB image.

There is a person at the edge of the RGB image, but in the Depth image the person is out of the field of view and cannot be seen.


Best regards,
Ryan.

  • erik replied to this.

    Hi

    Based on our test you can check the RGB-Depth angle of view difference by just running the demo code provided by Luxonis like below. How do you think?


    Best regards,
    Ryan.

    • erik replied to this.

      Hi RyanLee ,
      The OV9782 color image has much wider FOV than the depth (as mono cameras need to be rectified for the depth) - so that's expected.
      Thanks, Erik

          Hi RyanLee ,
          It should be possible to rgb-detph align the model, but you would need to undistort the rgb, as calibration uses undistorted rgb when calculating intrinsics.
          Thanks, Erik

              18 days later

              HI erik

              We tried the code undistort the rgb image. But we got some error result like below.

              Please check it out and let us know how to handle this issue.

              Best regards,
              Ryan.

              • erik replied to this.

                  Hi RyanLee ,
                  I have developed rgb-depth alignemnt demo that also does rgb undistortion;
                  I tried it with the OAK-D W and it works as expected.
                  Thanks, Erik

                      HI erik

                      I checked the test code you shared us above.
                      It has been confirmed that the alignment between RGB/Depth has been significantly improved compared to before.

                      However, the alignment between the two images still seems insufficient at the lens periphery for use in Spatial Detection.

                      Is it possible to further improve the alignment between the two images more precisely?

                      <mismathing 1>

                      <mismathing 2>

                      <No depth information>

                      Best regards,
                      Ryan.

                      • erik replied to this.

                          Hi RyanLee ,
                          Have you modified the script so it uses 800P color camera (which is max of OV9782), and potentially disabled isp scaling?
                          So the first few lines of the script would be:

                          import cv2
import depthai as dai
import numpy as np

camRes = dai.ColorCameraProperties.SensorResolution.THE_800_P
camSocket = dai.CameraBoardSocket.RGB
ispScale = (1,1)
                              6 days later

                              Hi erik

                              This issue is cleared. So it is working now. And there was one of our mistake.

                              Actually, this issue is cleared with above solution you shared. But we did some mistake (Bug) by ourselves. Thank you for your patience always.

                              Best regards,
                              Ryan.

                                8 months later

                                I also experience the same depth-rgb missmatch.

                                Hello,

                                I'm unsure if it's appropriate to reply to this post, When I run the example code above I'm face the same issue as RyanLee regarding the depth-RGB alignment mismatch.

                                In my case I'm using an OAK-D Pro Wide with the IMX378 sensor, which has a 12MP resolution as opposed to 800P resolution on the OV972. Could this resolution difference be the cause of the misalignment between the depth and RGB images?

                                If not, could you please guide me on the camera resolution settings that would enable pixel-wise alignment between the depth and RGB images?

                                Thank you for your assistance.

                                • erik replied to this.

                                  Hello erik,

                                  No, when I run this example it does not work as expected.

                                  As you can see in this picture the whiteboard depth mask is unaligned with the rgb image, this can be clearly seen in the right hand-side of the picture.

                                  Here is another example:

                                  • erik replied to this.
                                    11 days later

                                    Hello Erik,

                                    I am using an OAKD PRO Wide camera, as stated by the documentation there is no need to calibrate this camera since it is a new model.

                                    Do you think I should still try to calibrate it?

                                      Hi repena
                                      Yes. Shouldn't take to long and it could fix the issue you are having.

                                      Thanks,
                                      Jaka

                                        Hi erik,

                                        I am also using the wide camera (OV9782), and I have tried your script, but I have a delay of 3 seconds. How to fix it?