OV9782 has pink/purple hue

silo

I have an Oak-D Pro Wide with the OV9782 central camera and I am having an issue where the video frames have a pink or purple hue. I have been using the undistort camera script provided here https://docs.luxonis.com/software/depthai/examples/camera_undistort/. When using the native resolution of 1280x800 everything appears normal. However, if I lower the resolution to 640x400 then the entire image is covered in a pink hue. The native resolution is too large for my use case and previously I was using the .IspScale function to downsize the image, but it looks like with the new camera node the .setSize method is the preferred approach now.

I also tested the IR illumination light and confirmed that is not what is causing the image to appear discolored.

I was reading about possible issues with the OV9782 where some of the sensors being shipped were missing the IR filter. The pink hue that I'm seeing seems to be basically identical to the users that had that issue. But the really strange part is that I only get this problem when I downscale from the 1280x800 native res.

I was originally 100% sure that this was some kind of software issue, but after seeing the images from users with defective units that were missing the IR filter and seeing how eerily similar my images look I'm starting to second guess myself.

I would greatly appreciate any help or insight into what might be happening.

jakaskerl

silo
This is incredibly pink and should probably not happen due to IR.
If you are using Camera node, there is a bug that autoselects the camera as MONO instead of RGB at certain resolutions. Should be fixed in depthai v3.

Thanks,
Jaka

silo

jakaskerl Thanks for the clarification. I am using the Camera node in place of the older ColorCamera node.

Do you know if there is a workaround for this in 2.29? Is there a way to force it to select the RGB camera instead of mono?

jakaskerl

luxonis/depthai-core1145

Thanks,
Jaka

silo

jakaskerl

Sorry for reviving a solved thread, but I am running into an issue with RGB depth alignment that I think may be due to the way I am undistorting the color image.

I would like the pipeline to output RGB and disparity frames that are aligned. I am trying to do this within the StereoDepth node since the documentation states it has better performance than the ImageManip node. But when I try to use stereo.setDepthAlign() I get an error for incompatible function arguments. Is there a simple way to accomplish the RGB depth alignment within the StereoDepth node while still undistorting the RGB image?

I attached the pipeline I'm trying to use for this.

Apologies if this is something simple that I overlooked but I just can't seem to get it working.


import depthai as dai
import numpy as np
import cv2
from datetime import timedelta


def main():
    # Create the pipeline
    pipeline = dai.Pipeline()

    # Create the nodes
    monoLeft = pipeline.create(dai.node.MonoCamera)
    monoRight = pipeline.create(dai.node.MonoCamera)
    color = pipeline.create(dai.node.Camera)
    stereo = pipeline.create(dai.node.StereoDepth)
    sync = pipeline.create(dai.node.Sync)
    xoutGrp = pipeline.create(dai.node.XLinkOut)

    # Set the properties for the color camera
    color.setBoardSocket(dai.CameraBoardSocket.CAM_A)
    color.setSize((1280, 800))
    color.setCalibrationAlpha(1.0)  # Prevents the crop when resizing
    color.setVideoSize((640, 400))  # Resize workaround due to bug

    # Set the properties for the mono cameras
    monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
    monoLeft.setCamera("left")
    monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
    monoRight.setCamera("right")

    # Set the properties for the stereo
    stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.DEFAULT)
    stereo.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_3x3)
    sub_pixel_active = True
    stereo.initialConfig.setSubpixel(sub_pixel_active)
    stereo.initialConfig.setSubpixelFractionalBits(3)
    stereo.initialConfig.setLeftRightCheck(True)
    stereo.initialConfig.setConfidenceThreshold(230)
    stereo.initialConfig.setDepthAlign(dai.CameraBoardSocket.CAM_A)

    # Set the properties for the sync
    sync.setSyncThreshold(timedelta(milliseconds=30))
    sync.setSyncAttempts(-1)  # Default, ensures node only sends message if successfully synchronized

    # Set the properties for the xlinkout
    xoutGrp.setStreamName("xout")

    # Link the nodes
    monoLeft.out.link(stereo.left)
    monoRight.out.link(stereo.right)
    stereo.disparity.link(sync.inputs["disparity"])
    color.video.link(sync.inputs["video"])
    sync.out.link(xoutGrp.input)

    disparityMultiplier = 255.0 / stereo.initialConfig.getMaxDisparity()
    with dai.Device(pipeline) as device:
        queue = device.getOutputQueue("xout", 1, False)
        while True:
            msgGrp = queue.get()
            for name, msg in msgGrp:
                frame = msg.getCvFrame()
                if name == "disparity":
                    frame = (frame * disparityMultiplier).astype(np.uint8)
                    frame = cv2.applyColorMap(frame, cv2.COLORMAP_JET)
                cv2.imshow(name, frame)
            if cv2.waitKey(1) == ord("q"):
                break



if __name__ == '__main__':
    main()

jakaskerl

silo But when I try to use stereo.setDepthAlign() I get an error for incompatible function arguments.

Can you paste that error? Looks like it could be syntactical.

Thanks,
Jaka

silo

jakaskerl

This is the error I'm getting.

Traceback (most recent call last):
  File "C:\**************************\OAKDProW\sync_testing_modified_example.py", line 74, in <module>
    main()
  File "C:\**************************\OAKDProW\sync_testing_modified_example.py", line 41, in main
    stereo.initialConfig.setDepthAlign(dai.CameraBoardSocket.CAM_A)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: setDepthAlign(): incompatible function arguments. The following argument types are supported:
    1. (self: depthai.StereoDepthConfig, align: depthai.RawStereoDepthConfig.AlgorithmControl.DepthAlign) -> depthai.StereoDepthConfig

Invoked with: <depthai.StereoDepthConfig object at 0x000002B5A637E2B0>, <CameraBoardSocket.CAM_A: 0>

silo

I was able to do a little bit more troubleshooting today and found a way around the error by using RGB instead like this:

stereo.setDepthAlign(dai.CameraBoardSocket.RGB)

This will allow the code (attached below) to run but it has the following issues:

Now I get a deprecation warning saying to use CAM_A instead of RGB, but at least it runs.
I have had to add a stereo.setOutputSize() command otherwise the disparity images are coming out as 1280x800 instead of the desired 640x400 that I set for the rgb images.
The rgb and disparity images are still not aligned perfectly. Just doing some quick measuring it seems like the rgb image has about 10 or so degrees more horizontal FOV, and a corresponding amount in the vertical direction, when compared to the disparity image.
In the disparity image there is a vertical band of missing values that starts at the left side and is about 20 pixels wide.


import depthai as dai
import numpy as np
import cv2
from datetime import timedelta


def main():
    # Create the pipeline
    pipeline = dai.Pipeline()
    device = dai.Device()

    # Create the nodes
    monoLeft = pipeline.create(dai.node.MonoCamera)
    monoRight = pipeline.create(dai.node.MonoCamera)
    color = pipeline.create(dai.node.Camera)
    stereo = pipeline.create(dai.node.StereoDepth)
    sync = pipeline.create(dai.node.Sync)
    xoutGrp = pipeline.create(dai.node.XLinkOut)

    # Set the properties for the color camera
    color.setBoardSocket(dai.CameraBoardSocket.CAM_A)
    color.setSize((1280, 800))
    color.setCalibrationAlpha(1.0)  # Prevents the crop when resizing
    color.setVideoSize((640, 400))  # Resize workaround due to bug

    # Set the properties for the mono cameras
    monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
    monoLeft.setCamera("left")
    monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
    monoRight.setCamera("right")

    # Set the properties for the stereo
    stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.DEFAULT)
    stereo.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_3x3)
    # sub_pixel_active = True
    # stereo.setSubpixel(sub_pixel_active)
    stereo.initialConfig.setSubpixelFractionalBits(3)
    stereo.initialConfig.setLeftRightCheck(True)
    stereo.initialConfig.setConfidenceThreshold(230)
    stereo.setDepthAlign(dai.CameraBoardSocket.RGB)
    stereo.setOutputSize(640, 400)

    # Set the properties for the sync
    sync.setSyncThreshold(timedelta(milliseconds=30))
    sync.setSyncAttempts(-1)  # Default, ensures node only sends message if successfully synchronized

    # Set the properties for the xlinkout
    xoutGrp.setStreamName("xout")

    # Link the nodes
    monoLeft.out.link(stereo.left)
    monoRight.out.link(stereo.right)
    stereo.disparity.link(sync.inputs["disparity"])
    color.video.link(sync.inputs["video"])
    sync.out.link(xoutGrp.input)

    disparityMultiplier = 255.0 / stereo.initialConfig.getMaxDisparity()
    with device:
        device.startPipeline(pipeline)
        queue = device.getOutputQueue("xout", 1, False)
        while True:
            msgGrp = queue.get()
            for name, msg in msgGrp:
                frame = msg.getCvFrame()
                if name == "disparity":
                    frame = (frame * disparityMultiplier).astype(np.uint8)
                    frame = cv2.applyColorMap(frame, cv2.COLORMAP_JET)
                cv2.imshow(name, frame)
            if cv2.waitKey(1) == ord("q"):
                break



if __name__ == '__main__':
    main()

jakaskerl

silo
Use stereo.setDepthAlign(rgbCamSocket) like in the example. Looks like initialConfig.setDepthAlign doesn't accept CAM_A.

silo I have had to add a stereo.setOutputSize() command otherwise the disparity images are coming out as 1280x800 instead of the desired 640x400 that I set for the rgb images.

That is expected since the alignment upscales depth to RGB size (and introduces a lot of latency). Use decimationFactor =2.

silo The rgb and disparity images are still not aligned perfectly. Just doing some quick measuring it seems like the rgb image has about 10 or so degrees more horizontal FOV, and a corresponding amount in the vertical direction, when compared to the disparity image.

They can't be aligned perfectly since mono and RGB sensors don't have the same FOV. Either some FOV is lost or there are missing RGB pixels. Anyhow, the images in the middle (where both depth and RGB are present) should be aligned.

Thanks,
Jaka

silo

jakaskerl

I found another example of rgb depth alignment on the github that seems to work so I'll start a new thread using that.