Hi, I want to generate depth and align it with a still frame generated from the OAK-D Pro W. The issue is that I have an offset and can't understand why. I'm using the same code as for ISP frames, and the ISP frames are aligned. I referred to this example for generating 12MP depth alignment: [https://gist.github.com/Erol444/25f374fa18efa7939ec9bb848b39249a](https://gist.github.com/Erol444/25f374fa18efa7939ec9bb848b39249a) One thing I suspect might be a problem is the difference between the resolutions. The ISP frame is 4056x3040, and the still frame is 4032x3040. - ISP sample: [upl-image-preview url=https://discuss.luxonis.com/assets/files/2024-03-25/1711373985-120835-dai-0-depth.png] [upl-image-preview url=https://discuss.luxonis.com/assets/files/2024-03-25/1711373985-214770-dai-0-color.jpg] - Still sample: [upl-image-preview url=https://discuss.luxonis.com/assets/files/2024-03-25/1711374119-612431-dai-0-depth.png] [upl-image-preview url=https://discuss.luxonis.com/assets/files/2024-03-25/1711374119-711752-dai-0-color.jpg] - ISP Blended: [upl-image-preview url=https://discuss.luxonis.com/assets/files/2024-03-25/1711377525-993273-dai-0-colorb.jpg] - Still Blended: [upl-image-preview url=https://discuss.luxonis.com/assets/files/2024-03-25/1711377562-510712-dai-0-colorb.jpg] Elements where you can see the offset: 1. The cup 2. The wet wipes 3. The camera next to the wet wipes 4. The edge of the tables FIY: I undistort the ISP color frame (referred to this example): [https://docs.luxonis.com/projects/api/en/latest/samples/StereoDepth/stereo_depth_custom_mesh/](https://docs.luxonis.com/projects/api/en/latest/samples/StereoDepth/stereo_depth_custom_mesh/) **Any thoughts?**

Depth alignment with still frames

vkmNuller

Hi,

I want to generate depth and align it with a still frame generated from the OAK-D Pro W. The issue is that I have an offset and can't understand why.

I'm using the same code as for ISP frames, and the ISP frames are aligned. I referred to this example for generating 12MP depth alignment: https://gist.github.com/Erol444/25f374fa18efa7939ec9bb848b39249a

One thing I suspect might be a problem is the difference between the resolutions. The ISP frame is 4056x3040, and the still frame is 4032x3040.

ISP sample:
Still sample:
ISP Blended:
Still Blended:

Elements where you can see the offset:

The cup
The wet wipes
The camera next to the wet wipes
The edge of the tables

FIY: I undistort the ISP color frame (referred to this example):
https://docs.luxonis.com/projects/api/en/latest/samples/StereoDepth/stereo_depth_custom_mesh/

Any thoughts?

jakaskerl

Hi @vkmNuller
Are these scaled to match the still video size?

M1 = np.array(calibData.getCameraIntrinsics(dai.CameraBoardSocket.CAM_B, resolution['w'], resolution['h']))
    d1 = np.array(calibData.getDistortionCoefficients(dai.CameraBoardSocket.CAM_B))
    R1 = np.array(calibData.getStereoLeftRectificationRotation())
    M2 = np.array(calibData.getCameraIntrinsics(dai.CameraBoardSocket.CAM_C, resolution['w'], resolution['h']))
    d2 = np.array(calibData.getDistortionCoefficients(dai.CameraBoardSocket.CAM_C))
    R2 = np.array(calibData.getStereoRightRectificationRotation())
    mapXL, mapYL = cv2.initUndistortRectifyMap(M1, d1, R1, M2, (resolution['w'], resolution['h']), cv2.CV_32FC1)
    mapXR, mapYR = cv2.initUndistortRectifyMap(M2, d2, R2, M2, (resolution['w'], resolution['h']), cv2.CV_32FC1)

vkmNuller the still frame is 4032x3040.

Unless you are upscaling the still frame, it should be smaller than 4K, since it is derived from video output.

Thanks,
Jaka

GeorgiDunchev

Hi jakaskerl

jakaskerl Unless you are upscaling the still frame, it should be smaller than 4K, since it is derived from video output.

according to https://docs.luxonis.com/projects/api/en/latest/components/nodes/camera/#inputs-and-outputs:

still (when a capture is triggered) and isp work at the max camera resolution, while video and preview are limited to max 4K (3840 x 2160) resolution, which is cropped from isp

The way I understand it is that a still should contain the whole 4/3 area of the image sensor and video crops at 4K. Am I wrong?

We were not modifying the rectification of the mono cameras. We'll test it and get back with the results or if we have more questions.

I'm wondering why does the alignment work well with the isp as we are not modifying the mono camera processing for the isp as well. What does the in-build rectification resolution default to?

jakaskerl

Hi @GeorgiDunchev
Right. My thoughts is it defaults to sensor resolution set to stereo cameras. If using external rectification using a mesh that isn't the case.
Could you add the code you are using so I can try to repro? I fail to wrap my head around why the still resolution would be smaller.

Thanks,
Jaka

vkmNuller

Hi @jakaskerl ,

The code for getting still frames is pretty simple and i don't think we upscale the frame anywere:

        self.pipeline = dai.Pipeline()

        self.color     = self.pipeline.create(dai.node.Camera)
        self.monoLeft  = self.pipeline.create(dai.node.MonoCamera)
        self.monoRight = self.pipeline.create(dai.node.MonoCamera)

        self.xoutColor = self.pipeline.create(dai.node.XLinkOut)
        self.xoutLeft  = self.pipeline.create(dai.node.XLinkOut)
        self.xoutRight = self.pipeline.create(dai.node.XLinkOut)

        self.xoutColor.setStreamName("color")

        self.xoutLeft.setStreamName("left")
        self.xoutRight.setStreamName("right")

        colorCameraSocket = dai.CameraBoardSocket.CAM_A
        leftCameraSocket  = dai.CameraBoardSocket.CAM_B
        rightCameraSocket = dai.CameraBoardSocket.CAM_C

        self.color.setPreviewSize(1600, 1200)
        self.color.setVideoSize(3840, 2160)
        self.color.setSize(4056, 3040)
        self.color.setBoardSocket(colorCameraSocket)
        self.color.setFps(30)
        self.color.setMeshSource(dai.CameraProperties.WarpMeshSource.CALIBRATION)

        self.monoLeft.setBoardSocket(leftCameraSocket)
        self.monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)
        self.monoLeft.setCamera("left")
        self.monoLeft.setFps(30)

        self.monoRight.setBoardSocket(rightCameraSocket)
        self.monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)
        self.monoRight.setCamera("right")
        self.monoRight.setFps(30)

        self.color.still.link(self.xoutColor.input)

        self.monoLeft.out.link(self.xoutLeft.input)
        self.monoRight.out.link(self.xoutRight.input)

        self.xin = self.pipeline.create(dai.node.XLinkIn)
        self.xin.setStreamName("control")
        self.xin.out.link(self.color.inputControl)

        self.device.startPipeline(self.pipeline)

        ...

        self.colorQueue = self.device.getOutputQueue(name="color", maxSize=1, blocking=True)
        self.leftQueue  = self.device.getOutputQueue(name="left",  maxSize=1, blocking=False)
        self.rightQueue = self.device.getOutputQueue(name="right", maxSize=1, blocking=False)
        self.qControl = self.device.getInputQueue(name="control", maxSize=1, blocking=True)

        # Getting the frame (already in 4032x3040)
         colorFrame = self.colorQueue.tryGet()

Also how do i use mapXL, mapYL and mapXR, mapYR? I can not find a function that overrides the left/right x/y maps. I only found:

    def setStereoLeft(self, cameraId: CameraBoardSocket, rectifiedRotation: List[List[float]]) -> None: ...
    def setStereoRight(self, cameraId: CameraBoardSocket, rectifiedRotation: List[List[float]]) -> None: ...

jakaskerl

Hi @vkmNuller
I see you are actually using the Camera node instead of ColorCamera.

vkmNuller self.color.setMeshSource(dai.CameraProperties.WarpMeshSource.CALIBRATION)

This part ^^^ changes the resolution as it applies undistortion. AFAIK, the undistortion doesn't apply to ISP stream but does to all other streams.

vkmNuller

Hi @jakaskerl,

I found this issue in your GitHub repo: https://github.com/luxonis/depthai-core/issues/741

We're trying to achieve the same thing. We also found a solution to why we get 4032x3040. If you set self.color.setStillSize(4056, 3040), then you get a 4056x3040 resolution. For some reason, if you only use self.color.setSize(4056, 3040), you will get a 4032x3040 frame.

jakaskerl

Hi @vkmNuller
The camera node is weird in that way as it tries to abstract the size setting in order to optimize downscaling. 4032 is a multiple of 16 which allows for faster downscaling.

Does correcting the size fix the alignment issue?

Thanks,
Jaka