Warp node vs. setWarpMesh

annaelise

Is there a known quality difference between v2 ImageManip.setWarpMesh() and v3 Warp node, or if is there a recommended approach for high-quality undistortion in v3? I'm using the new Warp node in my v3 pipeline and the images are noticeably less crisp than the setWarpMesh from v2, so I'm wondering what might be causing this.

get_mesh() function used by both:

def get_mesh(
        self, calibData: dai.CalibrationHandler, ispSize: Tuple[int, int]
    ) -> Tuple:
        """Get mesh to correct distortion.

        Adapted from:
        https://github.com/luxonis/depthai-python/blob/
        develop/examples/ColorCamera/rgb_undistort.py

        Args:
            calibData: the calibration data from the camera
            ispSize: isp resolution

        Returns:
            Tuple containing the mesh, mesh width, and mesh height.
        """
        camSocket = dai.CameraBoardSocket.RGB
        M1 = np.array(calibData.getCameraIntrinsics(camSocket, ispSize[0], ispSize[1]))
        d1 = np.array(calibData.getDistortionCoefficients(camSocket))
        R1 = np.identity(3)
        mapX, mapY = cv2.initUndistortRectifyMap(M1, d1, R1, M1, ispSize, cv2.CV_32FC1)

        meshCellSize = 16
        mesh0 = []
        # Creates subsampled mesh which will be loaded on
        # the device to undistort the image
        for y in range(mapX.shape[0] + 1):  # iterating over height of the image
            if y % meshCellSize == 0:
                rowLeft = []
                for x in range(mapX.shape[1]):  # iterating over width of the image
                    if x % meshCellSize == 0:
                        if y == mapX.shape[0] and x == mapX.shape[1]:
                            rowLeft.append(mapX[y - 1, x - 1])
                            rowLeft.append(mapY[y - 1, x - 1])
                        elif y == mapX.shape[0]:
                            rowLeft.append(mapX[y - 1, x])
                            rowLeft.append(mapY[y - 1, x])
                        elif x == mapX.shape[1]:
                            rowLeft.append(mapX[y, x - 1])
                            rowLeft.append(mapY[y, x - 1])
                        else:
                            rowLeft.append(mapX[y, x])
                            rowLeft.append(mapY[y, x])
                if (mapX.shape[1] % meshCellSize) % 2 != 0:
                    rowLeft.append(0)
                    rowLeft.append(0)

                mesh0.append(rowLeft)

        mesh0 = np.array(mesh0)
        meshWidth = mesh0.shape[1] // 2
        meshHeight = mesh0.shape[0]
        mesh0.resize(meshWidth * meshHeight, 2)

        mesh = list(map(tuple, mesh0))

Warp node:

# Request RGB output in RGB888 format for warping
            rgb_output_rgb888 = rgb_camera.requestOutput(
                size=tuple(self.rgb.resolution),
                type=dai.ImgFrame.Type.RGB888p,
                fps=self.rgb.fps,
            )

            warp_node = self.pipeline.create(dai.node.Warp)
            warp_node.setOutputSize(tuple(self.rgb.resolution))
            max_frame_size = self.rgb.resolution[0] * self.rgb.resolution[1] * 3
            warp_node.setMaxOutputFrameSize(max_frame_size)
            warp_node.setInterpolation(dai.Interpolation.BILINEAR)

            # Convert warped output to NV12 for encoder
            manip_post_warp = self.pipeline.create(dai.node.ImageManip)
            manip_post_warp.initialConfig.setFrameType(dai.ImgFrame.Type.NV12)
            manip_post_warp.setMaxOutputFrameSize(
                self.rgb.resolution[0] * self.rgb.resolution[1] * 3 // 2
            )

            rgb_output_rgb888.link(warp_node.inputImage)
            warp_node.out.link(manip_post_warp.inputImage)
            manip_post_warp.out.link(encoder.input)

setWarpMesh:

# Create ImageManip node to correct distortion
            manip = self.pipeline.createImageManip()
            mesh, meshWidth, meshHeight = self.get_mesh(
                self.device.readCalibration(), rgb_camera.getIspSize()
            )
            manip.setWarpMesh(mesh, meshWidth, meshHeight)
            manip.setMaxOutputFrameSize(
                rgb_camera.getIspWidth() * rgb_camera.getIspHeight() * 3 // 2,
            )

            rgb_camera.video.link(manip.inputImage)
            manip.out.link(encoder.input)
            encoder.bitstream.link(rgb_out.input)

OskarSonc

Hey annaelise - on RVC2, Warp uses the same hardware block as ImageManip.setWarpMesh(), so there shouldn’t be an inherent quality drop. If the output looks softer, it’s almost always due to pipeline details. Few things that could cause it: the mesh was generated for the ISP size, but the Warp input is a resized/cropped stream, the interpolation is set to BILINEAR instead of BICUBIC, or the v3 path introduces extra format conversions compared to the v2 path.

If you only need undistortion, the recommended v3 approach is Camera.requestOutput(..., enableUndistortion=True)
See example here

If you’re still seeing worse results, can you share a MRE for both the v2 and v3 pipelines so I can try it on my side?

Thanks,
Oskar

annaelise

One thing I noticed when using enableUndistortion was that there was quite a bit of fish-eye effect to the images that wasn't there before, which is why I'm trying the Warp node now. I will try to create the MRE but in the meantime here is more of the pipeline so you can see how everything is set up:

v3:

def start(self) -> bool:
        self.pipeline = dai.Pipeline()
        
        camera_exposure_us = self.rgb.config.get("camera_exposure_us", 1000)
        camera_iso_sensitivity = self.rgb.config.get("camera_iso_sensitivity", 500)

        if not self.rgb:
            logging.error(
                f"OakCameraDevice[{self.camera_index}]"
                " requires an RGB camera configuration"
            )
            return False

        rgb_camera = self.pipeline.create(dai.node.Camera)
        rgb_camera.initialControl.setManualExposure(
            camera_exposure_us, camera_iso_sensitivity
        )
        rgb_camera.initialControl.setSharpness(0)
        rgb_camera.initialControl.setLumaDenoise(0)
        rgb_camera.initialControl.setChromaDenoise(0)
        rgb_camera.build(dai.CameraBoardSocket.RGB)

        # RGB video encoding node
        encoder = self.pipeline.create(dai.node.VideoEncoder)
        encoder.setDefaultProfilePreset(1, dai.VideoEncoderProperties.Profile.MJPEG)

        if self.is_wide:
            # Request RGB output in RGB888 format for warping
            rgb_output_rgb888 = rgb_camera.requestOutput(
                size=tuple(self.rgb.resolution),
                type=dai.ImgFrame.Type.RGB888p,
                fps=self.rgb.fps,
            )

            warp_node = self.pipeline.create(dai.node.Warp)
            warp_node.setOutputSize(tuple(self.rgb.resolution))
            max_frame_size = self.rgb.resolution[0] * self.rgb.resolution[1] * 3
            warp_node.setMaxOutputFrameSize(max_frame_size)
            warp_node.setInterpolation(dai.Interpolation.BILINEAR)

            # Convert warped output to NV12 for encoder
            manip_post_warp = self.pipeline.create(dai.node.ImageManip)
            manip_post_warp.initialConfig.setFrameType(dai.ImgFrame.Type.NV12)
            manip_post_warp.setMaxOutputFrameSize(
                self.rgb.resolution[0] * self.rgb.resolution[1] * 3 // 2
            )

            rgb_output_rgb888.link(warp_node.inputImage)
            warp_node.out.link(manip_post_warp.inputImage)
            manip_post_warp.out.link(encoder.input)
        else:
            # Request RGB output in NV12 format for encoder
            rgb_output_nv12 = rgb_camera.requestOutput(
                size=tuple(self.rgb.resolution),
                type=dai.ImgFrame.Type.NV12,
                fps=self.rgb.fps,
            )
            rgb_output_nv12.link(encoder.input)

        # If depth is enabled, add a stereo depth node to the pipeline
        if self.depth:
            brightness = self.depth.config.get("brightness", 0)
            contrast = self.depth.config.get("contrast", 0)
            camera_exposure_us = self.depth.config.get("camera_exposure_us")
            camera_iso_sensitivity = self.depth.config.get("camera_iso_sensitivity")

            spatial_filter_hole_filling_radius = self.depth.config.get(
                "spatial_filter_hole_filling_radius", 3
            )
            spatial_filter_num_iterations = self.depth.config.get(
                "spatial_filter_num_iterations", 3
            )

            if self.depth.resolution in self.OAK_DEPTH_RESOLUTIONS:
                depth_camera_resolution = self.OAK_DEPTH_RESOLUTIONS[
                    self.depth.resolution
                ]
            else:
                logging.warn(
                    f"Unsupported depth resolution {self.depth.resolution}."
                    " Defaulting to 400p."
                )
                depth_camera_resolution = (
                    dai.MonoCameraProperties.SensorResolution.THE_400_P
                )

            white_balance = self.depth.config.get("white_balance")

            # Create left/right cameras using Camera node
            left_camera = self.pipeline.create(dai.node.Camera)
            left_camera.initialControl.setBrightness(brightness)
            left_camera.initialControl.setContrast(contrast)
            if camera_exposure_us and camera_iso_sensitivity:
                left_camera.initialControl.setManualExposure(
                    camera_exposure_us, camera_iso_sensitivity
                )
            if white_balance:
                left_camera.initialControl.setManualWhiteBalance(white_balance)

            # Build with LEFT socket
            left_camera.build(dai.CameraBoardSocket.LEFT)

            # Request output at the depth resolution
            left_output = left_camera.requestOutput(
                size=self.depth.resolution,
                type=dai.ImgFrame.Type.GRAY8,
                fps=self.depth.fps,
            )

            right_camera = self.pipeline.create(dai.node.Camera)
            right_camera.initialControl.setBrightness(brightness)
            right_camera.initialControl.setContrast(contrast)
            if camera_exposure_us and camera_iso_sensitivity:
                right_camera.initialControl.setManualExposure(
                    camera_exposure_us, camera_iso_sensitivity
                )
            if white_balance:
                right_camera.initialControl.setManualWhiteBalance(white_balance)

            # Build with RIGHT socket
            right_camera.build(dai.CameraBoardSocket.RIGHT)

            # Request output at the depth resolution
            right_output = right_camera.requestOutput(
                size=self.depth.resolution,
                type=dai.ImgFrame.Type.GRAY8,
                fps=self.depth.fps,
            )

            # Create a Stereo node that will produce the depth map
            stereo = self.pipeline.create(dai.node.StereoDepth)
            disparity_shift = self.depth.config.get("disparity_shift", 0)
            stereo.initialConfig.setDisparityShift(disparity_shift)

            median_filter_str = self.depth.config.get("median_filter", "KERNEL_7x7")
            if median_filter_str == "KERNEL_3x3":
                median_filter = dai.MedianFilter.KERNEL_3x3
            elif median_filter_str == "KERNEL_5x5":
                median_filter = dai.MedianFilter.KERNEL_5x5
            elif median_filter_str == "KERNEL_7x7":
                median_filter = dai.MedianFilter.KERNEL_7x7
            else:
                median_filter = dai.MedianFilter.MEDIAN_OFF
            stereo.initialConfig.setMedianFilter(median_filter)

            left_right_check = self.depth.config.get("left_right_check", True)
            stereo.setLeftRightCheck(left_right_check)

            extended_disparity = self.depth.config.get("extended_disparity", True)
            stereo.setExtendedDisparity(extended_disparity)

            subpixel = self.depth.config.get("subpixel", False)
            stereo.setSubpixel(subpixel)

            stereo.setDepthAlign(dai.CameraBoardSocket.RGB)

            # Set output size to match RGB resolution (must be multiple of 16)
            # Round down to nearest multiple of 16 if needed
            depth_width = (self.rgb.resolution[0] // 16) * 16
            depth_height = (self.rgb.resolution[1] // 16) * 16
            stereo.setOutputSize(depth_width, depth_height)

            # Log warning if RGB resolution is not a multiple of 16
            if (
                depth_width != self.rgb.resolution[0]
                or depth_height != self.rgb.resolution[1]
            ):
                logging.warning(
                    f"RGB resolution {self.rgb.resolution} is not a multiple of 16. "
                    f"Depth output will be {depth_width}x{depth_height}. "
                    f"This may cause pixel-to-meter calibration mismatches. "
                    f"Consider using a resolution that's a multiple of 16 "
                    f"(e.g., 1280x720, 1920x1080)."
                )

            # Set depth post-processing

            # Spatial Edge-Preserving Filter will fill invalid depth pixels with
            # valid neighboring depth pixels. It performs a series of 1D
            # horizontal and vertical passes or iterations, to enhance the
            # smoothness of the reconstructed data.
            spatial_filter = self.depth.config.get("spatial_filter", {})
            if spatial_filter:
                stereo.initialConfig.postProcessing.spatialFilter.enable = True

                # An in-place heuristic symmetric hole-filling mode applied
                # horizontally during the filter passes. Intended to rectify
                # minor artifacts with minimal performance impact. Search radius
                # for hole filling.
                spatial_filter_hole_filling_radius = self.depth.config[
                    "spatial_filter"
                ].get("spatial_filter_hole_filling_radius", 3)
                stereo.initialConfig.postProcessing.spatialFilter.holeFillingRadius = (
                    spatial_filter_hole_filling_radius
                )
                # Number of iterations over the image in both horizontal and
                # vertical direction.
                spatial_filter_num_iterations = self.depth.config["spatial_filter"].get(
                    "spatial_filter_num_iterations", 3
                )
                stereo.initialConfig.postProcessing.spatialFilter.numIterations = (
                    spatial_filter_num_iterations
                )

            # Minimum pixel brightness. If input pixel is less or equal
            # than this value the depth value is invalidated.
            brightness_filter_min = self.depth.config.get(
                "brightness_filter_min_brightness", 0
            )
            stereo.initialConfig.postProcessing.brightnessFilter.minBrightness = (
                brightness_filter_min
            )
            # Maximum range in depth units. If input pixel is less or equal
            # than this value the depth value is invalidated.
            brightness_filter_max = self.depth.config.get(
                "brightness_filter_max_brightness", 255
            )
            stereo.initialConfig.postProcessing.brightnessFilter.maxBrightness = (
                brightness_filter_max
            )

            # Threshold Filter filters out all disparity/depth pixels outside
            # the configured min/max threshold values.
            threshold_filter_min = self.depth.config.get(
                "threshold_filter_min_range",
                200,  # Defaults to 200mm
            )
            threshold_filter_max = self.depth.config.get(
                "threshold_filter_max_range",
                2000,  # Defaults to 2m
            )
            stereo.initialConfig.postProcessing.thresholdFilter.minRange = (
                threshold_filter_min
            )
            stereo.initialConfig.postProcessing.thresholdFilter.maxRange = (
                threshold_filter_max
            )

            # Speckle Filter is used to reduce the speckle noise. Speckle noise
            # is a region with huge variance between neighboring disparity/depth
            # pixels, and speckle filter tries to filter this region.
            speckle_filter = self.depth.config.get("speckle_filter", {})
            if speckle_filter:
                stereo.initialConfig.postProcessing.speckleFilter.enable = True
                speckle_range = self.depth.config["speckle_filter"].get(
                    "speckle_range", 4
                )
                stereo.initialConfig.postProcessing.speckleFilter.speckleRange = (
                    speckle_range
                )

            # Decimation Factor will sub-samples the depth map, which means it
            # reduces the depth scene complexity and allows other filters to run
            # faster.
            decimation_filter = self.depth.config.get("decimation_filter", {})
            if decimation_filter:
                decimation_factor = decimation_filter.get("decimation_factor", 4)
                stereo.initialConfig.postProcessing.decimationFilter.decimationFactor = decimation_factor
            left_output.link(stereo.left)
            right_output.link(stereo.right)

        # Create output queues directly from nodes
        self.rgb_queue = encoder.bitstream.createOutputQueue(maxSize=1, blocking=False)
        self.rgb_queue.addCallback(self.__add_rgb_packet)

        # If depth is enabled, create Depth Queue and callbacks for new frames
        if self.depth:
            self.depth_queue = stereo.depth.createOutputQueue(maxSize=1, blocking=False)
            self.depth_queue.addCallback(self.__add_depth_packet)

        # Start the pipeline
        self.pipeline.start()

        # Handle wide angle mesh after pipeline is started
        if self.is_wide:
            device = self.pipeline.getDefaultDevice()
            # For v3 Camera node, we need to get ISP size differently
            # Since we requested a specific output size, we use that
            isp_size = tuple(self.rgb.resolution)
            mesh, meshWidth, meshHeight = self.get_mesh(
                device.readCalibration(), isp_size
            )
            warp_node.setWarpMesh(mesh, meshWidth, meshHeight)

        if self.depth:
            ir_laser_dot_projector_brightness = self.depth.config.get(
                "ir_laser_dot_projector_brightness", 700
            )
            ir_flood_light_brightness = self.depth.config.get(
                "ir_flood_light_brightness", 700
            )
            # Note: v3 API expects intensity values in range [0.0, 1.0], but config
            # values are in range [0, 1500]. Division by 1500.0 normalizes the config
            # values.
            self.pipeline.getDefaultDevice().setIrLaserDotProjectorIntensity(
                ir_laser_dot_projector_brightness / 1500.0
            )
            self.pipeline.getDefaultDevice().setIrFloodLightIntensity(
                ir_flood_light_brightness / 1500.0
            )

        device = self.pipeline.getDefaultDevice()
        device_info = device.getDeviceInfo()
        camera_features = device.getConnectedCameraFeatures()

        return True

v2:

def start(self) -> bool:

        self.device = dai.Device(OAK_FIXED_IP)
        self.pipeline = dai.Pipeline()
        self.pipeline.setXLinkChunkSize(0)  # Disable XLink chunking to reduce latency

        # Get relevant RGB configurations
        camera_exposure_us = self.rgb.config.get("camera_exposure_us", 1000)
        camera_iso_sensitivity = self.rgb.config.get("camera_iso_sensitivity", 500)

        if not self.rgb:
            logging.error(
                f"OakCameraDevice[{self.camera_index}]"
                " requires an RGB camera configuration"
            )
            return False

        # RGB output node for clients
        rgb_out = self.pipeline.create(dai.node.XLinkOut)
        rgb_out.setStreamName("rgb")

        # Create color camera
        rgb_camera = self.pipeline.create(dai.node.ColorCamera)
        rgb_camera.setInterleaved(False)
        rgb_camera.setBoardSocket(dai.CameraBoardSocket.RGB)
        rgb_camera.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K)
        rgb_camera.setFps(self.rgb.fps)
        rgb_camera.initialControl.setManualExposure(
            camera_exposure_us, camera_iso_sensitivity
        )
        rgb_camera.initialControl.setSharpness(0)
        rgb_camera.initialControl.setLumaDenoise(0)
        rgb_camera.initialControl.setChromaDenoise(0)

        # Resize 4k to a lower resolution on the ISP if needed
        if tuple(self.rgb.resolution) == (1920, 1080):
            rgb_camera.setIspScale(1, 2)
        elif tuple(self.rgb.resolution) == (1280, 720):
            rgb_camera.setIspScale(1, 3)

        # RGB video encoding node
        encoder = self.pipeline.create(dai.node.VideoEncoder)
        encoder.setDefaultProfilePreset(1, dai.VideoEncoderProperties.Profile.MJPEG)

        if self.is_wide:
            # Create ImageManip node to correct distortion
            manip = self.pipeline.createImageManip()
            mesh, meshWidth, meshHeight = self.get_mesh(
                self.device.readCalibration(), rgb_camera.getIspSize()
            )
            manip.setWarpMesh(mesh, meshWidth, meshHeight)
            manip.setMaxOutputFrameSize(
                rgb_camera.getIspWidth() * rgb_camera.getIspHeight() * 3 // 2,
            )

            rgb_camera.video.link(manip.inputImage)
            manip.out.link(encoder.input)
            encoder.bitstream.link(rgb_out.input)

        else:
            rgb_camera.video.link(encoder.input)
            encoder.bitstream.link(rgb_out.input)

        # If depth is enabled, add a stereo depth node to the pipeline
        if self.depth:
            depth_out = self.pipeline.create(dai.node.XLinkOut)
            depth_out.setStreamName("depth")

            brightness = self.depth.config.get("brightness", 0)
            contrast = self.depth.config.get("contrast", 0)
            camera_exposure_us = self.depth.config.get("camera_exposure_us")
            camera_iso_sensitivity = self.depth.config.get("camera_iso_sensitivity")

            spatial_filter_hole_filling_radius = self.depth.config.get(
                "spatial_filter_hole_filling_radius", 3
            )
            spatial_filter_num_iterations = self.depth.config.get(
                "spatial_filter_num_iterations", 3
            )

            if self.depth.resolution in self.OAK_DEPTH_RESOLUTIONS:
                depth_camera_resolution = self.OAK_DEPTH_RESOLUTIONS[
                    self.depth.resolution
                ]
            else:
                logging.warn(
                    f"Unsupported depth resolution {self.depth.resolution}."
                    " Defaulting to 400p."
                )
                depth_camera_resolution = (
                    dai.MonoCameraProperties.SensorResolution.THE_400_P
                )

            white_balance = self.depth.config.get("white_balance")

            # Create left/right mono cameras for Stereo depth
            left_camera = self.pipeline.create(dai.node.MonoCamera)
            left_camera.setResolution(depth_camera_resolution)
            left_camera.setCamera("left")
            left_camera.setFps(self.depth.fps)
            if camera_exposure_us and camera_iso_sensitivity:
                left_camera.initialControl.setManualExposure(
                    camera_exposure_us, camera_iso_sensitivity
                )
            left_camera.initialControl.setBrightness(brightness)
            left_camera.initialControl.setContrast(contrast)
            if white_balance:
                left_camera.initialControl.setManualWhiteBalance(white_balance)

            right_camera = self.pipeline.create(dai.node.MonoCamera)
            right_camera.setResolution(depth_camera_resolution)
            right_camera.setCamera("right")
            right_camera.setFps(self.depth.fps)
            if camera_exposure_us and camera_iso_sensitivity:
                right_camera.initialControl.setManualExposure(
                    camera_exposure_us, camera_iso_sensitivity
                )
            right_camera.initialControl.setBrightness(brightness)
            right_camera.initialControl.setContrast(contrast)
            if white_balance:
                right_camera.initialControl.setManualWhiteBalance(white_balance)

            # Create a Stereo node that will produce the depth map
            stereo = self.pipeline.create(dai.node.StereoDepth)
            disparity_shift = self.depth.config.get("disparity_shift", 0)
            stereo.initialConfig.setDisparityShift(disparity_shift)
            depth_preset_mode = dai.node.StereoDepth.PresetMode.HIGH_DENSITY
            stereo.setDefaultProfilePreset(depth_preset_mode)

            # This is a non-edge preserving Median filter, which can be used to
            # reduce noise and smoothen the depth map. Median filter is
            # implemented in hardware, so it’s the fastest filter.
            median_filter_str = self.depth.config.get("median_filter", "KERNEL_7x7")
            if median_filter_str == "KERNEL_3x3":
                median_filter = dai.MedianFilter.KERNEL_3x3
            elif median_filter_str == "KERNEL_5x5":
                median_filter = dai.MedianFilter.KERNEL_5x5
            elif median_filter_str == "KERNEL_7x7":
                median_filter = dai.MedianFilter.KERNEL_7x7
            else:
                median_filter = dai.MedianFilter.MEDIAN_OFF
            stereo.initialConfig.setMedianFilter(median_filter)

            # Computes and combines disparities in both L-R and R-L directions,
            # and combine them. For better occlusion handling, discarding
            # invalid disparity values
            left_right_check = self.depth.config.get("left_right_check", True)
            stereo.setLeftRightCheck(left_right_check)

            # Disparity range increased from 0-95 to 0-190, combined from full
            # resolution and downscaled images. Suitable for short range
            # objects. We always set it to True.
            extended_disparity = self.depth.config.get("extended_disparity", True)
            stereo.setExtendedDisparity(extended_disparity)

            # Subpixel mode improves the precision and is especially useful for
            # long-range measurements. We always set it to False.
            subpixel = self.depth.config.get("subpixel", False)
            stereo.setSubpixel(subpixel)

            # Align the depth map to the RGB image
            stereo.setDepthAlign(dai.CameraBoardSocket.RGB)

            # Set depth post-processing
            config = stereo.initialConfig.get()

            # Spatial Edge-Preserving Filter will fill invalid depth pixels with
            # valid neighboring depth pixels. It performs a series of 1D
            # horizontal and vertical passes or iterations, to enhance the
            # smoothness of the reconstructed data.
            spatial_filter = self.depth.config.get("spatial_filter", {})
            if spatial_filter:
                config.postProcessing.spatialFilter.enable = True

                # An in-place heuristic symmetric hole-filling mode applied
                # horizontally during the filter passes. Intended to rectify
                # minor artifacts with minimal performance impact. Search radius
                # for hole filling.
                spatial_filter_hole_filling_radius = self.depth.config[
                    "spatial_filter"
                ].get("spatial_filter_hole_filling_radius", 3)
                config.postProcessing.spatialFilter.holeFillingRadius = (
                    spatial_filter_hole_filling_radius
                )
                # Number of iterations over the image in both horizontal and
                # vertical direction.
                spatial_filter_num_iterations = self.depth.config["spatial_filter"].get(
                    "spatial_filter_num_iterations", 3
                )
                config.postProcessing.spatialFilter.numIterations = (
                    spatial_filter_num_iterations
                )

            # Minimum pixel brightness. If input pixel is less or equal
            # than this value the depth value is invalidated.
            brightness_filter_min = self.depth.config.get(
                "brightness_filter_min_brightness", 0
            )
            config.postProcessing.brightnessFilter.minBrightness = brightness_filter_min
            # Maximum range in depth units. If input pixel is less or equal
            # than this value the depth value is invalidated.
            brightness_filter_max = self.depth.config.get(
                "brightness_filter_max_brightness", 255
            )
            config.postProcessing.brightnessFilter.maxBrightness = brightness_filter_max

            # Threshold Filter filters out all disparity/depth pixels outside
            # the configured min/max threshold values.
            threshold_filter_min = self.depth.config.get(
                "threshold_filter_min_range",
                200,  # Defaults to 200mm
            )
            threshold_filter_max = self.depth.config.get(
                "threshold_filter_max_range",
                2000,  # Defaults to 2m
            )
            config.postProcessing.thresholdFilter.minRange = threshold_filter_min
            config.postProcessing.thresholdFilter.maxRange = threshold_filter_max

            # Speckle Filter is used to reduce the speckle noise. Speckle noise
            # is a region with huge variance between neighboring disparity/depth
            # pixels, and speckle filter tries to filter this region.
            speckle_filter = self.depth.config.get("speckle_filter", {})
            if speckle_filter:
                config.postProcessing.speckleFilter.enable = True
                speckle_range = self.depth.config["speckle_filter"].get(
                    "speckle_range", 4
                )
                config.postProcessing.speckleFilter.speckleRange = speckle_range

            # Decimation Factor will sub-samples the depth map, which means it
            # reduces the depth scene complexity and allows other filters to run
            # faster.
            decimation_filter = self.depth.config.get("decimation_filter", {})
            if decimation_filter:
                decimation_factor = decimation_filter.get("decimation_factor", 4)
                config.postProcessing.decimationFilter.decimationFactor = (
                    decimation_factor
                )

            stereo.initialConfig.set(config)

            left_camera.out.link(stereo.left)
            right_camera.out.link(stereo.right)
            stereo.depth.link(depth_out.input)

            # Set the flood light and dot projector brightness [0-1500]
            ir_laser_dot_projector_brightness = self.depth.config.get(
                "ir_laser_dot_projector_brightness", 700
            )
            self.device.setIrLaserDotProjectorBrightness(
                ir_laser_dot_projector_brightness
            )
            ir_flood_light_brightness = self.depth.config.get(
                "ir_flood_light_brightness", 700
            )
            self.device.setIrFloodLightBrightness(ir_flood_light_brightness)

        self.device.startPipeline(self.pipeline)

        device_info = self.device.getDeviceInfo()

        camera_features = self.device.getConnectedCameraFeatures()

        # Create RGB Queue and callbacks for new frames
        self.rgb_queue = self.device.getOutputQueue(
            name="rgb", maxSize=1, blocking=False
        )
        self.rgb_queue.addCallback(self.__add_rgb_packet)

        # If depth is enabled, create Depth Queue and callbacks for new frames
        if self.depth:
            self.depth_queue = self.device.getOutputQueue(
                name="depth", maxSize=1, blocking=False
            )
            self.depth_queue.addCallback(self.__add_depth_packet)

        return True

OskarSonc

Hm annaelise I tested the MRE and I don't really see the difference between the v2 and v3. Could you provide example images? Also which camera do you have?

annaelise One thing I noticed when using enableUndistortion was that there was quite a bit of fish-eye effect to the images that wasn't there before, which is why I'm trying the Warp node now.

Hm the undistortion shouldn't have that effect. Did do you try the example, since I don't get any fisheye effect... You should use resizeMode=dai.ImgResizeMode.CROP, enableUndistortion=True and there should be a big difference between the normal and undistorted stream as can be seen in demo.

Thanks,
Oskar

annaelise

MREs:

v2:

#!/usr/bin/env python3
"""
MRE (Minimal Reproducible Example) - DepthAI v2 Warp Pipeline

This script demonstrates the v2 approach using ImageManip.setWarpMesh()
for undistortion on wide OAK cameras.

Requirements: depthai==2.28.0.0 (or similar v2 version), opencv-python, numpy

Usage:
    python mre_v2_warp.py
"""

import cv2
import depthai as dai
import numpy as np

OAK_FIXED_IP = "169.254.1.222"
RESOLUTION = (1920, 1080)
FPS = 30


def get_mesh(calibData, ispSize):
    """Get mesh to correct distortion.

    Adapted from:
    https://github.com/luxonis/depthai-python/blob/develop/examples/ColorCamera/rgb_undistort.py
    """
    camSocket = dai.CameraBoardSocket.RGB
    M1 = np.array(calibData.getCameraIntrinsics(camSocket, ispSize[0], ispSize[1]))
    d1 = np.array(calibData.getDistortionCoefficients(camSocket))
    R1 = np.identity(3)
    mapX, mapY = cv2.initUndistortRectifyMap(M1, d1, R1, M1, ispSize, cv2.CV_32FC1)

    meshCellSize = 16
    mesh0 = []
    for y in range(mapX.shape[0] + 1):
        if y % meshCellSize == 0:
            rowLeft = []
            for x in range(mapX.shape[1]):
                if x % meshCellSize == 0:
                    if y == mapX.shape[0] and x == mapX.shape[1]:
                        rowLeft.append(mapX[y - 1, x - 1])
                        rowLeft.append(mapY[y - 1, x - 1])
                    elif y == mapX.shape[0]:
                        rowLeft.append(mapX[y - 1, x])
                        rowLeft.append(mapY[y - 1, x])
                    elif x == mapX.shape[1]:
                        rowLeft.append(mapX[y, x - 1])
                        rowLeft.append(mapY[y, x - 1])
                    else:
                        rowLeft.append(mapX[y, x])
                        rowLeft.append(mapY[y, x])
            if (mapX.shape[1] % meshCellSize) % 2 != 0:
                rowLeft.append(0)
                rowLeft.append(0)
            mesh0.append(rowLeft)

    mesh0 = np.array(mesh0)
    meshWidth = mesh0.shape[1] // 2
    meshHeight = mesh0.shape[0]
    mesh0.resize(meshWidth * meshHeight, 2)
    mesh = list(map(tuple, mesh0))
    return mesh, meshWidth, meshHeight


def main():
    print(f"DepthAI version: {dai.__version__}")
    print("Creating v2 pipeline with ImageManip.setWarpMesh()...")

    # Create pipeline
    pipeline = dai.Pipeline()

    # Create color camera
    cam_rgb = pipeline.createColorCamera()
    cam_rgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
    cam_rgb.setVideoSize(RESOLUTION[0], RESOLUTION[1])
    cam_rgb.setFps(FPS)
    cam_rgb.setInterleaved(False)
    cam_rgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)

    # Camera controls - use faster manual exposure to reduce motion blur
    # Increase exposure time and ISO for brighter images while keeping motion sharp
    cam_rgb.initialControl.setManualExposure(5000, 1600)  # 5ms exposure, ISO 1600
    cam_rgb.initialControl.setSharpness(0)
    cam_rgb.initialControl.setLumaDenoise(0)
    cam_rgb.initialControl.setChromaDenoise(0)

    # Create ImageManip for warp
    manip = pipeline.createImageManip()
    manip.setMaxOutputFrameSize(RESOLUTION[0] * RESOLUTION[1] * 3)

    # Create encoder
    encoder = pipeline.createVideoEncoder()
    encoder.setDefaultProfilePreset(1, dai.VideoEncoderProperties.Profile.MJPEG)

    # Link: cam_rgb.video -> manip -> encoder
    cam_rgb.video.link(manip.inputImage)
    manip.out.link(encoder.input)

    # Create output
    xout = pipeline.createXLinkOut()
    xout.setStreamName("mjpeg")
    encoder.bitstream.link(xout.input)

    # Connect to device
    device_info = dai.DeviceInfo(OAK_FIXED_IP)
    print(f"Connecting to OAK camera at {OAK_FIXED_IP}...")

    with dai.Device(pipeline, device_info) as device:
        # Get calibration and set mesh
        calibData = device.readCalibration()
        mesh, meshWidth, meshHeight = get_mesh(calibData, RESOLUTION)
        manip.setWarpMesh(mesh, meshWidth, meshHeight)

        print("Pipeline started. Press 'q' to quit, 's' to save image.")

        q = device.getOutputQueue("mjpeg", maxSize=4, blocking=False)

        frame_count = 0
        while True:
            packet = q.tryGet()
            if packet is not None:
                frame = cv2.imdecode(packet.getData(), cv2.IMREAD_COLOR)
                if frame is not None:
                    frame_count += 1
                    cv2.imshow("v2 Warp (ImageManip.setWarpMesh)", frame)

                    if frame_count == 1:
                        cv2.imwrite("mre_v2_output.png", frame)
                        print("Saved first frame to mre_v2_output.png")

            key = cv2.waitKey(1)
            if key == ord("q"):
                break
            elif key == ord("s"):
                if frame is not None:
                    cv2.imwrite("mre_v2_output.png", frame)
                    print("Saved frame to mre_v2_output.png")

    cv2.destroyAllWindows()
    print("Done.")


if __name__ == "__main__":
    main()

v3:

#!/usr/bin/env python3
"""
MRE (Minimal Reproducible Example) - DepthAI v3 Warp Pipeline

This script demonstrates the v3 approach using the Warp node
for undistortion on wide OAK cameras.

Requirements: depthai>=3.0.0, opencv-python, numpy

Usage:
    python mre_v3_warp.py
"""

import cv2
import depthai as dai
import numpy as np

OAK_FIXED_IP = "169.254.1.222"
RESOLUTION = (1920, 1080)
FPS = 30


def get_mesh(calibData, ispSize):
    """Get mesh to correct distortion.

    Adapted from:
    https://github.com/luxonis/depthai-python/blob/develop/examples/ColorCamera/rgb_undistort.py
    """
    camSocket = dai.CameraBoardSocket.RGB
    M1 = np.array(calibData.getCameraIntrinsics(camSocket, ispSize[0], ispSize[1]))
    d1 = np.array(calibData.getDistortionCoefficients(camSocket))
    R1 = np.identity(3)
    mapX, mapY = cv2.initUndistortRectifyMap(M1, d1, R1, M1, ispSize, cv2.CV_32FC1)

    meshCellSize = 16
    mesh0 = []
    for y in range(mapX.shape[0] + 1):
        if y % meshCellSize == 0:
            rowLeft = []
            for x in range(mapX.shape[1]):
                if x % meshCellSize == 0:
                    if y == mapX.shape[0] and x == mapX.shape[1]:
                        rowLeft.append(mapX[y - 1, x - 1])
                        rowLeft.append(mapY[y - 1, x - 1])
                    elif y == mapX.shape[0]:
                        rowLeft.append(mapX[y - 1, x])
                        rowLeft.append(mapY[y - 1, x])
                    elif x == mapX.shape[1]:
                        rowLeft.append(mapX[y, x - 1])
                        rowLeft.append(mapY[y, x - 1])
                    else:
                        rowLeft.append(mapX[y, x])
                        rowLeft.append(mapY[y, x])
            if (mapX.shape[1] % meshCellSize) % 2 != 0:
                rowLeft.append(0)
                rowLeft.append(0)
            mesh0.append(rowLeft)

    mesh0 = np.array(mesh0)
    meshWidth = mesh0.shape[1] // 2
    meshHeight = mesh0.shape[0]
    mesh0.resize(meshWidth * meshHeight, 2)
    mesh = list(map(tuple, mesh0))
    return mesh, meshWidth, meshHeight


def main():
    print(f"DepthAI version: {dai.__version__}")
    print("Creating v3 pipeline with Warp node...")

    # Connect to device first to get calibration
    device_info = dai.DeviceInfo(OAK_FIXED_IP)
    print(f"Connecting to OAK camera at {OAK_FIXED_IP}...")

    device = dai.Device(device_info)

    # Get calibration data
    calibData = device.readCalibration()
    mesh, meshWidth, meshHeight = get_mesh(calibData, RESOLUTION)

    # Create pipeline
    pipeline = dai.Pipeline(device)

    # Create color camera using new Camera node
    rgb_camera = pipeline.create(dai.node.Camera)

    # Set camera controls before building - use faster manual exposure to reduce motion blur
    # Increase exposure time and ISO for brighter images while keeping motion sharp
    rgb_camera.initialControl.setManualExposure(5000, 1600)  # 5ms exposure, ISO 1600
    rgb_camera.initialControl.setSharpness(0)
    rgb_camera.initialControl.setLumaDenoise(0)
    rgb_camera.initialControl.setChromaDenoise(0)

    # Build the camera
    rgb_camera.build(dai.CameraBoardSocket.RGB)

    # Request RGB output in RGB888 format for warping
    rgb_output = rgb_camera.requestOutput(
        size=RESOLUTION,
        type=dai.ImgFrame.Type.RGB888p,
        fps=FPS,
    )

    # Create Warp node
    warp_node = pipeline.create(dai.node.Warp)
    warp_node.setOutputSize(RESOLUTION)
    max_frame_size = RESOLUTION[0] * RESOLUTION[1] * 3
    warp_node.setMaxOutputFrameSize(max_frame_size)
    warp_node.setInterpolation(dai.Interpolation.BILINEAR)
    warp_node.setWarpMesh(mesh, meshWidth, meshHeight)

    # Convert warped output to NV12 for encoder
    manip_post_warp = pipeline.create(dai.node.ImageManip)
    manip_post_warp.initialConfig.setFrameType(dai.ImgFrame.Type.NV12)
    manip_post_warp.setMaxOutputFrameSize(RESOLUTION[0] * RESOLUTION[1] * 3 // 2)

    # Create encoder
    encoder = pipeline.create(dai.node.VideoEncoder)
    encoder.setDefaultProfilePreset(1, dai.VideoEncoderProperties.Profile.MJPEG)

    # Link pipeline
    rgb_output.link(warp_node.inputImage)
    warp_node.out.link(manip_post_warp.inputImage)
    manip_post_warp.out.link(encoder.input)

    # Create output queue
    q = encoder.out.createOutputQueue(maxSize=4, blocking=False)

    # Start pipeline
    pipeline.start()
    print("Pipeline started. Press 'q' to quit, 's' to save image.")

    frame_count = 0
    frame = None
    while pipeline.isRunning():
        packet = q.tryGet()
        if packet is not None:
            frame = cv2.imdecode(packet.getData(), cv2.IMREAD_COLOR)
            if frame is not None:
                frame_count += 1
                cv2.imshow("v3 Warp (Warp node)", frame)

                if frame_count == 1:
                    cv2.imwrite("mre_v3_output.png", frame)
                    print("Saved first frame to mre_v3_output.png")

        key = cv2.waitKey(1)
        if key == ord("q"):
            break
        elif key == ord("s"):
            if frame is not None:
                cv2.imwrite("mre_v3_output.png", frame)
                print("Saved frame to mre_v3_output.png")

    pipeline.stop()
    cv2.destroyAllWindows()
    print("Done.")


if __name__ == "__main__":
    main()

annaelise

I can try the dai.ImgResizeMode.CROP and see if that fixes anything. The camera I have is a wide oak camera. One other issue I've noticed with the v3 code is that there always seems to be a strip of blackness on the right side of the image where in the v2 images there is not. I don't know if this has to do with some processing being done, but here is an example of the images:

{V2 image?}

and

{V3 image?}

where the v2 image is first and the v3 image is second. This is also after changing to use enableUndistortion and CROP image resize mode. I will attach another image of the v3 version with what they looked like running the code I posted previously.

Also I apologize for the darkness of the images. An external light went out and hasn't been able to be replaced. But things are more visible if the brightness of the images are turned up.

OskarSonc

annaelise with which script did you get the black strips? It's usually alignment/padding (warp/encoder need widths divisible by 16) or a resize/undistort aspect mismatch.

annaelise

I used the script I sent you.

I understand the /16 issue, but I'm curious about why that wouldn't be an issue in the v2 version.

OskarSonc

annaelise I am a bit confused now. What is the 3rd image you sent? Is that v3 or v2?
I tested the v3 script on OAK-D PRO W and I get 0 strips:

annaelise

OskarSonc The third image I sent was v3 but with the Warp node before I made the CROP and enableUndistortion changes

The camera I'm having these issues with is an older wide oak camera. Is there any chance there have been any hardware changes between older and newer versions of the wide oak that could be responsible for the discrepancies I'm seeing in the image?

OskarSonc

annaelise Oh I see, since I don't see any strips there. Can you share device model + MXID and script with which you got that strips? It's possible if you are on very old OAK with sensor that isn't supported well in daiv3.
Also, are you on latest depthaiv3?

annaelise

OskarSonc

This is what I have

DeviceInfo(mxid=14442C10F16AF2D600, X_LINK_BOOTED, X_LINK_TCP_IP, X_LINK_MYRIAD_X, X_LINK_SUCCESS). Features: [{socket: CAM_A, sensorName: IMX378, width: 4056, height: 3040, orientation: AUTO, supportedTypes: [COLOR], hasAutofocus: 0, hasAutofocusIC: 0, name: color}, {socket: CAM_B, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO], hasAutofocus: 0, hasAutofocusIC: 0, name: left}, {socket: CAM_C, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO], hasAutofocus: 0, hasAutofocusIC: 0, name: right}]

as for the depthai version, I have 3.2.1 (though this printout is from the v2.28 version because I reverted back for good quality images)

OskarSonc

annaelise the hardware you have shouldn't be a problem, since I tested the script on the same camera as you have... Also, you can update to the latest version of software (v3.3.0) and (v2.32.0).
If you run this slightly changed v3 undistortion example, you should see what undistortion is actually doing:

#!/usr/bin/env python3

import cv2
import depthai as dai

# Create pipeline
with dai.Pipeline() as pipeline:
    # Define source and output
    cam = pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_A)
    croppedQueue = cam.requestOutput((640,360), resizeMode=dai.ImgResizeMode.CROP, enableUndistortion=True).createOutputQueue()
    LetterBoxQueue = cam.requestOutput((640,360), resizeMode=dai.ImgResizeMode.LETTERBOX, enableUndistortion=True).createOutputQueue()

    # Connect to device and start pipeline
    pipeline.start()
    while pipeline.isRunning():
        croppedIn = croppedQueue.get()
        assert isinstance(croppedIn, dai.ImgFrame)
        cv2.imshow("cropped undistorted", croppedIn.getCvFrame())

        stretchedIn = LetterBoxQueue.get()
        assert isinstance(stretchedIn, dai.ImgFrame)
        cv2.imshow("stretched undistorted", stretchedIn.getCvFrame())

        if cv2.waitKey(1) == ord("q"):
            break

You should get something like this and then see what undistortion actually does and from where the black strip is coming from:

Also what is your use-case, since from the pictures you sent it doesn't look needed. With it you also lose a bit of FOV as you can see in the picture aboove...