Stereo alignment error: 1. Stereo rectification error.

Jay-jay

I get this error and I do not know how to fix. I was searching for answers in the community, but have not got any luck yet.

[29.481] [StereoDepth(3)] [error] Stereo alignment error: 1, trying to recover. ] [192.168.200.9] [29.536] [StereoDepth(3)] [error] Stereo rectification error: 1, trying to recover. This error keeps coming

I have Oak-D S2 depth camera and I will share my code running.

If anything to know more, please make comments. Thank you in advance.

`class depth_cam:`

`    def __init__(self):`

`        # Create pipeline`

`        self.pipeline = dai.Pipeline()`

`        self.device = dai.Device()`

`        self.fps = 30`

`        # The disparity is computed at this resolution, then upscaled to RGB resolution`

`        #monoResolution = dai.MonoCameraProperties.SensorResolution.THE_720_P`

`        self.monoResolution = dai.MonoCameraProperties.SensorResolution.THE_800_P`

`        # Define sources and outputs`

`        self.camRgb = self.pipeline.create(dai.node.ColorCamera)`

`        self.left = self.pipeline.create(dai.node.MonoCamera)`

`        self.right = self.pipeline.create(dai.node.MonoCamera)`

`        self.stereo = self.pipeline.create(dai.node.StereoDepth)`

`        try:`

`            calibData = self.device.readCalibration2()`

`            lensPosition = calibData.getLensPosition(dai.CameraBoardSocket.CAM_A)`

`            if lensPosition:`

`                self.camRgb.initialControl.setManualFocus(lensPosition)`

`        except:`

`            raise`

`        #Properties`

`        self.camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)`

`        #camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)`

`        self.camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP) # 4056x3040`

`        self.camRgb.setFps(self.fps)`

`        self.left.setResolution(self.monoResolution)`

`        self.left.setBoardSocket(dai.CameraBoardSocket.CAM_B)`

`        self.left.setFps(self.fps)`

`        self.right.setResolution(self.monoResolution)`

`        self.right.setBoardSocket(dai.CameraBoardSocket.CAM_C)`

`        self.right.setFps(self.fps)`

`        self.stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)`

`        # LR-check is required for depth alignment`

`        self.stereo.setLeftRightCheck(True)`

`        self.stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A)`

`        # self.h, self.w = 1248*2, 936*2`

`        self.h, self.w = 4048, 3040`

`        # # 4056x3040`

`        self.stereo.setOutputSize(self.h, self.w)`

`        #stereo.setOutputSize(1280, 720)`

`        # stereo.setOutputSize(4056, 3040)`

`        self.sync = self.pipeline.create(dai.node.Sync)`

`        self.sync.setSyncThreshold(timedelta(milliseconds=70))`

`        # Linking`

`        self.camRgb.isp.link(self.sync.inputs["rgb"])`

`        self.left.out.link(self.stereo.left)`

`        self.right.out.link(self.stereo.right)`

`        self.stereo.depth.link(self.sync.inputs["depth"])`

`        self.sync_out = self.pipeline.createXLinkOut()`

`        self.sync_out.setStreamName("rgbd")`

`        self.sync.out.link(self.sync_out.input)`

`depth_camera = depth_cam()`

`with depth_camera.device:`

`        depth_camera.device.startPipeline(depth_camera.pipeline)`

`        while True:`

`                      depth_camera.messages = depth_camera.device.getOutputQueue("rgbd", maxSize=1).tryGetAll()`

`                       start = time.time()`

`                        if not depth_camera.messages:`

`                            continue`

`                        for message_group in depth_camera.messages:`

`                            for name, frame in message_group:`

`                                print(f"{name}: {frame.getSequenceNum()}")`

`                                if name == 'depth':`

`                                    # If the packet from RGB camera is present, we're retrieving the frame in OpenCV format using getCvFrame`

`                                    depth_frame = frame.getCvFrame()##.astype(np.uint16)`

`                                    # cv2.imwrite(f"/home/sdt/Workspace/onvif/python-onvif-zeep/hojun/image_test/depth_depth/depthFrame_{currentDateAndTime}.jpg", depth_frame)`

`                                    depth_time = datetime.datetime.now()`

`                                    with open(f"/home/sdt/Workspace/onvif/python-onvif-zeep/hojun/image_test/depth_depth_array/depthFrameArray_{depth_time}_{vseq_no}.pickle", 'wb') as f:`

`                                        pickle.dump(depth_frame, f)`

`                                elif name == 'rgb':`

`                                    rgb_frame = frame.getCvFrame()##.astype(np.uint16)`

`                                    current_time_depth_rgb = datetime.datetime.now()`

`                                   logger.info(f"Depth rgb img saved at : {current_time_depth_rgb}")`

`                                    rgb_frame = cv2.resize(rgb_frame, (depth_camera.h, depth_camera.w), interpolation=cv2.INTER_NEAREST)`


`                                    cv2.imwrite(f"/home/sdt/Workspace/onvif/python-onvif-zeep/hojun/image_test/depth_rgb/rgb_frame_{currentDateAndTime}.jpg", rgb_frame)`

jakaskerl

Jay-jay
I'll test this and get back to you.

jakaskerl

Jay-jay
I think the main issue is you are upscaling the depth from 800p to 12+MP. You should be getting this error:
[error] Custom disparity/depth width must be multiple of 16.

Change output size to 4000, 3000 and try again. Or lower it if you can.
Essentially, the device is running out of resources. Also make sure you are using latest depthai.

Thanks,
Jaka

Jay-jay

jakaskerl
Thank you!. I do not get [error] Custom disparity/depth width must be multiple of 16 error because 4048 and 3040 I have as height and width are divisible by 16 as below. I wonder why you think I would get the error?
self.h, self.w = 4048, 3040
And,
https://docs.luxonis.com/hardware/platform/depth/depth-accuracy/ tells if we wanna have better depth accuracy,
Resolution of stereo pair: Higher resolution results in more accurate depth. should be observed. Doesnt your suggestion sacrifice the depth accuracy?
To summarize,

I do not get [error] Custom disparity/depth width must be multiple of 16. error with self.h, self.w = 4048, 3040 and self.h, self.w = 1248*2, 936*2. Where do you think we get the multiple of 16 error?
Will I lose depth accuracy if I change to 800p as suggested? And do you think the 12M is the main cause of Stereo alignment error: 1. Stereo rectification error.? What else do you think we should look at?

Thank you in advance again.
Hojun

jakaskerl

Jay-jay

Jay-jay Custom disparity/depth width must be multiple of 16 error because 4048 and 3040 I have as height and width are divisible by 16 as below. I wonder why you think I would get the error?

Oh, looks like the GPT formated your code and just plain ignored your settings.. I didn't bother to check you had different w, h specified.

Jay-jay Resolution of stereo pair: Higher resolution results in more accurate depth. should be observed. Doesnt your suggestion sacrifice the depth accuracy?

No. Depth accuracy is dependent on the the resolution of the sensors used in stereo (so 800p). Merely upscaling the resulting depth map will not result in a better accuracy.

Jay-jay Will I lose depth accuracy if I change to 800p as suggested? And do you think the 12M is the main cause of Stereo alignment error: 1. Stereo rectification error.? What else do you think we should look at?

Change RGB to 1080p or 4K. And don't upscale the depth map. You don't get anything from it other than more pixels which are generally meaningless and a lot of additional latency.

Thanks,
Jaka