Hello,

me and my team are working on a grasping application where we want to be able to grasp objects whose geometry can be inferred by the pointclouds captured using some depth camera. We bought an OAK-d pro which be compare to a Realsense D435i depth camera.

We were preliminarily happy with the depth map produced by the RealSense camera (with unaltered, default settings) but unhappy with their color camera, which is why we switched over to the OAK-d pro. Now we find however that the 3d reconstructions we build using the OAK-d pro is insufficient after trying out various settings. The code we use to produce the actual depth image is, color image is:

import os

import json

import numpy as np

import depthai as dai

import matplotlib.pyplot as plt
cwd = os.path.join(os.path.dirname(os.path.realpath(__file__)), "images")
# Create pipelinepipeline = dai.Pipeline()
FPS = 1
pipeline = dai.Pipeline()

camRgb = pipeline.create(dai.node.ColorCamera)

monoLeft = pipeline.create(dai.node.MonoCamera)

monoRight = pipeline.create(dai.node.MonoCamera)

depth = pipeline.create(dai.node.StereoDepth)

# pointcloud = pipeline.create(dai.node.PointCloud)

sync = pipeline.create(dai.node.Sync)

xOut = pipeline.create(dai.node.XLinkOut)

xOut.input.setBlocking(False)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)

camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)

camRgb.setIspScale(1,3)camRgb.setFps(FPS)
monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)

monoLeft.setCamera("left")

monoLeft.setFps(FPS)

monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)

monoRight.setCamera("right")

monoRight.setFps(FPS)
config = depth.initialConfig.get()

config.postProcessing.speckleFilter.enable = True

config.postProcessing.speckleFilter.speckleRange = 60

config.postProcessing.temporalFilter.enable = True

config.postProcessing.decimationFilter.decimationFactor = 1

config.postProcessing.spatialFilter.holeFillingRadius = 2

config.postProcessing.spatialFilter.numIterations = 1

config.postProcessing.thresholdFilter.minRange = 900 # mm

config.postProcessing.thresholdFilter.maxRange = 1500 # mm

config.censusTransform.enableMeanMode = True

config.costMatching.linearEquationParameters.alpha = 0

config.costMatching.linearEquationParameters.beta = 2

depth.initialConfig.set(config)

depth.initialConfig.setConfidenceThreshold(200)

depth.setLeftRightCheck(True)

depth.setExtendedDisparity(False)

depth.setSubpixel(True)

depth.setSubpixelFractionalBits(4)

depth.setRectifyEdgeFillColor(0) # Black, to better see the cutout

# configure color/depth image outputsmonoLeft.out.link(depth.left)

monoRight.out.link(depth.right)

camRgb.isp.link(sync.inputs["rgb"])

depth.depth.link(sync.inputs["dep"])

sync.out.link(xOut.input)

xOut.setStreamName("out")
# Add outputs for mono framesxOutLeft = pipeline.create(dai.node.XLinkOut)

xOutLeft.setStreamName("mono_left")monoLeft.out.link(xOutLeft.input)
xOutRight = pipeline.create(dai.node.XLinkOut)

xOutRight.setStreamName("mono_right")

monoRight.out.link(xOutRight.input)

# Connect to device and start pipeline

with dai.Device(pipeline, device_info) as device:

print("Connected to device")

q = device.getOutputQueue(name="out", maxSize=4, blocking=False)

qLeft = device.getOutputQueue(name="mono_left", maxSize=4, blocking=False)

qRight = device.getOutputQueue(name="mono_right", maxSize=4, blocking=False)

# Obtain intrinsic parameters from the calibration data

calib_data = device.readCalibration()

K_color = calib_data.getCameraIntrinsics(dai.CameraBoardSocket.CAM_A, dai.Size2f(1920, 1080)) dist_color = calib_data.getDistortionCoefficients(dai.CameraBoardSocket.CAM_A)

K_depth = calib_data.getCameraIntrinsics(dai.CameraBoardSocket.CAM_C, dai.Size2f(640, 400))

# find extrinsics

extrinsics_color = np.eye(4)

color_to_depth = calib_data.getCameraExtrinsics(dai.CameraBoardSocket.CAM_A, dai.CameraBoardSocket.CAM_C) extrinsics_depth = np.array(color_to_depth)

i = 0

while True:

i += 1 # Get frames

message = q.get()

color_frame = message["rgb"].getCvFrame()

depth_frame = message["dep"].getCvFrame().astype(np.float32)

mono_left_frame = qLeft.get().getCvFrame()

mono_right_frame = qRight.get().getCvFrame()
depth_array = np.array(depth_frame)

color_array = np.array(color_frame)[:,:,::-1]

mask = np.ones_like(depth_array)

Here is a link to some image material:

https://drive.google.com/drive/folders/1eog_DYnHkH_4OMERsfkqux5-aag8-8qh?usp=sharing

It contains a reconstruction using realsense data, and then two images of a reconstruction done using the oakd-pro along with a depth map used to produce the reconstruction.

The questions are as follows:

Are we using suboptimal settings for our purpose? We would like material to be well-defined to the point where antipodal grasps can be proposed using the depth map alone.
Are we using a suboptimal choice of device? The Oak-d PRO has a baseline of 75mm and is intended for distances 0.7m to 12m, but we are realistically operating in the range 0.6-1.2m. Realsense d435i has a baseline of 50mm and an intended range of 0.3-3m. Could it be that we should select a camera from your line-up which has a shorter baseline but the same good color camera?
We expect all objects of interest to move a constant velocity (speed and direction). What pre/post-processing would mitigate noise related to this?

I appreciate your response,

Enodo-Oscar