Using spatialLocationCalculator when depth is aligned with RGB

katrinemn

Hello, I have aligned the depth with RGB as in yout python example rgb_depth_aligned.py, and I wish to obtain the depth of certain ROIs using the spatialLocationCalculator. However, when I have set it up and I try to extract the data from the spatialLocationCalculator queue it freezes.
Do you have alternatives to obtaining the x,y,z data from ROIs?

My setup is:

Create pipeline

stepSize = 0.05

newConfig = False

pipeline = dai.Pipeline()

# Define sources and outputs
monoLeft = pipeline.createMonoCamera()
monoRight = pipeline.createMonoCamera()
stereo = pipeline.createStereoDepth()
spatialLocationCalculator = pipeline.createSpatialLocationCalculator()
cam_rgb = pipeline.createColorCamera()
cam_rgb.setPreviewSize(300, 300)
cam_rgb.setInterleaved(False)


xout_rgb = pipeline.createXLinkOut()
xoutDepth = pipeline.createXLinkOut()
xoutSpatialData = pipeline.createXLinkOut()
xinSpatialCalcConfig = pipeline.createXLinkIn()

xoutDepth.setStreamName("depth")
xoutSpatialData.setStreamName("spatialData")
xinSpatialCalcConfig.setStreamName("spatialCalcConfig")
xout_rgb.setStreamName("rgb")
queueNames = []

queueNames.append("depth")
queueNames.append("rgb")
# Properties
monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)
monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)
cam_rgb.setBoardSocket(dai.CameraBoardSocket.RGB)
cam_rgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
cam_rgb.initialControl.setManualFocus(130)
lrcheck = True
subpixel = False

stereo.setConfidenceThreshold(255)
stereo.setLeftRightCheck(lrcheck)
stereo.setSubpixel(subpixel)

# Config
topLeft = dai.Point2f(0.0, 0.0)
bottomRight = dai.Point2f(0.1, 0.1)

config = dai.SpatialLocationCalculatorConfigData()
config.depthThresholds.lowerThreshold = 100
config.depthThresholds.upperThreshold = 10000
config.roi = dai.Rect(topLeft, bottomRight)

spatialLocationCalculator.setWaitForConfigInput(False)
spatialLocationCalculator.initialConfig.addROI(config)
stereo.setDepthAlign(dai.CameraBoardSocket.RGB)

# Linking
cam_rgb.isp.link(xout_rgb.input)
monoLeft.out.link(stereo.left)
monoRight.out.link(stereo.right)

spatialLocationCalculator.passthroughDepth.link(xoutDepth.input)
#stereo.depth.link(spatialLocationCalculator.inputDepth) ### FAILS WHEN THIS IS NOT COMMENTED
stereo.disparity.link(xoutDepth.input)

spatialLocationCalculator.out.link(xoutSpatialData.input)
xinSpatialCalcConfig.out.link(spatialLocationCalculator.inputConfig)

device_cam = dai.Device(pipeline)

erik

Hello katrinemn, code/pipeline usually freezes due to packets not being properly read (docs here), so you would have to copy the while True loop as well. Yes, besides spatialLocationCalculator you can calculate spatial coordinates from depth on the host, very similar demo (rgb+depth alignment, getting spatial from ROI on host) can be found here - more specifically this function. Hope this helps🙂
Thanks, Erik

katrinemn

Thank you Erik! It was a very helpful answer, and it runs easily with the code from the example. I do however experience an approximate 10% error in the depth values. Do you normally recommend re-calibration when the depth is aligned with the RGB?

Brandon

The 10% error may come from the jumps in the standard 0 to 95 disparity range. At what distance are you measuring?

So you could try subpixel w/out RGB and NN inference to see if this is the case, say by running this example:
https://github.com/luxonis/depthai-experiments/tree/master/gen2-camera-demo.

And then looking to see if the same error is present.