No longer necessary for the depth frame size to match rgb for rgb aligned depth?

AdamPolak

In this example: https://github.com/luxonis/depthai-experiments/blob/master/gen2-social-distancing/depthai_utils.py

It shows the depth set up with the rgb:
camRgb = pipeline.create(dai.node.ColorCamera)

    spatialDetectionNetwork = pipeline.create(dai.node.MobileNetSpatialDetectionNetwork)

    monoLeft = pipeline.create(dai.node.MonoCamera)

    monoRight = pipeline.create(dai.node.MonoCamera)

    stereo = pipeline.create(dai.node.StereoDepth)

    xoutRgb = pipeline.create(dai.node.XLinkOut)

    camRgb.preview.link(xoutRgb.input)

    xoutNN = pipeline.create(dai.node.XLinkOut)

    xoutRgb.setStreamName("rgb")

    xoutNN.setStreamName("detections")

    # Properties

    camRgb.setPreviewSize(544, 320)

    camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)

    camRgb.setInterleaved(False)

    camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)

    monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)

    monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)

    monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)

    monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)

    # Setting node configs

    stereo.setConfidenceThreshold(255)

    spatialDetectionNetwork.setBlobPath(blobconverter.from_zoo(name=model_name, shaves=6))

    spatialDetectionNetwork.setConfidenceThreshold(0.5)

    spatialDetectionNetwork.input.setBlocking(False)

    spatialDetectionNetwork.setBoundingBoxScaleFactor(0.5)

    spatialDetectionNetwork.setDepthLowerThreshold(100)

    spatialDetectionNetwork.setDepthUpperThreshold(5000)

    # Linking

    monoLeft.out.link(stereo.left)

    monoRight.out.link(stereo.right)

    camRgb.preview.link(spatialDetectionNetwork.input)

    spatialDetectionNetwork.out.link(xoutNN.input)

    stereo.depth.link(spatialDetectionNetwork.inputDepth)

    stereo.setDepthAlign(dai.CameraBoardSocket.RGB)

The rgb frame is set to 544,320 into the SpatialDetectionNetwork.
The depth resolution is set, but never the size.
The only alignment or "matching" comes from this line: stereo.setDepthAlign(dai.CameraBoardSocket.RGB)

Are the previous alignment checks no longer necessary?

stereo->setLeftRightCheck(true);
stereo->setOutputSize(st->rgb_dai_preview_x, st->depth_rgb_preview_y);

jakaskerl

Hi AdamPolak
I copied this from the RGB depth alignment:

By default, the depth map will get scaled to match the resolution of the camera sensor we want to align to. In other words, if depth is aligned to the 1080P color sensor, StereoDepth will upscale depth to 1080P as well. Depth scaling can be avoided by configuring StereoDepth’s stereo.setOutputSize(width, height).

As far as lrcheck goes; it is used to improve the depth map. You can usually calculate depth by checking the disparity between left and right camera. Since the algorithm sometimes gives you invalid disparity values, to improve such image, you do a reverse check (if disparity was calculated L ---> R, you could benefit from also doing a R ---> L check to fix invalid disparities).

Hope this helps,
Jaka

AdamPolak

jakaskerl

@jakaskerl I am trying to figure out how to get the full field of view which seems to require ISP.

I then need to resize that full frame into my NN, but at least I will have the full field of view.

How can I make that happen for it to be RGB aligned depth? There doesn't seem to be an example of this in the docs or the experiments?

jakaskerl

Hi AdamPolak
I guess you could link isp through manip node, set the size, set keepAspectRatio(False) and link the output to NN. Also, link the other end of isp to xlink for display.

manip = pipeline.create(dai.node.ImageManip)
manip.initialConfig.setResize(300, 300)
manip.initialConfig.setFrameType(dai.RawImgFrame.Type.BGR888p)
manip.setKeepAspectRatio(False)

Hope this helps,
Jaka