Hi, I have used multiple cameras instruction and I am able to get multiple videos, I was wondering how I can view two depth map from two cameras using the same approach

Is this the correct approach? I tried to combine multiple device and calculate spatials on host here to get two depth map and two rgb output from two camera. I am also getting an error that stereo is not a function. Please let me know if i am heading in the right direction


import cv2
import depthai as dai
import contextlib

from calc import HostSpatialsCalc
from utility import *
import numpy as np
import math

def createPipeline():
    # Start defining a pipeline
    pipeline = dai.Pipeline()
    # Define a source - color camera
    camRgb = pipeline.create(dai.node.ColorCamera)

    camRgb.setPreviewSize(300, 300)
    camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
    camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
    camRgb.setInterleaved(False)

    # Create output
    xoutRgb = pipeline.create(dai.node.XLinkOut)
    xoutRgb.setStreamName("rgb")
    camRgb.preview.link(xoutRgb.input)
    # Define sources and outputs
    monoLeft = pipeline.create(dai.node.MonoCamera)
    monoRight = pipeline.create(dai.node.MonoCamera)
    stereo = pipeline.create(dai.node.StereoDepth)

    # Properties
    monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
    monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)
    monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
    monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)

    stereo.initialConfig.setConfidenceThreshold(255)
    stereo.setLeftRightCheck(True)
    stereo.setSubpixel(False)

    # Linking
    monoLeft.out.link(stereo.left)
    monoRight.out.link(stereo.right)

    xoutDepth = pipeline.create(dai.node.XLinkOut)
    xoutDepth.setStreamName("depth")
    stereo.depth.link(xoutDepth.input)

    xoutDepth = pipeline.create(dai.node.XLinkOut)
    xoutDepth.setStreamName("disp")
    stereo.disparity.link(xoutDepth.input)

    return pipeline


with contextlib.ExitStack() as stack:
    deviceInfos = dai.Device.getAllAvailableDevices()
    usbSpeed = dai.UsbSpeed.SUPER
    openVinoVersion = dai.OpenVINO.Version.VERSION_2021_4

    qRgbMap = []
    devices = []

    for deviceInfo in deviceInfos:
        deviceInfo: dai.DeviceInfo
        device: dai.Device = stack.enter_context(dai.Device(openVinoVersion, deviceInfo, usbSpeed))
        devices.append(device)
    
        mxId = device.getMxId()
        cameras = device.getConnectedCameras()
        usbSpeed = device.getUsbSpeed()
        eepromData = device.readCalibration2().getEepromData()
       

        pipeline = createPipeline()
        device.startPipeline(pipeline)

        # Output queue will be used to get the rgb frames from the output defined above
        depthQueue = device.getOutputQueue(name="depth")
        dispQ = device.getOutputQueue(name="disp")
        text = TextHelper()
        hostSpatials = HostSpatialsCalc(device)
        y = 200
        x = 300
        step = 3
        delta = 5
        hostSpatials.setDeltaRoi(delta)

        q_rgb = device.getOutputQueue(name="rgb", maxSize=4, blocking=False)
        stream_name = "rgb-" + mxId + "-" + eepromData.productName
        qRgbMap.append((q_rgb, stream_name, depthQueue,dispQ))
      

    while True:
        
        for q_rgb, stream_name ,depthQueue, dispQ in qRgbMap:
            depthData = depthQueue.get()
            spatials, centroid = hostSpatials.calc_spatials(depthData, (x,y)) # centroid == x/y in our case
            # Get disparity frame for nicer depth visualization
            disp = dispQ.get().getFrame()
            disp = (disp * (255 / stereo.initialConfig.getMaxDisparity())).astype(np.uint8)
            disp = cv2.applyColorMap(disp, cv2.COLORMAP_JET)
        

            # Show the frame
            cv2.imshow("depth", disp)


            if q_rgb.has():
                cv2.imshow(stream_name, q_rgb.get().getCvFrame())

        if cv2.waitKey(1) == ord('q'):
            break

    Hi @SadiaC
    No need to run spatial calculations on host. Each device is capable of doing that and then sending the results to the host.

    SadiaC # Show the frame
    cv2.imshow("depth", disp)

    This wont work, since you are essentially showing the depth from both devices inside the same frame (named "depth"). Make sure to set distinct names for the streams.

    Thanks,
    Jaka

      jakaskerl Hi thank you, would it just work if I have two names for the streams? Also how to take the stereo information out of the pipeline? Currently, stereo function is not being recognized. Can I output return pipeline and stereo together?

        SadiaC Hi I was able to do so by returning both the pipeline and stereo. Just wanted to make sure if this looks okay! Or if there's a better approach to save the disparity maps from two cameras thank you!

        #!/usr/bin/env python3
        
        import cv2
        
        import depthai as dai
        
        import contextlib
        
        from calc import HostSpatialsCalc
        
        from utility import *
        
        import numpy as np
        
        import math
        
        def createPipeline():
        # Start defining a pipeline
        
        pipeline = dai.Pipeline()
        
        # Define a source - color camera
        
        camRgb = pipeline.create(dai.node.ColorCamera)
        
        camRgb.setPreviewSize(300, 300)
        
        camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
        
        camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
        
        camRgb.setInterleaved(False)
        
        # Create output
        
        xoutRgb = pipeline.create(dai.node.XLinkOut)
        
        xoutRgb.setStreamName("rgb")
        
        camRgb.preview.link(xoutRgb.input)
        
        # Define sources and outputs
        
        monoLeft = pipeline.create(dai.node.MonoCamera)
        
        monoRight = pipeline.create(dai.node.MonoCamera)
        
        stereo = pipeline.create(dai.node.StereoDepth)
        
        # Properties
        
        monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
        
        monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)
        
        monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
        
        monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)
        
        stereo.initialConfig.setConfidenceThreshold(255)
        
        stereo.setLeftRightCheck(True)
        
        stereo.setSubpixel(False)
        
        # Linking
        
        monoLeft.out.link(stereo.left)
        
        monoRight.out.link(stereo.right)
        
        xoutDepth = pipeline.create(dai.node.XLinkOut)
        
        xoutDepth.setStreamName("depth")
        
        stereo.depth.link(xoutDepth.input)
        
        xoutDepth = pipeline.create(dai.node.XLinkOut)
        
        xoutDepth.setStreamName("disp")
        
        stereo.disparity.link(xoutDepth.input)
        
        return pipeline, stereo
        with contextlib.ExitStack() as stack:
        deviceInfos = dai.Device.getAllAvailableDevices()
        
        usbSpeed = dai.UsbSpeed.SUPER
        
        openVinoVersion = dai.OpenVINO.Version.VERSION_2021_4
        
        qRgbMap = []
        
        devices = []
        
        for deviceInfo in deviceInfos:
        
            deviceInfo: dai.DeviceInfo
        
            device: dai.Device = stack.enter_context(dai.Device(openVinoVersion, deviceInfo, usbSpeed))
        
            devices.append(device)
        
        
        
            mxId = device.getMxId()
        
            cameras = device.getConnectedCameras()
        
            usbSpeed = device.getUsbSpeed()
        
            eepromData = device.readCalibration2().getEepromData()
        
           
        
            pipeline, stereo= createPipeline()
        
            device.startPipeline(pipeline)
        
            # Output queue will be used to get the rgb frames from the output defined above
        
            depthQueue = device.getOutputQueue(name="depth")
        
            dispQ = device.getOutputQueue(name="disp")
        
            text = TextHelper()
        
            hostSpatials = HostSpatialsCalc(device)
        
            y = 200
        
            x = 300
        
            step = 3
        
            delta = 5
        
            hostSpatials.setDeltaRoi(delta)
        
            q_rgb = device.getOutputQueue(name="rgb", maxSize=4, blocking=False)
        
            stream_name = "rgb-" + mxId + "-" + eepromData.productName
        
            stream_name_depth = "depth-" + mxId + "-" + eepromData.productName
        
            qRgbMap.append((q_rgb, stream_name, stream_name_depth, depthQueue,dispQ))
        
          
        
        while True:
        
            
        
            for q_rgb, stream_name , stream_name_depth, depthQueue, dispQ in qRgbMap:
        
                depthData = depthQueue.get()
        
                spatials, centroid = hostSpatials.calc_spatials(depthData, (x,y)) # centroid == x/y in our case
        
                # Get disparity frame for nicer depth visualization
        
                disp = dispQ.get().getFrame()
        
                disp = (disp \* (255 / stereo.initialConfig.getMaxDisparity())).astype(np.uint8)
        
                disp = cv2.applyColorMap(disp, cv2.COLORMAP_JET)
        
            
        
                # Show the frame
        
                cv2.imshow(stream_name_depth, disp)
        
                if q_rgb.has():
        
                    cv2.imshow(stream_name, q_rgb.get().getCvFrame())
        
            if cv2.waitKey(1) == ord('q'):
        
                break

          Also wondering how I can get distance from the disparity maps. Do i need to calibrate the cameras again and the put then into an equation by using focal length and disparity values? Or if theres a better approach?

            SadiaC ust wanted to make sure if this looks okay! Or if there's a better approach to save the disparity maps from two cameras thank you!

            The code looks good on first glance, but it does not save the disparity maps unless I have missed something.

            SadiaC Also wondering how I can get distance from the disparity maps. Do i need to calibrate the cameras again and the put then into an equation by using focal length and disparity values? Or if theres a better approach?

            Can you explain your setup, I'm not sure I follow. If you are using depth/disparity from each device, then no recalibration is needed and you can simply use the SpatialCalculator node. If you intend to use one camera from one device and the other for the second camera, then the process is a bit more complicated.

            Thanks,
            Jaka

              7 days later

              jakaskerl Hi, I want to see the depth map and and corresponding rgb video feed. From the depth map, I want to figure out some sort of distance of an object that is seen. Previously, I was using depth map generated from monocular cameras which give less accurate results. However, since the oak d lite cameras have the stereo properties, I was wondering if that can be utilized with the depth map to give a better distance estimation. I hope I could make it a bit clear! Thanks.

              Hi @SadiaC
              It's possible to align the RGB frame to depth within the depthai API. This will make it so pixels on RGB correspond with pixels on depth image, which you can use to calculate the distance from an object detected on RGB.

              https://docs.luxonis.com/projects/api/en/latest/samples/StereoDepth/rgb_depth_aligned/
              https://docs.luxonis.com/projects/api/en/latest/samples/SpatialDetection/spatial_tiny_yolo/

              Thanks,
              Jaka

                Hi @SadiaC
                What do you mean exactly? The alignment works with all three cameras on the Oak-D series of devices, therefore it is possible no matter which camera you choose as the detections source.

                Thanks,
                Jaka