Hi, I have used multiple cameras instruction and I am able to get multiple videos, I was wondering how I can view two depth map from two cameras using the same approach
Multiple Camera Depth Map
- Edited
Is this the correct approach? I tried to combine multiple device and calculate spatials on host here to get two depth map and two rgb output from two camera. I am also getting an error that stereo is not a function. Please let me know if i am heading in the right direction
import cv2
import depthai as dai
import contextlib
from calc import HostSpatialsCalc
from utility import *
import numpy as np
import math
def createPipeline():
# Start defining a pipeline
pipeline = dai.Pipeline()
# Define a source - color camera
camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setPreviewSize(300, 300)
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRgb.setInterleaved(False)
# Create output
xoutRgb = pipeline.create(dai.node.XLinkOut)
xoutRgb.setStreamName("rgb")
camRgb.preview.link(xoutRgb.input)
# Define sources and outputs
monoLeft = pipeline.create(dai.node.MonoCamera)
monoRight = pipeline.create(dai.node.MonoCamera)
stereo = pipeline.create(dai.node.StereoDepth)
# Properties
monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)
monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)
stereo.initialConfig.setConfidenceThreshold(255)
stereo.setLeftRightCheck(True)
stereo.setSubpixel(False)
# Linking
monoLeft.out.link(stereo.left)
monoRight.out.link(stereo.right)
xoutDepth = pipeline.create(dai.node.XLinkOut)
xoutDepth.setStreamName("depth")
stereo.depth.link(xoutDepth.input)
xoutDepth = pipeline.create(dai.node.XLinkOut)
xoutDepth.setStreamName("disp")
stereo.disparity.link(xoutDepth.input)
return pipeline
with contextlib.ExitStack() as stack:
deviceInfos = dai.Device.getAllAvailableDevices()
usbSpeed = dai.UsbSpeed.SUPER
openVinoVersion = dai.OpenVINO.Version.VERSION_2021_4
qRgbMap = []
devices = []
for deviceInfo in deviceInfos:
deviceInfo: dai.DeviceInfo
device: dai.Device = stack.enter_context(dai.Device(openVinoVersion, deviceInfo, usbSpeed))
devices.append(device)
mxId = device.getMxId()
cameras = device.getConnectedCameras()
usbSpeed = device.getUsbSpeed()
eepromData = device.readCalibration2().getEepromData()
pipeline = createPipeline()
device.startPipeline(pipeline)
# Output queue will be used to get the rgb frames from the output defined above
depthQueue = device.getOutputQueue(name="depth")
dispQ = device.getOutputQueue(name="disp")
text = TextHelper()
hostSpatials = HostSpatialsCalc(device)
y = 200
x = 300
step = 3
delta = 5
hostSpatials.setDeltaRoi(delta)
q_rgb = device.getOutputQueue(name="rgb", maxSize=4, blocking=False)
stream_name = "rgb-" + mxId + "-" + eepromData.productName
qRgbMap.append((q_rgb, stream_name, depthQueue,dispQ))
while True:
for q_rgb, stream_name ,depthQueue, dispQ in qRgbMap:
depthData = depthQueue.get()
spatials, centroid = hostSpatials.calc_spatials(depthData, (x,y)) # centroid == x/y in our case
# Get disparity frame for nicer depth visualization
disp = dispQ.get().getFrame()
disp = (disp * (255 / stereo.initialConfig.getMaxDisparity())).astype(np.uint8)
disp = cv2.applyColorMap(disp, cv2.COLORMAP_JET)
# Show the frame
cv2.imshow("depth", disp)
if q_rgb.has():
cv2.imshow(stream_name, q_rgb.get().getCvFrame())
if cv2.waitKey(1) == ord('q'):
break
Hi @SadiaC
No need to run spatial calculations on host. Each device is capable of doing that and then sending the results to the host.
SadiaC # Show the frame
cv2.imshow("depth", disp)
This wont work, since you are essentially showing the depth from both devices inside the same frame (named "depth"). Make sure to set distinct names for the streams.
Thanks,
Jaka
- Edited
SadiaC Hi I was able to do so by returning both the pipeline and stereo. Just wanted to make sure if this looks okay! Or if there's a better approach to save the disparity maps from two cameras thank you!
#!/usr/bin/env python3
import cv2
import depthai as dai
import contextlib
from calc import HostSpatialsCalc
from utility import *
import numpy as np
import math
def createPipeline():
# Start defining a pipeline
pipeline = dai.Pipeline()
# Define a source - color camera
camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setPreviewSize(300, 300)
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRgb.setInterleaved(False)
# Create output
xoutRgb = pipeline.create(dai.node.XLinkOut)
xoutRgb.setStreamName("rgb")
camRgb.preview.link(xoutRgb.input)
# Define sources and outputs
monoLeft = pipeline.create(dai.node.MonoCamera)
monoRight = pipeline.create(dai.node.MonoCamera)
stereo = pipeline.create(dai.node.StereoDepth)
# Properties
monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)
monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)
stereo.initialConfig.setConfidenceThreshold(255)
stereo.setLeftRightCheck(True)
stereo.setSubpixel(False)
# Linking
monoLeft.out.link(stereo.left)
monoRight.out.link(stereo.right)
xoutDepth = pipeline.create(dai.node.XLinkOut)
xoutDepth.setStreamName("depth")
stereo.depth.link(xoutDepth.input)
xoutDepth = pipeline.create(dai.node.XLinkOut)
xoutDepth.setStreamName("disp")
stereo.disparity.link(xoutDepth.input)
return pipeline, stereo
with contextlib.ExitStack() as stack:
deviceInfos = dai.Device.getAllAvailableDevices()
usbSpeed = dai.UsbSpeed.SUPER
openVinoVersion = dai.OpenVINO.Version.VERSION_2021_4
qRgbMap = []
devices = []
for deviceInfo in deviceInfos:
deviceInfo: dai.DeviceInfo
device: dai.Device = stack.enter_context(dai.Device(openVinoVersion, deviceInfo, usbSpeed))
devices.append(device)
mxId = device.getMxId()
cameras = device.getConnectedCameras()
usbSpeed = device.getUsbSpeed()
eepromData = device.readCalibration2().getEepromData()
pipeline, stereo= createPipeline()
device.startPipeline(pipeline)
# Output queue will be used to get the rgb frames from the output defined above
depthQueue = device.getOutputQueue(name="depth")
dispQ = device.getOutputQueue(name="disp")
text = TextHelper()
hostSpatials = HostSpatialsCalc(device)
y = 200
x = 300
step = 3
delta = 5
hostSpatials.setDeltaRoi(delta)
q_rgb = device.getOutputQueue(name="rgb", maxSize=4, blocking=False)
stream_name = "rgb-" + mxId + "-" + eepromData.productName
stream_name_depth = "depth-" + mxId + "-" + eepromData.productName
qRgbMap.append((q_rgb, stream_name, stream_name_depth, depthQueue,dispQ))
while True:
for q_rgb, stream_name , stream_name_depth, depthQueue, dispQ in qRgbMap:
depthData = depthQueue.get()
spatials, centroid = hostSpatials.calc_spatials(depthData, (x,y)) # centroid == x/y in our case
# Get disparity frame for nicer depth visualization
disp = dispQ.get().getFrame()
disp = (disp \* (255 / stereo.initialConfig.getMaxDisparity())).astype(np.uint8)
disp = cv2.applyColorMap(disp, cv2.COLORMAP_JET)
# Show the frame
cv2.imshow(stream_name_depth, disp)
if q_rgb.has():
cv2.imshow(stream_name, q_rgb.get().getCvFrame())
if cv2.waitKey(1) == ord('q'):
break
Also wondering how I can get distance from the disparity maps. Do i need to calibrate the cameras again and the put then into an equation by using focal length and disparity values? Or if theres a better approach?
SadiaC ust wanted to make sure if this looks okay! Or if there's a better approach to save the disparity maps from two cameras thank you!
The code looks good on first glance, but it does not save the disparity maps unless I have missed something.
SadiaC Also wondering how I can get distance from the disparity maps. Do i need to calibrate the cameras again and the put then into an equation by using focal length and disparity values? Or if theres a better approach?
Can you explain your setup, I'm not sure I follow. If you are using depth/disparity from each device, then no recalibration is needed and you can simply use the SpatialCalculator node. If you intend to use one camera from one device and the other for the second camera, then the process is a bit more complicated.
Thanks,
Jaka
jakaskerl Hi, I want to see the depth map and and corresponding rgb video feed. From the depth map, I want to figure out some sort of distance of an object that is seen. Previously, I was using depth map generated from monocular cameras which give less accurate results. However, since the oak d lite cameras have the stereo properties, I was wondering if that can be utilized with the depth map to give a better distance estimation. I hope I could make it a bit clear! Thanks.
Hi @SadiaC
It's possible to align the RGB frame to depth within the depthai API. This will make it so pixels on RGB correspond with pixels on depth image, which you can use to calculate the distance from an object detected on RGB.
https://docs.luxonis.com/projects/api/en/latest/samples/StereoDepth/rgb_depth_aligned/
https://docs.luxonis.com/projects/api/en/latest/samples/SpatialDetection/spatial_tiny_yolo/
Thanks,
Jaka