Hi, I bought OAK-D a day before to create my custom dataset. I used the following code to save rectified Left and Right Image, RGB, depth...The code was working before i have inserted the save code for disparity. Now if i run the program, I dont find any error and it is running as an infinite loop with getting any frames. Can you tell me where i was wrong please

from pathlib import Path
import cv2
import depthai as dai
import time
import numpy as np

Create pipeline
pipeline = dai.Pipeline()

monoLeft = pipeline.create(dai.node.MonoCamera)
monoRight = pipeline.create(dai.node.MonoCamera)
depth = pipeline.create(dai.node.StereoDepth)
xoutL = pipeline.create(dai.node.XLinkOut)
xoutR = pipeline.create(dai.node.XLinkOut)
xout = pipeline.create(dai.node.XLinkOut)
xoutL.setStreamName("rectL")
xoutR.setStreamName("rectR")
xout.setStreamName("disparity")
camRgb = pipeline.create(dai.node.ColorCamera)
videoEnc = pipeline.create(dai.node.VideoEncoder)
xoutJpeg = pipeline.create(dai.node.XLinkOut)
xoutRgb = pipeline.create(dai.node.XLinkOut)

xoutJpeg.setStreamName("jpeg")
xoutRgb.setStreamName("rgb")

Properties
monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)
monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)
camRgb.setBoardSocket(dai.CameraBoardSocket.RGB)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K)
videoEnc.setDefaultProfilePreset(camRgb.getFps(), dai.VideoEncoderProperties.Profile.MJPEG)

Linking
monoLeft.out.link(depth.left)
monoRight.out.link(depth.right)
depth.rectifiedLeft.link(xoutL.input)
depth.rectifiedRight.link(xoutR.input)

camRgb.video.link(xoutRgb.input)
camRgb.video.link(videoEnc.input)
videoEnc.bitstream.link(xoutJpeg.input)

frameno=0

Connect to device and start pipeline
with dai.Device(pipeline) as device:

qRight = device.getOutputQueue(name="rectR", maxSize=4, blocking=False)
qLeft  = device.getOutputQueue(name="rectL", maxSize=4, blocking=False)

qRgb = device.getOutputQueue(name="rgb", maxSize=30, blocking=False)
qJpeg = device.getOutputQueue(name="jpeg", maxSize=30, blocking=True)
q = device.getOutputQueue(name="disparity", maxSize=4, blocking=False)



dirNameL = "leftimage"
Path(dirNameL).mkdir(parents=True, exist_ok=True)
dirNameR = "rightimage"
Path(dirNameR).mkdir(parents=True, exist_ok=True)
dirName = "rgb_data"
Path(dirName).mkdir(parents=True, exist_ok=True)

dirNameL = "disparity"
Path(dirNameL).mkdir(parents=True, exist_ok=True)


while True:
    inRight = qRight.get()  # Blocking call, will wait until a new data has arrived
    inLeft  = qLeft.get()
    inRgb = qRgb.tryGet()
    inDisparity = q.get()  # blocking call, will wait until a new data has arrived
    frame = inDisparity.getFrame() 
 
    cv2.imshow("right", inRight.getCvFrame())
    cv2.imshow("left", inLeft.getCvFrame())
    if inRgb is not None:
        cv2.imshow("rgb", inRgb.getCvFrame())
   
   
    frame = (frame * (255 / depth.initialConfig.getMaxDisparity())).astype(np.uint8)
    cv2.imwrite(f"{dirNameL}/{frameno}.png", frame)
    cv2.imshow("disparity", inDisparity.getFrame() )


    cv2.imwrite(f"{dirNameR}/{frameno}.png", inRight.getFrame())
    cv2.imwrite(f"{dirNameL}/{frameno}.png", inLeft.getFrame())
    frameno += 1
    for encFrame in qJpeg.tryGetAll():
            with open(f"{dirName}/{frameno}.jpeg", "wb") as f:
            f.write(bytearray(encFrame.getData()))

    if cv2.waitKey(1) == ord('q'):
        break

Hi now i can save the images. But I have the following doubts

  1. The disparity map is too noisy .Can we get a better disparity map.
    Another doubt is disparity map is inversely proportional to depth map.. So how to obtain the depth map?
    1. How to save rectified depth and rgb images? Thanks in advance
  • erik replied to this.

    Hello Subetha !

    1. Regarding the noisy disparity, it looks like you are too close to the camera. OAK-D by default will have min depth perception of 70cm (at 800P), and 35cm if 400P/extended disparity (more info here).
    2. I would suggest checking depthai-record-replay demo. These tools allow you to record encoded and synced camera streams and replay them, including reconstructing the stereo depth perception.
      Thoughts?
      Thanks, Erik
    5 days later

    Hi @erik

    I saved 16 bit depth image using depthFrame = depthQueue.get().getFrame(). Currently I am working on facial landmark detection and i have to extract the 106 landmarks along with the depth values. But most of the landmark points are 0. Any suggestions that would help me to rectify this? Are there any other ways to save depth images and get the Z values?

    • erik replied to this.

      Subetha I would work with the ROI around the landmark (eg. 9x9 square around point) to avoid invalid depth (Z=0) and to average out depth points. I would suggest looking at similar demo here. Thoughts?
      Thanks, Erik

        Hi erik

        Thanks for your reply. But averaging the points will make the system unable to differentiate between nose points and cheek points of a face . because all the points will have the same value. Anything else we can do in this part will helps a lot?Any suggestions?

        Thanks

        Subetha

        • erik replied to this.

          Hello Subetha , maybe just average in 3x3/5x5? I assume there are enough depth points so eg. nose landmark will be at least a few pixels away from cheek landmark, so you could do some averaging. You could also try setting depth confidence to 255, which will result in much less invalid depth pixels (but lower accuracy):
          stereoDepthNode.initialConfig.setConfidenceThreshold(255)
          Thanks, Erik

            Hi erik

            Thanks

            Can we do something like Sparse RLOF using OAK-d to acquire the optical flow points

              Hi erik

              Thanks Erik...Will try that....Thanks for your immediate response

              Hi Erik

              Can we save unrectified Mono camera images? Thanks in advance

              • erik replied to this.

                Hi Subetha ,
                Of course you can, DepthAI is very flexible🙂 You could use either depthai-record, which also syncs frames so you can later reconstruct the depth perception, or use a simpler example here that will just save mono frames directly (+ RGB frames).
                Thanks, Erik