How to save rectifiedLeft,rectifiedRight, rectifiedRGB, depth frames

Subetha

Hi, I bought OAK-D a day before to create my custom dataset. I used the following code to save rectified Left and Right Image, RGB, depth...The code was working before i have inserted the save code for disparity. Now if i run the program, I dont find any error and it is running as an infinite loop with getting any frames. Can you tell me where i was wrong please

from pathlib import Path
import cv2
import depthai as dai
import time
import numpy as np

Create pipeline
pipeline = dai.Pipeline()

monoLeft = pipeline.create(dai.node.MonoCamera)
monoRight = pipeline.create(dai.node.MonoCamera)
depth = pipeline.create(dai.node.StereoDepth)
xoutL = pipeline.create(dai.node.XLinkOut)
xoutR = pipeline.create(dai.node.XLinkOut)
xout = pipeline.create(dai.node.XLinkOut)
xoutL.setStreamName("rectL")
xoutR.setStreamName("rectR")
xout.setStreamName("disparity")
camRgb = pipeline.create(dai.node.ColorCamera)
videoEnc = pipeline.create(dai.node.VideoEncoder)
xoutJpeg = pipeline.create(dai.node.XLinkOut)
xoutRgb = pipeline.create(dai.node.XLinkOut)

xoutJpeg.setStreamName("jpeg")
xoutRgb.setStreamName("rgb")

Properties
monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)
monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)
camRgb.setBoardSocket(dai.CameraBoardSocket.RGB)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K)
videoEnc.setDefaultProfilePreset(camRgb.getFps(), dai.VideoEncoderProperties.Profile.MJPEG)

Linking
monoLeft.out.link(depth.left)
monoRight.out.link(depth.right)
depth.rectifiedLeft.link(xoutL.input)
depth.rectifiedRight.link(xoutR.input)

camRgb.video.link(xoutRgb.input)
camRgb.video.link(videoEnc.input)
videoEnc.bitstream.link(xoutJpeg.input)

frameno=0

Connect to device and start pipeline
with dai.Device(pipeline) as device:

qRight = device.getOutputQueue(name="rectR", maxSize=4, blocking=False)
qLeft  = device.getOutputQueue(name="rectL", maxSize=4, blocking=False)

qRgb = device.getOutputQueue(name="rgb", maxSize=30, blocking=False)
qJpeg = device.getOutputQueue(name="jpeg", maxSize=30, blocking=True)
q = device.getOutputQueue(name="disparity", maxSize=4, blocking=False)



dirNameL = "leftimage"
Path(dirNameL).mkdir(parents=True, exist_ok=True)
dirNameR = "rightimage"
Path(dirNameR).mkdir(parents=True, exist_ok=True)
dirName = "rgb_data"
Path(dirName).mkdir(parents=True, exist_ok=True)

dirNameL = "disparity"
Path(dirNameL).mkdir(parents=True, exist_ok=True)


while True:
    inRight = qRight.get()  # Blocking call, will wait until a new data has arrived
    inLeft  = qLeft.get()
    inRgb = qRgb.tryGet()
    inDisparity = q.get()  # blocking call, will wait until a new data has arrived
    frame = inDisparity.getFrame() 
 
    cv2.imshow("right", inRight.getCvFrame())
    cv2.imshow("left", inLeft.getCvFrame())
    if inRgb is not None:
        cv2.imshow("rgb", inRgb.getCvFrame())
   
   
    frame = (frame * (255 / depth.initialConfig.getMaxDisparity())).astype(np.uint8)
    cv2.imwrite(f"{dirNameL}/{frameno}.png", frame)
    cv2.imshow("disparity", inDisparity.getFrame() )


    cv2.imwrite(f"{dirNameR}/{frameno}.png", inRight.getFrame())
    cv2.imwrite(f"{dirNameL}/{frameno}.png", inLeft.getFrame())
    frameno += 1
    for encFrame in qJpeg.tryGetAll():
            with open(f"{dirName}/{frameno}.jpeg", "wb") as f:
            f.write(bytearray(encFrame.getData()))

    if cv2.waitKey(1) == ord('q'):
        break

Subetha

Hi now i can save the images. But I have the following doubts

The disparity map is too noisy .Can we get a better disparity map.
Another doubt is disparity map is inversely proportional to depth map.. So how to obtain the depth map?
1. How to save rectified depth and rgb images? Thanks in advance

erik

Hello Subetha !

Regarding the noisy disparity, it looks like you are too close to the camera. OAK-D by default will have min depth perception of 70cm (at 800P), and 35cm if 400P/extended disparity (more info here).
I would suggest checking depthai-record-replay demo. These tools allow you to record encoded and synced camera streams and replay them, including reconstructing the stereo depth perception.
Thoughts?
Thanks, Erik

Subetha

Thanks Erik for your reply.

Subetha

Hi @erik

I saved 16 bit depth image using depthFrame = depthQueue.get().getFrame(). Currently I am working on facial landmark detection and i have to extract the 106 landmarks along with the depth values. But most of the landmark points are 0. Any suggestions that would help me to rectify this? Are there any other ways to save depth images and get the Z values?

erik

Subetha I would work with the ROI around the landmark (eg. 9x9 square around point) to avoid invalid depth (Z=0) and to average out depth points. I would suggest looking at similar demo here. Thoughts?
Thanks, Erik

Subetha

Hi erik

Thanks for your reply. But averaging the points will make the system unable to differentiate between nose points and cheek points of a face . because all the points will have the same value. Anything else we can do in this part will helps a lot?Any suggestions?

Thanks

Subetha

erik

Hello Subetha , maybe just average in 3x3/5x5? I assume there are enough depth points so eg. nose landmark will be at least a few pixels away from cheek landmark, so you could do some averaging. You could also try setting depth confidence to 255, which will result in much less invalid depth pixels (but lower accuracy):
stereoDepthNode.initialConfig.setConfidenceThreshold(255)
Thanks, Erik

Subetha

Hi erik

Thanks

Can we do something like Sparse RLOF using OAK-d to acquire the optical flow points

erik

Hi Subetha ,
I would suggest checking the feature tracker example. Would that work?
Thanks, Erik

Subetha

Hi erik

Thanks Erik...Will try that....Thanks for your immediate response

Subetha

Hi Erik

Can we save unrectified Mono camera images? Thanks in advance

erik

Hi Subetha ,
Of course you can, DepthAI is very flexible🙂 You could use either depthai-record, which also syncs frames so you can later reconstruct the depth perception, or use a simpler example here that will just save mono frames directly (+ RGB frames).
Thanks, Erik

Subetha

Thanks Erik