Moduar cameras. Stereoscopy and RGB image. OAK-FFC-4P and IMX378 modules

MartnTous

Hi everybody!

I am using an OAK-FFC-4P, and it has two IMX378 modules connected.

My goal is to be able to perform stereoscopy with both modules and at the same time use the RGB image of one of them. The reason why I opted for these IMX378 modules is because of their high resolution of 12 MP, since I need to have a high quality RGB image.

The problem is that for stereoscopy the maximum quality supported is 1080 P, that is, I have to set both modules to 1080 P, but this way I do not have the high quality RGB image I need. If I set the resolution to a higher value, I get the following error:

I wonder if there is a way to configure the modules in 12 MP (to have a high quality RGB image), and take from it a lower resolution image to perform the stereoscopy.
After a moment, the code freezes and stops working. I do not understand why. I think it has to do with connecting cam_a and cam_d to two different outputs (one for rgb image and one for stereoscopy)

# LINKING

cam_a.isp.link(outcam_a.input)

cam_d.isp.link(outcam_d.input)
cam_a.isp.link(stereo.right)

cam_d.isp.link(stereo.left)

because if I test each system in different codes (on one side only rgb image and on the other side only stereoscopy) it works fine. What I find strange is that for a moment it allows me to work with both at the same time and then it freezes.

Below I show the code used:

import cv2
import depthai as dai
import numpy as np

def getFrame(queue):
    # Get frame from queue
    frame = queue.get()
    # Convert frame to OpenCV format and return
    return frame.getCvFrame()

def StereoConfig(stereo):

    # Prioritize fill-rate, sets Confidence threshold to 255
    # Prioritize accuracy, sets Confidence threshold to 200
    stereo.initialConfig.setConfidenceThreshold(245)
    # LR-check is required for depth alignment. Better handling for occlusions
    stereo.setLeftRightCheck(True)  
    # Closer-in minimum depth, disparity range is doubled (from 95 to 190):
    stereo.setExtendedDisparity(False)
    # Better accuracy for longer distance, fractional disparity 32-levels:
    stereo.setSubpixel(False)

    # FILTERS CONFIGURATION
    # Options: MEDIAN_OFF, KERNEL_3x3, KERNEL_5x5, KERNEL_7x7 (default)
    stereo.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_7x7)
    config = stereo.initialConfig.get()
    config.postProcessing.speckleFilter.enable = True
    config.postProcessing.speckleFilter.speckleRange = 5
    config.postProcessing.temporalFilter.enable = True
    config.postProcessing.temporalFilter.alpha = 0.01
    config.postProcessing.spatialFilter.enable = True
    config.postProcessing.spatialFilter.holeFillingRadius = 10
    config.postProcessing.spatialFilter.numIterations = 3
    # config.postProcessing.thresholdFilter.minRange = 400
    # config.postProcessing.thresholdFilter.maxRange = 1000
    config.postProcessing.decimationFilter.decimationFactor = 3
    stereo.initialConfig.set(config)


if name == 'main':

     fps = 30
     disparityMultiplier=2.125
     enableRectified = True

     # Define a pipeline
     pipeline = dai.Pipeline()
     device = dai.Device()

     # DEFINE SOURCES AND OUTPUTS
     cam_a = pipeline.create(dai.node.ColorCamera)
     cam_d = pipeline.create(dai.node.ColorCamera)
     stereo = pipeline.create(dai.node.StereoDepth)

     cam_a.setBoardSocket(dai.CameraBoardSocket.CAM_A)  # IMX378    
     cam_d.setBoardSocket(dai.CameraBoardSocket.CAM_D)  # IMX378

     cam_a.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)    
     cam_d.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)

     cam_a.setFps(fps)
     cam_d.setFps(fps)

     cam_a.setIspScale(2, 3)
     cam_d.setIspScale(2, 3)

     StereoConfig(stereo)

     # Set output Xlink
     outcam_a = pipeline.create(dai.node.XLinkOut)
     outcam_d = pipeline.create(dai.node.XLinkOut)
     outstereo = pipeline.create(dai.node.XLinkOut)

     outcam_a.setStreamName("cam_a")
     outcam_d.setStreamName("cam_d")
     outstereo.setStreamName("stereo")

     if enableRectified:
          xoutl = pipeline.create(dai.node.XLinkOut)        
          xoutr = pipeline.create(dai.node.XLinkOut)        
          xoutl.setStreamName("rectifiedLeft")        
          xoutr.setStreamName("rectifiedRight")

      # LINKING
      cam_a.isp.link(outcam_a.input)
      cam_d.isp.link(outcam_d.input)
      cam_a.isp.link(stereo.right)
      cam_d.isp.link(stereo.left)
      stereo.disparity.link(outstereo.input)

      if enableRectified:        
            stereo.rectifiedLeft.link(xoutl.input)        
            stereo.rectifiedRight.link(xoutr.input)        
            stereo.depth.link(depthQueue.input)    

      # Pipeline is defined, now we can connect to the device    
      with device:        
            device.startPipeline(pipeline)
            CAM_a = device.getOutputQueue(name="cam_a", maxSize=1)        
            stereo1 = device.getOutputQueue(name="stereo", maxSize=1)                

            while True:
                  Framecam1a = getFrame(CAM_a)
                  Framestereo = getFrame(stereo1)

                  # Colormap disparity for display
                  Framestereo = (Framestereo \* disparityMultiplier).astype(np.uint8)            
                  Framestereo = cv2.applyColorMap(Framestereo, cv2.COLORMAP_JET)                        

                  # SHOW IMAGES            
                  cv2.imshow("Framecam1a", Framecam1a)            
                  cv2.imshow("Framestereo", Framestereo)

                  # Check for keyboard input
                  key = cv2.waitKey(1)
                  if key == ord('q'):
                        break

I would really appreciate it if you could help me solve these problems. Thank you

jakaskerl

Hi MartnTous
IMX378 is rolling shutter and needs to be perfectly in sync for the stereo to work properly, otherwise the depth is terrible.

MartnTous wonder if there is a way to configure the modules in 12 MP (to have a high quality RGB image), and take from it a lower resolution image to perform the stereoscopy.

You can downsize the frames using ImgManip node, or you can set the preview size, and pipe the preview output to the stereo node (I'm not entirely sure, what type of input the node expects - RGB/NV12).

MartnTous After a moment, the code freezes and stops working. I do not understand why

Can you make sure you are getting an output? Cam-d is sometimes causing issues.

Thanks,
Jaka

jakaskerl

Hi MartnTous
Create an RGB cam with high resolution, then output one end directly to xlink node and the other end (preview) to stereo depth node (along with the other RGB camera - also preview output).

MartnTous with device:
device.startPipeline(pipeline)
CAM_a = device.getOutputQueue(name="cam_a", maxSize=1)
stereo1 = device.getOutputQueue(name="stereo", maxSize=1)

There is cam_d queue missing... 🙂 Since you have created the Xlink node, but it's queue can't be read, it fill's up, backfeeding the rest of the pipeline and blocking it.

Thanks,
Jaka

MartnTous

Hi @jakaskerl thanks for your response.

You can downsize the frames using ImgManip node, or you can set the preview size, and pipe the preview output to the stereo node (I'm not entirely sure, what type of input the node expects - RGB/NV12).

So, using the ImgManip node can I get two different images from the same camera (a high-resolution image and a downscaled image that I can link to the stereo output)?

I wonder if there is any code example of this implementation, I would appreciate it very much.

Can you make sure you are getting an output? Cam-d is sometimes causing issues.

Yeah, I'm sure that all ports are working correctly, I had an issue with port-D but I solved it with "pip3 install depthai -U"

As mentioned above, I think the problem with the execution freezing has to do with connecting cam_a and cam_d to two different outputs (one for rgb image and one for stereoscopy)

# LINKING
cam_a.isp.link(outcam_a.input)
cam_d.isp.link(outcam_d.input)
cam_a.isp.link(stereo.right)
cam_d.isp.link(stereo.left)

because if I test each system in different codes (on one side only rgb image and on the other side only stereoscopy) it works. What I find strange is that for a moment it allows me to work with both at the same time and then it freezes.

MartnTous

jakaskerl

Create an RGB cam with high resolution, then output one end directly to xlink node and the other end (preview) to stereo depth node (along with the other RGB camera - also preview output).

When I do this, I get the following error:

There is cam_d queue missing... Since you have created the Xlink node, but it's queue can't be read, it fill's up, backfeeding the rest of the pipeline and blocking it.

There were two problems, one, the one you mention, and the other, the enableRectified option. Even adding the cam_d queue or even deleting the whole cam_d output node it still froze. It was solved by disabling this enableRectified option and adding the cam_d queue or deleting the cam_d output node (since I only need the output of one of the two RGB images and the stereo output).