Full 4/3 sensor binning video mode

Matthieu

Hi there,

We have tried all the wide angle module we can find but none of them is wide enough. The reason is not the lens but the video stream we get.

For exemple, on the OAK-FFC IMX378 W, the full sensor FOV is great. But once we use the video stream, it's cropped to 16:9 format and on wide angle camera, it cuts SO MUCH of the FOV !
From (H:108°, V:93°) we get only (H:79.8°, V:50°).

We can try all wide angle sensor out there, if it's cropped, even a little, we lose most of the FOV. For example, on the IMX378 W, sensor is only cropped by 5% horizontally but we lose 25% of the HFOV.

So what we need to achieve is rather getting a binning version of the 4/3 image (from 12 MP to 1 or 2. MP would be enough) to be encoded with H264 and streamed !

Is there a way to achieve that ? Or a particular sensor that allows it ?

I hope you have few idea because we have tried all workaround we thought about.

Best

erik

Hi Matthieu ,
To get full FOV, you'll need to use 12MP resolution when using IMX378 sensor (explanation docs here).

Code below will set it to 12MP, and do downscaling on-device (you can skip downscaling if you want full 12MP images):

#!/usr/bin/env python3

import cv2
import depthai as dai

# Create pipeline
pipeline = dai.Pipeline()

# Define source and output
camRgb = pipeline.create(dai.node.ColorCamera)
xout = pipeline.create(dai.node.XLinkOut)
xout.setStreamName("isp")

# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
camRgb.setIspScale(6, 13) # 4056x3040 -> 1872x1404

# Linking
camRgb.isp.link(xout.input)

# Connect to device and start pipeline
with dai.Device(pipeline) as device:

    ispQ = device.getOutputQueue(name="isp")

    while True:
        cv2.imshow("isp", ispQ.get().getCvFrame())

        if cv2.waitKey(1) == ord('q'):
            break

apirrone

Hi @erik

Two questions :

When using the 12MP resolution, even after rescaling the framerate is choppy and there is a lot of latency between the images we get (color stereo rig with IMX378W on a OAK-FFC-4P). Is there any trick to get a smoother framerate or is it expected with such large images ?
I wanted to try Letterboxing to get the full FOV of the 12MP sensor as explained here

However, the arguments of manip.setResizeThumbnail(x,y)(as shown above) don't seem compatible (see below) and if I use manip.initialConfig.setResizeThumbnail(x, y), it hangs.

Please see the piece of code below.

  manip = pipeline.createImageManip()
  
  # manip.setResizeThumbnail(1280, 720) # incompatible arguments : (self: depthai.node.ImageManip, arg0: int, arg1: int, arg2: int, arg3: int, arg4: int) -> None
  # manip.initialConfig.setResizeThumbnail(1280, 720)  # Hangs
  
  manip.initialConfig.setResize(1280, 720)  # Works but crops instead of padding
  manip.setMaxOutputFrameSize(1280 * 720 * 3)
  cam_node.isp.link(manip.inputImage)
  
  out = pipeline.createXLinkOut()
  out.setStreamName(cam_id)
  
  manip.out.link(out.input)

Thanks !

Antoine

jakaskerl

Hi apirrone

manip.initialConfig.setResizeThumbnail(1280, 720) # Hangs

Should not hang.

#!/usr/bin/env python3

import cv2
import depthai as dai

# Create pipeline
pipeline = dai.Pipeline()

# Define source and output
camRgb = pipeline.create(dai.node.ColorCamera)

# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)


manip = pipeline.createImageManip()

# manip.setResizeThumbnail(1280, 720) # incompatible arguments : (self: depthai.node.ImageManip, arg0: int, arg1: int, arg2: int, arg3: int, arg4: int) -> None
# manip.initialConfig.setResizeThumbnail(1280, 720)  # Hangs

manip.initialConfig.setResizeThumbnail(1280, 720)  # Works but crops instead of padding
manip.setMaxOutputFrameSize(1280 * 720 * 3)
camRgb.isp.link(manip.inputImage)

out = pipeline.createXLinkOut()
out.setStreamName("manip")

manip.out.link(out.input)

# Connect to device and start pipeline
with dai.Device(pipeline) as device:

    manipq = device.getOutputQueue(name="manip")

    while True:
        cv2.imshow("manip", manipq.get().getCvFrame())

        if cv2.waitKey(1) == ord('q'):
            break

Thanks,
Jaka

apirrone

jakaskerl

Thanks,

Indeed, it does not hang with a single camera, but it does when I try to access two cameras. Below is a minimal example:

import depthai as dai
import cv2

# Works if you comment out CAM_A or CAM_D
# Not with both uncommented
stringToCam = {
    "CAM_A": dai.CameraBoardSocket.CAM_A,
    "CAM_D": dai.CameraBoardSocket.CAM_D,
}

fps = 30
pipeline = dai.Pipeline()
cams = {}
for cam_id in stringToCam.keys():
    cam_node = pipeline.createColorCamera()

    cam_node.initialControl.setManualFocus(135)
    cam_node.setBoardSocket(stringToCam[cam_id])
    cam_node.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
    cam_node.setFps(fps)
    cam_node.setImageOrientation(dai.CameraImageOrientation.ROTATE_180_DEG)
    cam_node.setInterleaved(False)

    manip = pipeline.createImageManip()
    manip.initialConfig.setResizeThumbnail(1280, 720)  # Hangs
    manip.setMaxOutputFrameSize(1280 * 720 * 3)
    cam_node.isp.link(manip.inputImage)

    out = pipeline.createXLinkOut()
    out.setStreamName(cam_id)

    manip.out.link(out.input)
    cams[cam_id] = cam_node

device = dai.Device()
device.startPipeline(pipeline)

queue = {}
for cam_id in cams.keys():
    queue[cam_id] = device.getOutputQueue(cam_id, maxSize=30, blocking=True)

i = 0
while True:
    frames = {}
    pkts = {}
    for cam in cams:
        pkts[cam] = queue[cam].get()
    for cam, pkt in pkts.items():
        cv2.imshow(cam, pkt.getCvFrame())

    cv2.waitKey(1)

Looking at the debug output, I see that the CMX memory is already at 2.49/2.5 with a single camera, maybe that's the issue ?

Thanks !

Antoine

jakaskerl

Hi apirrone
Most likely yes. Even when running 4K with both mono cameras at 720p, the device starts to slowly struggle with image capture. I would assume torturing it with 24MP would be problematic..

Thanks,
Jaka

apirrone

Thanks @jakaskerl

Would the new hardware present on the FFC-6P for example be able to handle two this smoothly ?

jakaskerl

Hi apirrone
No, the performance should be the same unfortunately.

Thanks,
Jaka

Matthieu

jakaskerl No, the performance should be the same unfortunately.

And with the futur RVC4 ?

jakaskerl

Hi Matthieu
To my knowledge, yes. The RVC4 should have higher processing power.

Thanks,
Jaka

Matthieu

When could we get an early beta of the RVC 4 ?

jakaskerl

Matthieu
Adding @erik, he should know more.

erik

@Matthieu likely Q4 this year (some dev board/kit), and official release Q2 next year.