• Hardware

  • Full 4/3 sensor binning video mode

Hi there,

We have tried all the wide angle module we can find but none of them is wide enough. The reason is not the lens but the video stream we get.

For exemple, on the OAK-FFC IMX378 W, the full sensor FOV is great. But once we use the video stream, it's cropped to 16:9 format and on wide angle camera, it cuts SO MUCH of the FOV !
From (H:108°, V:93°) we get only (H:79.8°, V:50°).

We can try all wide angle sensor out there, if it's cropped, even a little, we lose most of the FOV. For example, on the IMX378 W, sensor is only cropped by 5% horizontally but we lose 25% of the HFOV.

So what we need to achieve is rather getting a binning version of the 4/3 image (from 12 MP to 1 or 2. MP would be enough) to be encoded with H264 and streamed !

Is there a way to achieve that ? Or a particular sensor that allows it ?

I hope you have few idea because we have tried all workaround we thought about.

Best

  • erik replied to this.

    Hi Matthieu ,
    To get full FOV, you'll need to use 12MP resolution when using IMX378 sensor (explanation docs here).

    Code below will set it to 12MP, and do downscaling on-device (you can skip downscaling if you want full 12MP images):

    #!/usr/bin/env python3

import cv2
import depthai as dai

# Create pipeline
pipeline = dai.Pipeline()

# Define source and output
camRgb = pipeline.create(dai.node.ColorCamera)
xout = pipeline.create(dai.node.XLinkOut)
xout.setStreamName("isp")

# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
camRgb.setIspScale(6, 13) # 4056x3040 -> 1872x1404

# Linking
camRgb.isp.link(xout.input)

# Connect to device and start pipeline
with dai.Device(pipeline) as device:

    ispQ = device.getOutputQueue(name="isp")

    while True:
        cv2.imshow("isp", ispQ.get().getCvFrame())

        if cv2.waitKey(1) == ord('q'):
            break
    • Uce likes this.

      Hi @erik

      Two questions :

      • When using the 12MP resolution, even after rescaling the framerate is choppy and there is a lot of latency between the images we get (color stereo rig with IMX378W on a OAK-FFC-4P). Is there any trick to get a smoother framerate or is it expected with such large images ?

      • I wanted to try Letterboxing to get the full FOV of the 12MP sensor as explained here

      • However, the arguments of manip.setResizeThumbnail(x,y)(as shown above) don't seem compatible (see below) and if I use manip.initialConfig.setResizeThumbnail(x, y), it hangs.


        Please see the piece of code below.

          manip = pipeline.createImageManip()
  
  # manip.setResizeThumbnail(1280, 720) # incompatible arguments : (self: depthai.node.ImageManip, arg0: int, arg1: int, arg2: int, arg3: int, arg4: int) -> None
  # manip.initialConfig.setResizeThumbnail(1280, 720)  # Hangs
  
  manip.initialConfig.setResize(1280, 720)  # Works but crops instead of padding
  manip.setMaxOutputFrameSize(1280 * 720 * 3)
  cam_node.isp.link(manip.inputImage)
  
  out = pipeline.createXLinkOut()
  out.setStreamName(cam_id)
  
  manip.out.link(out.input)

        Thanks !

      Antoine

        Hi apirrone

        manip.initialConfig.setResizeThumbnail(1280, 720) # Hangs

        Should not hang.

        #!/usr/bin/env python3

import cv2
import depthai as dai

# Create pipeline
pipeline = dai.Pipeline()

# Define source and output
camRgb = pipeline.create(dai.node.ColorCamera)

# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)


manip = pipeline.createImageManip()

# manip.setResizeThumbnail(1280, 720) # incompatible arguments : (self: depthai.node.ImageManip, arg0: int, arg1: int, arg2: int, arg3: int, arg4: int) -> None
# manip.initialConfig.setResizeThumbnail(1280, 720)  # Hangs

manip.initialConfig.setResizeThumbnail(1280, 720)  # Works but crops instead of padding
manip.setMaxOutputFrameSize(1280 * 720 * 3)
camRgb.isp.link(manip.inputImage)

out = pipeline.createXLinkOut()
out.setStreamName("manip")

manip.out.link(out.input)

# Connect to device and start pipeline
with dai.Device(pipeline) as device:

    manipq = device.getOutputQueue(name="manip")

    while True:
        cv2.imshow("manip", manipq.get().getCvFrame())

        if cv2.waitKey(1) == ord('q'):
            break

        Thanks,
        Jaka

            jakaskerl

            Thanks,

            Indeed, it does not hang with a single camera, but it does when I try to access two cameras. Below is a minimal example:

            import depthai as dai
import cv2

# Works if you comment out CAM_A or CAM_D
# Not with both uncommented
stringToCam = {
    "CAM_A": dai.CameraBoardSocket.CAM_A,
    "CAM_D": dai.CameraBoardSocket.CAM_D,
}

fps = 30
pipeline = dai.Pipeline()
cams = {}
for cam_id in stringToCam.keys():
    cam_node = pipeline.createColorCamera()

    cam_node.initialControl.setManualFocus(135)
    cam_node.setBoardSocket(stringToCam[cam_id])
    cam_node.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
    cam_node.setFps(fps)
    cam_node.setImageOrientation(dai.CameraImageOrientation.ROTATE_180_DEG)
    cam_node.setInterleaved(False)

    manip = pipeline.createImageManip()
    manip.initialConfig.setResizeThumbnail(1280, 720)  # Hangs
    manip.setMaxOutputFrameSize(1280 * 720 * 3)
    cam_node.isp.link(manip.inputImage)

    out = pipeline.createXLinkOut()
    out.setStreamName(cam_id)

    manip.out.link(out.input)
    cams[cam_id] = cam_node

device = dai.Device()
device.startPipeline(pipeline)

queue = {}
for cam_id in cams.keys():
    queue[cam_id] = device.getOutputQueue(cam_id, maxSize=30, blocking=True)

i = 0
while True:
    frames = {}
    pkts = {}
    for cam in cams:
        pkts[cam] = queue[cam].get()
    for cam, pkt in pkts.items():
        cv2.imshow(cam, pkt.getCvFrame())

    cv2.waitKey(1)

            Looking at the debug output, I see that the CMX memory is already at 2.49/2.5 with a single camera, maybe that's the issue ?

            Thanks !

            Antoine

                Hi apirrone
                Most likely yes. Even when running 4K with both mono cameras at 720p, the device starts to slowly struggle with image capture. I would assume torturing it with 24MP would be problematic..

                Thanks,
                Jaka

                  Thanks @jakaskerl

                  Would the new hardware present on the FFC-6P for example be able to handle two this smoothly ?

                    Hi Matthieu
                    To my knowledge, yes. The RVC4 should have higher processing power.

                    Thanks,
                    Jaka

                      4 months later