Cannot get more than 4K 15fps

Seneral

I have an OAK-D Lite (backer version /wo IMU).
I remember having no issues with 4K28 when I used DepthAI V2 in the past.
Now, with DepthAI v3.5.0 (tested 3.3.0 before, too), I can't get anything above 4K15 out of the video node, even 4K16 just doesn't output any frames.
Here's a minimal test that works at 15fps but fails at anything above:

#!/usr/bin/env python3

importcv2
importdepthaiasdai

withdai.Pipeline()aspipeline:
   cam = pipeline.create(dai.node.Camera).build()
   videoQueue = cam.requestOutput(size=(1920*2,1080*2),fps=16).createOutputQueue()

   print('Usb speed:', pipeline.getDefaultDevice().getUsbSpeed().name)
   assert pipeline.getDefaultDevice().getUsbSpeed() == dai.UsbSpeed.SUPER

   pipeline.start()
   whilepipeline.isRunning():
       videoIn = videoQueue.get()
       cv2.imshow("video", videoIn.getCvFrame())
       ifcv2.waitKey(1) == ord("q"):
           break

I've also tried inserting the following since I thought the defaults looked odd:
cam.setMaxSizePoolIsp(3840*2160*3)
cam.setIspNumFramesPool(10)
cam.setMaxSizePoolRaw(3840*2160*3)
cam.setRawNumFramesPool(10)
cam.setOutputsMaxSizePool(3840*2160*3)
cam.setOutputsNumFramesPool(10)

But still, 15fps works, 16-28fps don't, simply no output from the video node.
In my actual use case, I feed it to a VideoEncoder node, and when purposefully setting the params wrong (e.g. not requesting NV12), the VideoEncoder complains up to 15fps, after that, nothing, meaning the VideoNode actually does not output anything above 4K15, even when it stays on the device.

Any clue why this might be, or any parameter I might need?

OskarSonc

Hey Seneral - in v3, I’d suggest forcing all of these explicitly:

CAM_A
native 3840x2160 sensor mode
requested FPS
NV12 output type

For example:

import time
import cv2
import depthai as dai

FPS = 28  # try 28 first

with dai.Pipeline() as pipeline:
    device = pipeline.getDefaultDevice()
    print("USB speed:", device.getUsbSpeed().name)

    cam = pipeline.create(dai.node.Camera).build(
        dai.CameraBoardSocket.CAM_A,
        sensorResolution=(3840, 2160),
        sensorFps=FPS,
    )

    q = cam.requestOutput(
        (3840, 2160),
        type=dai.ImgFrame.Type.NV12,
        fps=FPS,
    ).createOutputQueue()

    pipeline.start()

    while pipeline.isRunning():
        frame = q.tryGet()
        if frame is None:
            time.sleep(0.001)
            continue

        cv2.imshow("video", frame.getCvFrame())
        if cv2.waitKey(1) == ord("q"):
            break

If your real use case is VideoEncoder, this is the closest match, since VideoEncoder expects NV12.

Also, since you mentioned this is a Kickstarter OAK-D Lite without IMU, it may be an older board revision. If the script above still stalls above 4K15, please also print and post the CAM_A entry from:

print(device.getConnectedCameraFeatures())

That should show us all available mode.

Thanks,
Oskar

Seneral

Thanks, adding sensorResolution did the trick (others I already did in the full version).
BUT the reason I didn't use sensorResolution before is that this messes up 1080p.
If I specify sensorResolution=(1920,1080) in build, it spams the following errors:
… [Camera(0)] [error] Output image is bigger (4923360B) than maximum frame size specified in properties (3110400B) - skipping frame. Please use the setMaxOutputFrameSize API to explicitly config the [maximum] output size.
But camera does not have a setMaxOutputFrameSize method - though I already used that for the videoEncoder output (just to be safe, part of debugging), and manually set the other camera queues & buffers large enough.

Now my workaround is just to set sensorResolution when I want 4K, and to not do that when I want 1080p.
But ideally, both modes would need internal fixes, since I think there are some parameters automatically determined that don't quite work, and it feels like we lost SOME of the flexibility the manual setup of DepthAI V2 offered.

I could see how requesting 4K output accidentally uses the full sensor and the ISP crops it down instead of using the sensor cropping - resulting in 15fps max.
I could also see how the sensorResolution being set to something that requires binning somehow messes with the calculations of the buffer sizes, and afaik there is no way to override those.

Anyway, thanks for the help! Hope these can be fixed

OskarSonc

Seneral - thanks for providing more info. I tested the same v3 approach on an other RVC2 OAK-D camera (since I don't have this kickstarter revision on hand) and could not reproduce the 1080p buffer-size issue there, so this may be specific to the IMX214 / older Kickstarter OAK-D Lite revision.

What do you get as output of this?:
print(device.getConnectedCameraFeatures())

Seneral

OskarSonc
No worries, thanks!

Connected cameras: [
    {socket: CAM_A, sensorName: IMX214, width: 4208, height: 3120, supportedTypes: [COLOR], hasAutofocus: 0, hasAutofocusIC: 1, name: color},
    {socket: CAM_B, sensorName: OV7251, width: 640, height: 480, supportedTypes: [MONO], hasAutofocus: 0, hasAutofocusIC: 0, name: left},
    {socket: CAM_C, sensorName: OV7251, width: 640, height: 480, supportedTypes: [MONO], hasAutofocus: 0, hasAutofocusIC: 0, name: right}
]
Usb speed: SUPER 
Device name: OAK-D-LITE  Product name: OAK-D-LITE

From this code:

print('Connected cameras:', device.getConnectedCameraFeatures())
print('Usb speed:', device.getUsbSpeed().name)
print('Device name:', device.getDeviceName(), ' Product name:', device.getProductName())
if device.getBootloaderVersion() is not None:
    print('Bootloader version:', device.getBootloaderVersion())

Notably, no bootloader version.

OskarSonc

Thanks, that clarifies it.

This does look like an edge case specific to the early backer/prototype OAK-D Lite units. It still has the expected IMX214, but the older hardware revision likely explains why v3 auto mode selection behaves differently here.

I think your current workaround sounds fine for now. But will report this to the team ...

Thanks,
Oskar