FPS cut in half when adding multiple streams

karimkhater

Hello, im currently using the OAK-D PRO using USB 3.0 and was programming a script that activates the camera and records the color, left and right streams using UDP signals that will all be used later on for pose estimation. I ran into a problem where while giving as input the fps as 60 i get 40 fps for the Color and if i enable the creation of the Left and Right streams, that fps is dropped to 22 fps, I got a USB Controller just for the camera alone to try to see if it improves anything it didnt. I would like some help in what can be causing this issue. Also im using Depthai = 2.21.2.0 version
`
import time
from datetime import timedelta

import depthai as dai

extended_disparity = False
subpixel = True
lr_check = False
def getStereoCamera(pipeline):

stereo = pipeline.createStereoDepth()
stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)
stereo.setDepthAlign(dai.CameraBoardSocket.LEFT)
stereo.initialConfig.setConfidenceThreshold(175)
stereo.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_7x7)
stereo.setLeftRightCheck(lr_check)
stereo.setSubpixel(subpixel)

config = stereo.initialConfig.get()
stereo.initialConfig.set(config)

return stereo

def getMonoCamera(pipeline, left, mono_reso, fps,shape):
mono = pipeline.createMonoCamera()
mono.setResolution(mono_reso)

if left:
    mono.setBoardSocket(dai.CameraBoardSocket.LEFT)
else :
    mono.setBoardSocket(dai.CameraBoardSocket.RIGHT)

shape = (640,400)
if(mono_reso==dai.MonoCameraProperties.SensorResolution.THE_720_P):
    shape = (1280,720)

if(mono_reso==dai.MonoCameraProperties.SensorResolution.THE_800_P):
    shape = (1280,800)

mono.setFps(fps)

return mono, shape

def getColorCamera(pipeline, color_reso,fps_record):
color = pipeline.createColorCamera()

color.setBoardSocket(dai.CameraBoardSocket.CENTER)
color.setResolution(color_reso)

shape = (1920,1080)
if(color_reso==dai.ColorCameraProperties.SensorResolution.THE_4_K):
    shape = (3840,2160)

if(color_reso==dai.ColorCameraProperties.SensorResolution.THE_12_MP):
    shape = (4056,3040)

color.setFps(fps_record)   
return color , shape

def getFrame(queue):
frame = queue.get()
return frame.getCvFrame()

def getXLinkOut(pipeline, name):
link = pipeline.createXLinkOut()
link.setStreamName(name)
link.input.setBlocking(False)
link.input.setQueueSize(10) # Ensure this queue size is large enough
return link

def addImage(frames,msg, type):
frames[type] = msg[type].getCvFrame()

return frames

`
thats the utility file that i use to customize the pipeline.

`COLOR_RESOLUTION = dai.ColorCameraProperties.SensorResolution.THE_1080_P
COLOR_RESOLUTION_2 = dai.ColorCameraProperties.SensorResolution.THE_800_P
MONO_RESOLUTION = dai.MonoCameraProperties.SensorResolution.THE_800_P
MONO_RESOLUTION_2 = dai.MonoCameraProperties.SensorResolution.THE_400_P

def getPipeline(device):
pipeline = dai.Pipeline()

if len(device.getIrDrivers())>0:
    device.setIrLaserDotProjectorBrightness(1200)

global TARGET_COLOR_SHAPE
global TARGET_MONO_SHAPE
global fps_video

fps_video = 60
fps_mono = 60
colorCam, TARGET_COLOR_SHAPE = getColorCamera(pipeline,COLOR_RESOLUTION,fps_record=fps_video)
outColor = getXLinkOut(pipeline,"color")

if left_right :
    leftCam, TARGET_MONO_SHAPE = getMonoCamera(pipeline= pipeline , left = True, fps= fps_mono, mono_reso= MONO_RESOLUTION_2,shape=TARGET_COLOR_SHAPE)
    rightCam, TARGET_MONO_SHAPE = getMonoCamera(pipeline= pipeline , left = False, fps= fps_mono, mono_reso= MONO_RESOLUTION_2,shape=TARGET_COLOR_SHAPE)

    outLeft = getXLinkOut(pipeline= pipeline , name= "left")
    outRight = getXLinkOut(pipeline= pipeline , name= "right")


colorCam.video.link(outColor.input)

if left_right :
    leftCam.out.link(outLeft.input)
    rightCam.out.link(outRight.input)

return pipeline

and this is how i create the streams in the main pipeline. Is there a way to have every stream works in 60 fps or it will always drop since im using multiple streams ?

jakaskerl

Hi @karimkhater
I'll assume you are using 1080p for color and 400p for mono (stereo).

try adding pipeline.setXLinkChunkSize(0) to decrease latency
when creating device, add maxUsbSpeed to super_PLUS
check the FPS of color stream only

I don't think you can view all frames at 60FPS since stereo depth even at 400p introduces a lot of latency.

Thanks,
Jaka

karimkhater

Hello jakaskerl

Yes, I am using 1080p for color and 400p for mono right now.

Using pipeline.setXLinkChunkSize(0) increased the fps significantly from 23 fps to around 38 fps.
The maxUsbSpeed was already at super, but I added that line, and it did not change the fps.
Also, now I check only for the FPS of the color stream.

Is there anything else I can modify in the pipeline to increase the FPS more, or are there any other suggestions?

This is how i start the device and the streams :

def worker(dev_info,stack,dic):

    openvino_version = dai.OpenVINO.Version.VERSION_2021_4
    
    device: dai.Device = stack.enter_context(dai.Device(openvino_version, dev_info, maxUsbSpeed=dai.UsbSpeed.SUPER_PLUS))

    usb_speed = device.getUsbSpeed()

    
    name = "Cam"
    if usb_speed.name != "UNKNOWN" : name = "OAK-D_PRO"


    print("=== Connected to " + name + " " + dev_info.getMxId())
    mxid = device.getMxId()
    cameras = device.getConnectedCameras()

    print("   >>> MXID:", mxid)
    print("   >>> Cameras:", *[c.name for c in cameras])
    print("   >>> USB speed:", usb_speed.name)


    
    global calib
    calib = device.readCalibration()

    # Initialisation de la pipeline
    device.startPipeline(getPipeline(device))

   
    dic["color-" + name + "-" + mxid] = device.getOutputQueue(name="color", maxSize=5, blocking=False)
    if left_right :
        dic["right-" + name + "-" + mxid] = device.getOutputQueue(name="right", maxSize=1, blocking=False)
        dic["left-" + name + "-" + mxid] = device.getOutputQueue(name="left", maxSize=1, blocking=False)

also ran the oak_latency_test.pyand got this result : Average latency 1.57 ms, Std: 0.5
and for òak_bandwith_test.py` i got this :
Downlink 1993.6 mbps Uplink 1666.8 mbps

jakaskerl

karimkhater
You should be able to check how much each step takes, then decide which one is causing you the most latency.

https://docs.luxonis.com/software/depthai/optimizing/#Measuring%20operation%20times

Thanks,
Jaka

karimkhater

jakaskerl
I checked the latency and concluded that it seems fine; it went from 0.8 to 2.04 with an average latency of 1.67 ms, standard deviation of 0.5. I lowered the FPS of the mono cameras to 30 fps because I think it might be a bandwidth limit issue and the color camera is using too much of it. The LeonOS CPU Usage is at 90%, even when i lowered it to 60%, it didn't affect the FPS at all. The maximum FPS I managed to achieve was 50 FPS, that was with setXLinkChunkSize(0), using 400p for mono camera and 30 fps. I'm using the OAK-D PRO, which says a maximum framerate of 60 FPS for the 1080P resolution. I can't find the expected framerate mentioned in the documentation, though. Does that mean the 60 FPS is only achievable in the case where only 1 stream of color camera exists with no other functionality in the background? Or is it possible to achieve 60 FPS on the color camera and have two other streams of left and right mono cameras with it? Since I can't even get 60 FPS using the Depthai Viewer software as well?

Thanks,
Karim

karimkhater

jakaskerl I upgraded from OAK-D Pro AUTO-FOCUS to OAK-D PRO W OV9782 to have access to the 120FPS and i still get max 50 FPS if its only 1 stream and 40 FPS if its 3 streams. can i send you the full code by mail to view where the issue might be or do you have any other suggestions to help in my case from the code above ?

jakaskerl

Hi @karimkhater
Sorry, I missed the post;
Generally you cannot display frames faster than around 50FPS (depending on the OS) because the opencv.imshow() cannot work faster than that. This might be the case for you.
Maybe try without displaying frames and only timing the .get() functions on the queues to calculate the FPS.

You can send the code to support@luxonis.com as well but make it as short as you possibly can, but still reproducible.

Thanks,
Jaka

karimkhater

jakaskerl

The problem is that even without displaying the frames the output at the end is noticeable. If i for example record for 15 sec, the RGB output video is 8 seconds long with faster movement in the video than it should have been. It seems for me that the number of frames captured is less than what it should have been.

jakaskerl

karimkhater
That's usually because frames can not be produced fast enough or the conversion FPS is not set correctly.

karimkhater

i read more and yes its the getCvFrame function that is limiting the FPS when using getFrame i get more fps but its a 2d array as the output so will need to change that thanks for the help.