FPS Drop

LcicC

Hi there,

I need to get both color and depth streams, aligned and in sync, with medium resolution and a good fps. I'm running the same code, with depthai 2.28.0.0, with different cameras and getting inconsistent outcomes. The main problem is the drop of fps - understood as the time to get both frames. Specifically, I have three issues:

1. For different cameras, same setting (i.e. literally same computer and same cable), I get frames with a resolution different than the one that was set.

2. The setFPS does not set FPS.

3. For different settings, the FPS drops from like 300 to 1, with literally the same code and the same camera (same object, not version), but different computer/cables.

I am always using depthai 2.28 and cables cat 6 A. I tested the network, checked the devices, updated firmware and tested different scripts from github depthai-experiments; including bandwidth and latency tests as per https://github.com/luxonis/depthai-experiments/tree/master/random-scripts.

Partial code:

#################################################################

# CREATE nodes

monoLeft = pipeline.create(dai.node.MonoCamera)

monoRight = pipeline.create(dai.node.MonoCamera)

stereo = pipeline.create(dai.node.StereoDepth)

rgb = pipeline.create(dai.node.Camera)

sync = pipeline.create(dai.node.Sync)

xoutGrp = pipeline.create(dai.node.XLinkOut)

xoutGrp.setStreamName("xout")

# set Properties RGB

rgbCamSocket = dai.CameraBoardSocket.CAM_A

rgb.setBoardSocket(rgbCamSocket)

rgb.setSize(1280, 720)

rgb.setFps(FPS)

# set Properties LEFT

monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)

monoLeft.setCamera("left") #monoLeft.setBoardSocket(dai.CameraBoardSocket.CAM_B)

monoLeft.setFps(FPS)

# set Properties RIGHT

monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)

monoRight.setCamera("right") #monoRight.setBoardSocket(dai.CameraBoardSocket.CAM_C)

monoRight.setFps(FPS)

# set Properties STEREO

stereo.setDepthAlign(rgbCamSocket)

stereo.initialConfig.setConfidenceThreshold(confidenceThreshold)

stereo.setLeftRightCheck(lr_check)

stereo.setSubpixel(subpixel)

stereo.setExtendedDisparity(extended_disparity)

stereo.initialConfig.setMedianFilter(dai.StereoDepthProperties.MedianFilter.KERNEL_7x7) # to reduce noise and smoothen the depth map

# set Properties SYNC

sync.setSyncThreshold(timedelta(milliseconds=100))

# Linking

monoLeft.out.link(stereo.left)

monoRight.out.link(stereo.right)

stereo.depth.link(sync.inputs["depth"])

rgb.video.link(sync.inputs["rgb"])

sync.out.link(xoutGrp.input)

if alpha is not None:

stereo.setAlphaScaling(alpha)

rgb.setCalibrationAlpha(alpha)

#####################################################

with dai.Device(pipeline) as device:

qSync = device.getOutputQueue(name="xout", maxSize=10, blocking=True)

while True:

s_time=time.time()

msgGrp = qSync.get()

for name, msg in msgGrp:

frame = msg.getCvFrame()

if name == "depth":

depth_frame = frame

depth_clock = msg.getTimestamp()

depth_bits = depth_frame.nbytes*8

depth_shape = depth_frame.shape

if name == "rgb":

rgb_frame = frame

rgb_clock = msg.getTimestamp()

rgb_bits = rgb_frame.nbytes*8

rgb_shape = rgb_frame.shape

print("Delta depth-rgb {} ms".format( (depth_clock - rgb_clock).total_seconds()*1000))

print("Bits depth {} and Bits rgb {}".format(depth_bits, rgb_bits))

print("Shape depth {} and Bits rgb {}".format(depth_shape, rgb_shape))

print("***fps sync : ",1/(time.time()-s_time))

#################################################################

Outcomes:

######################### Outcome 1 #########################

OAK-D PoE AF

Available camera sensors: {<CameraBoardSocket.CAM_B: 1>: 'OV9282', <CameraBoardSocket.CAM_A: 0>: 'IMX378', <CameraBoardSocket.CAM_C: 2>: 'OV9282'}Product name: OAK-D PoE AF, board name DM2088IMU type: BNO086

firmware version: 3.9.9, embedded firmware version: 3.9.9

USB speed: UsbSpeed.UNKNOWN

Delta depth-rgb 0.546 ms

Bits depth 33177600 and Bits rgb 49766400

Shape depth (1080, 1920) and rgb (1080, 1920, 3)

fps sync : 227.5679019044002

######################### Outcome 2 #########################

OAK-D-W-POE-97

Available camera sensors: {<CameraBoardSocket.CAM_B: 1>: 'OV9282', <CameraBoardSocket.CAM_A: 0>: 'OV9782', <CameraBoardSocket.CAM_C: 2>: 'OV9282'}Product name: OAK-D-W-POE-97, board name NG9097IMU type: BNO086,

firmware version: 3.9.9, embedded firmware version: 3.9.9

USB speed: UsbSpeed.UNKNOWN

Delta depth-rgb 95.834 ms

Bits depth 14745600 and Bits rgb 22118400

Shape depth (720, 1280) and rgb (720, 1280, 3)

fps sync : 269.03810134701735

######################## Outcome 3 #########################

OAK-D-PRO-W-POE

firmware version: 3.9.9, embedded firmware version: 3.9.9

USB speed: UsbSpeed.UNKNOWN

Delta depth-rgb 96.121 ms

Bits depth 14745600 and Bits rgb 22118400

Shape depth (720, 1280) and rgb (720, 1280, 3)

fps sync : 583.7583855254002

######################### Outcome 4 #########################

OAK-D-PRO-W-POE-97

Available camera sensors: {<CameraBoardSocket.CAM_B: 1>: 'OV9282', <CameraBoardSocket.CAM_A: 0>: 'OV9782', <CameraBoardSocket.CAM_C: 2>: 'OV9282'}

Product name: OAK-D-PRO-W-POE-97, board name NG9097IMU type: BNO086,

firmware version: 3.9.9, embedded firmware version: 3.9.9

USB speed: UsbSpeed.UNKNOWN

Delta depth-rgb 45.458999999999996 ms

Bits depth 14745600 and Bits rgb 22118400

Shape depth (720, 1280) and rgb (720, 1280, 3)

fps sync : 3.3333285649685727

######################### Outcome 5 #########################

OAK-D-PRO-W-POE (not the same as Outcome 3)

firmware version: 3.9.9, embedded firmware version: 3.9.9

USB speed: UsbSpeed.UNKNOWN

Delta depth-rgb 0.856 ms

Bits depth 33177600 and Bits rgb 49766400

Shape depth (1080, 1920) and rgb (1080, 1920, 3)

fps sync : 0.6608250229397115

jakaskerl

Hi LcicC
What does FPS mean in this context? If frames per second, something is very wrong with the calculation. Max expected FPS should be around 60.

Thanks,
Jaka

LcicC

Hi jakaskerl ,
Yes, fps means frames per second and I’m computing it with the time it takes to get both rgb and depth frames, like:

start_time with time.time()

get message with qSync.get() and frames with msg.getCvFrame()

compute fps as 1/(time.time()-s_time))

I’m trying to set the fps to reasonable values such as 10, 20 or 30, using cam.setFps(FPS).

However, I’m getting frames arriving at very high (100, 300) or very low (1, 0.5) rates.

jakaskerl

LcicC
The low rates are probably right (for POE device with gbps bandwidth). Tested with your script myself and I am getting around 23FPS on USB and 5FPS on POE.

GPT constructed script.

import depthai as dai
import time
from datetime import timedelta

# Set FPS and confidence threshold (replace with your desired values)
FPS = 30
confidenceThreshold = 200
lr_check = True
subpixel = True
extended_disparity = False
alpha = None

# Create pipeline
pipeline = dai.Pipeline()

# CREATE nodes
monoLeft = pipeline.create(dai.node.MonoCamera)
monoRight = pipeline.create(dai.node.MonoCamera)
stereo = pipeline.create(dai.node.StereoDepth)
rgb = pipeline.create(dai.node.ColorCamera)
sync = pipeline.create(dai.node.Sync)
xoutGrp = pipeline.create(dai.node.XLinkOut)

# Set stream name
xoutGrp.setStreamName("xout")

# set Properties RGB
rgbCamSocket = dai.CameraBoardSocket.CAM_A
rgb.setBoardSocket(rgbCamSocket)
rgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_720_P)
rgb.setFps(FPS)

# set Properties LEFT
monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)
monoLeft.setFps(FPS)

# set Properties RIGHT
monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)
monoRight.setFps(FPS)

# set Properties STEREO
stereo.setDepthAlign(rgbCamSocket)
stereo.initialConfig.setConfidenceThreshold(confidenceThreshold)
stereo.setLeftRightCheck(lr_check)
stereo.setSubpixel(subpixel)
stereo.setExtendedDisparity(extended_disparity)
stereo.initialConfig.setMedianFilter(dai.StereoDepthProperties.MedianFilter.KERNEL_7x7)

# set Properties SYNC
sync.setSyncThreshold(timedelta(milliseconds=100))

# Linking
monoLeft.out.link(stereo.left)
monoRight.out.link(stereo.right)
stereo.depth.link(sync.inputs["depth"])
rgb.video.link(sync.inputs["rgb"])
sync.out.link(xoutGrp.input)

# Optional alpha scaling
if alpha is not None:
    stereo.setAlphaScaling(alpha)
    rgb.setCalibrationAlpha(alpha)

# Running the pipeline
with dai.Device(pipeline) as device:
    qSync = device.getOutputQueue(name="xout", maxSize=10, blocking=True)

    while True:
        s_time = time.time()
        msgGrp = qSync.get()

        for name, msg in msgGrp.getData():
            frame = msg.getCvFrame()

            if name == "depth":
                depth_frame = frame
                depth_clock = msg.getTimestamp()
                depth_bits = depth_frame.nbytes * 8
                depth_shape = depth_frame.shape
            elif name == "rgb":
                rgb_frame = frame
                rgb_clock = msg.getTimestamp()
                rgb_bits = rgb_frame.nbytes * 8
                rgb_shape = rgb_frame.shape

            # Print timing and frame details
            print(f"Delta depth-rgb {((depth_clock - rgb_clock).total_seconds() * 1000):.3f} ms")
            print(f"Bits depth {depth_bits} and Bits rgb {rgb_bits}")
            print(f"Shape depth {depth_shape} and Shape rgb {rgb_shape}")

        # Print FPS
        print("***fps sync : ", 1 / (time.time() - s_time))

The high FPS outcomes are wrong.

Thanks,
Jaka

LcicC

Nice, your example reproduce problem 2: the setFps does not set Fps, since you set it to 30 and got 5 with POE. Any ideas how to get a higher fps in this case?

jakaskerl

LcicC
Lower performance cost.
Set rgb isp scale to a factor that will scale the color camera to depth size. This way the setDepthAlign won't need do upscale the depth to fit RGB.

Thanks,
Jaka

LcicC

Hi Jaka,

Even when we set rgb and depth to the same resolution, of 1280x720, using either setSize or ispScale depending on the case, we get inconsistent results. For example, using rgb_depth_aligned.py (from depthai-python/examples/StereoDepth/rgb_depth_aligned.py at main · luxonis/depthai-python · GitHub) with one camera we get the expected resolution but with another one the rgb frame comes in full HD regardless of setDepthAlign being set.

jakaskerl

LcicC
Well that makes sense since the cameras are different. You should be getting a warning saying "unsupported resolution.... defaulting to [some resolution]". This overrides the set resolution so the same code will not necessarily have the same output.

Thanks,
Jaka

LcicC

It is not unsupported as changing the code the resolution changes as well.

jakaskerl

LcicC
720p is an unsupported resolution on IMX378. ColorCamera would warn you, Camera doesn't because it should use the next best resolution (in this case 1080p) and apply IspScale automatically. But it doesn't do that cc @GergelySzabolcs perhaps the ISP output has to be initialized for it to work.

Anyway, setting camRgb.setVideoSize(1280, 720) and stereo.setOutputSize(1280, 720) should solve the issue on both devices.

Thanks,
Jaka

GergelySzabolcs

That has been addressed on v3 API, we are not planning to make changes to V2 Camera node.

LcicC

Indeed setting camRgb.setVideoSize(1280, 720) and stereo.setOutputSize(1280, 720) works. How do these methods work though? I mean, are the frames being taken with the resolution that was set and then getting resized in the camera after all the post-processing operations? And I guess I could put any reasonable size here, if color and depth are in alignment at least.

jakaskerl

LcicC

LcicC I mean, are the frames being taken with the resolution that was set and then getting resized in the camera after all the post-processing operations?

Correct. It applies to the last resize step before sending out a frame.

Thanks,
Jaka

LcicC

Hi Jaka,

Good to know, thanks! Now I’m facing another issue: when I set the same resolution for setVideoSize, setOutputSize, and setResolution, RGB and DEPTH are aligned. However, that is not the case if I choose a smaller size for setOutputSize and setVideoSize.

Example:

using resolution

monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P) monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)

with

camRgb.setVideoSize(1280, 720)

stereo.setOutputSize(1280, 720)

I get a reasonable alignment of RGB and DEPTH:

But with

monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P) monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)

and

camRgb.setVideoSize(320, 240)

stereo.setOutputSize(320, 240)

I get

jakaskerl

LcicC
This is because the aspect ratio of the image is changed.

Camera output is cropped since the default resize method for 16:9 --> 4/3 is CROP.
Depth output is squished because the resize method is STRETCH.

Keeping the aspect ratio will preserve the alignment.

Thanks
Jaka

LcicC

Hi Jaka,

with

monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
camRgb.setVideoSize(640, 360)
stereo.setOutputSize(640, 360)

output is not aligned.

Tried adding setPreviewSize or isp scale for the resolution of the color camera but couldn't make it work.

jakaskerl

Hi LcicC
I think preview output should work to align the cameras. With 320, 240 as well.

camRgb.setPreviewSize(640, 360)
camRgb.setFps(fps)
stereo.setOutputSize(640, 360)

Thanks,
Jaka

LcicC

Hi Jaka, tried with setPreviewSize and the output is still not aligned

jakaskerl

LcicC

make sure to link preview output instead of video, otherwise it won't affect anything.

# Linking
camRgb.preview.link(rgbOut.input)
left.out.link(stereo.left)
right.out.link(stereo.right)
stereo.disparity.link(disparityOut.input)

Thanks,
Jaka