100% CPU utilization on LeonOS causing dropped frames

bob

I noticed frames were being dropped and discovered LeonOS load increases to 100% in about 2s and stays there. Using an OAK FFC 4P PoE with 2 OV9282 and an AR0234 into an unmanaged switch with only the host and device on the network. I tried decreasing frames down to 2Hz, using jumbo frames and turning off auto focus and WB. This sounds sort of similar to a recent post by John194 posted 12 days ago. Any idea why this is happening?

MRE

#!/usr/bin/env python3

import cv2

import depthai as dai

from itertools import cycle

def printSystemInformation(info):

m = 1024 \* 1024 # MiB

print(f"Ddr used / total - {info.ddrMemoryUsage.used / m:.2f} / {info.ddrMemoryUsage.total / m:.2f} MiB")

print(f"Cmx used / total - {info.cmxMemoryUsage.used / m:.2f} / {info.cmxMemoryUsage.total / m:.2f} MiB")

print(f"LeonCss heap used / total - {info.leonCssMemoryUsage.used / m:.2f} / {info.leonCssMemoryUsage.total / m:.2f} MiB")

print(f"LeonMss heap used / total - {info.leonMssMemoryUsage.used / m:.2f} / {info.leonMssMemoryUsage.total / m:.2f} MiB")

t = info.chipTemperature

print(f"Chip temperature - average: {t.average:.2f}, css: {t.css:.2f}, mss: {t.mss:.2f}, upa: {t.upa:.2f}, dss: {t.dss:.2f}")

print(f"Cpu usage - Leon CSS: {info.leonCssCpuUsage.average \* 100:.2f}%, Leon MSS: {info.leonMssCpuUsage.average \* 100:.2f} %")

print("----------------------------------------")

# Create pipeline

pipeline = dai.Pipeline()

#jumbo frames and lower delays

config = dai.Device.Config()

config.board.network.mtu = 9000 # Jumbo frames. Default 1500

#config.board.network.xlinkTcpNoDelay = False # Default True

config.board.sysctl.append("net.inet.tcp.delayed_ack=1") # configure sysctl settings. 0 by default.

# Define sources and outputs

camRgb = pipeline.create(dai.node.ColorCamera)

monoLeft = pipeline.create(dai.node.MonoCamera)

monoRight = pipeline.create(dai.node.MonoCamera)

ve1 = pipeline.create(dai.node.VideoEncoder)

ve2 = pipeline.create(dai.node.VideoEncoder)

ve3 = pipeline.create(dai.node.VideoEncoder)

ve1Out = pipeline.create(dai.node.XLinkOut)

ve2Out = pipeline.create(dai.node.XLinkOut)

ve3Out = pipeline.create(dai.node.XLinkOut)

ve1Out.setStreamName('ve1Out')

ve2Out.setStreamName('ve2Out')

ve3Out.setStreamName('ve3Out')

sysLog = pipeline.create(dai.node.SystemLogger)

linkOut = pipeline.create(dai.node.XLinkOut)

linkOut.setStreamName("sysinfo")

controlIn = pipeline.create(dai.node.XLinkIn)

controlIn.setStreamName('control')

# Properties

camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)

camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1200_P) #using 1200_P camera

monoLeft.setCamera("left")

monoRight.setCamera("right")

sysLog.setRate(1) # 1 Hz

# Create encoders, one for each camera, consuming the frames and encoding them using H.264 / H.265 encoding

ve1.setDefaultProfilePreset(10, dai.VideoEncoderProperties.Profile.MJPEG)

ve2.setDefaultProfilePreset(10, dai.VideoEncoderProperties.Profile.MJPEG)

ve3.setDefaultProfilePreset(10, dai.VideoEncoderProperties.Profile.MJPEG)

ve1.setFrameRate(10)

ve2.setFrameRate(10)

ve3.setFrameRate(10)

ve1.setLossless(True)

ve3.setLossless(True)

# Linking

monoLeft.out.link(ve1.input)

camRgb.video.link(ve2.input)

monoRight.out.link(ve3.input)

ve1.bitstream.link(ve1Out.input)

ve2.bitstream.link(ve2Out.input)

ve3.bitstream.link(ve3Out.input)

sysLog.out.link(linkOut.input)

controlIn.out.link(camRgb.inputControl)

controlIn.out.link(monoLeft.inputControl)

controlIn.out.link(monoRight.inputControl)

# Connect to device and start pipeline

with dai.Device(pipeline) as dev:

# Output queues will be used to get the encoded data from the outputs defined above

outQ1 = dev.getOutputQueue(name='ve1Out', maxSize=30, blocking=True)

outQ2 = dev.getOutputQueue(name='ve2Out', maxSize=30, blocking=True)

outQ3 = dev.getOutputQueue(name='ve3Out', maxSize=30, blocking=True)



qSysInfo = dev.getOutputQueue(name="sysinfo", maxSize=4, blocking=False)

dev.setLogLevel(dai.LogLevel.DEBUG)

dev.setLogOutputLevel(dai.LogLevel.DEBUG)



controlQueue = dev.getInputQueue('control')

ctrl = dai.CameraControl()

ctrl.setAutoFocusMode(dai.CameraControl.AutoFocusMode.OFF)

ctrl.setAutoWhiteBalanceMode(dai.CameraControl.AutoWhiteBalanceMode.OFF)

controlQueue.send(ctrl) 



   



# The .h264 / .h265 files are raw stream files (not playable yet)

with open('mono1.h264', 'wb') as fileMono1H264, open('color.h265', 'wb') as fileColorH265, open('mono2.h264', 'wb') as fileMono2H264:

    print("Press Ctrl+C to stop encoding...")

    while True:

        try:

            # Empty each queue

            while outQ1.has():

                outQ1.get().getData().tofile(fileMono1H264)

            while outQ2.has():

                outQ2.get().getData().tofile(fileColorH265)

            while outQ3.has():

                outQ3.get().getData().tofile(fileMono2H264)

                

            while qSysInfo.has():

                sysInfo = qSysInfo.get()

                printSystemInformation(sysInfo)

                

        except KeyboardInterrupt:

            # Keyboard interrupt (Ctrl + C) detected

            break

erik

Hi bob ,
Have you already tried decreasing 3A FPS? More info here: https://docs.luxonis.com/projects/api/en/latest/tutorials/debugging/#cpu-usage

bob

I tried disabling Autofocus and white balance and decreasing frame rate to 2 Hz. I am skeptical the auto exposure calculation at a 2Hz frame rate is the problem. Using the Aug 31 release build.

bob

The 3A algorithm calculation runs even when Autofocus/WB are set to off?

Adding setIsp3aFps() did not change CPU utilization in the following MRE (just mono cameras, 30FPS, 100% CPU). Any additional ideas?

#!/usr/bin/env python3

FPS=30

import cv2

import depthai as dai

# Create pipeline

pipeline = dai.Pipeline()

# Define sources and outputs

monoLeftNode = pipeline.create(dai.node.MonoCamera)

monoRightNode = pipeline.create(dai.node.MonoCamera)

veMonoL = pipeline.create(dai.node.VideoEncoder)

veMonoR = pipeline.create(dai.node.VideoEncoder)

veMonoLOut = pipeline.create(dai.node.XLinkOut)

veMonoLOut.setStreamName('veMonoLOut')

veMonoROut = pipeline.create(dai.node.XLinkOut)

veMonoROut.setStreamName('veMonoROut')

# Properties

monoLeftNode.setBoardSocket(dai.CameraBoardSocket.CAM_C)

monoLeftNode.setFps(FPS)

monoLeftNode.setIsp3aFps(FPS)

monoRightNode.setBoardSocket(dai.CameraBoardSocket.CAM_B)

monoRightNode.setFps(FPS)

monoRightNode.setIsp3aFps(FPS)

# Configure encoders

veMonoL.setDefaultProfilePreset(FPS, dai.VideoEncoderProperties.Profile.MJPEG)

veMonoR.setDefaultProfilePreset(FPS, dai.VideoEncoderProperties.Profile.MJPEG)

veMonoL.setLossless(True)

veMonoR.setLossless(True)

# Linking

monoLeftNode.out.link(veMonoL.input)

monoRightNode.out.link(veMonoR.input)

veMonoL.bitstream.link(veMonoLOut.input)

veMonoR.bitstream.link(veMonoROut.input)

# Connect to device and start pipeline

with dai.Device(pipeline) as dev:

# Output queues will be used to get the encoded data from the outputs defined above

outQ2 = dev.getOutputQueue(name='veMonoLOut', maxSize=30, blocking=True)

outQ3 = dev.getOutputQueue(name='veMonoROut', maxSize=30, blocking=True)



dev.setLogLevel(dai.LogLevel.DEBUG)

dev.setLogOutputLevel(dai.LogLevel.DEBUG)   



# The .h264 / .h265 files are raw stream files (not playable yet)

with open('monoL.h264', 'wb') as fileMonoL, open('monoR.h264', 'wb') as fileMonoR:

    print("Press Ctrl+C to stop encoding...")

    while True:

        try:

            while outQ2.has():

                outQ2.get().getData().tofile(fileMonoL)

            while outQ3.has():

                outQ3.get().getData().tofile(fileMonoR)

jakaskerl

Hi bob
I am testing your code on an OAK-D-PRO with same cameras. With 0V9282 my usage is 40% at maximum. Since you are using a POE device and the fact that CSS handles the network stack as well, could it be possible your network setup requires additional cpu resources?
Anything specific you are using?

Thanks,
Jaka

bob

The network consists of an unmanaged PoE switch which operates at 2.5Gbit which is faster than the 1G connection of the OAK and a new workstation running windows 10/depthai; it is essentially directly connected and only limited by the OAK. I have tried two different brands of PoE switches, TRENDnet and tp-link just to make sure. The code in the last post above runs either continuously at 100% CPU or crashes so there must be a setup difference. The bootloader reports the card is running version 0.22. Is the bootloader just a bootloader which accepts code, starts it and cleanly exists or does it do more?

When I try to update the user bootloader over PoE, it always fails during verification. When I try to update it over USB, it lists the device state as unbooted as shown and the message: "Factory Bootloader Doesn't support User Bootloader flash Factory bootloader will be updated instead." Before I take the risk of bricking and loosing even more time on an exchange for this, please comment:

1) the bootloader contains FW or performs some initialization and setup for the next stage beyond just accepting an ethernet connection so updating the bootloader might be a valid troubleshooting step

2) the error message I am getting makes sense for an OAK-FFC 4P PoE shipped a few months ago and running 0.22; pushing the proceed button is not likely to damage it.

erik

Hi bob ,

100% CPU - I'd suggest checking Ping was missed troubleshooting docs, it provides some workarounds.
If you have presset the recovery button (see image below), then it won't go into bootloader (UNBOOTED), and it won't show whether bootloader has been flashed. Some info on recovery buttons/bootloader here.
You might soft-brick it, but it doesn't matter, as you can just use recovery button to directly connect (without bootloader) to the device. I'd suggest a factory reset though.

Thoughts?