FPS drop when detections in YoloSpatialDetectionNetwork + ObjectTracker pipeline

austy246

Hi,

I'm trying to troubleshoot now for weeks why my fps drops significantly when there are detections and tracklets when running pipeline which looks like this:

FPS normally is ~23 when there are no detections and trackings but once there is something detected, it drops to:

I tried to use pipeline_graph in debugging, but it does not say where slow down is located:

Do you have some suggestions how to debug the pipeline?

Thanks

austy246

My device is Oak-D and I'm using Raspberry PI4 as host.

jakaskerl

Hi austy246
Can you add a minimal repro code as well? Also, any chance you can try the script on a PC instead of raspberrypi to see if there is any difference?

Thanks,
Jaka

austy246

Currently I'm not able to try it on my machine, because it is deployed in customer location.

simplified pipeline looks like this:

pipeline = dai.Pipeline()

Define sources and outputs

camRgb = pipeline. Create(dai.node.ColorCamera)
manipSquare = pipeline. Create(dai.node.ImageManip)
spatialDetectionNetwork = pipeline.create(dai.node.YoloSpatialDetectionNetwork)
monoLeft = pipeline.create(dai.node.MonoCamera)
monoRight = pipeline. Create(dai.node.MonoCamera)
stereoDepth = pipeline. Create(dai.node.StereoDepth)
objectTracker = pipeline.create(dai.node.ObjectTracker)
manipOut = pipeline. Create(dai.node.ImageManip)
xoutPreview = pipeline. Create(dai.node.XLinkOut)
xoutTracklets = pipeline.create(dai.node.XLinkOut)

xoutPreview.setStreamName("preview")
xoutTracklets.setStreamName("tracklets")

Properties

camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P) # THE_1080_P / THE_4_K / THE_12_MP
camRgb.setIspScale(432, 1080) # 1920x1080 -> 576x324 or 432 or 540 or 648 or 756 or 864 or 972 or 1080
camRgb.setInterleaved(False)
camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
camRgb.setFps(20)

mono

monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT)
monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)

setting stereoDepth

stereoDepth.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)
stereoDepth.setDepthAlign(dai.CameraBoardSocket.RGB)
stereoDepth.setOutputSize(monoLeft.getResolutionWidth(), monoLeft.getResolutionHeight())

spatialDetectionNetwork

Custom trained yolo tiny model (416x416) with one class

spatialDetectionNetwork.setBlobPath(config['blob_path'])
spatialDetectionNetwork.input.setBlocking(False)
spatialDetectionNetwork.setBoundingBoxScaleFactor(0.5)
spatialDetectionNetwork.setDepthLowerThreshold(500) # 50 cm
spatialDetectionNetwork.setDepthUpperThreshold(5000) # 5 m

Yolo specific parameters

metadata = config.get("NN_specific_metadata", {})
spatialDetectionNetwork.setNumClasses(1)
spatialDetectionNetwork.setCoordinateSize(metadata['coordinates'])
spatialDetectionNetwork.setAnchors(metadata['anchors'])
spatialDetectionNetwork.setAnchorMasks(metadata['anchor_masks'])
spatialDetectionNetwork.setIouThreshold(0.1)
spatialDetectionNetwork.setConfidenceThreshold(0.5)

objectTracker

objectTracker.setDetectionLabelsToTrack(config['track_labels'])

other tracking types did not help:

ZERO_TERM_IMAGELESS
SHORT_TERM_IMAGELESS
SHORT_TERM_KCF

objectTracker.setTrackerType(dai.TrackerType.ZERO_TERM_COLOR_HISTOGRAM)
objectTracker.setTrackerIdAssignmentPolicy(dai.TrackerIdAssignmentPolicy.UNIQUE_ID)
objectTracker.inputTrackerFrame.setBlocking(False)
objectTracker.inputTrackerFrame.setQueueSize(4)

set input resize node

input_size = (416, 416)
manipSquare.initialConfig.setResize(int(input_size[0]), int(input_size[1]))
manipSquare.initialConfig.setFrameType(dai.ImgFrame.Type.BGR888p) # NV12,YUV420p,BGR888p
keep_aspect_ratio = False
manipSquare.initialConfig.setKeepAspectRatio(keep_aspect_ratio)

set output manip node

manipOut.initialConfig.setResize(int(int(input_size[0])*1920/1080), int(input_size[1]))
manipOut.initialConfig.setKeepAspectRatio(False)

linking

camRgb.isp.link(manipSquare.inputImage)
manipSquare.out.link(spatialDetectionNetwork.input)
spatialDetectionNetwork.passthrough.link(objectTracker.inputTrackerFrame)
spatialDetectionNetwork.passthrough.link(objectTracker.inputDetectionFrame)
spatialDetectionNetwork.out.link(objectTracker.inputDetections)
monoLeft.out.link(stereoDepth.left)
monoRight.out.link(stereoDepth.right)
stereoDepth.depth.link(spatialDetectionNetwork.inputDepth)
objectTracker.out.link(xoutTracklets.input)
objectTracker.passthroughTrackerFrame.link(manipOut.inputImage)
manipOut.out.link(xoutPreview.input)

austy246

Any suggestions how to debug this issue?

austy246

Described problem with dropping FPS was just mistake in FPS counting. But real problem is, pipeline miss some detections and it is also losing some already tracked objects, but NN for detection works fine when I run detection separately (on my machine 🙂)) ). But tracker lost them in the pipeline. Problem is probably somewhere between spatialDetectionNetwork and objectTracker. I'll close this issue and try to formulate the problem in other post.

jakaskerl

Hi austy246
You can usually fix the tracking problem with a different tracking policy and type. Also please when you paste code, do so inside a code block (same as github 3x ` on both ends). It makes it 100x easier to read and work with.
Thanks,
Jaka

Fakhrullo

Hey bro, I'm also working on a project like that, it should count the people who are entering and going out, I'm also facing a major problem which is the camera is so slow showing between 0-6 FPS and working very laggy

Like in this picture, Did you find solutions for that? And is there any way to keep in touch with you bro? I have some more questions regarding to this project, you seem to know a lot about it 🙂

austy246

Fakhrullo Hi, lets connect my email is hausterlitz@gmail.com. I did not figure it out. It looks like your issue is different than mine. I don't have POE camera and my FPS is around 23. But for some reason some detections are missed or tracked object is lost. I dent know for sure now.

Fakhrullo

Hi austy246 I emailed you bro, If you didn't receive my email contact me asap dombit1999@gmail.com