Neural Network Detection Optimization on Oak-D-Cm4 POE camera

GurdeepakSidhu
First this I see is that the boxes seem to not be synced to the video stream. I guess this is not a problem if you don't intend to ship detections coupled with the actual frame.

The other is the issue of camera motion blur - docs here.

Thanks,
Jaka

GurdeepakSidhu

GurdeepakSidhu What would be the best approach in syncing the detections with the video frame? Is it to use the neural network passthrough node? Unfortunately the issue with this is that I won't be able to display the video frame in higher resolution.

Generally if you only want to count the objects and don't need RGB frame for production, it is not necessary to sync it. It is only for visualization. You can use a sync node.

GurdeepakSidhu Is increasing the sensitivity a good approach as well? Am I understanding this correctly?

Increasing the sensitivity will only brighten the image. If you wish to unblur the images, you need lower exposure time. Then increase the ISO to brighten the image without needing to increase the lightning. Keep in mind this will increase noise on the image.

Thanks,
Jaka

GurdeepakSidhu

import depthai as dai
from datetime import timedelta

# Create pipeline
pipeline = dai.Pipeline()

# Create nodes
nn = pipeline.create(dai.node.YoloDetectionNetwork)
nnOut = pipeline.create(dai.node.XLinkOut)
nnOut.setStreamName("nn")

camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRgb.setIspScale(2, 3)
camRgb.setInterleaved(False)
camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
camRgb.setPreviewKeepAspectRatio(False)
camRgb.setFps(15)
camRgb.setPreviewSize(640, 640)

xoutRgb = pipeline.create(dai.node.XLinkOut)
xoutRgb.setStreamName("rgb")

nnNetworkOut = pipeline.create(dai.node.XLinkOut)
nnNetworkOut.setStreamName("nnNetwork")

# Create sync node
sync = pipeline.create(dai.node.Sync)
sync.setSyncThreshold(timedelta(milliseconds=50))  # Adjust sync threshold if needed

# Set NN properties
nn.setConfidenceThreshold(0.30)
nn.setNumClasses(1)
nn.setCoordinateSize(4)
nn.setAnchors([])  # Add your anchors here
nn.setAnchorMasks({})  # Add your anchor masks here
nn.setIouThreshold(0.50)
nn.setBlobPath(nnPath)
nn.setNumInferenceThreads(2)
nn.input.setBlocking(False)
nn.input.setQueueSize(1)

# Link nodes
camRgb.video.link(sync.inputs["video"])  # Link the RGB output to the sync node
camRgb.preview.link(nn.input)  # Link the camera preview to NN input
nn.out.link(sync.inputs["nn"])  # Link the NN output to the sync node

# Link sync output to XLinkOut
sync.out.link(xoutRgb.input)

# If you want to also output the NN metadata separately, link as follows:
nn.outNetwork.link(nnNetworkOut.input)

# Now, when running the pipeline, the RGB frames and NN output will be synchronized.

Thanks,
Jaka

GurdeepakSidhu
It's easier to use the output of the Sync node (the messageGroup) since for each iteration of the host side loop, the messages are synced; it is prone to error.

If you sync, then demux the messages before sending each one separately to the host side, there could be issues with syncing as the frames might be read in separate iteration. But I guess .get would prevent that since it blocks the queues.

Thanks,
Jaka

Hi GurdeepakSidhu
Use get() on both queues. This should read synced messages one by one.

Thanks,
Jaka