### Description: I am performing inference on an MP4 video and following a demo where frames are converted to BGR before being fed into the object detector and tracker. However, when I attempt to feed RGB images directly and change the frame type, I encounter the following error: `Unsupported frame type 7! Supported frame types are: NV12, YUV420p, BGR888p. Skipping.` Here's the context: * I am training the model using Ultralytics YOLOv8, with training images in RGB format. * I want to maintain the same RGB color order during inference, as I have concerns that converting to BGR might affect accuracy. When using the Ultralytics `predict` function on the same MP4 video, the detections are accurate. However, when running the same inference on an OpenAI Kit (OAK) device, the detection accuracy is noticeably lower. **Issue:** The OAK device only supports frame types NV12, YUV420p, and BGR888p, not RGB. This forced conversion might be causing the observed decrease in accuracy. I am seeking guidance on how to maintain color consistency or any other potential solutions to improve detection accuracy on the OAK device.

Inconsistent Object Detection Accuracy with YOLOv8 and frame color order

AmunTimilsina

Description:

I am performing inference on an MP4 video and following a demo where frames are converted to BGR before being fed into the object detector and tracker. However, when I attempt to feed RGB images directly and change the frame type, I encounter the following error:

Unsupported frame type 7! Supported frame types are: NV12, YUV420p, BGR888p. Skipping.

Here's the context:

I am training the model using Ultralytics YOLOv8, with training images in RGB format.
I want to maintain the same RGB color order during inference, as I have concerns that converting to BGR might affect accuracy.

When using the Ultralytics predict function on the same MP4 video, the detections are accurate. However, when running the same inference on an OpenAI Kit (OAK) device, the detection accuracy is noticeably lower.

Issue: The OAK device only supports frame types NV12, YUV420p, and BGR888p, not RGB. This forced conversion might be causing the observed decrease in accuracy. I am seeking guidance on how to maintain color consistency or any other potential solutions to improve detection accuracy on the OAK device.

jakaskerl

Hi @AmunTimilsina
Can you add a short MRE please? I don't think the NN forces you to above three image types. If object tracker is the problem you can always just link a different stream to it. Also make sure you are using latest depthai.

Thanks,
Jaka

AmunTimilsina

Defining Source Node

source_node = self.pipeline.create(dai.node.ColorCamera)

source_node.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)

source_node.setPreviewSize((1920, 1080))

source_node.setInterleaved(False)

source_node.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)

Defining Manip Node

manip_node = self.pipeline.create(dai.node.ImageManip)

max_output_frame_size = int(640 x 640 x 3)

manip_node.setMaxOutputFrameSize(max_output_frame_size)

manip_node.setResizeThumbnail(640, 640, 0, 0, 0)

manip_node.initialConfig.setFrameType(dai.ImgFrame.Type.BGR888p)

manip_node.inputImage.setBlocking(True)

Defining Yolov8 Node (Trained through Ultralytics)

detector_network = self.pipeline.create(dai.node.YoloDetectionNetwork) detector_network*.setBlobPath('model.blob'*)

detector_network.setConfidenceThreshold(0.5)

detector_network.setNumClasses(3)

detector_network.setCoordinateSize(4)

detector_network.setIouThreshold(0.5)

detector_network.input.setBlocking(False)

Defining Tracker Node

object_tracker = self.pipeline.create(dai.node.ObjectTracker)

object_tracker.setDetectionLabelsToTrack([0, 1, 2])

object_tracker.setTrackerType(tracker_type)

object_tracker.setTrackerIdAssignmentPolicy(dai.TrackerIdAssignmentPolicy.UNIQUE_ID)

object_tracker.inputTrackerFrame.setBlocking(True)

object_tracker.inputDetectionFrame.setBlocking(True)

object_tracker.inputDetections.setBlocking(True)

object_tracker.setStreamName("tracklets")

Defining XLinkOut Node

xlink_out = self.pipeline.create(dai.node.XLinkOut)
xlink_out.setStreamName("outFrame")

Linking Frame

source_node.preview.link(manip_node.inputImage)

manip_node.out.link(detector_network.input)

detector_network.input.setBlocking(False)

detector_network.out.link(object_tracker.inputDetections)

detector_network.passthrough.link(object_tracker.inputDetectionFrame)

source_node.video.link(object_tracker.inputTrackerFrame)

object_tracker.out.link(tracker_out.input)

object_tracker.passthroughTrackerFrame.link(xlink_out.input)

Execute Pipeline

pipeline = dai.Pipeline()

with dai.Device(pipeline=pipeline ) as device: 
     q_frame_out = device.getOutputQueue(name="outFrame", maxSize=4, blocking=False)
     q_tracklets = device.getOutputQueue(name="tracklets", maxSize=4)
     While True:
       track = q_tracklets.get() 
       track_frame = q_frame_out.get()     
       tracklets_data = track.tracklets 
       tracker_frame = track_frame.getCvFrame()

I am not getting anything from tracklets_data. Its returning empty list.

erik

Hi @AmunTimilsina , please provide the full MRE. Zip everything and upload to eg. google drive, and share the link.