Hello, I'm trying to use my OAK-1 camera to do inference on a video from host. However, it seems that there is a problem with detection, nothing is detected correctly on the video, even if we tried from different hosts and different models. I'm using this [sample code](https://discuss.luxonis.com/d/306-is-it-possible-to-feed-video-stream-from-host-instead-of-sensors-on-board) from the docs. It's like the MobileNet is not synced with the frames, or that there is a problem during the image transfer to the model. We tried to set the model with blocking queue also, but nothing changed. Here is an example of what we get on [this youtube video](https://www.youtube.com/watch?v=MNn9qKG2UFI): [URL=https://i.imgur.com/AoiS1PA.jpg][IMG]https://i.imgur.com/AoiS1PAh.jpg[/IMG][/URL] [URL=https://i.imgur.com/dX1ihBQ.jpg][IMG]https://i.imgur.com/dX1ihBQh.jpg[/IMG][/URL] [URL=https://i.imgur.com/QkFiSqn.jpg][IMG]https://i.imgur.com/QkFiSqnh.jpg[/IMG][/URL] Has anyone any idea on what could be the problem here? We really hope to get more than 30 fps with satisfying detection results on video from host... Thank you, Pierre

Hi @"pierreia"#500 , This is sort of a common issue, the culprit is that NN expects different aspect ratio input (1:1?) than what the frame is (16:9?). We have [docs on this here](https://docs.luxonis.com/projects/api/en/latest/tutorials/maximize_fov/), hopefully these will be useful! Thanks, Erik

Thank you for you answer. I just realize that I put the wrong link in the main topic, so it may not be clear that I'm trying to use MobileNet on a video file from my computer. I'm [using this code](https://docs.luxonis.com/projects/api/en/latest/samples/MobileNet/video_mobilenet/#video-mobilenetssd): ```python #!/usr/bin/env python3 from pathlib import Path import sys import cv2 import depthai as dai import numpy as np from time import monotonic # Get argument first parentDir = Path(__file__).parent nnPath = str((parentDir / Path('../models/mobilenet-ssd_openvino_2021.4_8shave.blob')).resolve().absolute()) videoPath = str((parentDir / Path('../models/construction_vest.mp4')).resolve().absolute()) if len(sys.argv) > 2: nnPath = sys.argv[1] videoPath = sys.argv[2] if not Path(nnPath).exists() or not Path(videoPath).exists(): import sys raise FileNotFoundError(f'Required file/s not found, please run "{sys.executable} install_requirements.py"') # MobilenetSSD label texts labelMap = ["background", "aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable", "dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor"] # Create pipeline pipeline = dai.Pipeline() # Define sources and outputs nn = pipeline.create(dai.node.MobileNetDetectionNetwork) xinFrame = pipeline.create(dai.node.XLinkIn) nnOut = pipeline.create(dai.node.XLinkOut) xinFrame.setStreamName("inFrame") nnOut.setStreamName("nn") # Properties nn.setConfidenceThreshold(0.5) nn.setBlobPath(nnPath) nn.setNumInferenceThreads(2) nn.input.setBlocking(False) # Linking xinFrame.out.link(nn.input) nn.out.link(nnOut.input) # Connect to device and start pipeline with dai.Device(pipeline) as device: # Input queue will be used to send video frames to the device. qIn = device.getInputQueue(name="inFrame") # Output queue will be used to get nn data from the video frames. qDet = device.getOutputQueue(name="nn", maxSize=4, blocking=False) frame = None detections = [] # nn data, being the bounding box locations, are in range - they need to be normalized with frame width/height def frameNorm(frame, bbox): normVals = np.full(len(bbox), frame.shape[0]) normVals[::2] = frame.shape[1] return (np.clip(np.array(bbox), 0, 1) * normVals).astype(int) def to_planar(arr: np.ndarray, shape: tuple) -> np.ndarray: return cv2.resize(arr, shape).transpose(2, 0, 1).flatten() def displayFrame(name, frame): for detection in detections: bbox = frameNorm(frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax)) cv2.putText(frame, labelMap[detection.label], (bbox[0] + 10, bbox[1] + 20), cv2.FONT_HERSHEY_TRIPLEX, 0.5, 255) cv2.putText(frame, f"{int(detection.confidence * 100)}%", (bbox[0] + 10, bbox[1] + 40), cv2.FONT_HERSHEY_TRIPLEX, 0.5, 255) cv2.rectangle(frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), (255, 0, 0), 2) # Show the frame cv2.imshow(name, frame) cap = cv2.VideoCapture(videoPath) while cap.isOpened(): read_correctly, frame = cap.read() if not read_correctly: break img = dai.ImgFrame() img.setData(to_planar(frame, (300, 300))) img.setTimestamp(monotonic()) img.setWidth(300) img.setHeight(300) qIn.send(img) inDet = qDet.tryGet() if inDet is not None: detections = inDet.detections if frame is not None: displayFrame("rgb", frame) if cv2.waitKey(1) == ord('q'): break ``` so I thought that these lines would rescale the frame correctly: ```python img = dai.ImgFrame() img.setData(to_planar(frame, (300, 300))) img.setTimestamp(monotonic()) img.setWidth(300) img.setHeight(300) qIn.send(img) ``` Should I rescale the video before using this script?

I've added some lines to check the frame that was going into the NN, and to ensure the image is resized: ```python img = dai.ImgFrame() resized_planar = to_planar(frame, (300, 300)) img.setType(dai.RawImgFrame.Type.BGR888p) img.setSize(300, 300) img.setTimestamp(monotonic()) img.setFrame(resized_planar) cv2.imshow("Img Frame Output", img.getCvFrame()) qIn.send(img) ``` It seems that the frame is properly sized and the model should be able to detect cars on the image. However, I still got no detections: [URL=https://i.imgur.com/P5u6NTj.jpg][IMG]https://i.imgur.com/P5u6NTjh.jpg[/IMG][/URL] Any idea on what could be the problem?

Hi @"pierreia"#p4683 Interesting. Could you provide a [full MRE please](https://docs.luxonis.com/en/latest/pages/support/#support)? Thanks, Erik

Hi @"erik"#p4694 , here is my MRE. https://filetransfer.io/data-package/oCBNoI32#link I've opened an Issue on Github: https://github.com/luxonis/depthai/issues/787 Thank you, Pierre

Hi @"pierreia"#p4700 , The code looks good to me (normalization of bounding boxes), I think the problem is with the model and/or deocding (so anchors/masks). I would also check for incorrect color order and/or incorrect channel layout ([both mentioned here](https://docs.luxonis.com/en/latest/pages/tutorials/deploying-custom-model/)). Thanks, Erik

@"erik"#p4701 We tried with the latest MobileNet "from_zoo", and a YOLO model, we got the same results. So I'm not sure it comes from the model itself. Regarding the channel layout, since we put the frame though an OpenCv manipulation before passing it to the model, the frame should be in BGR channel layout, which is what we set in the `ImgFrame` . **Update**: it seems to be working with Yolo tiny, despite it is very slow

Hi @"pierreia"#p4702 , could you share the code with the mobilened from blobconverter? THnaks, Erik

Hi @"erik"#p4703 , you can download the code and the mobilenet here: https://filetransfer.io/data-package/oCBNoI32#link

Hi @"pierreia"#p4704 . The only problem in the code above is that it's not synced. You could either sync frames+detections with host-side syncing, or just use passthrough frame like I did below: ``` #!/usr/bin/env python3 from pathlib import Path import sys import cv2 import depthai as dai import numpy as np from time import monotonic import blobconverter # Get argument first nnPath = 'mobilenet-ssd_openvino_2021.4_8shave.blob' videoPath = 'traffic_5mn.mp4' if len(sys.argv) > 2: nnPath = sys.argv[1] videoPath = sys.argv[2] if not Path(nnPath).exists() or not Path(videoPath).exists(): import sys raise FileNotFoundError(f'Required file/s not found, please run "{sys.executable} install_requirements.py"') # MobilenetSSD label texts labelMap = ["background", "aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable", "dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor"] # Create pipeline pipeline = dai.Pipeline() # Define sources and outputs nn = pipeline.create(dai.node.MobileNetDetectionNetwork) xinFrame = pipeline.create(dai.node.XLinkIn) xinFrame.setStreamName("inFrame") xinFrame.out.link(nn.input) # Properties nn.setConfidenceThreshold(0.5) nn.setBlobPath(nnPath) nn.setNumInferenceThreads(2) nn.input.setBlocking(True) # Linking nnOut = pipeline.create(dai.node.XLinkOut) nnOut.setStreamName("nn") nn.out.link(nnOut.input) nnPass = pipeline.create(dai.node.XLinkOut) nnPass.setStreamName("pass") nn.passthrough.link(nnPass.input) # Connect to device and start pipeline with dai.Device(pipeline) as device: # Input queue will be used to send video frames to the device. qIn = device.getInputQueue(name="inFrame") # Output queue will be used to get nn data from the video frames. qDet = device.getOutputQueue(name="nn", maxSize=6, blocking=True) qPass = device.getOutputQueue("pass") frame = None detections = [] # nn data, being the bounding box locations, are in range - they need to be normalized with frame width/height def frameNorm(frame, bbox): normVals = np.full(len(bbox), frame.shape[0]) normVals[::2] = frame.shape[1] return (np.clip(np.array(bbox), 0, 1) * normVals).astype(int) def to_planar(arr: np.ndarray, shape: tuple) -> np.ndarray: return cv2.resize(arr, shape).transpose(2, 0, 1).flatten() def displayFrame(name, frame): for detection in detections: bbox = frameNorm(frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax)) cv2.putText(frame, labelMap[detection.label], (bbox[0] + 10, bbox[1] + 20), cv2.FONT_HERSHEY_TRIPLEX, 0.5, 255) cv2.putText(frame, f"{int(detection.confidence * 100)}%", (bbox[0] + 10, bbox[1] + 40), cv2.FONT_HERSHEY_TRIPLEX, 0.5, 255) cv2.rectangle(frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), (255, 0, 0), 2) # Show the frame cv2.imshow(name, frame) cap = cv2.VideoCapture(videoPath) while cap.isOpened(): read_correctly, frame = cap.read() if not read_correctly: break img = dai.ImgFrame() resized = to_planar(frame, (300, 300)) img.setTimestamp(monotonic()) img.setType(dai.RawImgFrame.Type.BGR888p) img.setSize(300, 300) img.setData(resized) qIn.send(img) inDet = qDet.tryGet() if inDet is not None: detections = inDet.detections frame = qPass.get().getCvFrame() displayFrame("passthrough", frame) if cv2.waitKey(1) == ord('q'): break ```

Detection problem with Mobile Net on video from host

Unknown

@erik

Hi, can you please provide the code to run the inference on a video file using yolov5?

erik

@Unknown perhaps look at
https://docs-old.luxonis.com/projects/sdk/en/latest/features/replaying/
SDK also has good support for YOLO models.

Unknown

Thankyou @erik

There are modules for object Tracking using mobilenet ssd but it did not work well for yolov5.
can you please give module for object tracker using yolov5.

Unknown

@erik

Whether we can run dlib tracker on-device together with yolov5 object detection?

erik

@Unknown it's exactly the same, you can link YoloDetectionNetwork the same way you link MobileNetDetectionNetwork to the ObjectTracker node.

Unknown

@erik okay sure, Thank you.
Can you please let me know about the query given below.
Whether we can run dlib tracker on-device together with yolov5 object detection?

jakaskerl

Unknown
Depthai has proprietary tracking algorithms that can run on RVC. Addding custom tracking (on-device) is not possible on RVC2.
You would have to port the tracking to host side. The rest of the pipeline (yolo and others) should still be run on device.

Thanks,
Jaka

Unknown

Thankyou @jakaskerl

« Previous Page