L
li_you_chen

  • 6 days ago
  • Joined Mar 31, 2024
  • 1 best answer
  • @li_you_chen this isn't reproducible, as I do not have your model. Please upload whole zip to drive and share the link.

  • @li_you_chen please provide minimal repro example that's formatted correctly (use ```)

  • @li_you_chen this is because you're doing inference on 320x320 frames (1:1 aspect ratio) and drawing bounding boxes on full 640x400 frame (not 1:1 aspect ratio). Perhaps crop 640x400 into 400x400 (so 120 pixels on left/right), or just visualise boxes on imagemanip output frame (320x320).

  • @li_you_chen ,

    objectTracker.passthroughTrackerFrame.link(trackerOut.input)
    objectTracker.out.link(trackerOut.input)

    So you are sending 2 different messages to the same queue - likely this is the problem.

  • Hi @li_you_chen
    If you get no tracklets, the problem is likely the detection part. Please create another XLINKOUT node and pass detectionNetwork.out to it, and print it on the host. This will tell you if detections are made. Then we can debug further.

    Thanks,
    Jaka

  • Hi @li_you_chen
    We have some here but we ditched MBNet a while ago and mostly use YOLO now. If you wish you can train YOLO as well.

    Thanks,
    Jaka

  • Hi @li_you_chen
    Yes, of course you can. The model may be different but the on-device decoding is done in a way that ensures the output structure is the same, so they can both be linked to an object tracker.

    If you train your own mobilenet model, the example you sent should be working without any changes (well, except for the model path).

    Thanks,
    Jaka

  • Hi @li_you_chen

    li_you_chen full of wrong boundingbox , even I raise threshold

    This suggest a problem with the model. Did you test it off depthai?

    Thanks,
    Jaka

  • Hi @li_you_chen
    This is the exact same script as tiny_yolo in the examples. I only changed the model to catdog and the label names respectively. Can you recheck please?

    Thanks
    Jaka

  • Hi @li_you_chen
    No, it's yolo. I'm just saying you can not use a yolo blob inside MobileNetDetectionNetwork.

    #!/usr/bin/env python3
    
    """
    The code is the same as for Tiny Yolo V3 and V4, the only difference is the blob file
    - Tiny YOLOv3: https://github.com/david8862/keras-YOLOv3-model-set
    - Tiny YOLOv4: https://github.com/TNTWEN/OpenVINO-YOLOV4
    """
    
    from pathlib import Path
    import sys
    import cv2
    import depthai as dai
    import numpy as np
    import time
    
    # tiny yolo v4 label texts
    labelMap = [
        "cat", "dog"
    ]
    
    syncNN = True
    
    # Create pipeline
    pipeline = dai.Pipeline()
    
    # Define sources and outputs
    camRgb = pipeline.create(dai.node.ColorCamera)
    detectionNetwork = pipeline.create(dai.node.YoloDetectionNetwork)
    xoutRgb = pipeline.create(dai.node.XLinkOut)
    nnOut = pipeline.create(dai.node.XLinkOut)
    
    xoutRgb.setStreamName("rgb")
    nnOut.setStreamName("nn")
    
    # Properties
    camRgb.setPreviewSize(320, 320)
    camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
    camRgb.setInterleaved(False)
    camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)
    camRgb.setFps(40)
    
    # Network specific settings
    detectionNetwork.setConfidenceThreshold(0.5)
    detectionNetwork.setNumClasses(2)
    detectionNetwork.setIouThreshold(.5)
    detectionNetwork.setBlobPath("catdog320.blob")
    detectionNetwork.setNumInferenceThreads(2)
    detectionNetwork.input.setBlocking(False)
    
    # Linking
    camRgb.preview.link(detectionNetwork.input)
    if syncNN:
        detectionNetwork.passthrough.link(xoutRgb.input)
    else:
        camRgb.preview.link(xoutRgb.input)
    
    detectionNetwork.out.link(nnOut.input)
    
    # Connect to device and start pipeline
    with dai.Device(pipeline) as device:
    
        # Output queues will be used to get the rgb frames and nn data from the outputs defined above
        qRgb = device.getOutputQueue(name="rgb", maxSize=4, blocking=False)
        qDet = device.getOutputQueue(name="nn", maxSize=4, blocking=False)
    
        frame = None
        detections = []
        startTime = time.monotonic()
        counter = 0
        color2 = (255, 255, 255)
    
        # nn data, being the bounding box locations, are in <0..1> range - they need to be normalized with frame width/height
        def frameNorm(frame, bbox):
            normVals = np.full(len(bbox), frame.shape[0])
            normVals[::2] = frame.shape[1]
            return (np.clip(np.array(bbox), 0, 1) * normVals).astype(int)
    
        def displayFrame(name, frame):
            color = (255, 0, 0)
            for detection in detections:
                bbox = frameNorm(frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))
                cv2.putText(frame, labelMap[detection.label], (bbox[0] + 10, bbox[1] + 20), cv2.FONT_HERSHEY_TRIPLEX, 0.5, 255)
                cv2.putText(frame, f"{int(detection.confidence * 100)}%", (bbox[0] + 10, bbox[1] + 40), cv2.FONT_HERSHEY_TRIPLEX, 0.5, 255)
                cv2.rectangle(frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), color, 2)
            # Show the frame
            cv2.imshow(name, frame)
    
        while True:
            if syncNN:
                inRgb = qRgb.get()
                inDet = qDet.get()
            else:
                inRgb = qRgb.tryGet()
                inDet = qDet.tryGet()
    
            if inRgb is not None:
                frame = inRgb.getCvFrame()
                cv2.putText(frame, "NN fps: {:.2f}".format(counter / (time.monotonic() - startTime)),
                            (2, frame.shape[0] - 4), cv2.FONT_HERSHEY_TRIPLEX, 0.4, color2)
    
            if inDet is not None:
                detections = inDet.detections
                counter += 1
    
            if frame is not None:
                displayFrame("rgb", frame)
    
            if cv2.waitKey(1) == ord('q'):
                break

    Does this work for you? Cause it doesn't for me and I think the model is not created correctly.

    Thanks ,
    Jaka

  • Hi @li_you_chen
    I tested it on images of cats and dogs, with the swapped model inside this example: https://docs.luxonis.com/projects/api/en/latest/samples/Yolo/tiny_yolo/

    Could you post the output of detections:

    inDet = qDet.tryGet()
    
    if inDet is not None:
                detections = inDet.detections
                print(detections)

    You can't interchange Yolo blobs and Mbnet, they have completely different decoding functions.

    Thanks,
    Jaka

  • Hi @li_you_chen
    I tested with the model and I am getting no detections (to pictures of cats and dogs). I looks like detection problem.

  • Hi @li_you_chen
    Hmm, could you send the model over, along with MRE so we can check please?
    Try changing the IOU threshold and the confidence to see if you get any improvements in detections.
    Also important make sure you update to the latest depthai version (2.25.1).

    Thanks,
    Jaka

  • Hi @li_you_chen
    If you train with yolov8, it won't work on mobilenetSSD. Here should be a more up to date yolov8 network notebook which you can use directly.

    Thanks,
    Jaka