MobileNetDetection Node synch question

TylerD

If you retrieve detections and passthrough from MobileNetDetection node queues at the same time are they guaranteed to be synchronized ? (i.e. detections were generated by image retrieved from the passthrough queue) I am saving passthrough images after drawing their detection boxes. I have notice when the target is moving I will occasionally get images saved that do not have any detectable objects in them, yet bounding boxes are drawn as if there were. This leads me to believe that the passthrough image is not the same one that the detections were generated with. I read from the detection queue and on the next line I read from the passthrough queue (as close to the same time as possible).

erik

Hello TylerD , yes, if you get a new detection you will always get a new passthrough frame as well. Could you share the pipeline, so we can debug this?
Thanks, Erik

TylerD

This is a modified version of the license plate OCR example. I'm reading producer id "tattoos" placed on pigs that are processed at the facility where I work. I use one MobileNet to detect the tattoo and pass the roi to another MobileNet that locates the individual characters and sorts them left to right. Works great on still images, but moving object are where I experience the problem. `import argparse
import numpy as np # numpy - manipulate the packet data returned by depthai
import cv2 # opencv - display the video stream
import depthai # depthai - access the camera and its data packets
from pathlib import Path
import os # os for file operations
import time
import threading
import PyPigVision as ppv
from PyPigVision import file_handling as fh
from PyPigVision import tattoo_handling as th
from pylogix import PLC

parser = argparse.ArgumentParser()
parser.add_argument('-cam', '--camera', action="store_true",
help="Use DepthAI 4K RGB camera for inference (conflicts with -vid)")
parser.add_argument('-vid', '--video', type=str,
help="Path to video file to be used for inference (conflicts with -cam)")
parser.add_argument('-rot', '--rotate', action="store_true",
help="Rotates video 90° so camera can be mounted normally and accomodate a network that is trained with horizontal images.")
parser.add_argument('-save', '--Save_Data', type=str,
help="Write data to file")
parser.add_argument('-plc', '--Use_PLC', action="store_true",
help="Write data to PLC")
parser.add_argument('-ocr_box', '--display_OCR_bboxes', action="store_true",
help="Display OCR images with bboxes (creates images that cant be used for training)")
args = parser.parse_args()

if not args.camera and not args.video:
raise RuntimeError(
"No source selected. Use either \"-cam\" to run on RGB camera as a source or \"-vid <path>\" to run on video"
)

Handle frames per second

class FPSHandler:
def init(self, cap=None):
self.timestamp = time.time()
self.start = time.time()
self.framerate = cap.get(cv2.CAP_PROP_FPS) if cap is not None else None

    self.frame_cnt = 0
    self.ticks = {}
    self.ticks_cnt = {}

def next_iter(self):
    if not args.camera:
        frame_delay = 1.0 / self.framerate
        delay = (self.timestamp + frame_delay) - time.time()
        if delay > 0:
            time.sleep(delay)
    self.timestamp = time.time()
    self.frame_cnt += 1

def tick(self, name):
    if name in self.ticks:
        self.ticks_cnt[name] += 1
    else:
        self.ticks[name] = time.time()
        self.ticks_cnt[name] = 0

def tick_fps(self, name):
    if name in self.ticks:
        if (time.time() - self.ticks[name]) > 0:
            return self.ticks_cnt[name] / (time.time() - self.ticks[name])
        else:
            return 0
    else:
        return 0

def fps(self):
    if (self.timestamp - self.start) > 0:
        return self.frame_cnt / (self.timestamp - self.start)
    else:
        return 0

def frameNorm(frame, bbox):
normVals = np.full(len(bbox), frame.shape[0])
normVals[::2] = frame.shape[1]
return (np.clip(np.array(bbox), 0, 1) * normVals).astype(int)

def to_planar(arr: np.ndarray, shape: tuple) -> list: # Could possibly be how you have to load images into nn
return cv2.resize(arr, shape).transpose(2, 0, 1).flatten()

Handles loading images into a frame sequence queue dictionary

def get_frame():
global frame_det_seq

if args.video:
    read_correctly, frame = cap.read()
    if read_correctly:
        frame_seq_map[frame_det_seq] = frame
        frame_det_seq += 1
    return read_correctly, frame
else:
    in_rgb = cam_out.get()
    frame = in_rgb.getCvFrame()
    frame_seq_map[in_rgb.getSequenceNum()] = frame

    return True, frame

if args.camera:
fps = FPSHandler()
else:
cap = cv2.VideoCapture(str(Path(args.video).resolve().absolute()))
fps = FPSHandler(cap)

define image storage variables

frame_seq_map = {}
frame_det_seq = 0
tattoo_detections = []
rec_results = []
tat_last_seq = 0
tat_last_img = np.empty([300, 300, 3])
labels = {1: '0', 2: '1', 3: '2', 4: '3', 5: '4', 6: '6', 7: '7', 8: '8', 9: '9', 10: 'X'}
decoded_text = ""
running = True
carc_dict = {}
found = 0
read = 0
offset = 20
chars_thresh = 4

pipeline = depthai.Pipeline() # create blank pipeline

if args.camera:
cam_rgb = pipeline.create(depthai.node.ColorCamera) # create color camera object
cam_rgb.setPreviewSize(576, 576) # set camera preview size
cam_rgb.setInterleaved(False)
cam_rgb.initialControl.setManualFocus(110)
cam_rgb.initialControl.setAutoWhiteBalanceMode(depthai.CameraControl.AutoWhiteBalanceMode.AUTO)

--DETECTION--

det_nn = pipeline.createMobileNetDetectionNetwork() # create tattoo detection mobilenet network
det_nn.setBlobPath("C:\Luxonis\DETECT_BLOBS\Detect_2_17_2022.blob") # configure path to blob
det_nn.setConfidenceThreshold(0.5) # set confidence threshold
det_nn.input.setQueueSize(1)
det_nn.input.setBlocking(False)

if args.rotate:

if args.camera:
    manipRgb = pipeline.createImageManip()
    rgbRr = depthai.RotatedRect()
    rgbRr.center.x, rgbRr.center.y = cam_rgb.getPreviewWidth() // 2, cam_rgb.getPreviewHeight() // 2
    rgbRr.size.width, rgbRr.size.height = cam_rgb.getPreviewHeight(), cam_rgb.getPreviewWidth()
    rgbRr.angle = 90
    manipRgb.initialConfig.setCropRotatedRect(rgbRr, False)
    cam_rgb.preview.link(manipRgb.inputImage)

    # Resize camera preview and map it to tattoo detection nn input
    manip = pipeline.createImageManip()
    manip.initialConfig.setResize(300, 300)
    manip.initialConfig.setFrameType(depthai.RawImgFrame.Type.RGB888p)
    manipRgb.out.link(manip.inputImage)
    manip.out.link(det_nn.input)

    # Create output que for rgb camera images
    cam_xout = pipeline.createXLinkOut()
    cam_xout.setStreamName("cam_out")
    manipRgb.out.link(cam_xout.input)
else:
    det_xin = pipeline.createXLinkIn()
    det_xin.setStreamName("det_in")
    det_xin.out.link(det_nn.input)

else:

if args.camera:
    # Resize camera preview and map it to tattoo detection nn input
    manip = pipeline.createImageManip()
    manip.initialConfig.setResize(300, 300)
    manip.initialConfig.setFrameType(depthai.RawImgFrame.Type.RGB888p)
    cam_rgb.preview.link(manip.inputImage)
    manip.out.link(det_nn.input)

    # Create output que for rgb camera images
    cam_xout = pipeline.createXLinkOut()
    cam_xout.setStreamName("cam_out")
    cam_rgb.preview.link(cam_xout.input)
else:
    det_xin = pipeline.createXLinkIn()
    det_xin.setStreamName("det_in")
    det_xin.out.link(det_nn.input)

--OCR--

rec_nn = pipeline.createMobileNetDetectionNetwork() # create tattoo ocr mobilenet network
rec_nn.setBlobPath("C:\Luxonis\READ_BLOBS\read_2_16_2022.blob") # configure path to blob
rec_nn.setConfidenceThreshold(0.4) # set confidence threshold
rec_nn.input.setQueueSize(1)
rec_nn.input.setBlocking(False)

rec_xin = pipeline.createXLinkIn()
rec_xin.setStreamName("rec_in")
rec_xin.out.link(rec_nn.input)

Create output queue for tattoo detection nn detections

det_nn_xout = pipeline.createXLinkOut()
det_nn_xout.setStreamName("det_nn")
det_nn.out.link(det_nn_xout.input)

Create output queue for tattoo detection nn image passthrough

det_pass = pipeline.createXLinkOut()
det_pass.setStreamName("det_pass")
det_nn.passthrough.link(det_pass.input)

Create output queue for tattoo ocr nn detections

rec_xout = pipeline.createXLinkOut()
rec_xout.setStreamName("rec_nn")
rec_nn.out.link(rec_xout.input)

Create output queue for tattoo ocr nn image passthrough

rec_pass = pipeline.createXLinkOut()
rec_pass.setStreamName("rec_pass")
rec_nn.passthrough.link(rec_pass.input)

Unloads tattoo detection nn output q and loads results into OCR nn input

def detect_thread(det_queue, det_pass, rec_queue):
global tattoo_detections, tat_last_seq, tat_last_img, found

while running:
    try:

        in_det = det_queue.get().detections
        in_pass = det_pass.get()
        orig_frame = frame_seq_map.get(in_pass.getSequenceNum(), None)

        tat_last_img = orig_frame

        if orig_frame is None:
            continue

        tat_last_seq = in_pass.getSequenceNum()
        tattoo_detections = in_det

        for detection in tattoo_detections:
            bbox = frameNorm(orig_frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))
            cropped_frame = orig_frame[bbox[1] - offset:bbox[3] + offset, bbox[0] - offset:bbox[2] + offset]
            found += 1
            shape = cropped_frame.shape

            if shape[0] > 0 and shape[1] > 0:
                tstamp = time.monotonic()
                img = depthai.ImgFrame()
                img.setTimestamp(tstamp)
                img.setType(depthai.RawImgFrame.Type.BGR888p)
                img.setData(to_planar(cropped_frame, (300, 300)))
                img.setWidth(300)
                img.setHeight(300)
                carc_dict[img.getSequenceNum()] = [bbox, orig_frame]
                rec_queue.send(img)

        fps.tick('detect')

    except RuntimeError:
        continue

Loads cropped tattoo images from queue loaded by the tattoo detection nn

def rec_thread(q_rec, q_pass):
global rec_results, decoded_text, read

with PLC('10.166.137.120') as comm:

    box_color_rec = (205, 0, 0)

    while running:

        try:
            # Get detections from queue of cropped frames from tattoo detection nn
            rec_data = q_rec.get().detections
            rec_frame = q_pass.get().getCvFrame()
            seq = q_pass.get().getSequenceNum()

            char_detections = [detection for detection in rec_data]

        except RuntimeError:
            continue

        # Get top four characters
        if len(char_detections) >= chars_thresh:
            raw_results = char_detections[:4]

            # Declare storage variables
            Xmin_Char = []
            ocr = []
            decoded_text = ''
            frame_copy = rec_frame.copy()

            # Create list of detections xmin position and detection label
            for detection in raw_results:
                Xmin_Char.append([detection.xmin, labels[detection.label]])
                bbox = frameNorm(frame_copy, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))
                ocr.append([bbox, labels[detection.label]])

                if args.display_OCR_bboxes:
                    cv2.rectangle(frame_copy, (bbox[0], bbox[1]), (bbox[2], bbox[3]), box_color_rec, 2)
                    cv2.putText(frame_copy, '{} ({}%)'.format(labels[detection.label], int(detection.confidence * 100)), (bbox[0] - 10, bbox[1] - 20), cv2.FONT_HERSHEY_TRIPLEX, 0.4, box_color_rec)

            # Sort previously created list by xmin position to provide left to right decoded text
            for detection in sorted(Xmin_Char, reverse=False):
                decoded_text += detection[1]

            # Create result image to stack
            rec_results = [(cv2.resize(frame_copy, (300, 300)), decoded_text)] + rec_results[:9]

            # Extract image and annotation information to save
            current_ocr_bbox_label = ocr
            current_ocr_img = frame_copy

        # Retrieve detection image and bounding box corresponding to current ocr image
        current_detect_data = carc_dict.get(seq)

        # Delete dictionary entry containing detection image and bounding box once they have been extracted
        if current_detect_data is not None:
            current_detect_bbox = current_detect_data[0]
            current_detect_img = current_detect_data[1]
            current_detect_bbox_label = [[current_detect_data[0], 'tattoo']]
            del carc_dict[seq]

            # Save files
            try:

                if len(char_detections) >= chars_thresh:
                    read += 1
                    if args.Save_Data is not None:
                        if os.path.isdir(args.Save_Data):
                            uid = fh.save_files(args.Save_Data, decoded_text, current_detect_bbox_label, current_ocr_bbox_label, current_detect_img, current_ocr_img)
                            if args.Use_PLC:
                                filtered_result = th.cond_tattoo(decoded_text)
                                request = [("Program:P01_MainPeriodicProgram.DINT_HotTattoo", filtered_result), ("Program:P01_MainPeriodicProgram.Tattoo_UID", int(uid))]
                                w_ret = comm.Write(request)
            except Exception as e:
                print(e)

        fps.tick('OCR')

intialize device with context manager using created pipeline

with depthai.Device(pipeline) as device:
if args.camera:
cam_out = device.getOutputQueue("cam_out", 1, True)
else:
det_in = device.getInputQueue("det_in")

rec_in = device.getInputQueue("rec_in")
det_nn = device.getOutputQueue("det_nn", 1, False)
det_pass = device.getOutputQueue("det_pass", 1, False)
rec_nn = device.getOutputQueue("rec_nn", 1, False)
rec_pass = device.getOutputQueue("rec_pass", 1, False)

# Start tattoo detection thread
det_t = threading.Thread(target=detect_thread, args=(det_nn, det_pass, rec_in))
det_t.start()

# Start tattoo ocr thread
rec_t = threading.Thread(target=rec_thread, args=(rec_nn, rec_pass))
rec_t.start()


def should_run():
    return cap.isOpened() if args.video else True

# Main loop
try:
    fps_rgb = 0
    ticks = 0
    start = time.time()

    # start main loop
    while should_run():

        read_correctly, frame = get_frame()  # Load frame from camera into queue

        if not read_correctly:  # Could be used later if video is incorporated
            break

        for map_key in list(filter(lambda item: item <= tat_last_seq, frame_seq_map.keys())):
            del frame_seq_map[map_key]

        fps.next_iter()  # Increment camera fps

        if not args.camera:
            if args.rotate:
                frame = cv2.rotate(frame, cv2.cv2.ROTATE_90_CLOCKWISE)

            tstamp = time.monotonic()
            tat_frame = depthai.ImgFrame()
            tat_frame.setData(to_planar(frame, (300, 300)))
            tat_frame.setTimestamp(tstamp)
            tat_frame.setSequenceNum(frame_det_seq)
            tat_frame.setType(depthai.RawImgFrame.Type.BGR888p)
            tat_frame.setWidth(300)
            tat_frame.setHeight(300)
            det_in.send(tat_frame)

        if tat_last_img is None:
            debug_frame = frame.copy()  # Copy frame to manipulate
        else:
            debug_frame = tat_last_img.copy()

        debug_frame = cv2.resize(debug_frame, (700, 900))

        box_color = (0, 0, 255)
        text_color = (0, 255, 255)

        for detection in tattoo_detections:  # Loop tattoo detections in current frame
            bbox = frameNorm(debug_frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))  # Normalize bounding box for current frame size
            cv2.rectangle(debug_frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), box_color, 2)  # Draw bounding box on current frame
            cv2.putText(debug_frame, '{} ({}%)'.format('tattoo', int(detection.confidence * 100)), (bbox[0] - 10, bbox[1] - 20), cv2.FONT_HERSHEY_TRIPLEX, 0.5, box_color)

        if ticks < 5:
            ticks += 1
        else:
            if time.time() - start > 0:
                fps_rgb = round(ticks * (1 / (time.time() - start)), 1)
            ticks = 0
            start = time.time()

        cv2.putText(debug_frame, f"RGB FPS: {round(fps.fps(), 1)}", (5, 15), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color)  # Display RGB fps on screen
        cv2.putText(debug_frame, f"DETECT FPS:  {round(fps.tick_fps('detect'), 1)}", (5, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color)  # Display Detection fps on screen
        cv2.putText(debug_frame, f"OCR FPS:  {round(fps.tick_fps('OCR'), 1)}", (5, 45), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color)  # Display OCR fps on screen
        cv2.putText(debug_frame, f"FOUND:  {found}", (5, 60), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color)  # Display tattoos found count
        cv2.putText(debug_frame, f"READ:  {read}", (5, 75), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color)  # Display tattoos read count
        cv2.imshow("rgb", debug_frame)  # Display main detection frame

        rec_stacked = None  # Create variable to house ocr result que

        for rec_img, rec_text in rec_results:  # Loop current tattoo ocr result queue
            rec_placeholder_img = np.zeros((300, 70, 3), np.uint8)
            cv2.putText(rec_placeholder_img, rec_text, (5, 25), cv2.FONT_HERSHEY_TRIPLEX, 0.5, (0, 255, 0))
            rec_combined = np.hstack((rec_img, rec_placeholder_img))

            if rec_stacked is None:
                rec_stacked = rec_combined
            else:
                rec_stacked = np.vstack((rec_stacked, rec_combined))

        if rec_stacked is not None:  # If result queue has contents, display them
            cv2.imshow("Recognized tattoos", rec_stacked)

        key = cv2.waitKey(1)
        if key == ord('q'):
            break

except KeyboardInterrupt:
    pass

running = False

det_t.join()
rec_t.join()
print("FPS: {:.2f}".format(fps.fps()))
`

erik

Hello TylerD , could you rather provide minimal reproducible code? As the code above is quite domain-specific, not purely issue related.
Thanks, Erik

TylerD

Sorry about that...here is the pipeline setup and threads used. The problem with detections and passthroughs not seeming to be synchronized happens with both MobileNets.

pipeline = depthai.Pipeline()

cam_rgb = pipeline.create(depthai.node.ColorCamera)  # create color camera object
cam_rgb.setPreviewSize(576, 576)  # set camera preview size
cam_rgb.setInterleaved(False)
cam_rgb.initialControl.setManualFocus(110)

det_nn = pipeline.createMobileNetDetectionNetwork()  # create tattoo detection mobilenet network
det_nn.setBlobPath("C:\\Luxonis\\DETECT_BLOBS\\Detect_2_17_2022.blob")  # configure path to blob
det_nn.setConfidenceThreshold(0.5)  # set confidence threshold
det_nn.input.setQueueSize(1)
det_nn.input.setBlocking(False)

manipRgb = pipeline.createImageManip()
rgbRr = depthai.RotatedRect()
rgbRr.center.x, rgbRr.center.y = cam_rgb.getPreviewWidth() // 2, cam_rgb.getPreviewHeight() // 2
rgbRr.size.width, rgbRr.size.height = cam_rgb.getPreviewHeight(), cam_rgb.getPreviewWidth()
rgbRr.angle = 90
manipRgb.initialConfig.setCropRotatedRect(rgbRr, False)
cam_rgb.preview.link(manipRgb.inputImage)

manip = pipeline.createImageManip()
manip.initialConfig.setResize(300, 300)
manip.initialConfig.setFrameType(depthai.RawImgFrame.Type.RGB888p)
manipRgb.out.link(manip.inputImage)
manip.out.link(det_nn.input)

cam_xout = pipeline.createXLinkOut()
cam_xout.setStreamName("cam_out")
manipRgb.out.link(cam_xout.input)

rec_nn = pipeline.createMobileNetDetectionNetwork()  # create tattoo ocr mobilenet network
rec_nn.setBlobPath("C:\\Luxonis\\READ_BLOBS\\read_2_16_2022.blob")  # configure path to blob
rec_nn.setConfidenceThreshold(0.4)  # set confidence threshold
rec_nn.input.setQueueSize(1)
rec_nn.input.setBlocking(False)

rec_xin = pipeline.createXLinkIn()
rec_xin.setStreamName("rec_in")
rec_xin.out.link(rec_nn.input)

det_nn_xout = pipeline.createXLinkOut()
det_nn_xout.setStreamName("det_nn")
det_nn.out.link(det_nn_xout.input)

det_pass = pipeline.createXLinkOut()
det_pass.setStreamName("det_pass")
det_nn.passthrough.link(det_pass.input)

rec_xout = pipeline.createXLinkOut()
rec_xout.setStreamName("rec_nn")
rec_nn.out.link(rec_xout.input)

rec_pass = pipeline.createXLinkOut()
rec_pass.setStreamName("rec_pass")
rec_nn.passthrough.link(rec_pass.input)


def detect_thread(det_queue, det_pass, rec_queue):
    global tattoo_detections, tat_last_seq, tat_last_img

    while running:
        try:

            in_det = det_queue.get().detections
            in_pass = det_pass.get()
            orig_frame = frame_seq_map.get(in_pass.getSequenceNum(), None)

            tat_last_img = orig_frame

            if orig_frame is None:
                continue

            tat_last_seq = in_pass.getSequenceNum()
            tattoo_detections = in_det

            for detection in tattoo_detections:
                bbox = frameNorm(orig_frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))
                cropped_frame = orig_frame[bbox[1] - offset:bbox[3] + offset, bbox[0] - offset:bbox[2] + offset]
                shape = cropped_frame.shape

                if shape[0] > 0 and shape[1] > 0:
                    tstamp = time.monotonic()
                    img = depthai.ImgFrame()
                    img.setTimestamp(tstamp)
                    img.setType(depthai.RawImgFrame.Type.BGR888p)
                    img.setData(to_planar(cropped_frame, (300, 300)))
                    img.setWidth(300)
                    img.setHeight(300)
                    rec_queue.send(img)

            fps.tick('detect')

        except RuntimeError:
            continue

def rec_thread(q_rec, q_pass):
    global rec_results, decoded_text

    while running:

        try:
            # Get detections from queue of cropped frames from tattoo detection nn
            rec_data = q_rec.get().detections
            rec_frame = q_pass.get().getCvFrame()
            seq = q_pass.get().getSequenceNum()

            char_detections = [detection for detection in rec_data]

        except RuntimeError:
            continue

        decoded_text = ''

        # Create list of detections xmin position and detection label
        for detection in rec_data:

            bbox = frameNorm(rec_frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))
            cv2.rectangle(rec_frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), (205, 0, 0), 2)
            cv2.putText(rec_frame, '{} ({}%)'.format(labels[detection.label], int(detection.confidence * 100)), (bbox[0] - 10, bbox[1] - 20), cv2.FONT_HERSHEY_TRIPLEX, 0.4, (205, 0, 0))

        # Create result image to stack
        rec_results = [(cv2.resize(rec_frame, (300, 300)), decoded_text)] + rec_results[:9]

    fps.tick('OCR')

with depthai.Device(pipeline) as device:

    cam_out = device.getOutputQueue("cam_out", 1, True)
    rec_in = device.getInputQueue("rec_in")
    det_nn = device.getOutputQueue("det_nn", 1, False)
    det_pass = device.getOutputQueue("det_pass", 1, False)
    rec_nn = device.getOutputQueue("rec_nn", 1, False)
    rec_pass = device.getOutputQueue("rec_pass", 1, False)


    det_t = threading.Thread(target=detect_thread, args=(det_nn, det_pass, rec_in))
    det_t.start()

    rec_t = threading.Thread(target=rec_thread, args=(rec_nn, rec_pass))
    rec_t.start()

erik

Hello TylerD , from my understanding of the code you are running first object detection (mobilenet), then crop the image on the host, send it back to the device and run another object detection model. Instead of sending imgs/NN results back to host for cropping, I would suggest using Script node instead (for ImageManipConfigs used for cropping), see demo here.
Thanks, Erik

TylerD

Understood, I'll give that a try.

TylerD

When using the script node it freezes. I used the existing warn diagnostic messages from the demo. The following is the output from the warn diagnostics and error message in the order they occured.

[Script(4)] [warning] Detection rect: 0.34814453125, 0.529296875, 0.45703125, 0.611328125
[Script(4)] [warning] 1 from nn_in: 0.34814453125, 0.529296875, 0.45703125, 0.611328125
[ImageManip(5)] [error] Processing failed, potentially unsupported config.

erik

Hello TylerD , could you share the current script? Do you send high-definition frame to the ImageManip node, together with the cropping config?

TylerD

cam_rgb = pipeline.create(depthai.node.ColorCamera)
cam_rgb.setPreviewSize(300, 300)
cam_rgb.setInterleaved(False)


det_nn = pipeline.createMobileNetDetectionNetwork()
det_nn.setBlobPath("C:\\Luxonis\\DETECT_BLOBS\\Detect_2_17_2022.blob")
det_nn.setConfidenceThreshold(0.5)
det_nn.input.setQueueSize(1)
det_nn.input.setBlocking(False)
cam_rgb.preview.link(det_nn.input)


image_manip_script = pipeline.create(depthai.node.Script)
det_nn.out.link(image_manip_script.inputs['nn_in'])
cam_rgb.preview.link(image_manip_script.inputs['frame'])
image_manip_script.setScript("""
import time
def limit_roi(det):
    if det.xmin <= 0: det.xmin = 0.001
    if det.ymin <= 0: det.ymin = 0.001
    if det.xmax >= 1: det.xmax = 0.999
    if det.ymax >= 1: det.ymax = 0.999
while True:
    frame = node.io['frame'].get()
    tat_dets = node.io['nn_in'].get().detections
    node.warn(f"Tats detected: {len(tat_dets)}")
    for det in tat_dets:
        limit_roi(det)
        node.warn(f"Detection rect: {det.xmin}, {det.ymin}, {det.xmax}, {det.ymax}")
        cfg = ImageManipConfig()
        cfg.setCropRect(det.xmin, det.ymin, det.xmax, det.ymax)
        cfg.setResize(300, 300)
        cfg.setKeepAspectRatio(False)
        node.io['manip_cfg'].send(cfg)
        node.io['manip_img'].send(frame)
        node.warn(f"1 from nn_in: {det.xmin}, {det.ymin}, {det.xmax}, {det.ymax}")
""")


manip_crop = pipeline.create(depthai.node.ImageManip)
image_manip_script.outputs['manip_img'].link(manip_crop.inputImage)
image_manip_script.outputs['manip_cfg'].link(manip_crop.inputConfig)
manip_crop.initialConfig.setResize(300, 300)
manip_crop.inputConfig.setWaitForMessage(True)

ocr_nn = pipeline.createMobileNetDetectionNetwork() 
ocr_nn.setBlobPath("C:\\Luxonis\\READ_BLOBS\\read_2_16_2022.blob") 
ocr_nn.setConfidenceThreshold(0.4)
ocr_nn.input.setQueueSize(1)
ocr_nn.input.setBlocking(False)
manip_crop.out.link(ocr_nn.input)

erik

Could you try setting both Script inputs to blocking=False and QueueSize=1?

image_manip_script.inputs['nn_in'].setBlocking(False)
image_manip_script.inputs['nn_in'].setQueueSize(1)
image_manip_script.inputs['frame'].setBlocking(False)
image_manip_script.inputs['frame'].setQueueSize(1)

Thanks, Erik

AdamPolak

erik

Erik why was this the recommendation here?

I am having an issue with my set up where a lot of frames are being dropped, I assume because some node is taking a while and then frames are being replaced. So when I try to grab synched messages on the host sometimes up to 4 seq numbers will skip.

I wanted to update to setQueueSize(10) on all my nodes and setBlocking(True) .

Is there some queue limit size I should be considering on the device?
it seems like NN's don't have setQueueSize they have setNumPoolFrames, is this equivalent
Does nn nodes have an equivalent for setBlocking?

Are all device nodes default setQueueSize(3) and setBlocking(False) ?

TylerD

Sorry for the delay...I made the recommended changes. I also found I had to replace the .get() and .get().detections with tryGets() and check that detections were not None before proceeding with the ImageManip. I am still get the warning "Processing failed, potentially unsupported config" error, but it doesn't cause a problem anymore. Setting the setKeepAspectRatio(True) stops this error from occurring, but has a negative effect on the second network's performance. The cropped image is being correctly sent to the second MobileNet and it is returning good results. I have a few more things to add, but will be doing more field testing soon. Thanks for the support (and the great product) !!!

erik

Hi AdamPolak ,

No, RAM is the limitation
No, pool is the output, queue is the input, see docs here: https://docs.luxonis.com/projects/api/en/latest/components/device/

Yes:

nn = pipeline.create(dai.node.MobileNetDetectionNetwork)
nn.input.setBlocking(False)

jakaskerl

Hi AdamPolak

AdamPolak Are all device nodes default setQueueSize(3) and setBlocking(False) ?

https://docs.luxonis.com/projects/api/en/latest/components/device/#blocking-behaviour

Though, I have found that on some nodes (object tracker) the default is non blocking.

Thanks,
Jaka