If you retrieve detections and passthrough from MobileNetDetection node queues at the same time are they guaranteed to be synchronized ? (i.e. detections were generated by image retrieved from the passthrough queue) I am saving passthrough images after drawing their detection boxes. I have notice when the target is moving I will occasionally get images saved that do not have any detectable objects in them, yet bounding boxes are drawn as if there were. This leads me to believe that the passthrough image is not the same one that the detections were generated with. I read from the detection queue and on the next line I read from the passthrough queue (as close to the same time as possible).

  • erik replied to this.

    Hello TylerD , yes, if you get a new detection you will always get a new passthrough frame as well. Could you share the pipeline, so we can debug this?
    Thanks, Erik

    This is a modified version of the license plate OCR example. I'm reading producer id "tattoos" placed on pigs that are processed at the facility where I work. I use one MobileNet to detect the tattoo and pass the roi to another MobileNet that locates the individual characters and sorts them left to right. Works great on still images, but moving object are where I experience the problem. `import argparse
    import numpy as np # numpy - manipulate the packet data returned by depthai
    import cv2 # opencv - display the video stream
    import depthai # depthai - access the camera and its data packets
    from pathlib import Path
    import os # os for file operations
    import time
    import threading
    import PyPigVision as ppv
    from PyPigVision import file_handling as fh
    from PyPigVision import tattoo_handling as th
    from pylogix import PLC

    parser = argparse.ArgumentParser()
    parser.add_argument('-cam', '--camera', action="store_true",
    help="Use DepthAI 4K RGB camera for inference (conflicts with -vid)")
    parser.add_argument('-vid', '--video', type=str,
    help="Path to video file to be used for inference (conflicts with -cam)")
    parser.add_argument('-rot', '--rotate', action="store_true",
    help="Rotates video 90° so camera can be mounted normally and accomodate a network that is trained with horizontal images.")
    parser.add_argument('-save', '--Save_Data', type=str,
    help="Write data to file")
    parser.add_argument('-plc', '--Use_PLC', action="store_true",
    help="Write data to PLC")
    parser.add_argument('-ocr_box', '--display_OCR_bboxes', action="store_true",
    help="Display OCR images with bboxes (creates images that cant be used for training)")
    args = parser.parse_args()

    if not args.camera and not args.video:
    raise RuntimeError(
    "No source selected. Use either \"-cam\" to run on RGB camera as a source or \"-vid <path>\" to run on video"
    )

    Handle frames per second

    class FPSHandler:
    def init(self, cap=None):
    self.timestamp = time.time()
    self.start = time.time()
    self.framerate = cap.get(cv2.CAP_PROP_FPS) if cap is not None else None

        self.frame_cnt = 0
        self.ticks = {}
        self.ticks_cnt = {}
    
    def next_iter(self):
        if not args.camera:
            frame_delay = 1.0 / self.framerate
            delay = (self.timestamp + frame_delay) - time.time()
            if delay > 0:
                time.sleep(delay)
        self.timestamp = time.time()
        self.frame_cnt += 1
    
    def tick(self, name):
        if name in self.ticks:
            self.ticks_cnt[name] += 1
        else:
            self.ticks[name] = time.time()
            self.ticks_cnt[name] = 0
    
    def tick_fps(self, name):
        if name in self.ticks:
            if (time.time() - self.ticks[name]) > 0:
                return self.ticks_cnt[name] / (time.time() - self.ticks[name])
            else:
                return 0
        else:
            return 0
    
    def fps(self):
        if (self.timestamp - self.start) > 0:
            return self.frame_cnt / (self.timestamp - self.start)
        else:
            return 0

    def frameNorm(frame, bbox):
    normVals = np.full(len(bbox), frame.shape[0])
    normVals[::2] = frame.shape[1]
    return (np.clip(np.array(bbox), 0, 1) * normVals).astype(int)

    def to_planar(arr: np.ndarray, shape: tuple) -> list: # Could possibly be how you have to load images into nn
    return cv2.resize(arr, shape).transpose(2, 0, 1).flatten()

    Handles loading images into a frame sequence queue dictionary

    def get_frame():
    global frame_det_seq

    if args.video:
        read_correctly, frame = cap.read()
        if read_correctly:
            frame_seq_map[frame_det_seq] = frame
            frame_det_seq += 1
        return read_correctly, frame
    else:
        in_rgb = cam_out.get()
        frame = in_rgb.getCvFrame()
        frame_seq_map[in_rgb.getSequenceNum()] = frame
    
        return True, frame

    if args.camera:
    fps = FPSHandler()
    else:
    cap = cv2.VideoCapture(str(Path(args.video).resolve().absolute()))
    fps = FPSHandler(cap)

    define image storage variables

    frame_seq_map = {}
    frame_det_seq = 0
    tattoo_detections = []
    rec_results = []
    tat_last_seq = 0
    tat_last_img = np.empty([300, 300, 3])
    labels = {1: '0', 2: '1', 3: '2', 4: '3', 5: '4', 6: '6', 7: '7', 8: '8', 9: '9', 10: 'X'}
    decoded_text = ""
    running = True
    carc_dict = {}
    found = 0
    read = 0
    offset = 20
    chars_thresh = 4

    pipeline = depthai.Pipeline() # create blank pipeline

    if args.camera:
    cam_rgb = pipeline.create(depthai.node.ColorCamera) # create color camera object
    cam_rgb.setPreviewSize(576, 576) # set camera preview size
    cam_rgb.setInterleaved(False)
    cam_rgb.initialControl.setManualFocus(110)
    cam_rgb.initialControl.setAutoWhiteBalanceMode(depthai.CameraControl.AutoWhiteBalanceMode.AUTO)

    --DETECTION--

    det_nn = pipeline.createMobileNetDetectionNetwork() # create tattoo detection mobilenet network
    det_nn.setBlobPath("C:\Luxonis\DETECT_BLOBS\Detect_2_17_2022.blob") # configure path to blob
    det_nn.setConfidenceThreshold(0.5) # set confidence threshold
    det_nn.input.setQueueSize(1)
    det_nn.input.setBlocking(False)

    if args.rotate:

    if args.camera:
        manipRgb = pipeline.createImageManip()
        rgbRr = depthai.RotatedRect()
        rgbRr.center.x, rgbRr.center.y = cam_rgb.getPreviewWidth() // 2, cam_rgb.getPreviewHeight() // 2
        rgbRr.size.width, rgbRr.size.height = cam_rgb.getPreviewHeight(), cam_rgb.getPreviewWidth()
        rgbRr.angle = 90
        manipRgb.initialConfig.setCropRotatedRect(rgbRr, False)
        cam_rgb.preview.link(manipRgb.inputImage)
    
        # Resize camera preview and map it to tattoo detection nn input
        manip = pipeline.createImageManip()
        manip.initialConfig.setResize(300, 300)
        manip.initialConfig.setFrameType(depthai.RawImgFrame.Type.RGB888p)
        manipRgb.out.link(manip.inputImage)
        manip.out.link(det_nn.input)
    
        # Create output que for rgb camera images
        cam_xout = pipeline.createXLinkOut()
        cam_xout.setStreamName("cam_out")
        manipRgb.out.link(cam_xout.input)
    else:
        det_xin = pipeline.createXLinkIn()
        det_xin.setStreamName("det_in")
        det_xin.out.link(det_nn.input)

    else:

    if args.camera:
        # Resize camera preview and map it to tattoo detection nn input
        manip = pipeline.createImageManip()
        manip.initialConfig.setResize(300, 300)
        manip.initialConfig.setFrameType(depthai.RawImgFrame.Type.RGB888p)
        cam_rgb.preview.link(manip.inputImage)
        manip.out.link(det_nn.input)
    
        # Create output que for rgb camera images
        cam_xout = pipeline.createXLinkOut()
        cam_xout.setStreamName("cam_out")
        cam_rgb.preview.link(cam_xout.input)
    else:
        det_xin = pipeline.createXLinkIn()
        det_xin.setStreamName("det_in")
        det_xin.out.link(det_nn.input)

    --OCR--

    rec_nn = pipeline.createMobileNetDetectionNetwork() # create tattoo ocr mobilenet network
    rec_nn.setBlobPath("C:\Luxonis\READ_BLOBS\read_2_16_2022.blob") # configure path to blob
    rec_nn.setConfidenceThreshold(0.4) # set confidence threshold
    rec_nn.input.setQueueSize(1)
    rec_nn.input.setBlocking(False)

    rec_xin = pipeline.createXLinkIn()
    rec_xin.setStreamName("rec_in")
    rec_xin.out.link(rec_nn.input)

    Create output queue for tattoo detection nn detections

    det_nn_xout = pipeline.createXLinkOut()
    det_nn_xout.setStreamName("det_nn")
    det_nn.out.link(det_nn_xout.input)

    Create output queue for tattoo detection nn image passthrough

    det_pass = pipeline.createXLinkOut()
    det_pass.setStreamName("det_pass")
    det_nn.passthrough.link(det_pass.input)

    Create output queue for tattoo ocr nn detections

    rec_xout = pipeline.createXLinkOut()
    rec_xout.setStreamName("rec_nn")
    rec_nn.out.link(rec_xout.input)

    Create output queue for tattoo ocr nn image passthrough

    rec_pass = pipeline.createXLinkOut()
    rec_pass.setStreamName("rec_pass")
    rec_nn.passthrough.link(rec_pass.input)

    Unloads tattoo detection nn output q and loads results into OCR nn input

    def detect_thread(det_queue, det_pass, rec_queue):
    global tattoo_detections, tat_last_seq, tat_last_img, found

    while running:
        try:
    
            in_det = det_queue.get().detections
            in_pass = det_pass.get()
            orig_frame = frame_seq_map.get(in_pass.getSequenceNum(), None)
    
            tat_last_img = orig_frame
    
            if orig_frame is None:
                continue
    
            tat_last_seq = in_pass.getSequenceNum()
            tattoo_detections = in_det
    
            for detection in tattoo_detections:
                bbox = frameNorm(orig_frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))
                cropped_frame = orig_frame[bbox[1] - offset:bbox[3] + offset, bbox[0] - offset:bbox[2] + offset]
                found += 1
                shape = cropped_frame.shape
    
                if shape[0] > 0 and shape[1] > 0:
                    tstamp = time.monotonic()
                    img = depthai.ImgFrame()
                    img.setTimestamp(tstamp)
                    img.setType(depthai.RawImgFrame.Type.BGR888p)
                    img.setData(to_planar(cropped_frame, (300, 300)))
                    img.setWidth(300)
                    img.setHeight(300)
                    carc_dict[img.getSequenceNum()] = [bbox, orig_frame]
                    rec_queue.send(img)
    
            fps.tick('detect')
    
        except RuntimeError:
            continue

    Loads cropped tattoo images from queue loaded by the tattoo detection nn

    def rec_thread(q_rec, q_pass):
    global rec_results, decoded_text, read

    with PLC('10.166.137.120') as comm:
    
        box_color_rec = (205, 0, 0)
    
        while running:
    
            try:
                # Get detections from queue of cropped frames from tattoo detection nn
                rec_data = q_rec.get().detections
                rec_frame = q_pass.get().getCvFrame()
                seq = q_pass.get().getSequenceNum()
    
                char_detections = [detection for detection in rec_data]
    
            except RuntimeError:
                continue
    
            # Get top four characters
            if len(char_detections) >= chars_thresh:
                raw_results = char_detections[:4]
    
                # Declare storage variables
                Xmin_Char = []
                ocr = []
                decoded_text = ''
                frame_copy = rec_frame.copy()
    
                # Create list of detections xmin position and detection label
                for detection in raw_results:
                    Xmin_Char.append([detection.xmin, labels[detection.label]])
                    bbox = frameNorm(frame_copy, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))
                    ocr.append([bbox, labels[detection.label]])
    
                    if args.display_OCR_bboxes:
                        cv2.rectangle(frame_copy, (bbox[0], bbox[1]), (bbox[2], bbox[3]), box_color_rec, 2)
                        cv2.putText(frame_copy, '{} ({}%)'.format(labels[detection.label], int(detection.confidence * 100)), (bbox[0] - 10, bbox[1] - 20), cv2.FONT_HERSHEY_TRIPLEX, 0.4, box_color_rec)
    
                # Sort previously created list by xmin position to provide left to right decoded text
                for detection in sorted(Xmin_Char, reverse=False):
                    decoded_text += detection[1]
    
                # Create result image to stack
                rec_results = [(cv2.resize(frame_copy, (300, 300)), decoded_text)] + rec_results[:9]
    
                # Extract image and annotation information to save
                current_ocr_bbox_label = ocr
                current_ocr_img = frame_copy
    
            # Retrieve detection image and bounding box corresponding to current ocr image
            current_detect_data = carc_dict.get(seq)
    
            # Delete dictionary entry containing detection image and bounding box once they have been extracted
            if current_detect_data is not None:
                current_detect_bbox = current_detect_data[0]
                current_detect_img = current_detect_data[1]
                current_detect_bbox_label = [[current_detect_data[0], 'tattoo']]
                del carc_dict[seq]
    
                # Save files
                try:
    
                    if len(char_detections) >= chars_thresh:
                        read += 1
                        if args.Save_Data is not None:
                            if os.path.isdir(args.Save_Data):
                                uid = fh.save_files(args.Save_Data, decoded_text, current_detect_bbox_label, current_ocr_bbox_label, current_detect_img, current_ocr_img)
                                if args.Use_PLC:
                                    filtered_result = th.cond_tattoo(decoded_text)
                                    request = [("Program:P01_MainPeriodicProgram.DINT_HotTattoo", filtered_result), ("Program:P01_MainPeriodicProgram.Tattoo_UID", int(uid))]
                                    w_ret = comm.Write(request)
                except Exception as e:
                    print(e)
    
            fps.tick('OCR')

    intialize device with context manager using created pipeline

    with depthai.Device(pipeline) as device:
    if args.camera:
    cam_out = device.getOutputQueue("cam_out", 1, True)
    else:
    det_in = device.getInputQueue("det_in")

    rec_in = device.getInputQueue("rec_in")
    det_nn = device.getOutputQueue("det_nn", 1, False)
    det_pass = device.getOutputQueue("det_pass", 1, False)
    rec_nn = device.getOutputQueue("rec_nn", 1, False)
    rec_pass = device.getOutputQueue("rec_pass", 1, False)
    
    # Start tattoo detection thread
    det_t = threading.Thread(target=detect_thread, args=(det_nn, det_pass, rec_in))
    det_t.start()
    
    # Start tattoo ocr thread
    rec_t = threading.Thread(target=rec_thread, args=(rec_nn, rec_pass))
    rec_t.start()
    
    
    def should_run():
        return cap.isOpened() if args.video else True
    
    # Main loop
    try:
        fps_rgb = 0
        ticks = 0
        start = time.time()
    
        # start main loop
        while should_run():
    
            read_correctly, frame = get_frame()  # Load frame from camera into queue
    
            if not read_correctly:  # Could be used later if video is incorporated
                break
    
            for map_key in list(filter(lambda item: item <= tat_last_seq, frame_seq_map.keys())):
                del frame_seq_map[map_key]
    
            fps.next_iter()  # Increment camera fps
    
            if not args.camera:
                if args.rotate:
                    frame = cv2.rotate(frame, cv2.cv2.ROTATE_90_CLOCKWISE)
    
                tstamp = time.monotonic()
                tat_frame = depthai.ImgFrame()
                tat_frame.setData(to_planar(frame, (300, 300)))
                tat_frame.setTimestamp(tstamp)
                tat_frame.setSequenceNum(frame_det_seq)
                tat_frame.setType(depthai.RawImgFrame.Type.BGR888p)
                tat_frame.setWidth(300)
                tat_frame.setHeight(300)
                det_in.send(tat_frame)
    
            if tat_last_img is None:
                debug_frame = frame.copy()  # Copy frame to manipulate
            else:
                debug_frame = tat_last_img.copy()
    
            debug_frame = cv2.resize(debug_frame, (700, 900))
    
            box_color = (0, 0, 255)
            text_color = (0, 255, 255)
    
            for detection in tattoo_detections:  # Loop tattoo detections in current frame
                bbox = frameNorm(debug_frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))  # Normalize bounding box for current frame size
                cv2.rectangle(debug_frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), box_color, 2)  # Draw bounding box on current frame
                cv2.putText(debug_frame, '{} ({}%)'.format('tattoo', int(detection.confidence * 100)), (bbox[0] - 10, bbox[1] - 20), cv2.FONT_HERSHEY_TRIPLEX, 0.5, box_color)
    
            if ticks < 5:
                ticks += 1
            else:
                if time.time() - start > 0:
                    fps_rgb = round(ticks * (1 / (time.time() - start)), 1)
                ticks = 0
                start = time.time()
    
            cv2.putText(debug_frame, f"RGB FPS: {round(fps.fps(), 1)}", (5, 15), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color)  # Display RGB fps on screen
            cv2.putText(debug_frame, f"DETECT FPS:  {round(fps.tick_fps('detect'), 1)}", (5, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color)  # Display Detection fps on screen
            cv2.putText(debug_frame, f"OCR FPS:  {round(fps.tick_fps('OCR'), 1)}", (5, 45), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color)  # Display OCR fps on screen
            cv2.putText(debug_frame, f"FOUND:  {found}", (5, 60), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color)  # Display tattoos found count
            cv2.putText(debug_frame, f"READ:  {read}", (5, 75), cv2.FONT_HERSHEY_SIMPLEX, 0.5, text_color)  # Display tattoos read count
            cv2.imshow("rgb", debug_frame)  # Display main detection frame
    
            rec_stacked = None  # Create variable to house ocr result que
    
            for rec_img, rec_text in rec_results:  # Loop current tattoo ocr result queue
                rec_placeholder_img = np.zeros((300, 70, 3), np.uint8)
                cv2.putText(rec_placeholder_img, rec_text, (5, 25), cv2.FONT_HERSHEY_TRIPLEX, 0.5, (0, 255, 0))
                rec_combined = np.hstack((rec_img, rec_placeholder_img))
    
                if rec_stacked is None:
                    rec_stacked = rec_combined
                else:
                    rec_stacked = np.vstack((rec_stacked, rec_combined))
    
            if rec_stacked is not None:  # If result queue has contents, display them
                cv2.imshow("Recognized tattoos", rec_stacked)
    
            key = cv2.waitKey(1)
            if key == ord('q'):
                break
    
    except KeyboardInterrupt:
        pass
    
    running = False

    det_t.join()
    rec_t.join()
    print("FPS: {:.2f}".format(fps.fps()))
    `

    • erik replied to this.

      Hello TylerD , could you rather provide minimal reproducible code? As the code above is quite domain-specific, not purely issue related.
      Thanks, Erik

      Sorry about that...here is the pipeline setup and threads used. The problem with detections and passthroughs not seeming to be synchronized happens with both MobileNets.

      pipeline = depthai.Pipeline()
      
      cam_rgb = pipeline.create(depthai.node.ColorCamera)  # create color camera object
      cam_rgb.setPreviewSize(576, 576)  # set camera preview size
      cam_rgb.setInterleaved(False)
      cam_rgb.initialControl.setManualFocus(110)
      
      det_nn = pipeline.createMobileNetDetectionNetwork()  # create tattoo detection mobilenet network
      det_nn.setBlobPath("C:\\Luxonis\\DETECT_BLOBS\\Detect_2_17_2022.blob")  # configure path to blob
      det_nn.setConfidenceThreshold(0.5)  # set confidence threshold
      det_nn.input.setQueueSize(1)
      det_nn.input.setBlocking(False)
      
      manipRgb = pipeline.createImageManip()
      rgbRr = depthai.RotatedRect()
      rgbRr.center.x, rgbRr.center.y = cam_rgb.getPreviewWidth() // 2, cam_rgb.getPreviewHeight() // 2
      rgbRr.size.width, rgbRr.size.height = cam_rgb.getPreviewHeight(), cam_rgb.getPreviewWidth()
      rgbRr.angle = 90
      manipRgb.initialConfig.setCropRotatedRect(rgbRr, False)
      cam_rgb.preview.link(manipRgb.inputImage)
      
      manip = pipeline.createImageManip()
      manip.initialConfig.setResize(300, 300)
      manip.initialConfig.setFrameType(depthai.RawImgFrame.Type.RGB888p)
      manipRgb.out.link(manip.inputImage)
      manip.out.link(det_nn.input)
      
      cam_xout = pipeline.createXLinkOut()
      cam_xout.setStreamName("cam_out")
      manipRgb.out.link(cam_xout.input)
      
      rec_nn = pipeline.createMobileNetDetectionNetwork()  # create tattoo ocr mobilenet network
      rec_nn.setBlobPath("C:\\Luxonis\\READ_BLOBS\\read_2_16_2022.blob")  # configure path to blob
      rec_nn.setConfidenceThreshold(0.4)  # set confidence threshold
      rec_nn.input.setQueueSize(1)
      rec_nn.input.setBlocking(False)
      
      rec_xin = pipeline.createXLinkIn()
      rec_xin.setStreamName("rec_in")
      rec_xin.out.link(rec_nn.input)
      
      det_nn_xout = pipeline.createXLinkOut()
      det_nn_xout.setStreamName("det_nn")
      det_nn.out.link(det_nn_xout.input)
      
      det_pass = pipeline.createXLinkOut()
      det_pass.setStreamName("det_pass")
      det_nn.passthrough.link(det_pass.input)
      
      rec_xout = pipeline.createXLinkOut()
      rec_xout.setStreamName("rec_nn")
      rec_nn.out.link(rec_xout.input)
      
      rec_pass = pipeline.createXLinkOut()
      rec_pass.setStreamName("rec_pass")
      rec_nn.passthrough.link(rec_pass.input)
      
      
      def detect_thread(det_queue, det_pass, rec_queue):
          global tattoo_detections, tat_last_seq, tat_last_img
      
          while running:
              try:
      
                  in_det = det_queue.get().detections
                  in_pass = det_pass.get()
                  orig_frame = frame_seq_map.get(in_pass.getSequenceNum(), None)
      
                  tat_last_img = orig_frame
      
                  if orig_frame is None:
                      continue
      
                  tat_last_seq = in_pass.getSequenceNum()
                  tattoo_detections = in_det
      
                  for detection in tattoo_detections:
                      bbox = frameNorm(orig_frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))
                      cropped_frame = orig_frame[bbox[1] - offset:bbox[3] + offset, bbox[0] - offset:bbox[2] + offset]
                      shape = cropped_frame.shape
      
                      if shape[0] > 0 and shape[1] > 0:
                          tstamp = time.monotonic()
                          img = depthai.ImgFrame()
                          img.setTimestamp(tstamp)
                          img.setType(depthai.RawImgFrame.Type.BGR888p)
                          img.setData(to_planar(cropped_frame, (300, 300)))
                          img.setWidth(300)
                          img.setHeight(300)
                          rec_queue.send(img)
      
                  fps.tick('detect')
      
              except RuntimeError:
                  continue
      
      def rec_thread(q_rec, q_pass):
          global rec_results, decoded_text
      
          while running:
      
              try:
                  # Get detections from queue of cropped frames from tattoo detection nn
                  rec_data = q_rec.get().detections
                  rec_frame = q_pass.get().getCvFrame()
                  seq = q_pass.get().getSequenceNum()
      
                  char_detections = [detection for detection in rec_data]
      
              except RuntimeError:
                  continue
      
              decoded_text = ''
      
              # Create list of detections xmin position and detection label
              for detection in rec_data:
      
                  bbox = frameNorm(rec_frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))
                  cv2.rectangle(rec_frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), (205, 0, 0), 2)
                  cv2.putText(rec_frame, '{} ({}%)'.format(labels[detection.label], int(detection.confidence * 100)), (bbox[0] - 10, bbox[1] - 20), cv2.FONT_HERSHEY_TRIPLEX, 0.4, (205, 0, 0))
      
              # Create result image to stack
              rec_results = [(cv2.resize(rec_frame, (300, 300)), decoded_text)] + rec_results[:9]
      
          fps.tick('OCR')
      
      with depthai.Device(pipeline) as device:
      
          cam_out = device.getOutputQueue("cam_out", 1, True)
          rec_in = device.getInputQueue("rec_in")
          det_nn = device.getOutputQueue("det_nn", 1, False)
          det_pass = device.getOutputQueue("det_pass", 1, False)
          rec_nn = device.getOutputQueue("rec_nn", 1, False)
          rec_pass = device.getOutputQueue("rec_pass", 1, False)
      
      
          det_t = threading.Thread(target=detect_thread, args=(det_nn, det_pass, rec_in))
          det_t.start()
      
          rec_t = threading.Thread(target=rec_thread, args=(rec_nn, rec_pass))
          rec_t.start()
      • erik replied to this.

        Hello TylerD , from my understanding of the code you are running first object detection (mobilenet), then crop the image on the host, send it back to the device and run another object detection model. Instead of sending imgs/NN results back to host for cropping, I would suggest using Script node instead (for ImageManipConfigs used for cropping), see demo here.
        Thanks, Erik

        Understood, I'll give that a try.

        When using the script node it freezes. I used the existing warn diagnostic messages from the demo. The following is the output from the warn diagnostics and error message in the order they occured.

        [Script(4)] [warning] Detection rect: 0.34814453125, 0.529296875, 0.45703125, 0.611328125
        [Script(4)] [warning] 1 from nn_in: 0.34814453125, 0.529296875, 0.45703125, 0.611328125
        [ImageManip(5)] [error] Processing failed, potentially unsupported config.

        • erik replied to this.

          Hello TylerD , could you share the current script? Do you send high-definition frame to the ImageManip node, together with the cropping config?

          cam_rgb = pipeline.create(depthai.node.ColorCamera)
          cam_rgb.setPreviewSize(300, 300)
          cam_rgb.setInterleaved(False)
          
          
          det_nn = pipeline.createMobileNetDetectionNetwork()
          det_nn.setBlobPath("C:\\Luxonis\\DETECT_BLOBS\\Detect_2_17_2022.blob")
          det_nn.setConfidenceThreshold(0.5)
          det_nn.input.setQueueSize(1)
          det_nn.input.setBlocking(False)
          cam_rgb.preview.link(det_nn.input)
          
          
          image_manip_script = pipeline.create(depthai.node.Script)
          det_nn.out.link(image_manip_script.inputs['nn_in'])
          cam_rgb.preview.link(image_manip_script.inputs['frame'])
          image_manip_script.setScript("""
          import time
          def limit_roi(det):
              if det.xmin <= 0: det.xmin = 0.001
              if det.ymin <= 0: det.ymin = 0.001
              if det.xmax >= 1: det.xmax = 0.999
              if det.ymax >= 1: det.ymax = 0.999
          while True:
              frame = node.io['frame'].get()
              tat_dets = node.io['nn_in'].get().detections
              node.warn(f"Tats detected: {len(tat_dets)}")
              for det in tat_dets:
                  limit_roi(det)
                  node.warn(f"Detection rect: {det.xmin}, {det.ymin}, {det.xmax}, {det.ymax}")
                  cfg = ImageManipConfig()
                  cfg.setCropRect(det.xmin, det.ymin, det.xmax, det.ymax)
                  cfg.setResize(300, 300)
                  cfg.setKeepAspectRatio(False)
                  node.io['manip_cfg'].send(cfg)
                  node.io['manip_img'].send(frame)
                  node.warn(f"1 from nn_in: {det.xmin}, {det.ymin}, {det.xmax}, {det.ymax}")
          """)
          
          
          manip_crop = pipeline.create(depthai.node.ImageManip)
          image_manip_script.outputs['manip_img'].link(manip_crop.inputImage)
          image_manip_script.outputs['manip_cfg'].link(manip_crop.inputConfig)
          manip_crop.initialConfig.setResize(300, 300)
          manip_crop.inputConfig.setWaitForMessage(True)
          
          ocr_nn = pipeline.createMobileNetDetectionNetwork() 
          ocr_nn.setBlobPath("C:\\Luxonis\\READ_BLOBS\\read_2_16_2022.blob") 
          ocr_nn.setConfidenceThreshold(0.4)
          ocr_nn.input.setQueueSize(1)
          ocr_nn.input.setBlocking(False)
          manip_crop.out.link(ocr_nn.input)

          Could you try setting both Script inputs to blocking=False and QueueSize=1?

          image_manip_script.inputs['nn_in'].setBlocking(False)
          image_manip_script.inputs['nn_in'].setQueueSize(1)
          image_manip_script.inputs['frame'].setBlocking(False)
          image_manip_script.inputs['frame'].setQueueSize(1)

          Thanks, Erik

            Sorry for the delay...I made the recommended changes. I also found I had to replace the .get() and .get().detections with tryGets() and check that detections were not None before proceeding with the ImageManip. I am still get the warning "Processing failed, potentially unsupported config" error, but it doesn't cause a problem anymore. Setting the setKeepAspectRatio(True) stops this error from occurring, but has a negative effect on the second network's performance. The cropped image is being correctly sent to the second MobileNet and it is returning good results. I have a few more things to add, but will be doing more field testing soon. Thanks for the support (and the great product) !!!

            a year later

            erik

            Erik why was this the recommendation here?

            I am having an issue with my set up where a lot of frames are being dropped, I assume because some node is taking a while and then frames are being replaced. So when I try to grab synched messages on the host sometimes up to 4 seq numbers will skip.

            I wanted to update to setQueueSize(10) on all my nodes and setBlocking(True) .

            1. Is there some queue limit size I should be considering on the device?
            2. it seems like NN's don't have setQueueSize they have setNumPoolFrames, is this equivalent
            3. Does nn nodes have an equivalent for setBlocking?

            Are all device nodes default setQueueSize(3) and setBlocking(False) ?

              Hi AdamPolak ,

              1. No, RAM is the limitation
              2. No, pool is the output, queue is the input, see docs here: https://docs.luxonis.com/projects/api/en/latest/components/device/
              3. Yes:
                nn = pipeline.create(dai.node.MobileNetDetectionNetwork)
                nn.input.setBlocking(False)