• DepthAI-v2
  • Image Manip node, incoming configuration messages lagging

Hello. I have a simple scenario where I detect an object in an image on the host and I want to get cropped frames around that object. So I enlarge the bounding box of that object and send it back to the device, to the ImageManip node (I assume the object moves at slow speeds, so adding a small factor of enlargement to the bounding box, it will always we included). What I see is that it takes a considerable amount of frames before the configuration sent is applied.

Another experiment is to just send configurations on keys pressed, still I get some lag there.

Is this expected an expected behaviour?
How can I get around it?

    VladimirosSterzentsenko What I see is that it takes a considerable amount of frames before the configuration sent is applied.

    Can you make a demo of that. I'm not sure what lag you are talking about. Usually there shouldn't be any.

    Thanks,
    Jaka

    Yeah, "lag" maybe is not the proper term here.

    So the idea of the code is that we use an ImageManip node to crop the incoming video frame and when 'p' is pressed, we change the crop parameters and start saving the frames. When I do this, the first 10-20 frames saved are with the initial image manip configuration and after that many frames I start to get crops with the new ImageManip configuration. Is this normal behaviour? Is there a way to get this as low as 1 frame latency?

    • Windows 11
    • OAK-D PRO W POE
    • POE connection (1G cable and switch)
    • Depthai 2.25

    Tried at 15,30,60 fps same results, an offset of 10 (or more) frames before I get the new configuration cropped frames

    import cv2
    import depthai as dai
    import random
    import os
    
    available_devices = dai.Device.getAllAvailableDevices()
    p = dai.Pipeline()
    p.setOpenVINOVersion(dai.OpenVINO.VERSION_2022_1)
    
    camera = p.create(dai.node.ColorCamera)
    camera.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
    camera.setPreviewSize(320, 180)
    camera.setInterleaved(False)
    camera.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
    camera.setFps(60.0)
    
    
    image_manip = p.create(dai.node.ImageManip)
    image_manip.initialConfig.setResize(256,256)
    image_manip.inputConfig.setWaitForMessage(False)
    image_manip.inputConfig.setQueueSize(1)
    image_manip.inputImage.setQueueSize(1)
    image_manip.inputConfig.setBlocking(True)
    
    
    video_encoder = p.create(dai.node.VideoEncoder)
    video_encoder.setProfile(dai.node.VideoEncoder.Properties.Profile.MJPEG)
    
    video_encoder_crop = p.create(dai.node.VideoEncoder)
    video_encoder_crop.setProfile(dai.node.VideoEncoder.Properties.Profile.MJPEG)
    
    camera_xout = p.create(dai.node.XLinkOut)
    camera_xout.setStreamName("rgb_preview")
    
    camera_xout_crop = p.create(dai.node.XLinkOut)
    camera_xout_crop.setStreamName("rgb_crop")
    
    
    camera.video.link(image_manip.inputImage)
    image_manip.out.link(video_encoder_crop.input)
    
    params = p.create(dai.node.XLinkIn)
    params.setStreamName("params")
    
    params.out.link(image_manip.inputConfig)
    
    video_encoder_crop.bitstream.link(camera_xout_crop.input)
    camera.preview.link(camera_xout.input)
    
    device = dai.Device(p)
    
    device.setLogLevel(dai.LogLevel.DEBUG)
    device.setLogOutputLevel(dai.LogLevel.DEBUG)
    
    sz ,blck = 10, False
    
    camera_stream = device.getOutputQueue("rgb_preview", maxSize=sz, blocking=blck)
    camera_stream_cropped = device.getOutputQueue("rgb_crop", maxSize=sz, blocking=blck)
    params_q = device.getInputQueue("params", maxSize=sz, blocking=blck)
    
    path = r"test"
    os.makedirs(path, exist_ok=True)
    counter = 0
    press = False
    while True:
    
        if press:
            frame_cropped = camera_stream_cropped.get().getRaw().data
            image = cv2.imdecode(frame_cropped, cv2.IMREAD_COLOR)
            cv2.imwrite(f"{path}\\{counter}.png", image)
            counter += 1
            if counter > 100:
                break
    
        frame = camera_stream.get().getCvFrame()
        cv2.imshow(f"rgb", frame)
    
        key = cv2.waitKey(1)
        if key == ord('q'):
            break
        elif key == ord('p'):
            press = True
            config = dai.ImageManipConfig()
            rr = dai.RotatedRect()
        
            rr.center = dai.Point2f(random.randint(0, 1920), random.randint(0, 1080))
            rr.size = dai.Size2f(800, 600)
            rr.angle = (random.random() * 2 - 1) * 10
    
            config.setCropRotatedRect(rr, False)
            config.setResize(256,256)
            config.setKeepAspectRatio(False)
            
            params_q.send(config)
    [19443010C1D17E1300] [192.168.1.27] [5.298] [VideoEncoder(3)] [debug] Bitstream frame size: 108544, num frames in pool: 4
    [19443010C1D17E1300] [192.168.1.27] [6.065] [system] [info] Memory Usage - DDR: 73.51 / 333.39 MiB, CMX: 2.49 / 2.50 MiB, LeonOS Heap: 40.53 / 82.31 MiB, LeonRT Heap: 4.04 / 40.50 MiB / NOC ddr: 607 MB/s
    [19443010C1D17E1300] [192.168.1.27] [6.066] [system] [info] Temperatures - Average: 50.08C, CSS: 52.02C, MSS 48.25C, UPA: 49.37C, DSS: 50.70C
    [19443010C1D17E1300] [192.168.1.27] [6.066] [system] [info] Cpu Usage - LeonOS 88.88%, LeonRT: 17.02%
    [19443010C1D17E1300] [192.168.1.27] [7.069] [system] [info] Memory Usage - DDR: 73.51 / 333.39 MiB, CMX: 2.49 / 2.50 MiB, LeonOS Heap: 40.78 / 82.31 MiB, LeonRT Heap: 4.04 / 40.50 MiB / NOC ddr: 2050 MB/s
    [19443010C1D17E1300] [192.168.1.27] [7.070] [system] [info] Temperatures - Average: 50.47C, CSS: 52.02C, MSS 49.37C, UPA: 49.37C, DSS: 51.14C
    [19443010C1D17E1300] [192.168.1.27] [7.070] [system] [info] Cpu Usage - LeonOS 99.48%, LeonRT: 13.62%
    [19443010C1D17E1300] [192.168.1.27] [8.073] [system] [info] Memory Usage - DDR: 73.51 / 333.39 MiB, CMX: 2.49 / 2.50 MiB, LeonOS Heap: 40.78 / 82.31 MiB, LeonRT Heap: 4.04 / 40.50 MiB / NOC ddr: 2023 MB/s
    [19443010C1D17E1300] [192.168.1.27] [8.074] [system] [info] Temperatures - Average: 50.36C, CSS: 52.24C, MSS 49.81C, UPA: 49.37C, DSS: 50.03C
    [19443010C1D17E1300] [192.168.1.27] [8.074] [system] [info] Cpu Usage - LeonOS 99.89%, LeonRT: 13.15%
    [19443010C1D17E1300] [192.168.1.27] [9.077] [system] [info] Memory Usage - DDR: 73.51 / 333.39 MiB, CMX: 2.49 / 2.50 MiB, LeonOS Heap: 40.78 / 82.31 MiB, LeonRT Heap: 4.04 / 40.50 MiB / NOC ddr: 2072 MB/s
    [19443010C1D17E1300] [192.168.1.27] [9.078] [system] [info] Temperatures - Average: 50.53C, CSS: 52.46C, MSS 49.37C, UPA: 50.03C, DSS: 50.25C
    [19443010C1D17E1300] [192.168.1.27] [9.078] [system] [info] Cpu Usage - LeonOS 99.49%, LeonRT: 14.60%
    [19443010C1D17E1300] [192.168.1.27] [10.081] [system] [info] Memory Usage - DDR: 73.51 / 333.39 MiB, CMX: 2.49 / 2.50 MiB, LeonOS Heap: 40.78 / 82.31 MiB, LeonRT Heap: 4.04 / 40.50 MiB / NOC ddr: 2065 MB/s
    [19443010C1D17E1300] [192.168.1.27] [10.081] [system] [info] Temperatures - Average: 50.80C, CSS: 52.90C, MSS 49.59C, UPA: 50.25C, DSS: 50.47C
    [19443010C1D17E1300] [192.168.1.27] [10.081] [system] [info] Cpu Usage - LeonOS 99.45%, LeonRT: 13.62%
    [19443010C1D17E1300] [192.168.1.27] [11.082] [system] [info] Memory Usage - DDR: 73.51 / 333.39 MiB, CMX: 2.49 / 2.50 MiB, LeonOS Heap: 40.78 / 82.31 MiB, LeonRT Heap: 4.04 / 40.50 MiB / NOC ddr: 2042 MB/s
    [19443010C1D17E1300] [192.168.1.27] [11.082] [system] [info] Temperatures - Average: 50.58C, CSS: 52.24C, MSS 49.59C, UPA: 49.59C, DSS: 50.92C
    [19443010C1D17E1300] [192.168.1.27] [11.082] [system] [info] Cpu Usage - LeonOS 99.47%, LeonRT: 15.38%
    [19443010C1D17E1300] [192.168.1.27] [12.089] [system] [info] Memory Usage - DDR: 73.51 / 333.39 MiB, CMX: 2.49 / 2.50 MiB, LeonOS Heap: 40.78 / 82.31 MiB, LeonRT Heap: 4.04 / 40.50 MiB / NOC ddr: 2089 MB/s
    [19443010C1D17E1300] [192.168.1.27] [12.089] [system] [info] Temperatures - Average: 51.30C, CSS: 52.90C, MSS 50.25C, UPA: 50.47C, DSS: 51.58C
    [19443010C1D17E1300] [192.168.1.27] [12.089] [system] [info] Cpu Usage - LeonOS 99.26%, LeonRT: 14.87%
    [19443010C1D17E1300] [192.168.1.27] [13.092] [system] [info] Memory Usage - DDR: 73.51 / 333.39 MiB, CMX: 2.49 / 2.50 MiB, LeonOS Heap: 40.78 / 82.31 MiB, LeonRT Heap: 4.04 / 40.50 MiB / NOC ddr: 2106 MB/s
    [19443010C1D17E1300] [192.168.1.27] [13.093] [system] [info] Temperatures - Average: 51.08C, CSS: 52.68C, MSS 50.25C, UPA: 49.81C, DSS: 51.58C
    [19443010C1D17E1300] [192.168.1.27] [13.093] [system] [info] Cpu Usage - LeonOS 99.61%, LeonRT: 13.96%
    [19443010C1D17E1300] [192.168.1.27] [14.096] [system] [info] Memory Usage - DDR: 73.51 / 333.39 MiB, CMX: 2.49 / 2.50 MiB, LeonOS Heap: 40.78 / 82.31 MiB, LeonRT Heap: 4.04 / 40.50 MiB / NOC ddr: 2077 MB/s
    [19443010C1D17E1300] [192.168.1.27] [14.097] [system] [info] Temperatures - Average: 51.25C, CSS: 52.68C, MSS 49.81C, UPA: 50.92C, DSS: 51.58C
    [19443010C1D17E1300] [192.168.1.27] [14.097] [system] [info] Cpu Usage - LeonOS 99.69%, LeonRT: 14.18%
    [19443010C1D17E1300] [192.168.1.27] [15.100] [system] [info] Memory Usage - DDR: 73.51 / 333.39 MiB, CMX: 2.49 / 2.50 MiB, LeonOS Heap: 40.78 / 82.31 MiB, LeonRT Heap: 4.04 / 40.50 MiB / NOC ddr: 2035 MB/s
    [19443010C1D17E1300] [192.168.1.27] [15.101] [system] [info] Temperatures - Average: 51.08C, CSS: 52.90C, MSS 49.59C, UPA: 50.47C, DSS: 51.36C
    [19443010C1D17E1300] [192.168.1.27] [15.101] [system] [info] Cpu Usage - LeonOS 99.38%, LeonRT: 13.94%
    [19443010C1D17E1300] [192.168.1.27] [16.102] [system] [info] Memory Usage - DDR: 73.51 / 333.39 MiB, CMX: 2.49 / 2.50 MiB, LeonOS Heap: 40.78 / 82.31 MiB, LeonRT Heap: 4.04 / 40.50 MiB / NOC ddr: 2072 MB/s
    [19443010C1D17E1300] [192.168.1.27] [16.103] [system] [info] Temperatures - Average: 51.19C, CSS: 52.68C, MSS 50.25C, UPA: 50.25C, DSS: 51.58C
    [19443010C1D17E1300] [192.168.1.27] [16.103] [system] [info] Cpu Usage - LeonOS 99.74%, LeonRT: 15.25%
    [19443010C1D17E1300] [192.168.1.27] [17.107] [system] [info] Memory Usage - DDR: 73.51 / 333.39 MiB, CMX: 2.49 / 2.50 MiB, LeonOS Heap: 40.78 / 82.31 MiB, LeonRT Heap: 4.04 / 40.50 MiB / NOC ddr: 1944 MB/s
    [19443010C1D17E1300] [192.168.1.27] [17.107] [system] [info] Temperatures - Average: 51.25C, CSS: 52.46C, MSS 50.25C, UPA: 50.47C, DSS: 51.80C
    [19443010C1D17E1300] [192.168.1.27] [17.107] [system] [info] Cpu Usage - LeonOS 99.81%, LeonRT: 16.81%
    [19443010C1D17E1300] [192.168.1.27] [18.111] [system] [info] Memory Usage - DDR: 73.51 / 333.39 MiB, CMX: 2.49 / 2.50 MiB, LeonOS Heap: 40.78 / 82.31 MiB, LeonRT Heap: 4.04 / 40.50 MiB / NOC ddr: 1923 MB/s
    [19443010C1D17E1300] [192.168.1.27] [18.111] [system] [info] Temperatures - Average: 51.36C, CSS: 52.68C, MSS 50.03C, UPA: 50.70C, DSS: 52.02C
    [19443010C1D17E1300] [192.168.1.27] [18.111] [system] [info] Cpu Usage - LeonOS 99.54%, LeonRT: 16.52%

      VladimirosSterzentsenko sz ,blck = 10, False

      camera_stream = device.getOutputQueue("rgb_preview", maxSize=sz, blocking=blck)
      camera_stream_cropped = device.getOutputQueue("rgb_crop", maxSize=sz, blocking=blck)
      params_q = device.getInputQueue("params", maxSize=sz, blocking=blck)

      Here is the part that saves the frames inside a queue with a size of 10. Set this to 1.

      Definetly helped to get down to 5 - 7 frames difference. Do you think it can get any lower? It is really important for me to get it to 1-2 frames.

      Also, can I couple frames with the imageManip parameters? It would work for me at least to know, a given frame, with what ImageManipConfig it was processed.

      Thank you, Vlad

      **Edit
      when changing to 15 or 30 fps, it gets down to 2 - 3 frames difference, which is kind of acceptable. Can I get it to work at 60 fps?

      Hi @VladimirosSterzentsenko
      With the above mentioned fix I am able to get 1 frame difference:

      import cv2
      import depthai as dai
      import random
      import os
      
      available_devices = dai.Device.getAllAvailableDevices()
      p = dai.Pipeline()
      p.setOpenVINOVersion(dai.OpenVINO.VERSION_2022_1)
      
      camera = p.create(dai.node.ColorCamera)
      camera.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
      camera.setPreviewSize(320, 180)
      camera.setInterleaved(False)
      camera.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
      camera.setFps(5)
      
      
      image_manip = p.create(dai.node.ImageManip)
      image_manip.initialConfig.setResize(256,256)
      image_manip.inputConfig.setWaitForMessage(False)
      image_manip.inputConfig.setQueueSize(1)
      image_manip.inputImage.setQueueSize(1)
      image_manip.inputConfig.setBlocking(True)
      
      
      video_encoder = p.create(dai.node.VideoEncoder)
      video_encoder.setProfile(dai.node.VideoEncoder.Properties.Profile.MJPEG)
      
      video_encoder_crop = p.create(dai.node.VideoEncoder)
      video_encoder_crop.setProfile(dai.node.VideoEncoder.Properties.Profile.MJPEG)
      
      camera_xout = p.create(dai.node.XLinkOut)
      camera_xout.setStreamName("rgb_preview")
      
      camera_xout_crop = p.create(dai.node.XLinkOut)
      camera_xout_crop.setStreamName("rgb_crop")
      
      
      camera.video.link(image_manip.inputImage)
      image_manip.out.link(video_encoder_crop.input)
      
      params = p.create(dai.node.XLinkIn)
      params.setStreamName("params")
      
      params.out.link(image_manip.inputConfig)
      
      video_encoder_crop.bitstream.link(camera_xout_crop.input)
      camera.preview.link(camera_xout.input)
      
      device = dai.Device(p)
      
      sz ,blck = 1, False
      
      camera_stream = device.getOutputQueue("rgb_preview", maxSize=sz, blocking=blck)
      camera_stream_cropped = device.getOutputQueue("rgb_crop", maxSize=sz, blocking=blck)
      params_q = device.getInputQueue("params", maxSize=sz, blocking=blck)
      
      counter = 0
      press = False
      i = 0
      while True:
      
          if press:
          
              frame_cropped_msg = camera_stream_cropped.get()
              frame_cropped = frame_cropped_msg.getRaw().data
              image = cv2.imdecode(frame_cropped, cv2.IMREAD_COLOR)
              cv2.imshow(f"rgb_crop", image)
              
              print("Cropped time: ", frame_cropped_msg.getTimestamp())
              
              print("Cropped image shape: ", image.shape)
              if frame_cropped_msg.getTimestamp() < time:
                  i += 1
              else: 
                  print("number of frames before key press: ", i)
      
          frame = camera_stream.get().getCvFrame()
          print(frame.shape)
          cv2.imshow(f"rgb", frame)
      
          key = cv2.waitKey(1)
          if key == ord('q'):
              break
          elif key == ord('p'):
              press = True
              print("key press")
              time = dai.Clock.now()
              print("time: ", time)
              config = dai.ImageManipConfig()
              rr = dai.RotatedRect()
          
              rr.center = dai.Point2f(random.randint(0, 1920), random.randint(0, 1080))
              rr.size = dai.Size2f(800, 600)
              rr.angle = (random.random() * 2 - 1) * 10
      
              config.setCropRotatedRect(rr, False)
              config.setResize(256,256)
              config.setKeepAspectRatio(False)
              
              params_q.send(config)

      Made a script to show the timestamps.

      Thanks,
      Jaka

      Seems like to work around 1-2 frame at 5 fps but still goes to over 5 frames at 60 fps.

      Cropped time:  5 days, 13:58:31.361937
      number of frames before key press:  5
      Cropped time:  5 days, 13:58:31.378606
      number of frames before key press:  5
      Cropped time:  5 days, 13:58:31.395266
      number of frames before key press:  5

      Is there something better to be done here? Can you reproduce your results on your machine with 60 fps please?

      Hi @VladimirosSterzentsenko
      You can check the times with DEPTHAI_LEVEL=TRACE python3 main.py, but I think you can't go lower than 3-4 frames because it takes some time for trigger to complete.

      You can however just compare the timestamps of the trigger and if the frame is older, discard it. Kind of like it is done in above code.

      Thanks,
      Jaka