Device side queue size

Kacper

Hello,
I am working on an application where a user is able to set a desired crop of the camera sensor and pan that copped view around the sensor, a snippet of the code has been pasted below.
The issue I am running into is that there is major lag > 1s between sending a imageManip config and that config being applied and affecting the output image. It seems that when I send updated configs they get queued up and even after the user stops panning, the image continues to pan due the built up queue of configs. I have attempted to rate limit the updates but the delay between sending and the config being applied is inconsistent which is inconvenient because setting some max timeout leads to very jerky input.

Question1: Is it possible to set the size of the device side queue to 1(not host side, but the device for XLinkIn)?
Question2: is there a way to speed up the rate at which the updated configs get applied?
Question3: perhaps there is a simpler way to do this

def __init__(self):
        self.pipeline = dai.Pipeline()
 
        self.camRgb = self.pipeline.create(dai.node.ColorCamera)
        self.colorEnc = self.pipeline.create(dai.node.VideoEncoder)
        self.colorOut = self.pipeline.create(dai.node.XLinkOut)

        self.colorOut.setStreamName('color')

        self.camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
        self.camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
        self.camRgb.setInterleaved(False)
        self.camRgb.setFps(30)
        self.camRgb.setIspScale(1, 1)

        # Set the ISP pool size to 2 to reduce memory usage
        self.camRgb.setNumFramesPool(2, 2, 0, 0, 0) 

        # ImageManip node for crop and resize
        self.crop = self.pipeline.create(dai.node.ImageManip)
        # Calculate the exact output frame size needed
        max_frame_size = SCENE_SIZE[0] * SCENE_SIZE[1] * 3  # This is actually a NV12 so bit bigger than necessary
        self.crop.setMaxOutputFrameSize(max_frame_size)
        self.crop.initialConfig.setResize(*SCENE_SIZE)
        self.crop.initialConfig.setHorizontalFlip(False)
        self.crop.initialConfig.setFrameType(dai.ImgFrame.Type.NV12)
    
        # HERE IS THE INPUT CONFIG INPUT LINK
        self.configIn = self.pipeline.create(dai.node.XLinkIn)
        self.configIn.setStreamName('config')        
        self.configIn.out.link(self.crop.inputConfig)

        self.camRgb.isp.link(self.crop.inputImage)

        # Video encoder configuration
        self.colorEnc.setDefaultProfilePreset(30, dai.VideoEncoderProperties.Profile.H264_MAIN)
        self.colorEnc.setKeyframeFrequency(30)
        self.colorEnc.setQuality(80)
        self.colorEnc.setNumFramesPool(1)  # Minimize encoder memory usage
        
        # Link the cropped output to the encoder
        self.crop.out.link(self.colorEnc.input)
        self.colorEnc.bitstream.link(self.colorOut.input)
    
        # Disable any direct ISP output from the camera to save memory
        self.camRgb.setPreviewSize(0, 0)  # Disable preview stream
        self.camRgb.setVideoSize(0, 0)    # Disable video stream
        self.camRgb.setStillSize(0, 0)    # Disable still image
        
        # Create device and queues
        self.device = dai.Device(self.pipeline)
        self.color_q = self.device.getOutputQueue(name="color", maxSize=1, blocking=False)
        self.config_queue = self.device.getInputQueue(name='config', maxSize=1, blocking=False)

    def set_zoom(self, zoom_value, x, y):
        """Set the zoom level (0-100) and/or position (x, y as 0-100 percentages)"""
        # Apply zoom and position
        self.zoom_val = min(100, max(0, int(zoom_value)))
        self.X = min(100, max(0, int(x)))
        self.Y = min(100, max(0, int(y)))
        try:
            # Get normalized crop rectangle
            norm_left, norm_top, norm_right, norm_bottom = self.get_normalized_crop_rect()

            # Create a config that maintains current settings
            ctrl = dai.ImageManipConfig()
            
            # Set resize first
            ctrl.setResize(*SCENE_SIZE)
            
            # Use normalized coordinates (0-1 range) for the crop
            ctrl.setCropRect(norm_left, norm_top, norm_right, norm_bottom)
            
            # Set horizontal flip
            ctrl.setHorizontalFlip(self.mirror)
            
            # Set frame type
            ctrl.setFrameType(dai.ImgFrame.Type.NV12)
            
            # Send control to the config queue
            self.config_queue.send(ctrl)
        except Exception as e:
            logging.error(f"Error setting zoom/crop: {str(e)}")

lovro

Hi,

If the delay is due to config queue buildup, you could try throttling updates or only sending configs when values change. Also, any reason you're doing the crop/pan on the device? Might be simpler to just stream full frames and crop on the host with OpenCV — avoids all the queueing and is much more responsive.

Best, Lovro

Kacper

lovro Hello, thanks for the reply, as mentioned in the OP I do throttle the config updates, but due to the unpredictability of the device processing the updates, it sometimes works fine and others it lags. Also, I do only send updated on when the user input changes.

To answer your question about "why not do it on the host": My host is very low power, and would not be able to process the raw image and encode to h264 while ensuring low latency which my application requires. Containing the entire pipeline to the device ensures low latency and minimal host requirements.

If it would be possible to resize the input queue to a single item it would ensure that whenever the device goes to update the config it has the latest one.