Optimize "complex" pipeline for latency

apirrone

Hi !

We wrote a pipeline for stereo video h264 streaming with on board color stereo rectification. We use two IMX296 sensors from Arducam (1440x1080) plugged in a OAK FFC-4P board. The goal is to get two stereo-rectified h264 streams right out of the board.

We had quite a lot of trouble getting the latency of the system as low as possible, but finaly managed to get between 30 and 40ms at 720p 50fps, which is good !

Before getting there, we would get 130-150ms of latency with the following pipeline.

isp (1440x1080) -> rectify_manip (1440x1080) -> resize_manip (1280x720 letterboxed) -> videoEncoder -> out

(rectify manip is an ImageManip node with setWarpMesh())

With only one camera, we get a nice 30ms of latency at 50fps, but adding a second camera, and thus a second identical pipeline dropped the latency to 130-150ms.

We noticed that it was the rectify imagemanip nodes that were a bottleneck in the pipelines, applying the warp mesh to two 1440x1080 images seems to be too much.

So we first downscaled the isp to (920x720) and got out 30-40ms.

isp downscaled(920x720) -> rectify_manip (920x720) -> resize_manip (1280x720 letterboxed) -> videoEncoder -> out

So this works, but we are limited to a 720p output right now.

Are we doing something wrong with the warp imagemanip nodes ? Or do you think this is an expected limitation ? It would be great to do everything in the original resolution of 1440x1080 to keep as many pixels as possible

Below is a simplified version of our pipeline:


def create_manipRectify(pipeline, cam_name, resolution):
    manipRectify = pipeline.createImageManip()

    try:
        mesh, meshWidth, meshHeight = get_mesh(cam_name)
        manipRectify.setWarpMesh(mesh, meshWidth, meshHeight)
    except Exception as e:
        print(e)
        exit()

    manipRectify.setMaxOutputFrameSize(resolution[0] * resolution[1] * 3)
    return manipRectify


def create_manipResize(pipeline, resolution):
    manipResize = pipeline.createImageManip()
    manipResize.initialConfig.setResizeThumbnail(resolution[0], resolution[1])
    manipResize.setMaxOutputFrameSize(resolution[0] * resolution[1] * 3)

    return manipResize


pipeline = dai.Pipeline()

left = pipeline.createColorCamera()
left.setFps(cam_config.fps)
left.setBoardSocket(left_socket)
left.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1440X1080)
left.setIspScale(2, 3)  # -> 960, 720

right = pipeline.createColorCamera()
right.setFps(cam_config.fps)
right.setBoardSocket(right_socket)
right.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1440X1080)
right.setIspScale(2, 3)  # -> 960, 720

left_manipRectify = create_manipRectify(pipeline, "left", (920, 720))
right_manipRectify = create_manipRectify(pipeline, "right", (920, 720))

left_manipRescale = create_manipResize(pipeline, (1280, 720))
right_manipRescale = create_manipResize(pipeline, (1280, 720))

profile = dai.VideoEncoderProperties.Profile.H264_MAIN
left_encoder = pipeline.create(dai.node.VideoEncoder)
left_encoder.setDefaultProfilePreset(cam_config.fps, profile)
left_encoder.setKeyframeFrequency(cam_config.fps)  # every 1s
left_encoder.setNumBFrames(0)  # gstreamer recommends 0 B frames
left_encoder.setBitrateKbps(4000)

right_encoder = pipeline.create(dai.node.VideoEncoder)
right_encoder.setDefaultProfilePreset(cam_config.fps, profile)
right_encoder.setKeyframeFrequency(cam_config.fps)  # every 1s
right_encoder.setNumBFrames(0)  # gstreamer recommends 0 B frames
right_encoder.setBitrateKbps(4000)


xout_left = pipeline.createXLinkOut()
xout_left.setStreamName("left")

xout_right = pipeline.createXLinkOut()
xout_right.setStreamName("right")


left.isp.link(left_manipRectify.inputImage)
left_manipRectify.out.link(left_manipRescale.inputImage)
right.isp.link(right_manipRectify.inputImage)
right_manipRectify.out.link(right_manipRescale.inputImage)

left_manipRescale.out.link(left_encoder.input)
right_manipRescale.out.link(right_encoder.input)

left_encoder.bitstream.link(xout_left.input)
right_encoder.bitstream.link(xout_right.input)

Thanks !

Antoine

jakaskerl

Hi apirrone
Since you are using color cameras, could you try using setInterleaved(False)? I found that planar formats are much less computationally expensive for the image manip nodes. Perhaps this improves your latency and FPS.

Thanks,
Jaka

apirrone

Hi jakaskerl

I just tried and it does not seem to make any difference.

Thanks,

Antoine

jakaskerl

Hi apirrone
Hm, ok. It's possible you are reaching the limit of the device capability. Any chance you could send me the complete code on the off chance we can find some trick/compromise to achieve your criteria?

Thanks,
Jaka

FabPoll

Hi jakaskerl ,

I'm working with @apirrone on this. Any new thoughts on this topic?
Our pipeline hasn't changed, the provided code is still valid:

We still have ~35ms latency with two camera streams (720p 50fps)

[18443010B18A5B1200] [3.4] [15.714] [system] [info] Temperatures - Average: 48.30C, CSS: 50.25C, MSS 47.35C, UPA: 46.90C, DSS: 48.70C

[18443010B18A5B1200] [3.4] [15.714] [system] [info] Memory Usage - DDR: 77.99 / 334.18 MiB, CMX: 2.50 / 2.50 MiB, LeonOS Heap: 34.71 / 82.79 MiB, LeonRT Heap: 6.21 / 40.87 MiB

[18443010B18A5B1200] [3.4] [15.714] [system] [info] Temperatures - Average: 48.30C, CSS: 50.25C, MSS 47.35C, UPA: 46.90C, DSS: 48.70C

[18443010B18A5B1200] [3.4] [15.714] [system] [info] Cpu Usage - LeonOS 57.04%, LeonRT: 41.49%

INFO:root:{'left': datetime.timedelta(microseconds=34653), 'right': datetime.timedelta(microseconds=39697)}

According to the debug log, the board is underused so we are hopping reaching full resolution and max fps (1440x1080@60fps). Is there anything you think we can do here?

Thanks

jakaskerl

Hi @FabPoll

FabPoll CMX: 2.50 / 2.50 MiB

This is the issue since your manip nodes are using CMX slices which are maxed out it seems.

Thanks,
Jaka

FabPoll

jakaskerl

I've got better performances if I fuse the two imagemanip steps in each sub pipeline. BTW, what's the order of operation? I'm calling setResize and setWarpMesh.

The debug log shows this:

[2024-02-07 09:16:36.452] [depthai] [debug] Python bindings - version: 2.24.0.0 from build: 2023-12-13 23:43:56 +0000

[2024-02-07 09:16:36.452] [depthai] [debug] Library information - version: 2.24.0, commit: 6628488ef8956f73f1c7bf4c8f1da218ad327a6f from 2023-12-13 14:45:09 +0100, build: 2023-12-13 23:26:25 +0000, libusb enabled: true

[2024-02-07 09:16:36.455] [depthai] [debug] Initialize - finished

[2024-02-07 09:16:36.595] [depthai] [debug] Resources - Archive 'depthai-bootloader-fwp-0.0.26.tar.xz' open: 1ms, archive read: 140ms

[2024-02-07 09:16:36.708] [host] [debug] Found an actual device by given DeviceInfo: DeviceInfo(name=3.4, mxid=18443010B18A5B1200, X_LINK_BOOTLOADER, X_LINK_USB_VSC, X_LINK_MYRIAD_X, X_LINK_SUCCESS)

[2024-02-07 09:16:36.708] [host] [debug] Device - OpenVINO version: universal

[18443010B18A5B1200] [3.4] [1707293796.708] [host] [debug] Device - BoardConfig: {"camera":[],"emmc":null,"gpio":[],"imu":null,"logDevicePrints":null,"logPath":null,"logSizeMax":null,"logVerbosity":null,"network":{"mtu":0,"xlinkTcpNoDelay":true},"nonExclusiveMode":false,"pcieInternalClock":null,"sysctl":[],"uart":[],"usb":{"flashBootedPid":63037,"flashBootedVid":999,"manufacturer":"","maxSpeed":5,"pid":63035,"productName":"","vid":999},"usb3PhyInternalClock":null,"uvc":null,"watchdogInitialDelayMs":null,"watchdogTimeoutMs":null}

libnop:

0000: b9 12 b9 07 81 e7 03 81 3b f6 81 e7 03 81 3d f6 05 bd 00 bd 00 b9 02 00 01 ba 00 be be bb 00 bb

0020: 00 be be be be be be be 00 bb 00 be be

[2024-02-07 09:16:36.871] [depthai] [debug] Resources - Archive 'depthai-device-fwp-a95f582a61ec9bdbd0f72dec84822455872ffaf7.tar.xz' open: 1ms, archive read: 417ms

[2024-02-07 09:16:36.886] [depthai] [debug] Searching for booted device: DeviceInfo(name=3.4, mxid=18443010B18A5B1200, X_LINK_BOOTLOADER, X_LINK_USB_VSC, X_LINK_MYRIAD_X, X_LINK_SUCCESS), name used as hint only

[2024-02-07 09:16:36.890] [depthai] [debug] Connected bootloader version 0.0.26

[18443010B18A5B1200] [3.4] [1707293797.752] [host] [debug] Booting FW with Bootloader. Version 0.0.26, Time taken: 862ms

[2024-02-07 09:16:37.752] [depthai] [debug] DeviceBootloader about to be closed...

[2024-02-07 09:16:37.753] [depthai] [debug] XLinkResetRemote of linkId: (0)

[2024-02-07 09:16:39.002] [depthai] [debug] DeviceBootloader closed, 1249

[2024-02-07 09:16:39.005] [depthai] [debug] Searching for booted device: DeviceInfo(name=3.4, mxid=18443010B18A5B1200, X_LINK_BOOTED, X_LINK_USB_VSC, X_LINK_MYRIAD_X, X_LINK_SUCCESS), name used as hint only

[18443010B18A5B1200] [3.4] [4.882] [system] [info] Memory Usage - DDR: 0.12 / 333.46 MiB, CMX: 2.04 / 2.50 MiB, LeonOS Heap: 7.50 / 82.56 MiB, LeonRT Heap: 2.89 / 40.68 MiB / NOC ddr: 7 MB/s

[18443010B18A5B1200] [3.4] [4.882] [system] [info] Temperatures - Average: 41.32C, CSS: 43.04C, MSS 40.51C, UPA: 41.20C, DSS: 40.51C

[18443010B18A5B1200] [3.4] [4.882] [system] [info] Cpu Usage - LeonOS 10.68%, LeonRT: 0.53%

[18443010B18A5B1200] [3.4] [1707293803.013] [host] [debug] Schema dump: {"connections":[{"node1Id":5,"node1Output":"bitstream","node1OutputGroup":"","node2Id":7,"node2Input":"in","node2InputGroup":""},{"node1Id":4,"node1Output":"bitstream","node1OutputGroup":"","node2Id":6,"node2Input":"in","node2InputGroup":""},{"node1Id":3,"node1Output":"out","node1OutputGroup":"","node2Id":5,"node2Input":"in","node2InputGroup":""},{"node1Id":1,"node1Output":"isp","node1OutputGroup":"","node2Id":3,"node2Input":"inputImage","node2InputGroup":""},{"node1Id":2,"node1Output":"out","node1OutputGroup":"","node2Id":4,"node2Input":"in","node2InputGroup":""},{"node1Id":0,"node1Output":"isp","node1OutputGroup":"","node2Id":2,"node2Input":"inputImage","node2InputGroup":""}],"globalProperties":{"calibData":null,"cameraTuningBlobSize":null,"cameraTuningBlobUri":"","leonCssFrequencyHz":700000000.0,"leonMssFrequencyHz":700000000.0,"pipelineName":null,"pipelineVersion":null,"sippBufferSize":18432,"sippDmaBufferSize":16384,"xlinkChunkSize":-1},"nodes":[[0,{"id":0,"ioInfo":[[["","video"],{"blocking":false,"group":"","id":30,"name":"video","queueSize":8,"type":0,"waitForMessage":false}],[["","still"],{"blocking":false,"group":"","id":28,"name":"still","queueSize":8,"type":0,"waitForMessage":false}],[["","isp"],{"blocking":false,"group":"","id":27,"name":"isp","queueSize":8,"type":0,"waitForMessage":false}],[["","preview"],{"blocking":false,"group":"","id":29,"name":"preview","queueSize":8,"type":0,"waitForMessage":false}],[["","raw"],{"blocking":false,"group":"","id":26,"name":"raw","queueSize":8,"type":0,"waitForMessage":false}],[["","frameEvent"],{"blocking":false,"group":"","id":25,"name":"frameEvent","queueSize":8,"type":0,"waitForMessage":false}],[["","inputConfig"],{"blocking":false,"group":"","id":24,"name":"inputConfig","queueSize":8,"type":3,"waitForMessage":false}],[["","inputControl"],{"blocking":true,"group":"","id":23,"name":"inputControl","queueSize":8,"type":3,"waitForMessage":false}]],"name":"ColorCamera","properties":[185,26,185,30,0,3,0,0,0,185,3,0,0,0,185,5,0,0,0,0,0,185,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,185,3,0,0,0,185,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,189,0,255,0,1,0,129,44,1,129,44,1,255,255,255,255,11,136,0,0,112,66,0,136,0,0,128,191,136,0,0,128,191,1,185,4,2,3,2,3,3,3,4,4,4,190]}],[1,{"id":1,"ioInfo":[[["","video"],{"blocking":false,"group":"","id":22,"name":"video","queueSize":8,"type":0,"waitForMessage":false}],[["","still"],{"blocking":false,"group":"","id":20,"name":"still","queueSize":8,"type":0,"waitForMessage":false}],[["","isp"],{"blocking":false,"group":"","id":19,"name":"isp","queueSize":8,"type":0,"waitForMessage":false}],[["","preview"],{"blocking":false,"group":"","id":21,"name":"preview","queueSize":8,"type":0,"waitForMessage":false}],[["","raw"],{"blocking":false,"group":"","id":18,"name":"raw","queueSize":8,"type":0,"waitForMessage":false}],[["","frameEvent"],{"blocking":false,"group":"","id":17,"name":"frameEvent","queueSize":8,"type":0,"waitForMessage":false}],[["","inputConfig"],{"blocking":false,"group":"","id":16,"name":"inputConfig","queueSize":8,"type":3,"waitForMessage":false}],[["","inputControl"],{"blocking":true,"group":"","id":15,"name":"inputControl","queueSize":8,"type":3,"waitForMessage":false}]],"name":"ColorCamera","properties":[185,26,185,30,0,3,0,0,0,185,3,0,0,0,185,5,0,0,0,0,0,185,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,185,3,0,0,0,185,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,189,0,255,0,1,0,129,44,1,129,44,1,255,255,255,255,11,136,0,0,112,66,0,136,0,0,128,191,136,0,0,128,191,1,185,4,2,3,2,3,3,3,4,4,4,190]}],[2,{"id":2,"ioInfo":[[["","out"],{"blocking":false,"group":"","id":14,"name":"out","queueSize":8,"type":0,"waitForMessage":false}],[["","inputConfig"],{"blocking":true,"group":"","id":13,"name":"inputConfig","queueSize":8,"type":3,"waitForMessage":false}],[["","inputImage"],{"blocking":true,"group":"","id":12,"name":"inputImage","queueSize":8,"type":3,"waitForMessage":true}]],"name":"ImageManip","properties":[185,6,185,9,185,7,185,4,136,0,0,0,0,136,0,0,0,0,136,0,0,0,0,136,0,0,0,0,185,3,185,2,136,0,0,0,0,136,0,0,0,0,185,2,136,0,0,0,0,136,0,0,0,0,136,0,0,0,0,0,136,0,0,128,63,136,0,0,128,63,0,1,185,15,133,192,3,133,208,2,0,0,0,0,186,0,1,0,186,0,0,0,136,0,0,0,0,0,1,185,6,22,0,0,0,0,133,255,0,0,1,1,0,0,255,134,0,164,31,0,4,60,46,189,10,97,115,115,101,116,58,109,101,115,104]}],[3,{"id":3,"ioInfo":[[["","out"],{"blocking":false,"group":"","id":11,"name":"out","queueSize":8,"type":0,"waitForMessage":false}],[["","inputConfig"],{"blocking":true,"group":"","id":10,"name":"inputConfig","queueSize":8,"type":3,"waitForMessage":false}],[["","inputImage"],{"blocking":true,"group":"","id":9,"name":"inputImage","queueSize":8,"type":3,"waitForMessage":true}]],"name":"ImageManip","properties":[185,6,185,9,185,7,185,4,136,0,0,0,0,136,0,0,0,0,136,0,0,0,0,136,0,0,0,0,185,3,185,2,136,0,0,0,0,136,0,0,0,0,185,2,136,0,0,0,0,136,0,0,0,0,136,0,0,0,0,0,136,0,0,128,63,136,0,0,128,63,0,1,185,15,133,192,3,133,208,2,0,0,0,0,186,0,1,0,186,0,0,0,136,0,0,0,0,0,1,185,6,22,0,0,0,0,133,255,0,0,1,1,0,0,255,134,0,164,31,0,4,60,46,189,10,97,115,115,101,116,58,109,101,115,104]}],[4,{"id":4,"ioInfo":[[["","out"],{"blocking":false,"group":"","id":7,"name":"out","queueSize":8,"type":0,"waitForMessage":false}],[["","bitstream"],{"blocking":false,"group":"","id":8,"name":"bitstream","queueSize":8,"type":0,"waitForMessage":false}],[["","in"],{"blocking":true,"group":"","id":6,"name":"in","queueSize":4,"type":3,"waitForMessage":true}]],"name":"VideoEncoder","properties":[185,11,134,0,9,61,0,60,0,0,0,0,2,80,0,0,136,0,0,112,66]}],[5,{"id":5,"ioInfo":[[["","out"],{"blocking":false,"group":"","id":4,"name":"out","queueSize":8,"type":0,"waitForMessage":false}],[["","bitstream"],{"blocking":false,"group":"","id":5,"name":"bitstream","queueSize":8,"type":0,"waitForMessage":false}],[["","in"],{"blocking":true,"group":"","id":3,"name":"in","queueSize":4,"type":3,"waitForMessage":true}]],"name":"VideoEncoder","properties":[185,11,134,0,9,61,0,60,0,0,0,0,2,80,0,0,136,0,0,112,66]}],[6,{"id":6,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":2,"name":"in","queueSize":8,"type":3,"waitForMessage":true}]],"name":"XLinkOut","properties":[185,3,136,0,0,128,191,189,4,108,101,102,116,0]}],[7,{"id":7,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":1,"name":"in","queueSize":8,"type":3,"waitForMessage":true}]],"name":"XLinkOut","properties":[185,3,136,0,0,128,191,189,5,114,105,103,104,116,0]}]]}

[18443010B18A5B1200] [3.4] [1707293803.013] [host] [debug] Asset map dump: {"map":{"/node/2/mesh":{"alignment":64,"offset":22080,"size":22080},"/node/3/mesh":{"alignment":64,"offset":0,"size":22080}}}

[18443010B18A5B1200] [3.4] [5.024] [system] [info] SIPP (Signal Image Processing Pipeline) internal buffer size '18432'B, DMA buffer size: '16384'B

[18443010B18A5B1200] [3.4] [5.059] [system] [info] ImageManip internal buffer size '393408'B, shave buffer size '35840'B

[18443010B18A5B1200] [3.4] [5.059] [system] [info] ColorCamera allocated resources: no shaves; cmx slices: [13-15]

ImageManip allocated resources: shaves: [15-15] no cmx slices.

Does [15-15] means that shaves and CMX are running at full capacity?

jakaskerl

FabPoll BTW, what's the order of operation? I'm calling setResize and setWarpMesh.

Resize (cropping) is done first iirc, then setWarpMesh.

FabPoll Does [15-15] means that shaves and CMX are running at full capacity?

It means the Manip node is using the 16th CMX slice. If you look at other nodes you will see that eg. ColorCamera node uses slices with indices [13-15].

Thanks,
Jaka

FabPoll

jakaskerl

OK thanks. So only slices from 13 to 15 are used. But the CMX memory is full as you noticed. What I am missing here?
I am trying to optimize the image manip node which seems to be the bottleneck.
Also using the video output instead of isp is much more slower (even with interleaved(false)). Is it a normal behaviour?

jakaskerl

Hi @FabPoll
Let me correct myself on previous posts:

ImageManip does not use CMX slices. It uses the unified CMX memory.

FabPoll OK thanks. So only slices from 13 to 15 are used. But the CMX memory is full as you noticed. What I am missing here?

Other slices 0-16, are preallocated for HW blocks, that is why the allocator sees them as used even though they are not. The rest of the slices 17-20, are used by whatever (including ImageManip). Since the manip is stressed, it uses up all it's available memory. So 2MiB (slices) + 500Kib (for manip) = 2.5MiB. There is currently no way to use ImageManip on CMX slices.

Thanks,
Jaka

FabPoll

Great thanks!
Would it be better to undistort the image directly inside Camera.Node (setMeshSource)? You mention this possibility here, but the documentation lacks an example. How can I pass the computed mesh to this function as I do with SetWarpMesh?

jakaskerl

Hi @FabPoll
I don't think it would make a difference since the Camera node only abstracts the undistorting process. In the background ImageManip is used, just like you are using it currently.

Thanks,
Jaka