Hi !
We wrote a pipeline for stereo video h264 streaming with on board color stereo rectification. We use two IMX296 sensors from Arducam (1440x1080) plugged in a OAK FFC-4P board. The goal is to get two stereo-rectified h264 streams right out of the board.
We had quite a lot of trouble getting the latency of the system as low as possible, but finaly managed to get between 30 and 40ms at 720p 50fps, which is good !
Before getting there, we would get 130-150ms of latency with the following pipeline.
isp (1440x1080) -> rectify_manip (1440x1080) -> resize_manip (1280x720 letterboxed) -> videoEncoder -> out
(rectify manip is an ImageManip node with setWarpMesh())
With only one camera, we get a nice 30ms of latency at 50fps, but adding a second camera, and thus a second identical pipeline dropped the latency to 130-150ms.
We noticed that it was the rectify imagemanip nodes that were a bottleneck in the pipelines, applying the warp mesh to two 1440x1080 images seems to be too much.
So we first downscaled the isp to (920x720) and got out 30-40ms.
isp downscaled(920x720) -> rectify_manip (920x720) -> resize_manip (1280x720 letterboxed) -> videoEncoder -> out
So this works, but we are limited to a 720p output right now.
- Are we doing something wrong with the warp imagemanip nodes ? Or do you think this is an expected limitation ? It would be great to do everything in the original resolution of 1440x1080 to keep as many pixels as possible
Below is a simplified version of our pipeline:
def create_manipRectify(pipeline, cam_name, resolution):
manipRectify = pipeline.createImageManip()
try:
mesh, meshWidth, meshHeight = get_mesh(cam_name)
manipRectify.setWarpMesh(mesh, meshWidth, meshHeight)
except Exception as e:
print(e)
exit()
manipRectify.setMaxOutputFrameSize(resolution[0] * resolution[1] * 3)
return manipRectify
def create_manipResize(pipeline, resolution):
manipResize = pipeline.createImageManip()
manipResize.initialConfig.setResizeThumbnail(resolution[0], resolution[1])
manipResize.setMaxOutputFrameSize(resolution[0] * resolution[1] * 3)
return manipResize
pipeline = dai.Pipeline()
left = pipeline.createColorCamera()
left.setFps(cam_config.fps)
left.setBoardSocket(left_socket)
left.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1440X1080)
left.setIspScale(2, 3) # -> 960, 720
right = pipeline.createColorCamera()
right.setFps(cam_config.fps)
right.setBoardSocket(right_socket)
right.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1440X1080)
right.setIspScale(2, 3) # -> 960, 720
left_manipRectify = create_manipRectify(pipeline, "left", (920, 720))
right_manipRectify = create_manipRectify(pipeline, "right", (920, 720))
left_manipRescale = create_manipResize(pipeline, (1280, 720))
right_manipRescale = create_manipResize(pipeline, (1280, 720))
profile = dai.VideoEncoderProperties.Profile.H264_MAIN
left_encoder = pipeline.create(dai.node.VideoEncoder)
left_encoder.setDefaultProfilePreset(cam_config.fps, profile)
left_encoder.setKeyframeFrequency(cam_config.fps) # every 1s
left_encoder.setNumBFrames(0) # gstreamer recommends 0 B frames
left_encoder.setBitrateKbps(4000)
right_encoder = pipeline.create(dai.node.VideoEncoder)
right_encoder.setDefaultProfilePreset(cam_config.fps, profile)
right_encoder.setKeyframeFrequency(cam_config.fps) # every 1s
right_encoder.setNumBFrames(0) # gstreamer recommends 0 B frames
right_encoder.setBitrateKbps(4000)
xout_left = pipeline.createXLinkOut()
xout_left.setStreamName("left")
xout_right = pipeline.createXLinkOut()
xout_right.setStreamName("right")
left.isp.link(left_manipRectify.inputImage)
left_manipRectify.out.link(left_manipRescale.inputImage)
right.isp.link(right_manipRectify.inputImage)
right_manipRectify.out.link(right_manipRescale.inputImage)
left_manipRescale.out.link(left_encoder.input)
right_manipRescale.out.link(right_encoder.input)
left_encoder.bitstream.link(xout_left.input)
right_encoder.bitstream.link(xout_right.input)
Thanks !
Antoine
