FFC-4p should support four ov9282 cameras for hardware synchronization, but is there any code to test this? Can they synchronize in microseconds?

Hi RBJin , I believe the A & D ports and B & C ports have fsync connected together, so hardware syncing between these pairs is already there. Are you experiencing larger delays?

The following is my test code and the output device timestamp. If I only use left and right cameras, the output is always in pairs and the delay is less than 1ms. But when I turn on four cameras, the delay is obviously greater than 1ms. I guess it may be a problem with my code, so I want to find some test codes that can be referenced. import depthai as dai import time import cv2 import collections set_fps = 30 class FPS: def __init__(self, window_size=30): self.dq = collections.deque(maxlen=window_size) self.fps = 0 def update(self, timestamp=None): if timestamp == None: timestamp = time.monotonic() count = len(self.dq) if count > 0: self.fps = count / (timestamp - self.dq[0]) self.dq.append(timestamp) def get(self): return self.fps # cam_list = ['left', 'right'] # cam_socket_opts = { # 'left' : dai.CameraBoardSocket.LEFT, # Or CAM_B # 'right': dai.CameraBoardSocket.RIGHT, # Or CAM_C # } cam_list = ['rgb', 'left', 'right', 'camd'] cam_socket_opts = { 'rgb' : dai.CameraBoardSocket.RGB, # Or CAM_A 'left' : dai.CameraBoardSocket.LEFT, # Or CAM_B 'right': dai.CameraBoardSocket.RIGHT, # Or CAM_C 'camd' : dai.CameraBoardSocket.CAM_D, } pipeline = dai.Pipeline() cam = {} xout = {} for c in cam_list: cam[c] = pipeline.create(dai.node.MonoCamera) cam[c].setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P) cam[c].setBoardSocket(cam_socket_opts[c]) xout[c] = pipeline.create(dai.node.XLinkOut) xout[c].setStreamName(c) cam[c].out.link(xout[c].input) with dai.Device(pipeline) as device: q = {} fps_host = {} # FPS computed based on the time we receive frames in app fps_capt = {} # FPS computed based on capture timestamps from device for c in cam_list: q[c] = device.getOutputQueue(name=c, maxSize=1, blocking=False) cv2.namedWindow(c, cv2.WINDOW_NORMAL) cv2.resizeWindow(c, (640, 480)) fps_host[c] = FPS() fps_capt[c] = FPS() while True: frame_list = [] for c in cam_list: pkt = q[c].tryGet() if pkt is not None: fps_host[c].update() fps_capt[c].update(pkt.getTimestamp().total_seconds()) print(c+":",pkt.getTimestampDevice()) frame = pkt.getCvFrame() cv2.imshow(c, frame) print("-------------------------------") # print("\rFPS:", # *["{:6.2f}|{:6.2f}".format(fps_host[c].get(), fps_capt[c].get()) for c in cam_list], # end='', flush=True) key = cv2.waitKey(1) if key == ord('q'): break if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); }

The B & C (or Left & Right) ports are synced by sharing the same I2C bus. For the ports A & D (RGB and CamD) this is not the case, and even though their FSIN pins are tied together, hardware sync has to be explicitly enabled. Can be done for example by configuring in the pipeline: cam['rgb'] .initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.OUTPUT) cam['camd'] .initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } And to get all 4 sensors synchronized, need to configure also FSIN for L/R sensors: cam['left'] .initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) cam['right'].initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } AND importantly to tie the FSIN signals of A+D and B+C pairs, by setting a GPIO: # OAK-FFC-4P requires driving GPIO6 high (FSIN_MODE_SELECT) to connect together # the A+D FSIN group (4-lane pair) with the B+C group (2-lane pair) config = dai.Device.Config() config.board.gpio[6] = dai.BoardConfig.GPIO(dai.BoardConfig.GPIO.OUTPUT, dai.BoardConfig.GPIO.Level.HIGH) with dai.Device(config) as device: device.startPipeline(pipeline) if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); } Can use this commit as a reference for the changes to be made: https://github.com/alex-luxonis/depthai-python/commit/7db4ef58d3ca8b9b84ea0dca6a80e3e382fc2d1a

The delay is still greater than 1ms. I used the 2.17.3.1 version of depthai . Do I need to switch to a specific version? import depthai as dai import time import cv2 import collections set_fps = 30 class FPS: def __init__(self, window_size=30): self.dq = collections.deque(maxlen=window_size) self.fps = 0 def update(self, timestamp=None): if timestamp == None: timestamp = time.monotonic() count = len(self.dq) if count > 0: self.fps = count / (timestamp - self.dq[0]) self.dq.append(timestamp) def get(self): return self.fps # cam_list = ['left', 'right'] # cam_socket_opts = { # 'left' : dai.CameraBoardSocket.LEFT, # Or CAM_B # 'right': dai.CameraBoardSocket.RIGHT, # Or CAM_C # } cam_list = ['rgb', 'left', 'right', 'camd'] cam_socket_opts = { 'rgb' : dai.CameraBoardSocket.RGB, # Or CAM_A 'left' : dai.CameraBoardSocket.LEFT, # Or CAM_B 'right': dai.CameraBoardSocket.RIGHT, # Or CAM_C 'camd' : dai.CameraBoardSocket.CAM_D, } pipeline = dai.Pipeline() cam = {} xout = {} for c in cam_list: cam[c] = pipeline.create(dai.node.MonoCamera) cam[c].setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P) if c == 'rgb': cam[c].initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.OUTPUT) else: cam[c].initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) cam[c].setBoardSocket(cam_socket_opts[c]) xout[c] = pipeline.create(dai.node.XLinkOut) xout[c].setStreamName(c) cam[c].out.link(xout[c].input) config = dai.Device.Config() config.board.gpio[6] = dai.BoardConfig.GPIO(dai.BoardConfig.GPIO.OUTPUT, dai.BoardConfig.GPIO.Level.HIGH) with dai.Device(config) as device: device.startPipeline(pipeline) q = {} fps_host = {} # FPS computed based on the time we receive frames in app fps_capt = {} # FPS computed based on capture timestamps from device for c in cam_list: q[c] = device.getOutputQueue(name=c, maxSize=1, blocking=False) cv2.namedWindow(c, cv2.WINDOW_NORMAL) cv2.resizeWindow(c, (640, 480)) fps_host[c] = FPS() fps_capt[c] = FPS() while True: frame_list = [] for c in cam_list: pkt = q[c].tryGet() if pkt is not None: fps_host[c].update() fps_capt[c].update(pkt.getTimestamp().total_seconds()) print(c+":",pkt.getTimestampDevice()) frame = pkt.getCvFrame() cv2.imshow(c, frame) print("-------------------------------") # print("\rFPS:", # *["{:6.2f}|{:6.2f}".format(fps_host[c].get(), fps_capt[c].get()) for c in cam_list], # end='', flush=True) key = cv2.waitKey(1) if key == ord('q'): break if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); }

@RBJin Are your camera modules produced by Luxonis, or by Arducam? For the latter case, if they are used with Arducam UC-796 FFC cables, the FSIN pin isn't actually reaching the camera, it ends on the FFC cable as a testpoint. You could also post a picture with the setup, we'll confirm and see about potential workarounds (may need soldering wires to interconnect the sync pins).

@Luxonis-Alex The camera module I use is the Arducam OV9282 with UC-796 FFC cable. Do I just need to connect the FSIN testpoint on each cable to the FSIN pin on the corresponding camera board with a jump wire? As shown in the red circle. Is there anything else I need to do?

Yes, that would be a way to fix. You can use some thin wire, to reduce the risk of tearing the testpoint off the FFC cable. Or for a simpler and less riskier way, as the signal is generated by one camera and doesn't need to reach the OAK SoM, you can just interconnect the FSIN pins from all the cameras together, like in the picture below (can also solder header pins on the camera PCB and use jumper cables). In this case, no need to configure GPIO 6, but just sensor FSIN pins to be in OUTPUT or INPUT mode.

@Luxonis-Alex I chose the less risker way. Now the delay is less than 1ms successfully. Thank you so much!

Is the timing between the camera triggers and the triggering of reads from internal imu etc fixed in this case? What manages the triggering of the first camera(which drives sync to all the other camers) relative to imu reads?

Hi Asa , I don't think IMU/frames are hardware synced. But having IMU at higher frequency (eg 5x higher), even timestamp syncing could be good enough. More information here . Thoughts?

FFC-4p hardware synchronization

aiyangsky

jakaskerl Thanks for the suggestion. But I tried this way and he didn't improve.

erik

@aiyangsky the scoped signal doesn't look correct - it starts at 1V (should be 0v?) and goes to 1.8V. Is this a measurement error?

aiyangsky

erik I double-checked today that this is indeed not a measurement error.

aiyangsky

erik Is this reference voltage change caused by the FFC module? Or camera？What can I do to investigate this problem?

« Previous Page