How to properly sync RGB and Mono cameras on Oak-D Pro W PoE?

henry0987

I am attempting to synchronize the monochrome and RGB cameras on my Oak-D Pro W PoE.

The monochrome cameras are synchronized correctly, with the right monochrome being the sync master (fsync mode OUTPUT). However, when I attempt to synchronize the RGB camera by setting it to an INPUT with setFrameSyncMode, the RGB camera crashes. All cameras are set to 30 FPS. An external fsync is not currently an option. The RGB camera works fine when not attempting to be synchronized.

Have you seen this issue before? Is there other configuration that I have to set, or something else that I need to do?

jakaskerl

henry0987
What about other things you are running on the pipeline? Can you try lowering the FPS to 10?

Thanks
Jaka

henry0987

@jakaskerl I lowered the FPS to 10, but still get the same result. We're running the RGB/Color Camera Node, Stereo Camera Nodes, Stereo Depth Nodes, and the IMU Node.

Are you able to replicate this issue on your end?

jakaskerl

henry0987
I have not tried to repro this on V3. I know for a fact it works on V2 - will test tomorrow on or Monday to make sure.

Thanks,
Jaka

henry0987

Hi @jakaskerl,

Thank you. For the record, we're using depthai v2.30 with our Oak-D Pro W PoEs. Can you confirm that it works for you on our exact setup, including at 30 FPS?

jakaskerl

henry0987
Sure, I'll will test tomorrow to check for sure.

Thanks
Jaka

jakaskerl

Hi henry0987
I can get sub ms sync with this script on 2.30. And it doesn't really seem to matter if I set the FrameSyncMode. Might be a FW addition is later versions of depthai.

#!/usr/bin/env python3

import cv2
import depthai as dai
import time

config = dai.Device.Config()

with dai.Device(config) as device:
    print('Device name:', device.getDeviceName())
    if device.getBootloaderVersion() is not None:
        print('Bootloader version:', device.getBootloaderVersion())
    print('Usb speed:', device.getUsbSpeed().name)
    print('Connected cameras:', device.getConnectedCameraFeatures())

    pipeline = dai.Pipeline()
    streams = []

    # Helper to set sync based on socket
    def apply_sync(ctrl, sock):
        if str(sock) == "CameraBoardSocket.CAM_B":
            ctrl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.OUTPUT)
        else:
            ctrl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

    # Create nodes explicitly per socket/type:
    # CAM_A -> Color
    try:
        color = pipeline.create(dai.node.ColorCamera)
        color.setBoardSocket(dai.CameraBoardSocket.CAM_A)
        apply_sync(color.initialControl, dai.CameraBoardSocket.CAM_A)
        color.setFps(15)
        color.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
        color.setPreviewSize(640, 360)
        color.setInterleaved(False)
        color.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
        xoutA = pipeline.create(dai.node.XLinkOut)
        xoutA.setStreamName("CAM_A (color)")
        color.preview.link(xoutA.input)
        streams.append("CAM_A (color)")
    except Exception as e:
        print("Skipping CAM_A color setup:", e)

    # CAM_B -> Mono @ 800p
    try:
        monoB = pipeline.create(dai.node.MonoCamera)
        monoB.setBoardSocket(dai.CameraBoardSocket.CAM_B)
        apply_sync(monoB.initialControl, dai.CameraBoardSocket.CAM_B)
        monoB.setFps(15)
        monoB.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)
        xoutB = pipeline.create(dai.node.XLinkOut)
        xoutB.setStreamName("CAM_B (mono 800p)")
        monoB.out.link(xoutB.input)
        streams.append("CAM_B (mono 800p)")
    except Exception as e:
        print("Skipping CAM_B mono setup:", e)

    # CAM_C -> Mono @ 800p
    try:
        monoC = pipeline.create(dai.node.MonoCamera)
        monoC.setBoardSocket(dai.CameraBoardSocket.CAM_C)
        apply_sync(monoC.initialControl, dai.CameraBoardSocket.CAM_C)
        monoC.setFps(15)
        monoC.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)
        xoutC = pipeline.create(dai.node.XLinkOut)
        xoutC.setStreamName("CAM_C (mono 800p)")
        monoC.out.link(xoutC.input)
        streams.append("CAM_C (mono 800p)")
    except Exception as e:
        print("Skipping CAM_C mono setup:", e)

    # Start pipeline
    device.startPipeline(pipeline)

    # FPS tracking
    fpsCounter, lastFpsCount = {}, {}
    tfps = time.time()

    while not device.isClosed():
        queueNames = device.getQueueEvents(streams)
        for stream in queueNames:
            q = device.getOutputQueue(stream)
            messages = q.tryGetAll()
            if not messages:
                continue

            fpsCounter[stream] = fpsCounter.get(stream, 0.0) + len(messages)
            for msg in messages:
                if isinstance(msg, dai.ImgFrame):
                    frame = msg.getCvFrame()
                    fps = lastFpsCount.get(stream, 0.0)
                    print(f"Received frame from {stream} at {msg.getTimestamp().microseconds}")
                    cv2.putText(frame, f"Fps: {fps:.2f}", (10, 20),
                                cv2.FONT_HERSHEY_TRIPLEX, 0.5, (255, 255, 255), 1, cv2.LINE_AA)
                    cv2.imshow(stream, frame)

        # Update FPS once per second
        now = time.time()
        if now - tfps >= 1.0:
            scale = now - tfps
            for s in list(fpsCounter.keys()):
                lastFpsCount[s] = fpsCounter[s] / scale
            fpsCounter = {}
            tfps = now

        if cv2.waitKey(1) == ord('q'):
            break

    cv2.destroyAllWindows()

Thanks,
Jaka

henry0987

Thanks @jakaskerl for running this test, and I'm glad that it worked. I'll compare this to my code and figure out what I'm doing differently.